CSBStatB Spring  Statistical Learning Theory Lecture  Reproducing Kernel Hilbert Spaces Lecturer Peter Bartlett Scribe Chunhui Gu  Reproducing Kernel Hilbert Spaces

CSBStatB Spring Statistical Learning Theory Lecture Reproducing Kernel Hilbert Spaces Lecturer Peter Bartlett Scribe Chunhui Gu Reproducing Kernel Hilbert Spaces - Description

1 Hilbert Space and Kernel An inner product uv can be 1 a usual dot product uv 2 a kernel product uv vw where may have in64257nite dimensions However an inner product must satisfy the following conditions 1 Symmetry uv vu uv 8712 X 2 Bilinearity ID: 24763 Download Pdf

271K - views

CSBStatB Spring Statistical Learning Theory Lecture Reproducing Kernel Hilbert Spaces Lecturer Peter Bartlett Scribe Chunhui Gu Reproducing Kernel Hilbert Spaces

1 Hilbert Space and Kernel An inner product uv can be 1 a usual dot product uv 2 a kernel product uv vw where may have in64257nite dimensions However an inner product must satisfy the following conditions 1 Symmetry uv vu uv 8712 X 2 Bilinearity

Similar presentations


Download Pdf

CSBStatB Spring Statistical Learning Theory Lecture Reproducing Kernel Hilbert Spaces Lecturer Peter Bartlett Scribe Chunhui Gu Reproducing Kernel Hilbert Spaces




Download Pdf - The PPT/PDF document "CSBStatB Spring Statistical Learning Th..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.



Presentation on theme: "CSBStatB Spring Statistical Learning Theory Lecture Reproducing Kernel Hilbert Spaces Lecturer Peter Bartlett Scribe Chunhui Gu Reproducing Kernel Hilbert Spaces"— Presentation transcript:


Page 1
CS281B/Stat241B (Spring 2008) Statistical Learning Theory Lecture: 7 Reproducing Kernel Hilbert Spaces Lecturer: Peter Bartlett Scribe: Chunhui Gu 1 Reproducing Kernel Hilbert Spaces 1.1 Hilbert Space and Kernel An inner product u,v can be 1. a usual dot product: u,v 2. a kernel product: u,v v,w ) = ) (where ) may have infinite dimensions) However, an inner product must satisfy the following conditions: 1. Symmetry u,v v,u u,v ∈ X 2. Bilinearity αu βv,w u,w v,w u,v,w ∈ X α, 3. Positive definiteness u,u ∈ X u,u = 0 = 0 Now we can

define the notion of a Hilbert space. Definition. Hilbert Space is an inner product space that is complete and separable with respect to the norm defined by the inner product. Examples of Hilbert spaces include: 1. The vector space with a,b , the vector dot product of and 2. The space of square summable sequences, with inner product x,y =1 3. The space of square integrable functions (i.e., dx < ), with inner product f,g dx Definition. ) is a reproducing kernel of a Hilbert space if ∈ H ,f ) = x, ,f
Page 2
Reproducing Kernel Hilbert Spaces A Reproducing

Kernel Hilbert Space (RKHS) is a Hilbert space with a reproducing kernel whose span is dense in . We could equivalently define an RKHS as a Hilbert space of functions with all evaluation functionals bounded and linear. For instance, the space is a Hilbert space, but not an RKHS because the delta function which has the reproducing property ) = du does not satisfy the square integrable condition, that is, du thus the delta function is not in Now let us define a kernel. Definition. X  X is a kernel if 1. is symmetric: x,y ) = y,x ). 2. is positive semi-definite,

i.e., ,x ,...,x ∈ X , the ”Gram Matrix defined by ij ,x ) is positive semi-definite. (A matrix is positive semi-definite if Ma 0.) Here are some properties of a kernel that are worth noting: 1. x,x 0. (Think about the Gram matrix of = 1) 2. u,v u,u v,v ). (This is the Cauchy-Schwarz inequality.) To see why the second property holds, we consider the case when = 2: Let v,v u,v . The Gram matrix u,u u,v v,u v,v Ka v,v u,u u,v v,v 0. By the first property we know v,v 0, so v,v u,u u,v 1.2 Build an Reproducing Kernel Hilbert Space (RKHS) Given a kernel , define

the ”reproducing kernel feature map” Φ : X as: Φ( ) = ,x Consider the vector space: span( Φ( ) : ∈ X} ) = ) = =1 ,x ) : ,x ∈ X , For ,u ) and ,v ), define f,g i,j ,v ). Note that: f,k ,x x,u ) = ), i.e., has the reproducing property. We show that f,g is an inner product by checking the following conditions:
Page 3
Reproducing Kernel Hilbert Spaces 1. Symmetry: f,g i,j ,v ) = i,j ,u ) = g,f 2. Bilinearity: f,g ) = 3. Positive definiteness: f,f K 0 with equality i = 0. From 3 we can also derive: 1. f,g f,f g,g Proof. af g,af f,f + 2 f,g g,g 0.

This implies that the quadratic expression has a non-positive discriminant. Therefore, f,g f,f g,g 2. ,x ,f x,x f,f , which implies that if f,f = 0 then is identically zero. Now we have defined an inner product space . Complete it to give the Hilbert space. Definition. For a (compact) X , and a Hilbert space of functions X , we say is a Reproducing Kernel Hilbert Space if X , s.t. 1. has the reproducing property, i.e., ) = ,k ,x 2. spans span ,x ) : ∈ X} 1.3 Mercer’s Theorem Another way to characterize a symmetric positive semi-definite kernel is via the Mercer’s

Theorem. Theorem 1.1 (Mercer’s) Suppose is a continuous positive semi-definite kernel on a compact set , and the integral operator ) defined by )( ) = ,x dx is positive semi-definite, that is, ), u,v dudv Then there is an orthonormal basis of ) consisting of eigenfunctions of such that the correspond- ing sequence of eigenvalues are non-negative. The eigenfunctions corresponding to non-zero eigenvalues are continuous on and u,v ) has the representation u,v ) = =1 where the convergence is absolute and uniform, that is, lim sup u,v u,v =1 = 0
Page 4
Reproducing Kernel

Hilbert Spaces To take an analogue in the finite case, that is, ,...,x . Let ij ,x ), and X with ). Then, =1 ,x f, f Kf Hence, ,x ) = ij = ( ij =1 ki kj =1 ) = ( We summarize several equivalent conditions on continuous, symmetric defined on compact 1. Every Gram matrix is positive semi-definite. 2. is positive semi-definite. 3. can be expressed as u,v ) = ). 4. is the reproducing kernel of an RKHS of functions on