1 Hilbert Space and Kernel An inner product uv can be 1 a usual dot product uv 2 a kernel product uv vw where may have in64257nite dimensions However an inner product must satisfy the following conditions 1 Symmetry uv vu uv 8712 X 2 Bilinearity ID: 24763 Download Pdf

271K - views

Published byphoebe-click

1 Hilbert Space and Kernel An inner product uv can be 1 a usual dot product uv 2 a kernel product uv vw where may have in64257nite dimensions However an inner product must satisfy the following conditions 1 Symmetry uv vu uv 8712 X 2 Bilinearity

Download Pdf

Download Pdf - The PPT/PDF document "CSBStatB Spring Statistical Learning Th..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.

Page 1

CS281B/Stat241B (Spring 2008) Statistical Learning Theory Lecture: 7 Reproducing Kernel Hilbert Spaces Lecturer: Peter Bartlett Scribe: Chunhui Gu 1 Reproducing Kernel Hilbert Spaces 1.1 Hilbert Space and Kernel An inner product u,v can be 1. a usual dot product: u,v 2. a kernel product: u,v v,w ) = ) (where ) may have inﬁnite dimensions) However, an inner product must satisfy the following conditions: 1. Symmetry u,v v,u u,v ∈ X 2. Bilinearity αu βv,w u,w v,w u,v,w ∈ X α, 3. Positive deﬁniteness u,u ∈ X u,u = 0 = 0 Now we can

deﬁne the notion of a Hilbert space. Deﬁnition. Hilbert Space is an inner product space that is complete and separable with respect to the norm deﬁned by the inner product. Examples of Hilbert spaces include: 1. The vector space with a,b , the vector dot product of and 2. The space of square summable sequences, with inner product x,y =1 3. The space of square integrable functions (i.e., dx < ), with inner product f,g dx Deﬁnition. ) is a reproducing kernel of a Hilbert space if ∈ H ,f ) = x, ,f

Page 2

Reproducing Kernel Hilbert Spaces A Reproducing

Kernel Hilbert Space (RKHS) is a Hilbert space with a reproducing kernel whose span is dense in . We could equivalently deﬁne an RKHS as a Hilbert space of functions with all evaluation functionals bounded and linear. For instance, the space is a Hilbert space, but not an RKHS because the delta function which has the reproducing property ) = du does not satisfy the square integrable condition, that is, du thus the delta function is not in Now let us deﬁne a kernel. Deﬁnition. X X is a kernel if 1. is symmetric: x,y ) = y,x ). 2. is positive semi-deﬁnite,

i.e., ,x ,...,x ∈ X , the ”Gram Matrix deﬁned by ij ,x ) is positive semi-deﬁnite. (A matrix is positive semi-deﬁnite if Ma 0.) Here are some properties of a kernel that are worth noting: 1. x,x 0. (Think about the Gram matrix of = 1) 2. u,v u,u v,v ). (This is the Cauchy-Schwarz inequality.) To see why the second property holds, we consider the case when = 2: Let v,v u,v . The Gram matrix u,u u,v v,u v,v Ka v,v u,u u,v v,v 0. By the ﬁrst property we know v,v 0, so v,v u,u u,v 1.2 Build an Reproducing Kernel Hilbert Space (RKHS) Given a kernel , deﬁne

the ”reproducing kernel feature map” Φ : X as: Φ( ) = ,x Consider the vector space: span( Φ( ) : ∈ X} ) = ) = =1 ,x ) : ,x ∈ X , For ,u ) and ,v ), deﬁne f,g i,j ,v ). Note that: f,k ,x x,u ) = ), i.e., has the reproducing property. We show that f,g is an inner product by checking the following conditions:

Page 3

Reproducing Kernel Hilbert Spaces 1. Symmetry: f,g i,j ,v ) = i,j ,u ) = g,f 2. Bilinearity: f,g ) = 3. Positive deﬁniteness: f,f K 0 with equality i = 0. From 3 we can also derive: 1. f,g f,f g,g Proof. af g,af f,f + 2 f,g g,g 0.

This implies that the quadratic expression has a non-positive discriminant. Therefore, f,g f,f g,g 2. ,x ,f x,x f,f , which implies that if f,f = 0 then is identically zero. Now we have deﬁned an inner product space . Complete it to give the Hilbert space. Deﬁnition. For a (compact) X , and a Hilbert space of functions X , we say is a Reproducing Kernel Hilbert Space if X , s.t. 1. has the reproducing property, i.e., ) = ,k ,x 2. spans span ,x ) : ∈ X} 1.3 Mercer’s Theorem Another way to characterize a symmetric positive semi-deﬁnite kernel is via the Mercer’s

Theorem. Theorem 1.1 (Mercer’s) Suppose is a continuous positive semi-deﬁnite kernel on a compact set , and the integral operator ) deﬁned by )( ) = ,x dx is positive semi-deﬁnite, that is, ), u,v dudv Then there is an orthonormal basis of ) consisting of eigenfunctions of such that the correspond- ing sequence of eigenvalues are non-negative. The eigenfunctions corresponding to non-zero eigenvalues are continuous on and u,v ) has the representation u,v ) = =1 where the convergence is absolute and uniform, that is, lim sup u,v u,v =1 = 0

Page 4

Reproducing Kernel

Hilbert Spaces To take an analogue in the ﬁnite case, that is, ,...,x . Let ij ,x ), and X with ). Then, =1 ,x f, f Kf Hence, ,x ) = ij = ( ij =1 ki kj =1 ) = ( We summarize several equivalent conditions on continuous, symmetric deﬁned on compact 1. Every Gram matrix is positive semi-deﬁnite. 2. is positive semi-deﬁnite. 3. can be expressed as u,v ) = ). 4. is the reproducing kernel of an RKHS of functions on

Â© 2020 docslides.com Inc.

All rights reserved.