Singular V alue Decomposition SVD T rucco Appendix A - PDF document

Singular Value Decomposition (SVD)(Trucco, Appendix A.6)ƒDeŒnition-Any realmxnmatrixAcan be decomposed uniquely asA=UDVTUismxnand column orthogonal (its columns are eigenvectors ofAAT)(AAT=UDVTVDUT=UD2UT)Visnxnand orthogonal (its columns are eigenvectors ofATA)(ATA=VDUTUDVT=VD2VT)Disnxndiagonal (non-negative real values calledsingularvalues)D=diag(1,2,...,n)ordered so that1³2³...³n(ifis a singular value ofA,it'ssquare is an eigenvalue ofATA)-IfU=(u1u2...un)andV=(v1v2...vn), thenA=ni=1SiuivTi(actually,the sum goes from 1 torwhereris the rank ofA)ƒAn exampleA=éêêë121232121ùúúû,thenAAT=ATA=éêêë61061017106106ùúúûThe eigenvalues ofAAT,ATAare:éêêë
1
2
3ùúúû=éêêë28. 860. 140ùúúûThe eigenvectors ofAAT,ATAare:u1=v1=éêêë0. 4540. 7660. 454ùúúû,u2=v2=éêêë0. 542-0. 6430. 542ùúúû,u3=v3=éêêë-0. 7070-0. 707ùúúû -2-The expansion ofAisA=2i=1SiuivTiImportant:note that the second eigenvalue is much smaller than the Œrst; if weneglect it from the above summation, we can representAby introducing rela-tively small errors only:A=éêêë1. 111. 871. 111. 873. 151. 871. 111. 871. 11ùúúûƒComputing the rank using SVD-The rank of a matrix is equal to the number of non-zero singular values.ƒComputing the inverse of a matrix using SVD-Asquare matrixAis nonsingulariffi¹0for alli-IfAis anxnnonsingular matrix, then its inverse is givenbyA=UDVTorA-1=VD-1UTwhereD-1=diag(11,12,...,1n)-IfAis singular or ill-conditioned, then we can use SVD to approximate itsinverse by the following matrix:A-1=(UDVT)-1»VD-10UTD-10=ìíî1/i0ifi�totherwise(wheretis a small threshold) -3-ƒThe condition of a matrix-Consider the system of linear equationsAx=bIf small changes inbcan lead to relatively large changes in the solutionx,thenwe callAill-conditioned.-The ratio givenbelowisrelated to theconditionofAand measures the degreeof singularity ofA(the larger this value is, the closerAis to being singular)1/n(largest oversmallest singular values)ƒLeast Squares Solutions ofmxnSystems-Consider theover-determinedsystem of linear equationsAx=b,(Aismxnwithm�n)-Letrbe the residual vector for somex:r=Ax-b-The vectorx*which yields the smallest possible residual is called aleast-squaressolution (it is an approximate solution).||r||=||Ax*-b||£||Ax-b|| for allxÎRn-Although a least-squares solution always exist, it might not be unique !-The least-squares solutionxwith the smallest norm ||x|| is unique and it isgivenby:ATAx=ATborx=(ATA)-1ATb=A+bExample:éêêë-112223-1ùúúûéêëx1x2ùúû=éêêë075ùúúû -4-x=A+b=éêë-.148.164.180.189.246-.107ùúûéêêë075ùúúû=éêë2. 4920. 787ùúûƒComputingA+using SVD-IfATAis ill-conditioned or singular,wecan use SVD to obtain a least squaressolution as follows:x=A+b»VD-10UTbD-10=ìíî1/i0ifi�totherwise(wheretis a small threshold)ƒLeast Squares Solutions ofnxnSystems-IfAis ill-conditioned or singular,SVD can give usaworkable solution in thiscase too:x=A-1b»VD-10UTbƒHomogeneous Systems-Supposeb=0, then the linear system is called homogeneous:Ax=0(assumeAismxnandA=UDVT)-The minimum-norm solution in this case isx=0 (trivial solution).-For homogeneous linear systems, the meaning of a least-squares solution ismodiŒed byimposing the constraint:||x||=1-This is a "constrained" optimization problem:min||x||=1||Ax|| -5--The minimum-norm solution for homogeneous systems is not always unique.Special case:rank(A)=n-1(m³n-1,n=0)solution isx=avn(ais a constant)(vnis the last column ofV-- corresponds to the smallest)General case:rank(A)=n-k(m³n-k,n-k+1=...=n=0)solution isx=a1vn-k+a2vn-k-1+...+akvn(aisisaconstant)witha21+a22+...+a2k=1