Approximation and 64257tting norm approximation leastnorm problems regularized approximation robust approximation 61 brPage 2br Norm approximation minimize Ax with k57527k is a norm on interpretations of solution argmin Ax geometric Ax is poi ID: 49300
Download Pdf The PPT/PDF document "Convex Optimization Boyd Vandenberghe" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
ConvexOptimization|Boyd&Vandenberghe6.Approximationand ttingnormapproximationleast-normproblemsregularizedapproximationrobustapproximation6{1 NormapproximationminimizekAx bk(A2Rmnwithmn,kkisanormonRm)interpretationsofsolutionx=argminxkAx bk:geometric:AxispointinR(A)closesttobestimation:linearmeasurementmodely=Ax+vyaremeasurements,xisunknown,vismeasurementerrorgiveny=b,bestguessofxisxoptimaldesign:xaredesignvariables(input),Axisresult(output)xisdesignthatbestapproximatesdesiredresultbApproximationand tting6{2 examplesleast-squaresapproximation(kk2):solutionsatis esnormalequationsATAx=ATb(x=(ATA) 1ATbifrankA=n)Chebyshevapproximation(kk1):canbesolvedasanLPminimizetsubjectto t1Ax bt1sumofabsoluteresidualsapproximation(kk1):canbesolvedasanLPminimize1Tysubjectto yAx byApproximationand tting6{3 Penaltyfunctionapproximationminimize (r1)++ (rm)subjecttor=Ax b(A2Rmn, :R!Risaconvexpenaltyfunction)examplesquadratic: (u)=u2deadzone-linearwithwidtha: (u)=maxf0;juj aglog-barrierwithlimita: (u)= a2log(1 (u=a)2)juja1otherwise u (u)deadzone-linearquadraticlogbarrier 1:5 1 0:500:511:500:511:52Approximationand tting6{4 example(m=100,n=30):histogramofresidualsforpenalties (u)=juj; (u)=u2; (u)=maxf0;juj ag; (u)= log(1 u2) p=1p=2DeadzoneLogbarrierr 2 2 2 2 1 1 1 1000011112222040010020010shapeofpenaltyfunctionhaslargee ectondistributionofresidualsApproximationand tting6{5 Huberpenaltyfunction(withparameter) hub(u)=u2juj(2juj )juj�Mlineargrowthforlargeumakesapproximationlesssensitivetooutliers replacements u hub(u) 1:5 1 0:500:511:500:511:52 tf(t) 10 50510 20 1001020left:Huberpenaltyfor=1right:anefunctionf(t)= + t ttedto42pointsti,yi(circles)usingquadratic(dashed)andHuber(solid)penaltyApproximationand tting6{6 Least-normproblemsminimizekxksubjecttoAx=b(A2Rmnwithmn,kkisanormonRn)interpretationsofsolutionx=argminAx=bkxk:geometric:xispointinanesetfxjAx=bgwithminimumdistanceto0estimation:b=Axare(perfect)measurementsofx;xissmallest('mostplausible')estimateconsistentwithmeasurementsdesign:xaredesignvariables(inputs);barerequiredresults(outputs)xissmallest('mostecient')designthatsatis esrequirementsApproximationand tting6{7 examplesleast-squaressolutionoflinearequations(kk2):canbesolvedviaoptimalityconditions2x+AT=0;Ax=bminimumsumofabsolutevalues(kk1):canbesolvedasanLPminimize1Tysubjectto yxy;Ax=btendstoproducesparsesolutionxextension:least-penaltyproblemminimize (x1)++ (xn)subjecttoAx=b :R!RisconvexpenaltyfunctionApproximationand tting6{8 Regularizedapproximationminimize(w.r.t.R2+)(kAx bk;kxk)A2Rmn,normsonRmandRncanbedi erentinterpretation: ndgoodapproximationAxbwithsmallxestimation:linearmeasurementmodely=Ax+v,withpriorknowledgethatkxkissmalloptimaldesign:smallxischeaperormoreecient,orthelinearmodely=Axisonlyvalidforsmallxrobustapproximation:goodapproximationAxbwithsmallxislesssensitivetoerrorsinAthangoodapproximationwithlargexApproximationand tting6{9 ScalarizedproblemminimizekAx bk+\rkxksolutionfor\r�0tracesoutoptimaltrade-o curveothercommonmethod:minimizekAx bk2+kxk2with�0TikhonovregularizationminimizekAx bk22+kxk22canbesolvedasaleast-squaresproblemminimize\r\r\r\rAp Ix b0\r\r\r\r22solutionx=(ATA+I) 1ATbApproximationand tting6{10 Optimalinputdesignlineardynamicalsystemwithimpulseresponseh:y(t)=tX =0h( )u(t );t=0;1;:::;Ninputdesignproblem:multicriterionproblemwith3objectives1.trackingerrorwithdesiredoutputydes:Jtrack=PNt=0(y(t) ydes(t))22.inputmagnitude:Jmag=PNt=0u(t)23.inputvariation:Jder=PN 1t=0(u(t+1) u(t))2trackdesiredoutputusingasmallandslowlyvaryinginputsignalregularizedleast-squaresformulationminimizeJtrack+Jder+Jmagfor xed;,aleast-squaresprobleminu(0),...,u(N)Approximationand tting6{11 example:3solutionsonoptimaltrade-o surface(top)=0,small;(middle)=0,larger;(bottom)large tu(t)050100150200 10 505 ty(t)050100150200 1 0:500:51 tu(t)050100150200 4 2024 ty(t)050100150200 1 0:500:51 tu(t)050100150200 4 2024 ty(t)050100150200 1 0:500:51Approximationand tting6{12 Signalreconstructionminimize(w.r.t.R2+)(k^x xcork2; (^x))x2Rnisunknownsignalxcor=x+vis(known)corruptedversionofx,withadditivenoisevvariable^x(reconstructedsignal)isestimateofx :Rn!Risregularizationfunctionorsmoothingobjectiveexamples:quadraticsmoothing,totalvariationsmoothing: quad(^x)=n 1Xi=1(^xi+1 ^xi)2; tv(^x)=n 1Xi=1j^xi+1 ^xijApproximationand tting6{13 quadraticsmoothingexample ixxcor0010001000200020003000300040004000 0:5 0:5000:50:5 i^x^x^x000100010001000200020002000300030003000400040004000 0:5 0:5 0:50000:50:50:5originalsignalxandnoisysignalxcorthreesolutionsontrade-o curvek^x xcork2versus quad(^x)Approximationand tting6{14 totalvariationreconstructionexample ixxcor00500500100010001500150020002000 2 2 1 1001122 i^xi^xi^xi000500500500100010001000150015001500200020002000 2 2 2000222originalsignalxandnoisysignalxcorthreesolutionsontrade-o curvek^x xcork2versus quad(^x)quadraticsmoothingsmoothsoutnoiseandsharptransitionsinsignalApproximationand tting6{15 ixxcor00500500100010001500150020002000 2 2 1 1001122 i^x^x^x000500500500100010001000150015001500200020002000 2 2 2000222originalsignalxandnoisysignalxcorthreesolutionsontrade-o curvek^x xcork2versus tv(^x)totalvariationsmoothingpreservessharptransitionsinsignalApproximationand tting6{16 RobustapproximationminimizekAx bkwithuncertainAtwoapproaches:stochastic:assumeAisrandom,minimizeEkAx bkworst-case:setAofpossiblevaluesofA,minimizesupA2AkAx bktractableonlyinspecialcases(certainnormskk,distributions,setsA)example:A(u)=A0+uA1xnomminimizeskA0x bk22xstochminimizesEkA(u)x bk22withuuniformon[ 1;1]xwcminimizessup 1u1kA(u)x bk22 gureshowsr(u)=kA(u)x bk2 ur(u)xnomxstochxwc 2 1012024681012Approximationand tting6{17 stochasticrobustLSwithA=A+U,Urandom,EU=0,EUTU=PminimizeEk(A+U)x bk22explicitexpressionforobjective:EkAx bk22=EkAx b+Uxk22=kAx bk22+ExTUTUx=kAx bk22+xTPxhence,robustLSproblemisequivalenttoLSproblemminimizekAx bk22+kP12xk22forP=I,getTikhonovregularizedproblemminimizekAx bk22+kxk22Approximationand tting6{18 worst-caserobustLSwithA=fA+u1A1++upApjkuk21gminimizesupA2AkAx bk22=supu21kP(x)u+q(x)k22whereP(x)=A1xA2xApx,q(x)=Ax bfrompage5{14,strongdualityholdsbetweenthefollowingproblemsmaximizekPu+qk22subjecttokuk221minimizet+subjectto24IPqPTI0qT0t350hence,robustLSproblemisequivalenttoSDPminimizet+subjectto24IP(x)q(x)P(x)TI0q(x)T0t350Approximationand tting6{19 example:histogramofresidualsr(u)=k(A0+u1A1+u2A2)x bk2withuuniformlydistributedonunitdisk,forthreevaluesofx r(u)xlsxtikxrlsfrequency01234500:050:10:150:20:25xlsminimizeskA0x bk2xtikminimizeskA0x bk22+kxk22(Tikhonovsolution)xrlsminimizessupA2AkAx bk22+kxk22Approximationand tting6{20