MathematicalProgrammingmanuscriptNo.(willbeinsertedbytheeditor)ArnoldN - PDF document

MathematicalProgrammingmanuscriptNo.(willbeinsertedbytheeditor)ArnoldNeumaier¢OlegShcherbina¢WaltraudHuyer¢Tam¶asVink¶oAcomparisonofcompleteglobaloptimizationsolversthedateofreceiptandacceptanceshouldbeinsertedlaterAbstract.Resultsarereportedoftestinganumberofexistingstateoftheartsolversforglobalconstrainedoptimizationandconstraintsatisfactiononasetofover1000testproblemsinupto1000variables,collectedfromtheliterature.ThetestproblemsareavailableonlineinAMPLandweretranslatedintotheinputformatsofthevarioussolversusingroutinesfromtheCOCONUTenvironment.Thesetranslatorsareavailableonline,too.1.OverviewThispaperpresentstestresultsforsoftwareperformingacompletesearchtosolveglobaloptimizationorconstraintsatisfactionproblems.Incontrasttolocalorheuristicsearches,acompletesearchchecksallpointsinthesearchregionforfeasibility,andallfeasiblepointsforglobaloptimality.Asolverthatperformsacompletesearch{apartfromroundingerrorissues{iscalledacompletesolver.Sincethesearchregionforcontinuousglobaloptimizationproblemscontainsanin¯nitenumberofpoints,analytictechniquesareneededtomakedecisionsaboutin¯nitelymanypointssimultaneously.Thisisusually(butnotalways)doneinabranch-and-boundframework.AstherecentsurveyNeumaier[25]ofcompletesolutiontechniquesinglobaloptimizationdocuments,therearenowaboutadozensolversforconstrainedglobaloptimizationthatclaimtosolveglobaloptimizationand/orconstraintsatisfactionproblemstoglobaloptimalitybyperformingacompletesearch.WithintheCOCONUTproject[30,31],weevaluatedmanyoftheexistingsoftwarepackagesforglobaloptimizationandconstraintsatisfactionproblems.Thisisthe¯rsttimethatdi®erentconstrainedglobaloptimizationandconstraintsatisfactionalgorithmsarecomparedonasystematicbasisandwithatestsetthatallowstoderivestatisticallysigni¯cantconclusions.WetestedtheglobalsolversBARON,COCOS,GlobSol,ICOS,LGO,LINGO,OQNLP,PremiumSolver,andthelocalsolverMINOS.Thetestingprocessturnedouttobeextremelytime-consuming,duetovari-ousreasonsnotinitiallyanticipated.Alotofe®ortwentintocreatingappropriateArnoldNeumaier,OlegShcherbina,WaltraudHuyer:FakultÄatfÄurMathematik,UniversitÄatWien,Nordbergstr.15,A-1090Wien,AustriaTam¶asVink¶o:ResearchGrouponArti¯cialIntelligenceoftheHungarianAcademyofSciencesandUniversityofSzeged,H-6720Szeged,Aradiv¶ertan¶uktere1.,Hungary 2ArnoldNeumaieretal.interfaces,makingthecomparisonfairandreliable,andmakingitpossibletoprocessalargenumberoftestexamplesinasemiautomaticfashion.Inarecentpaperabouttestinglocaloptimizationsoftware,Dolan&Mor¶e[7,8]write:Werealizethattestingoptimizationsoftwareisanotoriouslydi±cultproblemandthattheremaybeobjectionstothetestingpresentedinthisreport.Forexample,performanceofaparticularsolvermayimprovesigni¯cantlyifnon-defaultoptionsaregiven.Anotherobjectionisthatweonlyuseonestartingpointperproblemandthattheperformanceofasolvermaybesensitivetothechoiceofstartingpoint.Wealsohaveusedthedefaultstoppingcriteriaofthesolvers.Thischoicemaybiasresultsbutshouldnota®ectcomparisonsthatrelyonlargetimedi®erences.Inspiteoftheseobjections,wefeelthatitisessentialthatweprovidesomeindicationoftheperformanceofoptimizationsolversoninterestingproblems.Thesedi±cultiesarealsopresentwithourbenchmarkingstudies.Section2describesourtestingmethodology.Weusealargetestsetofover1000problemsfromvariouscollections.Ourmainperformancecriteriumiscurrentlyhowoftentheattainmentoftheglobaloptimum,ortheinfeasibilityofaproblem,iscor-rectlyorincorrectlyclaimed(withinsometimelimit).Allsolversaretestedwiththedefaultoptionssuggestedbytheprovidersofthecodes,withtherequesttostopatatimelimitorafterthesolverbelievedthat¯rstglobalsolutionwasobtained.Theseareveryhighstandards,muchmoredemandingthanwhathadbeendonebyanyonebefore.Thoroughcomparisonsareindeedveryrare,duetothedi±cultyofperformingextensiveandmeaningfultesting.Indeed,weknowofonlytwocomparativestudies[18,23]inglobaloptimizationrangingovermorethanperhapsadozenexamples,andbotharelimitedtoboundconstrainedblackboxoptimization.(SeealsoHuyer[15]forsomefurthertests.)Onlyrecentlyrudimentarybeginningsweremadeelsewhereintestingcon-strainedglobaloptimization[12].Ontheotherhand,thereareanumberofre-portsaboutcomparingcodesinlocaloptimization[1,4,7,8,14,17,28],andthereisanextensivewebsite[22]withwide-rangingcomparativeresultsonlocalcon-strainedoptimizationcodes.Section3describesthetestsdoneonthemostimportantstateoftheartglobalsolvers.Shortlyexpressed,theresultisthefollowing:Amongthecurrentlyavailableglobalsolvers,BARONisthefastestandmostrobustone,withOQNLPbeingclose.Noneofthecurrentglobalsolversisfullyreliable,withoneexception:Forpureconstraintsatisfactionproblems,ICOS,whileslowerthanBARON,hasexcel-lentreliabilitypropertieswhenitisableto¯nishthecompletesearch.Modelsindimension100aresolvedwithasuccessrate(globalop-timumfound)ofover90%byBARONwhile(withinhalfanhourofCPUtime)lessthantwothirdsofthelargermodelsaresolved.OQNLP,thebestofthestochasticsolvers,hadsolvedthemaximalnumberofproblemsayearago,butisnowinmostrespectssecondtothenewestversionofBARON;moreover,itismuchslowerandcannot Acomparisonofcompleteglobaloptimizationsolvers3o®erinformationaboutwhenthesearchiscompleted.However,onthemodelswith�100variables,OQNLPstillsolves(withintheimposedtimelimit)thehighestpercentage(72%)ofproblems.Thebestsolver,BARON,wasabletocompletethesearchinovertwothirdofthemodelswithlessthan100variables(forlargerprob-lemsonlyaboutonethird,withinthetimelimitof30minutes),butitlosttheglobalminimuminabout4percentofthecases.The¯nalSection4concludeswithvariousremarks,includingguidelinesfordevelopersofglobaloptimizationcodesderivedfromourexperiencewiththeinitialversionsofthepackagestested.Muchmoredetailedresultsthancanbegivenhereareavailableonlineat[3].Acknowledgments.Wearehappytoacknowledgecooperationwithvari-ouspeople.DanFylstra(FrontlineSolver),TonyGau(LINGO),BakerKearfott(GlobSol),YahiaLebbah(ICOS),AlexMeeraus(GAMS),J¶anosPint¶er(LGO),andNickSahinidis(BARON)providedvaluablesupportinobtainingandrun-ningtheirsoftware.The¯rstroundoftestingGlobSolwasdonebyBogl¶arkaT¶oth(Szeged,Hungary).Mih¶alyMark¶ottestedanddebuggedpreliminaryver-sionsoftheCOCOSstrategy.TheresultspresentedarepartofworkdoneinthecontextoftheCOCONUTproject[30,31]sponsoredbytheEuropeanUnion,withthegoalofintegrat-ingvariousexistingcompleteapproachestoglobaloptimizationintoauniformwhole.FundingbytheEuropeanUnionundertheISTProjectReferenceNumberIST-2000-26063withintheFETOpenSchemeisgratefullyacknowledged.2.TestingIntroduction.WepresenttestresultsfortheglobaloptimizationsystemsBARON,COCOS,GlobSol,ICOS,LGO/GAMS,LINGO,OQNLPPremiumSolver,andforcomparisonthelocalsolverMINOS.AlltestsweremadeontheCOCONUTbenchmarkingsuitedescribedinShcherbinaetal.[33].Ourexperiencewiththesolverstestedandpreliminarytestresultswerecom-municatedtothedevelopersofthesolversandleadtosigni¯cantimprovementsintherobustnessanduser-friendlinessofseveralsolvers{thepresentresultsarebasedonthelastavailableversionofeachsolver.Forgeneralitiesonbenchmarkingandtheassociateddi±culties,inparticularforglobaloptimization,seeShcherbinaetal.[33].Hereweconcentrateonthedocumentationofthetestingconditionsusedandontheinterpretationoftheresultsobtained.Fortheinterpretationofthemainresults,seetheoverviewinSection1.Thetestset.Agoodbenchmarkmustbeonethatcanbeinterfacedwithallexistingsolvers,inawaythatasu±cientnumberofcomparativeresultscanbeobtained.Therearevarioussmaller-scalebenchmarkprojectsforpartial 4ArnoldNeumaieretal.domains,inparticularthebenchmarksforlocaloptimizationbyMittelmann[22].Averyrecentwebsite,theGAMSGlobalLibrary[13]startedcollectingreallifeglobaloptimizationproblemswithindustrialrelevance,butcurrentlymostproblemsonthissitearewithoutcomputationalresults.Ourbenchmark(describedinmoredetailin[33])includesmostoftheproblemsfromthesecollections.Thetestsetconsistsof1322modelsvaryingindimension(numberofvari-ables)between1andover1000,codedinthemodelinglanguageAMPL[9].Theyaresortedbysizeandsource(library).Sizekdenotesmodelswhosenumberofvariables(aftercreatingthecorrespondingDAGandsimplifyingit)haskdeci-maldigits.Library1(fromGlobalLibrary[13])andLibrary2(fromCUTE[5],intheversionofVanderbei[36])consistofglobal(andsomelocal)optimiza-tionproblemswithnonemptyfeasibledomain,whileLibrary3(fromEPFL[33])consistsofpureconstraintsatisfactionproblems(constantobjectivefunction),almostallbeingfeasible.Theresulting12modelclassesarelabeledaslib2s1(=size1modelsfromLibrary2),etc..NumberoftestmodelsNumberofvariables1¡910¡99100¡999¸1000anysize1size2size3size4totalLibrary184904448266Library234710093187727Library322576226329total6562661592411322Werestrictedtestingtomodelswithlessthan1000variablessincethemodelsofsize3alreadyposesomanydi±cultiesthatworkingonthe(muchmoreCPUtimeconsuming)largermodelsislikelytogivenoadditionalinsightforthecurrentgenerationofsolvers.Wealsoexcludedasmallnumberofmodelsfromthesetestsetsbecauseofdi±cultiesunrelatedtothesolvers.Inparticular,thefunctionsif,log10,tan,atan,asin,acosandacosharecurrentlynotsupportedbytheampl2dagcon-verterunderlyingallourtranslatorsintothevarioussolverinputformats.SincetheyareusedinthemodelsColumnDesign-original,FatigueDesign-original,djtl,hubfit(if),bearing(log10),yfit,yfitu(tan),artif,helix,s332,TrussDesign-full,TrussDesign01(atan),dallasl,dallasm,dallass(asin),chebyqad,cresc4,cresc50,cresc100,cresc132(acos),coshfun(acosh),thesemodelswereexcluded.AfewoftheproblemsinLibrary3(purecon-straintsatisfactionproblems)infactcontainedobjectivefunctions,andhencewereexcluded,too.Thiswasthecaseforthemodelsh78,h80,h81,logcheb,medianexp,mediannonconvex,robotarm,steenbre.Afewothermodels,namelyex8312,ex8314,concon,mconcon,osborneb,showedstrangebehavior,mak-ingussuspectthatitisduetounspottedbugsinourconverters.Themodelswherenoneofthesolversfoundafeasiblepointandsomeotherattemptstogetonefailed,areregardedinthefollowingasbeinginfeasible(thoughsomeofthesemightpossessundiscoveredfeasiblepoints). Acomparisonofcompleteglobaloptimizationsolvers5Thecomputers.Becauseofthelargenumberofmodelstobesolved,weper-formedourtestsonanumberofdi®erentcomputerscalledLisa,Hektor,Zenon,TheseusandBagend.Theirbrandandtheirgeneralperformancecharacteristicsaredisplayedbelow.ThestandardunittimeSTU,rede¯nedinShcherbinaetal.[33],isessentiallyequivalentto108standardunittimesaccordingtoDixon&SzegÄo[6].ForBogomipsandLinpack,seehttp://www.tldp.org/HOWTO/mini/BogoMips-2.html(ThehighLinpackentryforZenonisapparentlycausedbyanine±cientWin-dowsenvironment.)ComputerCPUtypeOSCPU/MHzBogomipsSTU/secLinpackLisaAMDAthlonLinux1678.863348.88507.42XP2000+HektorAMDAthlonLinux1544.513080.19536.66XP1800+ZenonAMDFamily6Windows1001|7446.78Model4NT4.0TheseusPentiumIIILinux1000.071992.291304.12BagendAMDAthlonLinux1666.723329.22365.68MP2000+Todecideonthebestwaytocompareacrosscomputers,weranthemodelsfromlib1s1withBARONonbothLisaandTheseus,andcomparedthere-sultingratiosofCPUtimeswiththeratiosofperformanceindices,givenbythefollowingtable.LisaTheseusRatiosInverseratiosFrequency1678.861000.071.680.60Bogomips3348.881992.291.680.59STU50.00130.000.382.60Linpack7.424.121.800.56AsFigure1withtheresultsshows,theappropriateindextouseisthefre-quency.Wethereforemeasuretimesinmultiplesof1000Mcycles,obtainedbymultiplyingtheCPUtimebythenominalfrequencyoftheCPUinMHz,anddividingtheresultby1000.Figure1alsoshowsthatsmalltimesarenotwellcomparable;wethereforedecidedtoroundtheresultingnumberstto1digitafterthedecimalpointift10,andtothenearestintegerift¸10.Fortinytimeswherethiswouldresultinazerotime,weuseinsteadt=0:05.Thesolvers.Thefollowingtablessummarizesomeofthemainpropertiesofthesesolvers,asfarasknowntous.Missinginformationisindicatedbyaquestionmark,andpartialapplicabilitybya+or¡inparentheses;thedominanttechnique(ifany)exploitedbythesolverisdenotedby++. 6ArnoldNeumaieretal.Fig.1.Timesandtimingratiosforlib1s1withBARON010203040506010-210-1100101102103TheseusLisatime in secondsproblem number, sorted by time for Theseus (1000MHz)01020304050600.20.40.60.811.21.41.6mean for time(Theseus) ³ 0.15frequency and Bogomips quotientsLinpack quotientstu quotientThe¯rsttworowsgivethenameofthesolversandtheaccesslanguageusedtopasstheproblemdescription.Thenexttworowsindicatewhetheritispossibletospecifyintegerconstraints(althoughwedon'ttestthisfeature),andwhetheritisnecessarytospecifya¯nitesearchboxwithinwhichallfunctionscanbeevaluatedwithout°oatingpointexceptions.Thenextthreerowsindicatewhetherblackboxfunctionevaluationissup-ported,whetherthesearchiscomplete(i.e.,isclaimedtocoverthewholesearchregionifthearithmeticisexactandsu±cientlyfast)orevenrigorous(i.e.,theresultsareclaimedtobevalidwithmathematicalcertaintyeveninthepresenceofroundingerrors). Acomparisonofcompleteglobaloptimizationsolvers7SolverMinosLGOBARONICOSGlobSolaccesslanguageGAMSGAMSGAMSAMPLFortran90optimization?+++¡+integerconstraints¡++¡¡searchbounds¡requiredrecommended¡requiredblackboxeval.++¡¡¡complete¡(¡)+++rigorous¡¡¡++local+++++(+)CP¡¡++++otherinterval¡¡¡+++convex/LP¡¡+++¡dual+¡+¡¡available+++++free¡¡¡(+)+SolverPremiumLINGO®BBGloptiPolyOQNLPSolverGlobalaccesslanguageVisualBasicLINGOMINOPTMatlabGAMSoptimization?+++(+)+integerconstraints+++¡+searchbounds+¡?¡+blackboxeval.¡¡¡¡+complete++++¡rigorous(+)¡¡¡¡local+++¡+CP++¡¡¡otherinterval++++¡¡convex/LP++++++¡dual¡+¡++¡available++¡++free¡¡¡+¡Notethatgeneraltheoremsforbidacomplete¯nitesearchifblackboxfunc-tionsarepartoftheproblemformulation,andthatarigoroussearchisneces-sarilycomplete.InviewofthegoalsoftheCOCONUTprojectweweremainlyinterestedincompletesolvers.However,wewerecurioushow(some)incompletesolversperform.Fivefurtherrowsindicatethemathematicaltechniquesusedtodotheglobalsearch.Wereportwhetherlocaloptimizationtechniques,constraintpropagation,otherintervaltechniques,convexanalysisandlinearprogramming(LP),ordual(multiplier)techniquesarepartofthetoolkitofthesolver.The¯naltworowsindicatewhetherthecodeisavailable(weincludeinthislistofpropertiesthesolver®BBbecauseofitsgoodreportedproperties,althoughwefailedtoobtainacopyofthecode),andwhetheritisfree(inthepublicdomain). 8ArnoldNeumaieretal.Inthispaper,westudythebehaviorofthesolversBARON/GAMS(version7.2,releasedJuly7,2004)[29,34,35],COCOS(betatestversion1.0,releasedSeptember20,2004),GlobSol(versionreleased11September2003)[19],ICOS(beta-testversion,releasedMarch29,2004)[20],LGO/GAMS[27],LINGO(ver-sion9.0,releasedOctober12,2004)[21],OQNLP/GAMS[11],PremiumSolver(IntervalGlobalSolverfromthePremiumSolverPlatformofFrontlineSystems,Version5)[10].(GloptiPolyislimitedtopolynomialsystemsofdimension20,andwasnottested.)Notethatourtestsapplytothecombinationofsolverplusinterface.ForLINGOweusedtheconverterfromGAMS.Inafewcases,thefailuresreportedareduetoproblemsintheGAMSinterfaceratherthanthesolver.Toenableustoassesshowdi±cultitis(i)to¯ndaglobalminimum,and(ii)toverifyitasglobal{inmanyinstances,part(ii)issigni¯cantlyharderthanpart(i){,results(withouttimings)fromthelocalsolverMINOS[24]arealsoincludedinourcomparison.ICOSonlyhandlespureconstraintsatisfactionproblems,andhencewastestedonlyonLibrary3.Twoofthesolvers(BARONandICOS)alsoallowthegenerationofmultiplesolutions,butduetothelackofareliablebasisforcomparison,thisfeaturehasnotbeentested.Twoofthesolvers(BARONandLINGO)allowonetoposeintegerconstraints,andtwo(LINGOandPremiumSolver)allowsnonsmoothexpressions.Neitherofthesefeatureshasbeentestedinthisstudy.Passingthemodels.TheaccesstoalltestmodelsisthroughanAMPLinterface,whichtranslatestheAMPLmodelde¯nitionintotheinternalformofadirectedacyclicgraph(DAG)whichislabelledinsuchawayastoprovideauniquedescriptionofthemodeltobesolved.Thisinternaldescriptioncouldbesimpli¯edbyaprogramdagsimplifywhichperformssimplepresolveandDAGsimpli¯cationtasks.Moreover,allmaximizationproblemsareconvertedtominimizationproblems,withobjectivemultipliedby¡1.Thispreprocessingensuresthatallsolversstartwithauniformlevelofdescriptionofthemodel.TheDAGisthentranslatedintotheinputformatrequiredbyeachsolver.(Forthetests,weswitchedo®themodelsimpli¯cationstage,sinceitisnotyete±cientlyimplemented.)Atestingenvironmentwascreatedtomakeasmuchaspossibleofthetestingworkautomatic.Wehadtorerunmanycalculationsformanymodelswheneverbugswere¯xed,newversionsofasolverbecameavailable,newsolverswereadded,improvementsinthetestingstatisticsweremade,etc.;thiswouldhavebeenimpossiblewithoutthesupportofsuchatestingenvironment.Thesimpli¯erandthetranslatorsfromAMPLintotheinputformatsforthesolverstestedareavailableintheCOCONUTenvironment(Schichl[32]).Theremainderofthetestenvironmentisnotfullyautomaticandhencenotpubliclyavailable.Performancecriteria.Allsolversaretestedwiththedefaultoptionssug-gestedbytheprovidersofthecodes.(Mostsolversmaybemadetoworksig- Acomparisonofcompleteglobaloptimizationsolvers9ni¯cantlybetterbytuningtheoptionstoparticularmodelclasses;hencetheviewgivenbyourcomparisonsmaylookmorepessimisticthantheviewusersgetwhospendtimeande®ortontuningasolvertotheirmodels.However,inalarge-scale,automatedcomparison,itisimpossibletodosuchtuning.)Thetimeoutlimitusedwas(scaledtoa1000MHzmachine)around180secondsCPUtimeformodelsofsize1,900secondsformodelsofsize2,and1800secondsformodelsofsize3(exceptforGlobSolandPremiumSolverwhichhadslightlydi®erenttimelimits,theresultsstemmingfromearlierruns).Theprecisevaluechangedbetweendi®erentrunssinceweexperimentedwithdif-ferentunitsformeasuringtimeondi®erentmachines.Butchanging(nottoomuch)thevalueforthetimeoutlimithardlya®ectsthecumulativeresults,sincetheoverwhelmingmajorityofthemodelswaseithercompletedveryquickly,orextremelyslow.ThesolversLGOandGlobSolrequiredaboundedsearchregion,andweboundedeachvariablebetween¡1000and1000,exceptinafewcaseswherethisleadstoalossoftheglobaloptimum.Thereliabilityofclaimedresultsisthemostpoorlydocumentedaspectofcurrentglobaloptimizationsoftware.Indeed,aswasshownbyNeumaier&Shcherbina[26]aspartofthecurrentproject,evenfamousstate-of-the-artsolverslikeCPLEX8.0(andmanyothercommercialMILPcodes)mayloseanintegralglobalsolutionofaninnocent-lookingmixedintegerlinearprogram.Weusethefollowing¯vecategoriestodescribethequalityclaimed:SignDescriptionXmodelnotacceptedbythesolverImodelclaimedinfeasiblebythesolverGresultclaimedtobeaglobaloptimizerLresultclaimedtobealocal(possiblyglobal)optimizerUunresolved(nosolutionfoundorerrormessage)Ttimeoutreached(quali¯esLandU)Notethattheunresolvedcasemaycontaincaseswhereafeasiblebutnonop-timalpointwasfound,butthesystemstoppedbeforeclaimingalocalorglobaloptimum.Checkingforbestfunctionvalue.TheprogramsolcheckfromtheCO-CONUTEnvironmentchecksthefeasibilityofputativesolutionsofsolverresults.Thiswasnecessarysincewefoundlotsofinconsistencieswheredi®erentsolversproduceddi®erentresults,andweneededawayofcheckingwhethertheproblemwasinthesolverorinourinterfacetoit.Apointwasconsideredtobe(nearly)feasibleifeachconstraintc(x)2[c;c]wassatis¯edwithinanabsoluteerroroftolforboundswithabsolutevalue1,andarelativeerroroftolforallotherbounds.Equalityconstraintswerehandledbythesamerecipewithc=c.Toevaluatethetestresults,thebestfunctionvalueisneededforeachmodel.Wecheckedinallcasesthenearfeasibilityofthebestpointsusedtoverifythe 10ArnoldNeumaieretal.claimofglobaloptimalityorfeasibility.Inalaterstageoftestingweintendtoproverigorouslytheexistenceofanearbyfeasiblepoint.Morespeci¯cally:Theresultsofallsolverstestedweretakenintoaccount,andthebestfunctionvaluewaschosenfromtheminimumofthe(nearly)feasiblesolutionsbyanysolver.Formodelswherethisdidnotgivea(nearly)feasiblepoint,wetriedto¯ndfeasiblepointsbyadhocmeans,whichweresometimessuccessful.Iftherewasstillnofeasiblesolutionforagivenmodel,the(local)solutionwiththeminimalresidualwaschosen(buttheresultmarkedasinfeasible).Totestwhichaccuracyrequirementsontheconstraintresidualswereade-quate,wecountedthenumberofsolutionsofBARONandLINGOonlib1s1whichwereacceptedasfeasiblewithvarioussolchecktolerancestol.Basedontheresultsgiveninthefollowingtable,itwasdecidedthatatoleranceoftol=10¡5wasadequate.(ThedefaulttolerancesusedforrunningBARONandLINGOwere10¡6.)solvertolallaccepted+GG!G?I?BARON1e-491887536101e-591887536101e-6918856241301e-791884919180LINGO1e-491918266301e-591918165401e-691917763601e-7919152412803.TestresultsNotationinthetables.Inthesummarystatistictablesthefollowingnotationisused:ColumnDescriptionlibrarydescribesthelibraryalllibrary/sizeacceptedthenumberofmodelsacceptedbythesolver+GnumberofmodelsforwhichtheglobalsolutionwasfoundG!numberofmodelsforwhichtheglobalsolutionwascorrectlyclaimedtobefoundG?numberofmodelsforwhichaglobalsolutionwasclaimedbutthetrueglobalsolutionwasinfactsigni¯cantlybetterortheglobalsolutionisreportedbutinfactthatisaninfeasiblepointI?numberofmodelsforwhichthemodelwasclaimedinfeasiblealthoughafeasiblepointexists Acomparisonofcompleteglobaloptimizationsolvers11Formodelswherealocaloptimizer¯ndstheglobaloptimum(withoutknow-ingit),thepurposeofaglobalcodeistocheckwhetherthereisindeednobetterpoint;thismaywellbethemosttime-consumingpartofacompletesearch.Fortheremainingmodelsthesearchfortheglobaloptimumisalreadyhard.Wethereforesplittheevaluationinto{`easylocationmodels',wherethelocaloptimizerMINOSfoundafeasiblepointwiththeglobalobjectivefunctionvalue,and{`hardlocationmodels',allotherswhereMINOSfailed.Fortheeasyandhardmodels(accordingtothisclassi¯cation),theclaimedstatusisgiveninthecolumns;therowscontainacomparisonwiththetruestatus:ColumnDescriptionwrongnumberofwrongclaims,i.e.thesumofG?andI?fromthesummarystatistictable+Ghowoftenthesolutionfoundwasinfactglobal¡GhowoftenitwasinfactnotglobalIhowmanymodelsareinfactinfeasibleForthepurposesofcomparisoninviewofroundo®,weroundedfunctionvaluesto3signi¯cantdecimaldigits,andregardedfunctionvaluesof10¡4aszero(butprinttheminthetablesasnonzeros)whenthebestknownfunctionvaluewaszero.Otherwise,weregardtheglobalminimumasachievediftheprinted(rounded)valuesagree.Foreachlibrarydetailedtablesforallmodelsandallsolverstested,andmanymore¯gures(ofthesametypeasthefewpresentedhere)arealsoavailable,forreasonsofspacetheyarepresentedonlineontheweb[3].Therewegiveacompletelistofresultswecurrentlyhaveontheninemodelclasses(i.e.,excludingthemodelswith1000ormorevariables,andthefewmodelsdescribedbefore.)GlobSolandPremiumSolver.TotestGlobSol,weusedanevaluationversionofLAHEYFortran95compiler.Notethatwehaddi±cultieswiththeIntelFortrancompiler.Inthe¯rstroundwetestedGlobSolonLibrary1size1problems(containing91problems)withthesametimelimitasusedfortheothersolvers.GlobSolfailedtosolvemostoftheproblemswithinthestricttimelimit.Forthisreasonwedecidedtouseaverypermissivetimelimit(eventhen,onlyhalftheacceptedproblemsweresolved).ThesametestsasforGlobSolwereperformedforPremiumSolver,withsimilarperformance.Figure2comparestheglobalsolversGlobSol,PremiumSolver,andBARONonthesize1problemsfromLibrary1.The¯gurecontainstimingresultsforthemodelsdescribedinthe¯gurecaption,sortedbythetimeusedbyBARON. 12ArnoldNeumaieretal.Conversiontimesforputtingthemodelsintotheformatrequiredbythesolversarenotincluded.Times(giveninunitsof1000Mcycles)below0.05areplacesonthebottomborderofeach¯gure,modelsforwhichtheglobalminimumwasnotfoundbythesolvergetadummytimeabovethetimeoutvalue,andareplacedatthetopborderofeach¯gure,inslightlydi®erentheightsforthedi®erentsolvers.Inthiswayonecanassessthesuccessfulcompletionoftheglobaloptimizationtask.Fig.2.Timesforlib1s1,allmodels,GlobSolandPremiumSolvervs.BARON010203040506070809010-1100101102103104times (unit = 1000 Mcycles)+=BARON7.2/GAMS x=GlobSol o=PremiumInafewcases,GlobSolandPremiumSolverfoundsolutionswhereBARONfailed,whichsuggeststhatBARONwouldbene¯tfromsomeoftheadvancedintervaltechniquesimplementedinGlobSolandPremiumSolver.However,GlobSolandPremiumSolveraremuchlesse±cientinbothtimeandsolvingcapacitythanBARON.ToalargeextentthismaybeduetothefactthatbothGlobSolandPremiumSolvestrivetoachievemathematicalrigor,resultinginsigni¯cantslowdownduetotheneedofrigorouslyvalidatedtech-niques.(Theremaybealsosystematicreasonsinthecomparison,sinceGlobSoldoesnotoutputabestapproximationtoaglobalsolutionbutboxes,fromwhichwehadtoextractatestpoint.) Acomparisonofcompleteglobaloptimizationsolvers13Moreover,thetheoreticallyrigorousperformanceguaranteesarenotborneout,inthatbothsolversmakesomewrongclaims,probablyduetolackofcareinprogramming.Inviewoftheseresults,werefrainedfromfurthertestingGlobSolandPre-miumSolver.Thesummarystatisticsonlib1s1canbefoundinthefollowingtables.GlobSolsummarystatisticslibraryallaccepted+GG!G?I?lib1s191773938200Premiumsummarystatisticslibraryallaccepted+GG!G?I?lib1s191754531101Amoredetailedtablegivesmoreinformation.Includedisanevaluationofthestatusclaimedaboutmodelfeasibilityandtheglobaloptimum,andthetruestatus(basedonthebestpointknowntous).GlobSolonlib1s1statusallwrongeasylocationhardlocation+G¡GI+G¡GIall9120303409180G582030130870U1900160120X140050090PremiumSolveronlib1s1statusallwrongeasylocationhardlocation+G¡GI+G¡GIall9111362809180G41102590610L120620220LT90430110U120120090X1600110050I11010000COCOS.TheCOCONUTenvironmentisanopendomainglobaloptimiza-tionplatform.Apartfromthetranslatorsusedforthepresentcomparison,itcontainsacon¯gurablesolverCOCOSwithmanymodulesthatcanbecom-binedtoyieldvariouscombinationstrategiesforglobaloptimization.Wetestedthestrategycalledby\cocos-hopt+bs+lpsolve&#xmode;&#xl000;". 14ArnoldNeumaieretal.COCOSsummarystatisticslibraryallaccepted+GG!G?I?lib1s191915136120lib1s280802611140lib1s341413020lib2s132432418298290lib2s29999371050lib2s3959511210lib3s12172171434820lib3s26969381900lib3s322224300ReliabilityanalysisforCOCOSglobalminimumfound/acceptedsize1376/632¼59%size2101/248¼41%size318/158¼11%all495/1038¼48%correctlyclaimedglobal/acceptedsize1182/632¼29%size240/248¼16%size35/158¼3%all227/1038¼22%wronglyclaimedglobal/claimedglobalsize143/225¼19%size219/59¼32%size33/8¼38%all65/292¼22%claimedinfeasible/acceptedandfeasiblesize10/626=0%size20/241=0%size30/147=0%all0/1014=0%ICOS.ICOSisapureconstraintsolver,whichcurrentlycannothandlemod-elswithanobjectivefunction,andhencewastestedonlyonLibrary3.(AnenhancedversionofICOS,capablealsoofrigorouslysolvingglobaloptimizationproblemsisunderdevelopment.)ICOSalsoclaimstoprovidemathematicallyrigorousresults,andindeed,itistheonlycompletesolvertestedthatdidnotmakeanyfalseclaimsinourtests. Acomparisonofcompleteglobaloptimizationsolvers15ICOSsummarystatisticslibraryallaccepted+GG!G?I?lib3s12172071456800lib3s26963341200lib3s322205000ReliabilityanalysisforICOS(onpureCSPsonly)globalminimumfound/acceptedsize1145/207¼70%size234/63¼54%size35/20¼25%all184/290¼63%correctlyclaimedglobal/acceptedsize168/207¼33%size212/63¼19%size30/20=0%all80/290¼28%wronglyclaimedglobal/claimedglobalsize10/68=0%size20/12=0%all0/80=0%claimedinfeasible/acceptedandfeasiblesize10/201=0%size20/59=0%size30/18=0%all0/278=0%BARON,LINGO,OQNLP,LGOandMINOS.Thefollowingtablescontainthesummarystatisticsfortheperformanceoftheothersolverstested,apartfromCOCOS.BARON7.2/GAMSsummarystatisticslibraryallaccepted+GG!G?I?lib1s19188886400lib1s28077714630lib1s3413323510lib2s1324296254206110lib2s29989824820lib2s39587512560lib3s121719518218033lib3s26963575721lib3s32220141310 16ArnoldNeumaieretal.LINGO9summarystatisticslibraryallaccepted+GG!G?I?lib1s19191847030lib1s280805342141lib1s3414112121lib2s1324324260232261lib2s299997145100lib2s395954926110lib3s1217217189189150lib3s26969555590lib3s3222210940OQNLP/GAMSsummarystatisticslibraryallaccepted+GG!G?I?lib1s1919183001lib1s2808070001lib1s3412812005lib2s1324315272001lib2s2999590000lib2s3958368001lib3s1217213196003lib3s2696747002lib3s3221911003LGO/GAMSsummarystatisticslibraryallaccepted+GG!G?I?lib1s1918565000lib1s2807839008lib1s3413140012lib2s1324309234004lib2s29994610015lib2s39557230010lib3s12172121550046lib3s26966350021lib3s322113004MINOS/GAMSsummarystatisticslibraryallaccepted+GG!G?I?lib1s1919164000lib1s2808047404lib1s3414119104lib2s132432324515112lib2s2999780423lib2s3959242108lib3s12172131553027lib3s26968350112lib3s3222111104 Acomparisonofcompleteglobaloptimizationsolvers17Thecorrespondingreliabilityanalysistablesareasfollows.ReliabilityanalysisforBARON7.2globalminimumfound/acceptedsize1524/579¼91%size2210/229¼92%size388/140¼63%all821/950¼86%correctlyclaimedglobal/acceptedsize1450/579¼78%size2151/229¼66%size343/140¼31%all644/950¼68%wronglyclaimedglobal/claimedglobalsize114/464¼3%size27/158¼4%size38/51¼16%all29/675¼4%claimedinfeasible/acceptedandfeasiblesize13/571¼1%size21/222¼0%size30/128=0%all4/921¼0.4%ReliabilityanalysisforLINGO9globalminimumfound/acceptedsize1533/632¼84%size2179/248¼72%size371/158¼45%all783/1038¼75%correctlyclaimedglobal/acceptedsize1491/632¼78%size2142/248¼57%size336/158¼23%all669/1038¼64%wronglyclaimedglobal/claimedglobalsize144/535¼8%size233/175¼19%size317/53¼32%all94/763¼12%claimedinfeasible/acceptedandfeasiblesize11/624¼0%size21/241¼0%size31/143¼0%all3/1008¼0.3% 18ArnoldNeumaieretal.ReliabilityanalysisforOQNLPglobalminimumfound/acceptedsize1551/619¼89%size2207/242¼86%size391/130¼72%all847/993¼86%claimedinfeasible/acceptedandfeasiblesize15/611¼1%size23/235¼1%size39/124¼8%all17/944¼2%ReliabilityanalysisforLGOglobalminimumfound/acceptedsize1454/606¼75%size2135/238¼57%size330/99¼30%all619/943¼66%claimedinfeasible/acceptedandfeasiblesize150/598¼8%size244/229¼18%size326/91¼30%all120/918¼13%ReliabilityanalysisforMINOSglobalminimumfound/acceptedsize1464/627¼74%size2162/245¼66%size372/154¼47%all698/1026¼68%correctlyclaimedglobal/acceptedsize118/627¼3%size28/245¼3%size33/154¼2%all29/1026¼3%wronglyclaimedglobal/claimedglobalsize11/19¼5%size23/11¼27%size30/3=0%all4/33¼12%claimedinfeasible/acceptedandfeasiblesize139/619¼6%size219/238¼8%size316/151¼11%all74/1008¼7% Acomparisonofcompleteglobaloptimizationsolvers19Finally,wecompareallsolversonlib1s1(problemsfromGlobalLibwithlessthan10variables).lib1s1summarystatisticslibraryallaccepted+GG!G?I?BARON9188886400LINGO9191837060OQNLP919183001LGO918565000MINOS919164000COCOS91915136120Premium91754531101GlobSol917739382004.ConclusionsTheresultsspeakforthemselves,andthemainconclusionswerealreadygivenintheopeningsection.Hereweaddafewmoreobservations.{ThemostremarkableobservationisthatthemodelsfromLibrary1,whichwerecollectedspeci¯callyastestproblemsforglobaloptimization,donotbehavemuchdi®erentlyfromthoseofLibrary2,whichwerecollectedastestproblemsforlocaloptimizationroutines.Inparticular,manyproblemsthatweresolvedinthepastonlyaslocaloptimizationproblemswereinfactglobalproblemswheretheglobalminimizerisnoteasilyfound.{TheGAMSsolversLGOandOQNLPareverycautious,neverclaimingaglobalminimum.Thisre°ectstheobservedunreliabilityoftheinternalclaims(asseenbystudyingthelog¯le)oftheLGOversionusedbyGAMS,andthedecisionofGAMSrathertoerrontheconservativeside.{Itison¯rstsightsurprisingthatunderGAMS,thelocalsolverMINOSsometimesclaimstohavefoundaglobalresult.Thisisthecase,e.g.,becausesomemodelsarerecognizedaslinearprogramsforwhicheverylocalsolutionisglobal.(ThecaseswithG?arecausedbytooinaccurateapproximatesolutions.){Inafewcases,solversreportedinfeasibility,althoughthepointtheyfoundwasconsideredfeasiblebysolcheck.{Conversely,anumberofthewrongclaimsofglobality(especiallyofLINGO)arecausedbythefactthatanapproximateminimizerwasfoundbutthattheconstraintswerenotsatis¯edwiththeaccuracyonecouldhaveexpectedfromthesolversettings{someresidualwaslargerthan10¡5,althoughtherequestedtolerancewas10¡6.{Inthemeantime,BARON/GAMShadsomebugs¯xed,whicheliminatesallwrongclaimstoinfeasibilityandreducestherateofwrongclaimsofglobaloptimalityto12=675=1:8%.Thisraisesthehopethatthenexto±cialreleasehasamuchimprovedreliability. 20ArnoldNeumaieretal.Fig.3.Performancepro¯lesforreachingtheglobaloptimum024681012141600.10.20.30.40.50.60.70.80.91 2log(time overhead factor over fastest solver) fraction of models solvedBARON 7.2Performancepro¯les.Uponrequestbyareviewer,wealsoaddsomeper-formancepro¯les(introducedbyDolan&Mor¶e[8]).However,thepro¯lesmustbeinterpretedwithmuchcautionsinceoftenaglobalsolver¯ndstheglobalminimumquiteearlyandthenspendsalotoftimecheckingwhetherthereisanotherone.Unlessatimelimitisreached,BARONandLINGOquitonlyaftertheycompletedthesearch,whileOQNLPandLGOquitaccordingtosomestatisticalcriterion,andMINOSquitsdirectlyafter¯nishingthelocaloptimization.Thisisre°ectedinaseveredependenceoftheperformancepro¯lesonthesolversselected;cf.Figures3and4.Figure3displaysthefractionofproblemssolvedtoglobaloptimalitywithinafactor2¿ofthetimethebestsolver(fromBARON,LINGO,OQNLP,LGOandMINOS)neededfortheproblem,amongallproblemsacceptedbyallthesesolvers.Figure4displaysthesame,butwithMINOSexcluded.(Thenumbersat¿=0adduptomorethan100%becauseofthewayweroundedtinytimes{asdiscussedinSection2{,whichresultedinmanytiesforthebesttimes,whichweremultiplycounted.Thestepsinthepro¯lesalsocomefromthisroundingprocedure.)Similarly,Figure5displaysthefractionofproblemswhereBARONorLINGO¯nishedtheglobalsearchsuccessfullywithinafactor2¿ofthetimethebetter Acomparisonofcompleteglobaloptimizationsolvers21Fig.4.Performancepro¯lesforreachingtheglobaloptimum0246810121400.10.20.30.40.50.60.70.80.91 2log(time overhead factor over fastest solver) fraction of models solvedBARON 7.2ofthetwosolversneededfor¯nishingtheglobalsearch,amongallproblemsacceptedbybothsolvers.Guidelinesforcodedevelopers.Ourextensivetestingrevealedanumberofdesirablepropertiesofsolvers,thatwouldhavesavedmuchofourtime,andthatcodedevelopersshouldconsidertakingintoaccount:{Solversshouldneverenterin¯niteloops;thereshouldbeanoptiontostopafter(approx.)agiventimelimit.{Aconstantobjectiveshouldnotcausedi±culties.{Solverswhichhaveaccesstothecomputationaltreeshouldnotfailbecauseofover°ow,under°ow,exceptions,exp(1000),1=0,0¤log(0),log(0).{Avoidconfusingmessages(e.g.,detectedinfeasibilityshouldnotbelabelled"success").{Outputmessagesshouldbemeaningfultotheuser(e.g.,"numericaldi±cul-tiesencountered"=)ResultsstillOK??).{Thereshouldbeaninformative¯nalqualityclaim(suchastheXIGLUclas-si¯cationusedhere).{Alargenumberofproblemshavenotallvariablesbounded;sosolversshouldbeabletoaddressthis.Ifasolverneeds¯niteboundstoperformwell,theseshouldbesetbydefaulttoreasonablevalueswherenoneareprovidedbythemodel,andawarningshouldbegiventotheuser. 22ArnoldNeumaieretal.Fig.5.Performancepro¯lesforreachingtheglobaloptimum012345678900.10.20.30.40.50.60.70.80.91 2log(time overhead factor over fastest solver) fraction of models solvedBARON 7.2References1.R.S.Barr,B.L.Golden,J.P.Kelly,M.G.C.Resende,andW.R.Stewart,Designingandreportingoncomputationalexperimentswithheuristicmethods,JournalofHeuristics1(1995),9{32.http://www.research.att.com/»mgcr/abstracts/guidelines.html2.F.BenhamouandF.Goualard,UniversallyQuanti¯edIntervalConstraints.InProceedingsofthe6thInternationalConferenceonPrinciplesandPracticeofConstraintProgram-ming(CP'2000),pages67{82,2000.3.COCONUTtestresults,WWW-directory,2004.http://www.mat.univie.ac.at/»neum/glopt/coconut/tests/figures/4.H.P.Crowder,R.S.DemboandJ.M.Mulvey,OnreportingComputationalExperimentswithMathematicalSoftware,ACMTransactionsonMathematicalSoftware,5(1979),193{203.5.N.I.M.Gould,D.OrbanandPh.L.Toint,CUTEr,aconstrainedandunconstrainedtestingenvironment,revisited,WWW-document,2001.http://cuter.rl.ac.uk/cuter-www/problems.html6.L.C.W.DixonandG.P.SzegÄo,TheGlobalOptimizationProblem:AnIntroduction,pp.1{15in:TowardsGlobalOptimization2,North-Holland,Amsterdam1978.7.E.D.DolanandJ.J.Mor¶e,BenchmarkingOptimizationSoftwarewithCOPS,Tech.ReportANL/MCS-246,ArgonneNat.Lab.,November2000.http://www-unix.mcs.anl.gov/»more/cops8.E.D.DolanandJ.J.Mor¶e,Benchmarkingoptimizationsoftwarewithperformancepro¯les,Math.Programming91(2002),201{213.http://www-unix.mcs.anl.gov/»more/cops9.R.Fourer,D.M.GayandB.W.Kernighan,AMPL:AModelingLanguageforMathemat-icalProgramming,DuxburyPress,Brooks/ColePublishingCompany,1993.http://www.ampl.com/cm/cs/what/ampl/ Acomparisonofcompleteglobaloptimizationsolvers2310.FrontlineSystems,Inc.,SolverTechnology-GlobalOptimization,WWW-document(2003).11.GAMSSolverdescriptions,GAMS/OQNLP,WWW-document(2003).http://www.gams.com/solvers/solvers.htm#OQNLP12.GAMSWorld,WWW-document,2002.http://www.gamsworld.org13.GLOBALLibrary,WWW-document,2002.http://www.gamsworld.org/global/globallib.htm14.H.J.Greenberg,Computationaltesting:Why,how,andhowmuch,ORSAJournalonComputing2(1990),94{97.15.W.Huyer,Acomparisonofsomealgorithmsforboundconstrainedglobaloptimization,WWW-document(2004).http://www.mat.univie.ac.at/»neum/glopt/contrib/compbound.pdf16.ILOG.ILOGSolver.ReferenceManual.2002.17.R.H.F.Jackson,P.T.Boggs,S.G.NashandS.Powell,Guidelinesforreportingresultsofcomputationalexperiments.Reportoftheadhoccommittee,MathematicalProgramming49(1990/91),413{426.18.E.Janka,VergleichstochastischerVerfahrenzurglobalenOptimierung,Diplomarbeit,MathematischesInst.,UniversitÄatWien,1999.AshorteronlineversioninEnglishlanguageisathttp://www.mat.univie.ac.at/»neum/glopt/janka/gopteng.html.19.R.B.Kearfott,RigorousGlobalSearch:ContinuousProblems,Kluwer,Dordrecht1996.www.mscs.mu.edu/»globsol20.Y.Lebbah,ICOS(IntervalCOnstraintsSolver),WWW-document(2003).http://www-sop.inria.fr/coprin/ylebbah/icos/21.LindoSystems,Inc.,NewLINGO8.0,WWW-document(2003).http://www.lindo.com/table/lgofeatures8t.html22.H.Mittelmann,Benchmarks.WWW-document,2002.http://plato.la.asu.edu/topics/benchm.html23.M.Mongeau,H.Karsenty,V.Rouz¶e,J.-B.Hiriart-Urruty,Comparisonofpublic-domainsoftwareforblackboxglobaloptimization,OptimizationMethodsandSoftware13(2000),203{226.24.B.A.MurtaghandM.A.Saunders,MINOS5.4User'sGuide,ReportSOL83-20R,SystemsOptimizationLaboratory,StanfordUniversity,December1983(revisedFebruary1995).http://www.sbsi-sol-optimize.com/Minos.htm25.A.Neumaier,CompleteSearchinContinuousGlobalOptimizationandConstraintSatis-faction,pp.271{369in:ActaNumerica2004(A.Iserles,ed.),CambridgeUniversityPress2004.26.A.NeumaierandO.Shcherbina,Safeboundsinlinearandmixed-integerprogramming,Math.ProgrammingA99(2004),283-296.http://www.mat.univie.ac.at/»neum/papers.html#mip27.J.D.Pinter,GlobalOptimizationinAction,Kluwer,Dordrecht1996.http://www.dal.ca/»jdpinter/lsd.html28.H.D.Ratli®andW.Pierskalla,ReportingComputationalExperienceinOperationsRe-search,OperationsResearch29(2)(1981),xi{xiv.29.H.S.RyooandN.V.Sahinidis,Abranch-and-reduceapproachtoglobaloptimization,J.GlobalOptim.8(1996),107{139.http://archimedes.scs.uiuc.edu/baron/baron.html30.H.Schichl,GlobaloptimizationintheCOCONUTproject.in:ProceedingsoftheDagstuhlSeminar\NumericalSoftwarewithResultVeri¯cation",SpringerLectureNotesinCom-puterScience2991,Springer,Berlin,2004.31.H.Schichl,MathematicalModelingandGlobalOptimization,Ha-bilitationThesis(2003),CambridgeUniv.Press,toappear.http://www.mat.univie.ac.at/»herman/papers/habil.ps32.H.Schichl,TheCOCONUTEnvironment.Website(2004).http://www.mat.univie.ac.at/coconut-environment/33.O.Shcherbina,A.Neumaier,DjamilaSam-Haroud,Xuan-HaVuandTuan-VietNguyen,Benchmarkingglobaloptimizationandconstraintsatisfactioncodes,pp.211{222in:Ch.Bliek,Ch.JermannandA.Neumaier(eds.),GlobalOptimizationandConstraintSatis-faction,Springer,Berlin2003. 24ArnoldNeumaieretal.:Acomparisonofcompleteglobaloptimizationsolvers34.M.TawarmalaniandN.V.Sahinidis,Convexi¯cationandGlobalOptimizationinCon-tinuousandMixed-IntegerNonlinearProgramming:Theory,Algorithms,Software,andApplications,Kluwer,Dordrecht2002.35.M.TawarmalaniandN.V.Sahinidis,Globaloptimizationofmixed-integernonlinearpro-grams:Atheoreticalandcomputationalstudy,Math.Programming99(2004),563{591.36.B.Vanderbei,NonlinearOptimizationModels,WWW-document.http://www.orfe.princeton.edu/»rvdb/ampl/nlmodels/