Thenon-interactivedownloadutilityUpdatedforWget1.17.1,10December2015by - PDF document

GNUWget1.20 Thenon-interactivedownloadutilityUpdatedforWget1.20,13November2018byHrvojeNiksicandothers ThisledocumentstheGNUWgetutilityfordownloadingnetworkdata.Copyrightc 1996-2011,2015,2018FreeSoftwareFoundation,Inc.Permissionisgrantedtomakeanddistributeverbatimcopiesofthismanualprovidedthecopy-rightnoticeandthispermissionnoticearepreservedonallcopies.Permissionisgrantedtocopy,distributeand/ormodifythisdocumentunderthetermsoftheGNUFreeDocumentationLicense,Version1.3oranylaterversionpublishedbytheFreeSoftwareFoundation;withnoInvariantSections,withnoFront-CoverTexts,andwithnoBack-CoverTexts.Acopyofthelicenseisincludedinthesectionentitled\GNUFreeDocumentationLicense". i TableofContents1Overview:::::::::::::::::::::::::::::::::::::::::::::::::::::::: 1 2Invoking::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 2 2.1URLFormat::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 2 2.2OptionSyntax::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 3 2.3BasicStartupOptions::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 3 2.4LoggingandInputFileOptions:::::::::::::::::::::::::::::::::::::::::::::::::::: 4 2.5DownloadOptions::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 6 2.6DirectoryOptions:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 14 2.7HTTPOptions::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 15 2.8HTTPS(SSL/TLS)Options:::::::::::::::::::::::::::::::::::::::::::::::::::::: 21 2.9FTPOptions::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 25 2.10FTPSOptions::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 26 2.11RecursiveRetrievalOptions:::::::::::::::::::::::::::::::::::::::::::::::::::::: 27 2.12RecursiveAccept/RejectOptions:::::::::::::::::::::::::::::::::::::::::::::::: 30 2.13ExitStatus:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 31 3RecursiveDownload:::::::::::::::::::::::::::::::::::::::::: 33 4FollowingLinks::::::::::::::::::::::::::::::::::::::::::::::: 34 4.1SpanningHosts::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 34 4.2TypesofFiles:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 34 4.3Directory-BasedLimits::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 36 4.4RelativeLinks:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 37 4.5FollowingFTPLinks::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 37 5Time-Stamping:::::::::::::::::::::::::::::::::::::::::::::::: 38 5.1Time-StampingUsage:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 38 5.2HTTPTime-StampingInternals:::::::::::::::::::::::::::::::::::::::::::::::::: 39 5.3FTPTime-StampingInternals:::::::::::::::::::::::::::::::::::::::::::::::::::: 39 6StartupFile::::::::::::::::::::::::::::::::::::::::::::::::::: 40 6.1WgetrcLocation:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 40 6.2WgetrcSyntax::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 40 6.3WgetrcCommands::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 40 6.4SampleWgetrc::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 47 7Examples:::::::::::::::::::::::::::::::::::::::::::::::::::::: 51 7.1SimpleUsage::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 51 7.2AdvancedUsage:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 51 7.3VeryAdvancedUsage::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 52 ii 8Various::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 54 8.1Proxies::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 54 8.2Distribution:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 55 8.3WebSite::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 55 8.4MailingLists::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 55 PrimaryList:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 55 ObsoleteLists:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 55 8.5InternetRelayChat:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 55 8.6ReportingBugs::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 55 8.7Portability:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 56 8.8Signals::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 57 9Appendices:::::::::::::::::::::::::::::::::::::::::::::::::::: 58 9.1RobotExclusion:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 58 9.2SecurityConsiderations::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 59 9.3Contributors:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 59 AppendixACopyingthismanual:::::::::::::::::::::::::::::: 61 A.1GNUFreeDocumentationLicense:::::::::::::::::::::::::::::::::::::::::::::::: 61 ConceptIndex:::::::::::::::::::::::::::::::::::::::::::::::::::: 68 1 1OverviewGNUWgetisafreeutilityfornon-interactivedownloadoflesfromtheWeb.Itsupportshttp,https,andftpprotocols,aswellasretrievalthroughhttpproxies.ThischapterisapartialoverviewofWget'sfeatures.Wgetisnon-interactive,meaningthatitcanworkinthebackground,whiletheuserisnotloggedon.Thisallowsyoutostartaretrievalanddisconnectfromthesystem,lettingWgetnishthework.Bycontrast,mostoftheWebbrowsersrequireconstantuser'spresence,whichcanbeagreathindrancewhentransferringalotofdata.Wgetcanfollowlinksinhtml,xhtml,andcsspages,tocreatelocalversionsofremotewebsites,fullyrecreatingthedirectorystructureoftheoriginalsite.Thisissometimesreferredtoas\recursivedownloading."Whiledoingthat,WgetrespectstheRobotExclusionStandard(/robots.txt).Wgetcanbeinstructedtoconvertthelinksindownloadedlestopointatthelocalles,foroineviewing.Filenamewildcardmatchingandrecursivemirroringofdirectoriesareavailablewhenre-trievingviaftp.Wgetcanreadthetime-stampinformationgivenbybothhttpandftpservers,andstoreitlocally.ThusWgetcanseeiftheremotelehaschangedsincelastretrieval,andautomaticallyretrievethenewversionifithas.ThismakesWgetsuitableformirroringofftpsites,aswellashomepages.Wgethasbeendesignedforrobustnessoversloworunstablenetworkconnections;ifadownloadfailsduetoanetworkproblem,itwillkeepretryinguntilthewholelehasbeenretrieved.Iftheserversupportsregetting,itwillinstructtheservertocontinuethedownloadfromwhereitlefto.Wgetsupportsproxyservers,whichcanlightenthenetworkload,speedupretrievalandprovideaccessbehindrewalls.Wgetusesthepassiveftpdownloadingbydefault,activeftpbeinganoption.WgetsupportsIPversion6,thenextgenerationofIP.IPv6isautodetectedatcompile-time,andcanbedisabledateitherbuildorruntime.BinariesbuiltwithIPv6supportworkwellinbothIPv4-onlyanddualfamilyenvironments.Built-infeaturesoermechanismstotunewhichlinksyouwishtofollow(see Chapter4[FollowingLinks],page34 ).Theprogressofindividualdownloadsistracedusingaprogressgauge.Interactivedownloadsaretrackedusinga\thermometer"-stylegauge,whereasnon-interactiveonesaretracedwithdots,eachdotrepresentingaxedamountofdatareceived(1KBbydefault).Eithergaugecanbecustomizedtoyourpreferences.Mostofthefeaturesarefullycongurable,eitherthroughcommandlineoptions,orviatheinitializationle.wgetrc(see Chapter6[StartupFile],page40 ).Wgetallowsyoutodeneglobalstartuples(/usr/local/etc/wgetrcbydefault)forsitesettings.Youcanalsospecifythelocationofastartuplewiththe{congoption.Todisablethereadingofcongles,use{no-cong.Ifboth{congand{no-congaregiven,{no-congisignored.Finally,GNUWgetisfreesoftware.Thismeansthateveryonemayuseit,redistributeitand/ormodifyitunderthetermsoftheGNUGeneralPublicLicense,aspublishedbytheFreeSoftwareFoundation(seetheleCOPYINGthatcamewithGNUWget,fordetails). 2 2InvokingBydefault,Wgetisverysimpletoinvoke.Thebasicsyntaxis:wget[option]...[URL]...Wgetwillsimplydownloadalltheurlsspeciedonthecommandline.URLisaUniformResourceLocator,asdenedbelow.However,youmaywishtochangesomeofthedefaultparametersofWget.Youcandoittwoways:permanently,addingtheappropriatecommandto.wgetrc(see Chapter6[StartupFile],page40 ),orspecifyingitonthecommandline.2.1URLFormatURLisanacronymforUniformResourceLocator.AuniformresourcelocatorisacompactstringrepresentationforaresourceavailableviatheInternet.Wgetrecognizestheurlsyntaxasperrfc1738.Thisisthemostwidelyusedform(squarebracketsdenoteoptionalparts):http://host[:port]/directory/fileftp://host[:port]/directory/fileYoucanalsoencodeyourusernameandpasswordwithinaurl:ftp://user:password@host/pathhttp://user:password@host/pathEitheruserorpassword,orboth,maybeleftout.Ifyouleaveouteitherthehttpusernameorpassword,noauthenticationwillbesent.Ifyouleaveouttheftpusername,`anonymous'willbeused.Ifyouleaveouttheftppassword,youremailaddresswillbesuppliedasadefaultpassword.1ImportantNote:ifyouspecifyapassword-containingurlonthecommandline,theuser-nameandpasswordwillbeplainlyvisibletoallusersonthesystem,bywayofps.Onmulti-usersystems,thisisabigsecurityrisk.Toworkaroundit,usewget-i-andfeedtheurlstoWget'sstandardinput,eachonaseparateline,terminatedbyC-d.Youcanencodeunsafecharactersinaurlas`%xy',xybeingthehexadecimalrepresentationofthecharacter'sasciivalue.Somecommonunsafecharactersinclude`%'(quotedas`%25'),`:'(quotedas`%3A'),and`@'(quotedas`%40').Refertorfc1738foracomprehensivelistofunsafecharacters.Wgetalsosupportsthetypefeatureforftpurls.Bydefault,ftpdocumentsareretrievedinthebinarymode(type`i'),whichmeansthattheyaredownloadedunchanged.Anotherusefulmodeisthe`a'(ASCII)mode,whichconvertsthelinedelimitersbetweenthedierentoperatingsystems,andisthususefulfortextles.Hereisanexample:ftp://host/directory/file;type=aTwoalternativevariantsofurlspecicationarealsosupported,becauseofhistorical(hys-terical?)reasonsandtheirwidespreadeduse.ftp-onlysyntax(supportedbyNcFTP):host:/dir/filehttp-onlysyntax(introducedbyNetscape):host[:port]/dir/fileThesetwoalternativeformsaredeprecated,andmayceasebeingsupportedinthefuture.Ifyoudonotunderstandthedierencebetweenthesenotations,ordonotknowwhichonetouse,justusetheplainordinaryformatyouusewithyourfavoritebrowser,likeLynxorNetscape. 1Ifyouhavea.netrcleinyourhomedirectory,passwordwillalsobesearchedforthere. Chapter2:Invoking3 2.2OptionSyntaxSinceWgetusesGNUgetopttoprocesscommand-linearguments,everyoptionhasalongformalongwiththeshortone.Longoptionsaremoreconvenienttoremember,buttaketimetotype.Youmayfreelymixdierentoptionstyles,orspecifyoptionsafterthecommand-linearguments.Thusyoumaywrite:wget-r--tries=10http://fly.srk.fer.hr/-ologThespacebetweentheoptionacceptinganargumentandtheargumentmaybeomitted.Insteadof`-olog'youcanwrite`-olog'.Youmayputseveraloptionsthatdonotrequireargumentstogether,like:wget-drcURLThisiscompletelyequivalentto:wget-d-r-cURLSincetheoptionscanbespeciedafterthearguments,youmayterminatethemwith`--'.Sothefollowingwilltrytodownloadurl`-x',reportingfailuretolog:wget-olog---xTheoptionsthatacceptcomma-separatedlistsallrespecttheconventionthatspecifyinganemptylistclearsitsvalue.Thiscanbeusefultoclearthe.wgetrcsettings.Forinstance,ifyour.wgetrcsetsexclude_directoriesto/cgi-bin,thefollowingexamplewillrstresetit,andthensetittoexclude/~nobodyand/~somebody.Youcanalsoclearthelistsin.wgetrc(see Section6.2[WgetrcSyntax],page40 ).wget-X''-X/~nobody,/~somebodyMostoptionsthatdonotacceptargumentsarebooleanoptions,sonamedbecausetheirstatecanbecapturedwithayes-or-no(\boolean")variable.Forexample,`--follow-ftp'tellsWgettofollowFTPlinksfromHTMLlesand,ontheotherhand,`--no-glob'tellsitnottoperformleglobbingonFTPURLs.Abooleanoptioniseitherarmativeornegative(beginningwith`--no').Allsuchoptionsshareseveralproperties.Unlessstatedotherwise,itisassumedthatthedefaultbehavioristheoppositeofwhattheoptionaccomplishes.Forexample,thedocumentedexistenceof`--follow-ftp'assumesthatthedefaultistonotfollowFTPlinksfromHTMLpages.Armativeoptionscanbenegatedbyprependingthe`--no-'totheoptionname;negativeoptionscanbenegatedbyomittingthe`--no-'prex.Thismightseemsuper uous|ifthedefaultforanarmativeoptionistonotdosomething,thenwhyprovideawaytoexplicitlyturnito?Butthestartuplemayinfactchangethedefault.Forinstance,usingfollow_ftp=onin.wgetrcmakesWgetfollowFTPlinksbydefault,andusing`--no-follow-ftp'istheonlywaytorestorethefactorydefaultfromthecommandline.2.3BasicStartupOptions`-V'`--version'DisplaytheversionofWget.`-h'`--help'PrintahelpmessagedescribingallofWget'scommand-lineoptions.`-b'`--background'Gotobackgroundimmediatelyafterstartup.Ifnooutputleisspeciedviathe`-o',outputisredirectedtowget-log. Chapter2:Invoking4 `-ecommand'`--executecommand'Executecommandasifitwereapartof.wgetrc(see Chapter6[StartupFile],page40 ).Acommandthusinvokedwillbeexecutedafterthecommandsin.wgetrc,thustakingprecedenceoverthem.Ifyouneedtospecifymorethanonewgetrccommand,usemultipleinstancesof`-e'.2.4LoggingandInputFileOptions`-ologfile'`--output-file=logfile'Logallmessagestologle.Themessagesarenormallyreportedtostandarderror.`-alogfile'`--append-output=logfile'Appendtologle.Thisisthesameas`-o',onlyitappendstologleinsteadofoverwritingtheoldlogle.Iflogledoesnotexist,anewleiscreated.`-d'`--debug'Turnondebugoutput,meaningvariousinformationimportanttothedevelopersofWgetifitdoesnotworkproperly.YoursystemadministratormayhavechosentocompileWgetwithoutdebugsupport,inwhichcase`-d'willnotwork.Pleasenotethatcompilingwithdebugsupportisalwayssafe|Wgetcompiledwiththedebugsupportwillnotprintanydebuginfounlessrequestedwith`-d'.See Section8.6[ReportingBugs],page55 ,formoreinformationonhowtouse`-d'forsendingbugreports.`-q'`--quiet'TurnoWget'soutput.`-v'`--verbose'Turnonverboseoutput,withalltheavailabledata.Thedefaultoutputisverbose.`-nv'`--no-verbose'Turnoverbosewithoutbeingcompletelyquiet(use`-q'forthat),whichmeansthaterrormessagesandbasicinformationstillgetprinted.`--report-speed=type'Outputbandwidthastype.Theonlyacceptedvalueis`bits'.`-ifile'`--input-file=file'Readurlsfromalocalorexternalle.If`-'isspeciedasle,urlsarereadfromthestandardinput.(Use`./-'toreadfromaleliterallynamed`-'.)Ifthisfunctionisused,nourlsneedbepresentonthecommandline.Ifthereareurlsbothonthecommandlineandinaninputle,thoseonthecommandlineswillbetherstonestoberetrieved.If`--force-html'isnotspecied,thenleshouldconsistofaseriesofURLs,oneperline.However,ifyouspecify`--force-html',thedocumentwillberegardedas`html'.Inthatcaseyoumayhaveproblemswithrelativelinks,whichyoucansolveeitherbyaddinghref="urlºse;&#x-333;"tothedocumentsorbyspecifying`--base=url'onthecommandline. Chapter2:Invoking5 Iftheleisanexternalone,thedocumentwillbeautomaticallytreatedas`html'iftheContent-Typematches`text/html'.Furthermore,thele'slocationwillbeimplicitlyusedasbasehrefifnonewasspecied.`--input-metalink=file'DownloadslescoveredinlocalMetalinkle.Metalinkversion3and4aresup-ported.`--keep-badhash'KeepsdownloadedMetalink'sleswithabadhash.Itappends.badhashtothenameofMetalink'sleswhichhaveachecksummismatch,exceptwithoutoverwritingexistingles.`--metalink-over-http'IssuesHTTPHEADrequestinsteadofGETandextractsMetalinkmetadatafromresponseheaders.ThenitswitchestoMetalinkdownload.IfnovalidMetalinkmeta-dataisfound,itfallsbacktoordinaryHTTPdownload.Enables`Content-Type:application/metalink4+xml'lesdownload/processing.`--metalink-index=number'SettheMetalink`application/metalink4+xml'metaurlordinalNUMBER.From1tothetotalnumberof\application/metalink4+xml"available.Specify0or`inf'tochoosetherstgoodone.Metaurls,suchasthosefroma`--metalink-over-http',mayhavebeensortedbyprioritykey'svalue;keepthisinmindtochoosetherightNUMBER.`--preferred-location'SetpreferredlocationforMetalinkresources.Thishaseectifmultipleresourceswithsamepriorityareavailable.`-F'`--force-html'Wheninputisreadfromale,forceittobetreatedasanhtmlle.Thisenablesyoutoretrieverelativelinksfromexistinghtmllesonyourlocaldisk,byaddinghref="urlºse;&#x-333;"tohtml,orusingthe`--base'command-lineoption.`-BURL'`--base=URL'ResolvesrelativelinksusingURLasthepointofreference,whenreadinglinksfromanHTMLlespeciedviathe`-i'/`--input-file'option(togetherwith`--force-html',orwhentheinputlewasfetchedremotelyfromaserverdescribingitashtml).ThisisequivalenttothepresenceofaBASEtaginthehtmlinputle,withURLasthevalueforthehrefattribute.Forinstance,ifyouspecify`http://foo/bar/a.html'forURL,andWgetreads`../baz/b.html'fromtheinputle,itwouldberesolvedto`http://foo/baz/b.html'.`--config=FILE'Specifythelocationofastartupleyouwishtouseinsteadofthedefaultone(s).Use{no-congtodisablereadingofcongles.Ifboth{congand{no-congaregiven,{no-congisignored.`--rejected-log=logfile'LogsallURLrejectionstologleascommaseparatedvalues.Thevaluesincludethereasonofrejection,theURLandtheparentURLitwasfoundin. Chapter2:Invoking6 2.5DownloadOptions`--bind-address=ADDRESS'WhenmakingclientTCP/IPconnections,bindtoADDRESSonthelocalmachine.ADDRESSmaybespeciedasahostnameorIPaddress.ThisoptioncanbeusefulifyourmachineisboundtomultipleIPs.`--bind-dns-address=ADDRESS'[libcaresonly]ThisaddressoverridestherouteforDNSrequests.Ifyoueverneedtocircumventthestandardsettingsfrom/etc/resolv.conf,thisoptiontogetherwith`--dns-servers'isyourfriend.ADDRESSmustbespeciedeitherasIPv4orIPv6address.Wgetneedstobebuiltwithlibcaresforthisoptiontobeavailable.`--dns-servers=ADDRESSES'[libcaresonly]Thegivenaddress(es)overridethestandardnameserveraddresses,e.g.asconguredin/etc/resolv.conf.ADDRESSESmaybespeciedeitherasIPv4orIPv6addresses,comma-separated.Wgetneedstobebuiltwithlibcaresforthisoptiontobeavailable.`-tnumber'`--tries=number'Setnumberoftriestonumber.Specify0or`inf'forinniteretrying.Thedefaultistoretry20times,withtheexceptionoffatalerrorslike\connectionrefused"or\notfound"(404),whicharenotretried.`-Ofile'`--output-document=file'Thedocumentswillnotbewrittentotheappropriateles,butallwillbeconcate-natedtogetherandwrittentole.If`-'isusedasle,documentswillbeprintedtostandardoutput,disablinglinkconversion.(Use`./-'toprinttoaleliterallynamed`-'.)Useof`-O'isnotintendedtomeansimply\usethenameleinsteadoftheoneintheURL;"rather,itisanalogoustoshellredirection:`wget-Ofilehttp://foo'isintendedtoworklike`wget-O-http://foo�file';filewillbetruncatedimmediately,andalldownloadedcontentwillbewrittenthere.Forthisreason,`-N'(fortimestamp-checking)isnotsupportedincombinationwith`-O':sinceleisalwaysnewlycreated,itwillalwayshaveaverynewtimestamp.Awarningwillbeissuedifthiscombinationisused.Similarly,using`-r'or`-p'with`-O'maynotworkasyouexpect:Wgetwon'tjustdownloadtherstletoleandthendownloadtheresttotheirnormalnames:alldownloadedcontentwillbeplacedinle.Thiswasdisabledinversion1.11,buthasbeenreinstated(withawarning)in1.11.2,astherearesomecaseswherethisbehaviorcanactuallyhavesomeuse.Acombinationwith`-nc'isonlyacceptedifthegivenoutputledoesnotexist.Notethatacombinationwith`-k'isonlypermittedwhendownloadingasingledocument,asinthatcaseitwilljustconvertallrelativeURIstoexternalones;`-k'makesnosenseformultipleURIswhenthey'reallbeingdownloadedtoasinglele;`-k'canbeusedonlywhentheoutputisaregularle.`-nc'`--no-clobber'Ifaleisdownloadedmorethanonceinthesamedirectory,Wget'sbehaviorde-pendsonafewoptions,including`-nc'.Incertaincases,thelocallewillbe Chapter2:Invoking7 clobbered,oroverwritten,uponrepeateddownload.Inothercasesitwillbepre-served.WhenrunningWgetwithout`-N',`-nc',`-r',or`-p',downloadingthesameleinthesamedirectorywillresultintheoriginalcopyoflebeingpreservedandthesecondcopybeingnamed`file.1'.Ifthatleisdownloadedyetagain,thethirdcopywillbenamed`file.2',andsoon.(Thisisalsothebehaviorwith`-nd',evenif`-r'or`-p'areineect.)When`-nc'isspecied,thisbehaviorissuppressed,andWgetwillrefusetodownloadnewercopiesof`file'.Therefore,\no-clobber"isactuallyamisnomerinthismode|it'snotclobberingthat'sprevented(asthenumericsuxeswerealreadypreventingclobbering),butratherthemultipleversionsavingthat'sprevented.WhenrunningWgetwith`-r'or`-p',butwithout`-N',`-nd',or`-nc',re-downloadingalewillresultinthenewcopysimplyoverwritingtheold.Adding`-nc'willpreventthisbehavior,insteadcausingtheoriginalversiontobepreservedandanynewercopiesontheservertobeignored.WhenrunningWgetwith`-N',withorwithout`-r'or`-p',thedecisionastowhetherornottodownloadanewercopyofaledependsonthelocalandremotetimestampandsizeofthele(see Chapter5[Time-Stamping],page38 ).`-nc'maynotbespeciedatthesametimeas`-N'.Acombinationwith`-O'/`--output-document'isonlyacceptedifthegivenoutputledoesnotexist.Notethatwhen`-nc'isspecied,leswiththesuxes`.html'or`.htm'willbeloadedfromthelocaldiskandparsedasiftheyhadbeenretrievedfromtheWeb.`--backups=backups'Before(over)writingale,backupanexistinglebyaddinga`.1'sux(`_1'onVMS)tothelename.Suchbackuplesarerotatedto`.2',`.3',andsoon,uptobackups(andlostbeyondthat).`--no-netrc'Donottrytoobtaincredentialsfrom.netrcle.Bydefault.netrcleissearchedforcredentialsincasenonehavebeenpassedoncommandlineandauthenticationisrequired.`-c'`--continue'Continuegettingapartially-downloadedle.ThisisusefulwhenyouwanttonishupadownloadstartedbyapreviousinstanceofWget,orbyanotherprogram.Forinstance:wget-cftp://sunsite.doc.ic.ac.uk/ls-lR.ZIfthereisalenamedls-lR.Zinthecurrentdirectory,Wgetwillassumethatitistherstportionoftheremotele,andwillasktheservertocontinuetheretrievalfromanosetequaltothelengthofthelocalle.Notethatyoudon'tneedtospecifythisoptionifyoujustwantthecurrentinvocationofWgettoretrydownloadingaleshouldtheconnectionbelostmidwaythrough.Thisisthedefaultbehavior.`-c'onlyaectsresumptionofdownloadsstartedpriortothisinvocationofWget,andwhoselocallesarestillsittingaround.Without`-c',thepreviousexamplewouldjustdownloadtheremoteletols-lR.Z.1,leavingthetruncatedls-lR.Zlealone.Ifyouuse`-c'onanon-emptyle,andtheserverdoesnotsupportcontinueddownloading,Wgetwillrestartthedownloadfromscratchandoverwritetheexistingleentirely. Chapter2:Invoking8 BeginningwithWget1.7,ifyouuse`-c'onalewhichisofequalsizeastheoneontheserver,Wgetwillrefusetodownloadtheleandprintanexplanatorymessage.Thesamehappenswhentheleissmallerontheserverthanlocally(presumablybecauseitwaschangedontheserversinceyourlastdownloadattempt)|because\continuing"isnotmeaningful,nodownloadoccurs.Ontheothersideofthecoin,whileusing`-c',anylethat'sbiggerontheserverthanlocallywillbeconsideredanincompletedownloadandonly(length(remote)-length(local))byteswillbedownloadedandtackedontotheendofthelocalle.Thisbehaviorcanbedesirableincertaincases|forinstance,youcanuse`wget-c'todownloadjustthenewportionthat'sbeenappendedtoadatacollectionorlogle.However,iftheleisbiggerontheserverbecauseit'sbeenchanged,asopposedtojustappendedto,you'llendupwithagarbledle.Wgethasnowayofverifyingthatthelocalleisreallyavalidprexoftheremotele.Youneedtobeespeciallycarefulofthiswhenusing`-c'inconjunctionwith`-r',sinceeverylewillbeconsideredasan"incompletedownload"candidate.Anotherinstancewhereyou'llgetagarbledleifyoutrytouse`-c'isifyouhavealamehttpproxythatinsertsa\transferinterrupted"stringintothelocalle.Inthefuturea\rollback"optionmaybeaddedtodealwiththiscase.Notethat`-c'onlyworkswithftpserversandwithhttpserversthatsupporttheRangeheader.`--start-pos=OFFSET'Startdownloadingatzero-basedpositionOFFSET.Osetmaybeexpressedinbytes,kilobyteswiththe`k'sux,ormegabyteswiththe`m'sux,etc.`--start-pos'hashigherprecedenceover`--continue'.When`--start-pos'and`--continue'arebothspecied,wgetwillemitawarningthenproceedasif`--continue'wasabsent.Serversupportforcontinueddownloadisrequired,otherwise`--start-pos'cannothelp.See`-c'fordetails.`--progress=type'Selectthetypeoftheprogressindicatoryouwishtouse.Legalindicatorsare\dot"and\bar".The\bar"indicatorisusedbydefault.Itdrawsanasciiprogressbargraphics(a.k.a\thermometer"display)indicatingthestatusofretrieval.IftheoutputisnotaTTY,the\dot"barwillbeusedbydefault.Use`--progress=dot'toswitchtothe\dot"display.Ittracestheretrievalbyprintingdotsonthescreen,eachdotrepresentingaxedamountofdownloadeddata.Theprogresstypecanalsotakeoneormoreparameters.Theparametersvarybasedonthetypeselected.Parameterstotypearepassedbyappendingthemtothetypesperatedbyacolon(:)likethis:`--progress=type:parameter1:parameter2'.Whenusingthedottedretrieval,youmaysetthestylebyspecifyingthetypeas`dot:style'.Dierentstylesassigndierentmeaningtoonedot.Withthedefaultstyleeachdotrepresents1K,therearetendotsinaclusterand50dotsinaline.Thebinarystylehasamore\computer"-likeorientation|8Kdots,16-dotsclustersand48dotsperline(whichmakesfor384Klines).Themegastyleissuitablefordownloadinglargeles|eachdotrepresents64Kretrieved,thereareeightdotsinacluster,and48dotsoneachline(soeachlinecontains3M).Ifmegaisnotenough Chapter2:Invoking9 thenyoucanusethegigastyle|eachdotrepresents1Mretrieved,thereareeightdotsinacluster,and32dotsoneachline(soeachlinecontains32M).With`--progress=bar',therearecurrentlytwopossibleparameters,forceandnoscroll.WhentheoutputisnotaTTY,theprogressbaralwaysfallsbackto\dot",evenif`--progress=bar'waspassedtoWgetduringinvocation.Thisbehaviourcanbeoverriddenandthe\bar"outputforcedbyusingthe\force"parameteras`--progress=bar:force'.Bydefault,the`bar'styleprogressbarscrollthenameofthelefromlefttorightforthelebeingdownloadedifthelenameexceedsthemaximumlengthallottedforitsdisplay.Incertaincases,suchaswith`--progress=bar:force',onemaynotwantthescrollinglenameintheprogressbar.Bypassingthe\noscroll"parameter,Wgetcanbeforcedtodisplayasmuchofthelenameaspossiblewithoutscrollingthroughit.Notethatyoucansetthedefaultstyleusingtheprogresscommandin.wgetrc.Thatsettingmaybeoverriddenfromthecommandline.Forexample,toforcethebaroutputwithoutscrolling,use`--progress=bar:force:noscroll'.`--show-progress'Forcewgettodisplaytheprogressbarinanyverbosity.Bydefault,wgetonlydisplaystheprogressbarinverbosemode.Onemayhowever,wantwgettodisplaytheprogressbaronscreeninconjunctionwithanyotherverbositymodeslike`--no-verbose'or`--quiet'.Thisisoftenadesiredapropertywheninvokingwgettodownloadseveralsmall/largeles.Insuchacase,wgetcouldsimplybeinvokedwiththisparametertogetamuchcleaneroutputonthescreen.Thisoptionwillalsoforcetheprogressbartobeprintedtostderrwhenusedalongsidethe`--logfile'option.`-N'`--timestamping'Turnontime-stamping.See Chapter5[Time-Stamping],page38 ,fordetails.`--no-if-modified-since'DonotsendIf-Modied-Sinceheaderin`-N'mode.SendpreliminaryHEADrequestinstead.Thishasonlyeectin`-N'mode.`--no-use-server-timestamps'Don'tsetthelocalle'stimestampbytheoneontheserver.Bydefault,whenaleisdownloaded,itstimestampsaresettomatchthosefromtheremotele.Thisallowstheuseof`--timestamping'onsubsequentinvocationsofwget.However,itissometimesusefultobasethelocalle'stimestamponwhenitwasactuallydownloaded;forthatpurpose,the`--no-use-server-timestamps'optionhasbeenprovided.`-S'`--server-response'Printtheheaderssentbyhttpserversandresponsessentbyftpservers.`--spider'Wheninvokedwiththisoption,WgetwillbehaveasaWebspider,whichmeansthatitwillnotdownloadthepages,justcheckthattheyarethere.Forexample,youcanuseWgettocheckyourbookmarks:wget--spider--force-html-ibookmarks.html Chapter2:Invoking10 ThisfeatureneedsmuchmoreworkforWgettogetclosetothefunctionalityofrealwebspiders.`-Tseconds'`--timeout=seconds'Setthenetworktimeouttosecondsseconds.Thisisequivalenttospecifying`--dns-timeout',`--connect-timeout',and`--read-timeout',allatthesametime.Wheninteractingwiththenetwork,Wgetcancheckfortimeoutandaborttheoperationifittakestoolong.Thispreventsanomalieslikehangingreadsandinniteconnects.Theonlytimeoutenabledbydefaultisa900-secondreadtimeout.Settingatimeoutto0disablesitaltogether.Unlessyouknowwhatyouaredoing,itisbestnottochangethedefaulttimeoutsettings.Alltimeout-relatedoptionsacceptdecimalvalues,aswellassubsecondvalues.Forexample,`0.1'secondsisalegal(thoughunwise)choiceoftimeout.Subsecondtimeoutsareusefulforcheckingserverresponsetimesorfortestingnetworklatency.`--dns-timeout=seconds'SettheDNSlookuptimeouttosecondsseconds.DNSlookupsthatdon'tcompletewithinthespeciedtimewillfail.Bydefault,thereisnotimeoutonDNSlookups,otherthanthatimplementedbysystemlibraries.`--connect-timeout=seconds'Settheconnecttimeouttosecondsseconds.TCPconnectionsthattakelongertoestablishwillbeaborted.Bydefault,thereisnoconnecttimeout,otherthanthatimplementedbysystemlibraries.`--read-timeout=seconds'Settheread(andwrite)timeouttosecondsseconds.The\time"ofthistimeoutreferstoidletime:if,atanypointinthedownload,nodataisreceivedformorethanthespeciednumberofseconds,readingfailsandthedownloadisrestarted.Thisoptiondoesnotdirectlyaectthedurationoftheentiredownload.Ofcourse,theremoteservermaychoosetoterminatetheconnectionsoonerthanthisoptionrequires.Thedefaultreadtimeoutis900seconds.`--limit-rate=amount'Limitthedownloadspeedtoamountbytespersecond.Amountmaybeexpressedinbytes,kilobyteswiththe`k'sux,ormegabyteswiththe`m'sux.Forexample,`--limit-rate=20k'willlimittheretrievalrateto20KB/s.Thisisusefulwhen,forwhateverreason,youdon'twantWgettoconsumetheentireavailablebandwidth.Thisoptionallowstheuseofdecimalnumbers,usuallyinconjunctionwithpowersuxes;forexample,`--limit-rate=2.5k'isalegalvalue.NotethatWgetimplementsthelimitingbysleepingtheappropriateamountoftimeafteranetworkreadthattooklesstimethanspeciedbytherate.EventuallythisstrategycausestheTCPtransfertoslowdowntoapproximatelythespeciedrate.However,itmaytakesometimeforthisbalancetobeachieved,sodon'tbesurprisediflimitingtheratedoesn'tworkwellwithverysmallles.`-wseconds'`--wait=seconds'Waitthespeciednumberofsecondsbetweentheretrievals.Useofthisoptionisrecommended,asitlightenstheserverloadbymakingtherequestslessfrequent.Insteadofinseconds,thetimecanbespeciedinminutesusingthemsux,inhoursusinghsux,orindaysusingdsux. Chapter2:Invoking11 Specifyingalargevalueforthisoptionisusefulifthenetworkorthedestinationhostisdown,sothatWgetcanwaitlongenoughtoreasonablyexpectthenetworkerrortobexedbeforetheretry.Thewaitingintervalspeciedbythisfunctionisin uencedby--random-wait,whichsee.`--waitretry=seconds'Ifyoudon'twantWgettowaitbetweeneveryretrieval,butonlybetweenretriesoffaileddownloads,youcanusethisoption.Wgetwilluselinearbacko,waiting1secondaftertherstfailureonagivenle,thenwaiting2secondsafterthesecondfailureonthatle,uptothemaximumnumberofsecondsyouspecify.Bydefault,Wgetwillassumeavalueof10seconds.`--random-wait'SomewebsitesmayperformloganalysistoidentifyretrievalprogramssuchasWgetbylookingforstatisticallysignicantsimilaritiesinthetimebetweenrequests.Thisoptioncausesthetimebetweenrequeststovarybetween0.5and1.5*waitseconds,wherewaitwasspeciedusingthe`--wait'option,inordertomaskWget'spresencefromsuchanalysis.A2001articleinapublicationdevotedtodevelopmentonapopularconsumerplatformprovidedcodetoperformthisanalysisonthe y.ItsauthorsuggestedblockingattheclassCaddressleveltoensureautomatedretrievalprogramswereblockeddespitechangingDHCP-suppliedaddresses.The`--random-wait'optionwasinspiredbythisill-advisedrecommendationtoblockmanyunrelatedusersfromawebsiteduetotheactionsofone.`--no-proxy'Don'tuseproxies,eveniftheappropriate*_proxyenvironmentvariableisdened.See Section8.1[Proxies],page54 ,formoreinformationabouttheuseofproxieswithWget.`-Qquota'`--quota=quota'Specifydownloadquotaforautomaticretrievals.Thevaluecanbespeciedinbytes(default),kilobytes(with`k'sux),ormegabytes(with`m'sux).Notethatquotawillneveraectdownloadingasinglele.Soifyouspecify`wget-Q10khttps://example.com/ls-lR.gz',allofthels-lR.gzwillbedownloaded.Thesamegoesevenwhenseveralurlsarespeciedonthecommand-line.However,quotaisrespectedwhenretrievingeitherrecursively,orfromaninputle.Thusyoumaysafelytype`wget-Q2m-isites'|downloadwillbeabortedwhenthequotaisexceeded.Settingquotato0orto`inf'unlimitsthedownloadquota.`--no-dns-cache'TurnocachingofDNSlookups.Normally,WgetrememberstheIPaddressesitlookedupfromDNSsoitdoesn'thavetorepeatedlycontacttheDNSserverforthesame(typicallysmall)setofhostsitretrievesfrom.Thiscacheexistsinmemoryonly;anewWgetrunwillcontactDNSagain.However,ithasbeenreportedthatinsomesituationsitisnotdesirabletocachehostnames,evenforthedurationofashort-runningapplicationlikeWget.WiththisoptionWgetissuesanewDNSlookup(moreprecisely,anewcalltogethostbynameorgetaddrinfo)eachtimeitmakesanewconnection.Pleasenotethatthisoptionwillnotaectcachingthatmightbeperformedbytheresolvinglibraryorbyanexternalcachinglayer,suchasNSCD. Chapter2:Invoking12 Ifyoudon'tunderstandexactlywhatthisoptiondoes,youprobablywon'tneedit.`--restrict-file-names=modes'ChangewhichcharactersfoundinremoteURLsmustbeescapedduringgenerationoflocallenames.Charactersthatarerestrictedbythisoptionareescaped,i.e.replacedwith`%HH',where`HH'isthehexadecimalnumberthatcorrespondstotherestrictedcharacter.Thisoptionmayalsobeusedtoforceallalphabeticalcasestobeeitherlower-oruppercase.Bydefault,Wgetescapesthecharactersthatarenotvalidorsafeaspartoflenamesonyouroperatingsystem,aswellascontrolcharactersthataretypicallyunprintable.Thisoptionisusefulforchangingthesedefaults,perhapsbecauseyouaredownloadingtoanon-nativepartition,orbecauseyouwanttodisableescapingofthecontrolcharacters,oryouwanttofurtherrestrictcharacterstoonlythoseintheasciirangeofvalues.Themodesareacomma-separatedsetoftextvalues.Theacceptablevaluesare`unix',`windows',`nocontrol',`ascii',`lowercase',and`uppercase'.Thevalues`unix'and`windows'aremutuallyexclusive(onewilloverridetheother),asare`lowercase'and`uppercase'.Thoselastarespecialcases,astheydonotchangethesetofcharactersthatwouldbeescaped,butratherforcelocallepathstobeconvertedeithertolower-oruppercase.When\unix"isspecied,Wgetescapesthecharacter`/'andthecontrolcharactersintheranges0{31and128{159.ThisisthedefaultonUnix-likeoperatingsystems.When\windows"isgiven,Wgetescapesthecharacters`\',`|',`/',`:',`?',`"',`*',`',`&#x]TJ/;ཕ ;.9;‘ ;&#xTf 5;&#x.728;&#x 0 T; [0;',andthecontrolcharactersintheranges0{31and128{159.Inaddi-tiontothis,WgetinWindowsmodeuses`+'insteadof`:'toseparatehostandportinlocallenames,anduses`@'insteadof`?'toseparatethequerypor-tionofthelenamefromtherest.Therefore,aURLthatwouldbesavedas`www.xemacs.org:4300/search.pl?input=blah'inUnixmodewouldbesavedas`www.xemacs.org+4300/search.pl@input=blah'inWindowsmode.ThismodeisthedefaultonWindows.Ifyouspecify`nocontrol',thentheescapingofthecontrolcharactersisalsoswitchedo.ThisoptionmaymakesensewhenyouaredownloadingURLswhosenamescontainUTF-8characters,onasystemwhichcansaveanddisplaylenamesinUTF-8(somepossiblebytevaluesusedinUTF-8bytesequencesfallintherangeofvaluesdesignatedbyWgetas\controls").The`ascii'modeisusedtospecifythatanybyteswhosevaluesareoutsidetherangeofasciicharacters(thatis,greaterthan127)shallbeescaped.Thiscanbeusefulwhensavinglenameswhoseencodingdoesnotmatchtheoneusedlocally.`-4'`--inet4-only'`-6'`--inet6-only'ForceconnectingtoIPv4orIPv6addresses.With`--inet4-only'or`-4',WgetwillonlyconnecttoIPv4hosts,ignoringAAAArecordsinDNS,andrefusingtoconnecttoIPv6addressesspeciedinURLs.Conversely,with`--inet6-only'or`-6',WgetwillonlyconnecttoIPv6hostsandignoreArecordsandIPv4addresses.Neitheroptionsshouldbeneedednormally.Bydefault,anIPv6-awareWgetwillusetheaddressfamilyspeciedbythehost'sDNSrecord.IftheDNSrespondswithbothIPv4andIPv6addresses,Wgetwilltrytheminsequenceuntilitndsoneitcanconnectto.(Alsosee--prefer-familyoptiondescribedbelow.) Chapter2:Invoking13 TheseoptionscanbeusedtodeliberatelyforcetheuseofIPv4orIPv6addressfamiliesondualfamilysystems,usuallytoaiddebuggingortodealwithbrokennetworkconguration.Onlyoneof`--inet6-only'and`--inet4-only'maybespeciedatthesametime.NeitheroptionisavailableinWgetcompiledwithoutIPv6support.`--prefer-family=none/IPv4/IPv6'Whengivenachoiceofseveraladdresses,connecttotheaddresseswithspeciedaddressfamilyrst.TheaddressorderreturnedbyDNSisusedwithoutchangebydefault.ThisavoidsspuriouserrorsandconnectattemptswhenaccessinghoststhatresolvetobothIPv6andIPv4addressesfromIPv4networks.Forexample,`www.kame.net'resolvesto`2001:200:0:8002:203:47ff:fea5:3085'andto`203.178.141.194'.WhenthepreferredfamilyisIPv4,theIPv4addressisusedrst;whenthepreferredfamilyisIPv6,theIPv6addressisusedrst;ifthespeciedvalueisnone,theaddressorderreturnedbyDNSisusedwithoutchange.Unlike`-4'and`-6',thisoptiondoesn'tinhibitaccesstoanyaddressfamily,itonlychangestheorderinwhichtheaddressesareaccessed.Alsonotethatthereorderingperformedbythisoptionisstable|itdoesn'taectorderofaddressesofthesamefamily.Thatis,therelativeorderofallIPv4addressesandofallIPv6addressesremainsintactinallcases.`--retry-connrefused'Consider\connectionrefused"atransienterrorandtryagain.NormallyWgetgivesuponaURLwhenitisunabletoconnecttothesitebecausefailuretoconnectistakenasasignthattheserverisnotrunningatallandthatretrieswouldnothelp.Thisoptionisformirroringunreliablesiteswhoseserverstendtodisappearforshortperiodsoftime.`--user=user'`--password=password'Specifytheusernameuserandpasswordpasswordforbothftpandhttpleretrieval.Theseparameterscanbeoverriddenusingthe`--ftp-user'and`--ftp-password'optionsforftpconnectionsandthe`--http-user'and`--http-password'optionsforhttpconnections.`--ask-password'Promptforapasswordforeachconnectionestablished.Cannotbespeciedwhen`--password'isbeingused,becausetheyaremutuallyexclusive.`--use-askpass=command'Promptforauserandpasswordusingthespeciedcommand.IfnocommandisspeciedthenthecommandintheenvironmentvariableWGET ASKPASSisused.IfWGET ASKPASSisnotsetthenthecommandintheenvironmentvariableSSH ASKPASSisused.Youcansetthedefaultcommandforuse-askpassinthe.wgetrc.Thatsettingmaybeoverriddenfromthecommandline.`--no-iri'TurnointernationalizedURI(IRI)support.Use`--iri'toturniton.IRIsupportisactivatedbydefault.YoucansetthedefaultstateofIRIsupportusingtheiricommandin.wgetrc.Thatsettingmaybeoverriddenfromthecommandline. Chapter2:Invoking14 `--local-encoding=encoding'ForceWgettouseencodingasthedefaultsystemencoding.ThataectshowWgetconvertsURLsspeciedasargumentsfromlocaletoutf-8forIRIsupport.Wgetusethefunctionnl_langinfo()andthentheCHARSETenvironmentvariabletogetthelocale.Ifitfails,asciiisused.Youcansetthedefaultlocalencodingusingthelocal_encodingcommandin.wgetrc.Thatsettingmaybeoverriddenfromthecommandline.`--remote-encoding=encoding'ForceWgettouseencodingasthedefaultremoteserverencoding.ThataectshowWgetconvertsURIsfoundinlesfromremoteencodingtoutf-8duringarecursivefetch.ThisoptionsisonlyusefulforIRIsupport,fortheinterpretationofnon-asciicharacters.ForHTTP,remoteencodingcanbefoundinHTTPContent-TypeheaderandinHTMLContent-Typehttp-equivmetatag.Youcansetthedefaultencodingusingtheremoteencodingcommandin.wgetrc.Thatsettingmaybeoverriddenfromthecommandline.`--unlink'ForceWgettounlinkleinsteadofclobberingexistingle.Thisoptionisusefulfordownloadingtothedirectorywithhardlinks.2.6DirectoryOptions`-nd'`--no-directories'Donotcreateahierarchyofdirectorieswhenretrievingrecursively.Withthisoptionturnedon,allleswillgetsavedtothecurrentdirectory,withoutclobbering(ifanameshowsupmorethanonce,thelenameswillgetextensions`.n').`-x'`--force-directories'Theoppositeof`-nd'|createahierarchyofdirectories,evenifonewouldnothavebeencreatedotherwise.E.g.`wget-xhttp://fly.srk.fer.hr/robots.txt'willsavethedownloadedletofly.srk.fer.hr/robots.txt.`-nH'`--no-host-directories'Disablegenerationofhost-prexeddirectories.Bydefault,invokingWgetwith`-rhttp://fly.srk.fer.hr/'willcreateastructureofdirectoriesbeginningwithfly.srk.fer.hr/.Thisoptiondisablessuchbehavior.`--protocol-directories'Usetheprotocolnameasadirectorycomponentoflocallenames.Forexample,withthisoption,`wget-rhttp://host'willsaveto`http/host/...'ratherthanjustto`host/...'.`--cut-dirs=number'Ignorenumberdirectorycomponents.Thisisusefulforgettingane-grainedcontroloverthedirectorywhererecursiveretrievalwillbesaved.Take,forexample,thedirectoryat`ftp://ftp.xemacs.org/pub/xemacs/'.Ifyouretrieveitwith`-r',itwillbesavedlocallyunderftp.xemacs.org/pub/xemacs/.Whilethe`-nH'optioncanremovetheftp.xemacs.org/part,youarestillstuckwithpub/xemacs.Thisiswhere`--cut-dirs'comesinhandy;itmakesWget Chapter2:Invoking15 not\see"numberremotedirectorycomponents.Hereareseveralexamplesofhow`--cut-dirs'optionworks.Nooptions�-ftp.xemacs.org/pub/xemacs/-nH�-pub/xemacs/-nH--cut-dirs=1�-xemacs/-nH--cut-dirs=2�-.--cut-dirs=1�-ftp.xemacs.org/xemacs/...Ifyoujustwanttogetridofthedirectorystructure,thisoptionissimilartoacombinationof`-nd'and`-P'.However,unlike`-nd',`--cut-dirs'doesnotlosewithsubdirectories|forinstance,with`-nH--cut-dirs=1',abeta/subdirectorywillbeplacedtoxemacs/beta,asonewouldexpect.`-Pprefix'`--directory-prefix=prefix'Setdirectoryprextoprex.Thedirectoryprexisthedirectorywhereallotherlesandsubdirectorieswillbesavedto,i.e.thetopoftheretrievaltree.Thedefaultis`.'(thecurrentdirectory).2.7HTTPOptions`--default-page=name'Usenameasthedefaultlenamewhenitisn'tknown(i.e.,forURLsthatendinaslash),insteadofindex.html.`-E'`--adjust-extension'Ifaleoftype`application/xhtml+xml'or`text/html'isdownloadedandtheURLdoesnotendwiththeregexp`\.[Hh][Tt][Mm][Ll]?',thisoptionwillcausethesux`.html'tobeappendedtothelocallename.Thisisuseful,forin-stance,whenyou'remirroringaremotesitethatuses`.asp'pages,butyouwantthemirroredpagestobeviewableonyourstockApacheserver.Anothergooduseforthisiswhenyou'redownloadingCGI-generatedmaterials.AURLlike`http://site.com/article.cgi?25'willbesavedasarticle.cgi?25.html.Notethatlenameschangedinthiswaywillbere-downloadedeverytimeyoure-mirrorasite,becauseWgetcan'ttellthatthelocalX.htmllecorrespondstoremoteURL`X'(sinceitdoesn'tyetknowthattheURLproducesoutputoftype`text/html'or`application/xhtml+xml'.Asofversion1.12,Wgetwillalsoensurethatanydownloadedlesoftype`text/css'endinthesux`.css',andtheoptionwasrenamedfrom`--html-extension',tobetterre ectitsnewbehavior.Theoldoptionnameisstillacceptable,butshouldnowbeconsidereddeprecated.Asofversion1.19.2,WgetwillalsoensurethatanydownloadedleswithaContent-Encodingof`br',`compress',`deflate'or`gzip'endinthesux`.br',`.Z',`.zlib'and`.gz'respectively.Atsomepointinthefuture,thisoptionmaywellbeexpandedtoincludesuxesforothertypesofcontent,includingcontenttypesthatarenotparsedbyWget. Chapter2:Invoking16 `--http-user=user'`--http-password=password'Specifytheusernameuserandpasswordpasswordonanhttpserver.Accordingtothetypeofthechallenge,Wgetwillencodethemusingeitherthebasic(insecure),thedigest,ortheWindowsNTLMauthenticationscheme.Anotherwaytospecifyusernameandpasswordisintheurlitself(see Section2.1[URLFormat],page2 ).Eithermethodrevealsyourpasswordtoanyonewhobotherstorunps.Topreventthepasswordsfrombeingseen,usethe`--use-askpass'orstorethemin.wgetrcor.netrc,andmakesuretoprotectthoselesfromotheruserswithchmod.Ifthepasswordsarereallyimportant,donotleavethemlyinginthoseleseither|editthelesanddeletethemafterWgethasstartedthedownload.See Section9.2[SecurityConsiderations],page59 ,formoreinformationaboutse-curityissueswithWget.`--no-http-keep-alive'Turnothe\keep-alive"featureforHTTPdownloads.Normally,Wgetaskstheservertokeeptheconnectionopensothat,whenyoudownloadmorethanonedocumentfromthesameserver,theygettransferredoverthesameTCPconnection.Thissavestimeandatthesametimereducestheloadontheserver.Thisoptionisusefulwhen,forsomereason,persistent(keep-alive)connectionsdon'tworkforyou,forexampleduetoaserverbugorduetotheinabilityofserver-sidescriptstocopewiththeconnections.`--no-cache'Disableserver-sidecache.Inthiscase,Wgetwillsendtheremoteserveranap-propriatedirective(`Pragma:no-cache')togetthelefromtheremoteservice,ratherthanreturningthecachedversion.Thisisespeciallyusefulforretrievingand ushingout-of-datedocumentsonproxyservers.Cachingisallowedbydefault.`--no-cookies'Disabletheuseofcookies.Cookiesareamechanismformaintainingserver-sidestate.TheserversendstheclientacookieusingtheSet-Cookieheader,andtheclientrespondswiththesamecookieuponfurtherrequests.Sincecookiesallowtheserverownerstokeeptrackofvisitorsandforsitestoexchangethisinformation,someconsiderthemabreachofprivacy.Thedefaultistousecookies;however,storingcookiesisnotonbydefault.`--load-cookiesfile'LoadcookiesfromlebeforetherstHTTPretrieval.leisatextualleintheformatoriginallyusedbyNetscape'scookies.txtle.Youwilltypicallyusethisoptionwhenmirroringsitesthatrequirethatyoubeloggedintoaccesssomeoralloftheircontent.Theloginprocesstypicallyworksbythewebserverissuinganhttpcookieuponreceivingandverifyingyourcredentials.Thecookieisthenresentbythebrowserwhenaccessingthatpartofthesite,andsoprovesyouridentity.MirroringsuchasiterequiresWgettosendthesamecookiesyourbrowsersendswhencommunicatingwiththesite.Thisisachievedby`--load-cookies'|simplypointWgettothelocationofthecookies.txtle,anditwillsendthesamecookiesyourbrowserwouldsendinthesamesituation.Dierentbrowserskeeptextualcookielesindierentlocations:Netscape4.x.Thecookiesarein~/.netscape/cookies.txt. Chapter2:Invoking17 MozillaandNetscape6.x.Mozilla'scookieleisalsonamedcookies.txt,locatedsomewhereunder~/.mozilla,inthedirectoryofyourprole.Thefullpathusu-allyendsuplookingsomewhatlike~/.mozilla/default/some-weird-string/cookies.txt.InternetExplorer.YoucanproduceacookieleWgetcanusebyusingtheFilemenu,ImportandExport,ExportCookies.ThishasbeentestedwithInternetExplorer5;itisnotguaranteedtoworkwithearlierversions.Otherbrowsers.Ifyouareusingadierentbrowsertocreateyourcookies,`--load-cookies'willonlyworkifyoucanlocateorproduceacookieleintheNetscapeformatthatWgetexpects.Ifyoucannotuse`--load-cookies',theremightstillbeanalternative.Ifyourbrowsersupportsa\cookiemanager",youcanuseittoviewthecookiesusedwhenaccessingthesiteyou'remirroring.Writedownthenameandvalueofthecookie,andmanuallyinstructWgettosendthosecookies,bypassingthe\ocial"cookiesupport:wget--no-cookies--header"Cookie:name=value"`--save-cookiesfile'Savecookiestolebeforeexiting.Thiswillnotsavecookiesthathaveex-piredorthathavenoexpirytime(so-called\sessioncookies"),butalsosee`--keep-session-cookies'.`--keep-session-cookies'Whenspecied,causes`--save-cookies'toalsosavesessioncookies.Sessioncook-iesarenormallynotsavedbecausetheyaremeanttobekeptinmemoryandfor-gottenwhenyouexitthebrowser.Savingthemisusefulonsitesthatrequireyoutologinortovisitthehomepagebeforeyoucanaccesssomepages.Withthisoption,multipleWgetrunsareconsideredasinglebrowsersessionasfarasthesiteisconcerned.Sincethecookieleformatdoesnotnormallycarrysessioncookies,Wgetmarksthemwithanexpirytimestampof0.Wget's`--load-cookies'recognizesthoseassessioncookies,butitmightconfuseotherbrowsers.Alsonotethatcookiessoloadedwillbetreatedasothersessioncookies,whichmeansthatifyouwant`--save-cookies'topreservethemagain,youmustuse`--keep-session-cookies'again.`--ignore-length'Unfortunately,somehttpservers(cgiprograms,tobemoreprecise)sendoutbogusContent-Lengthheaders,whichmakesWgetgowild,asitthinksnotallthedocumentwasretrieved.YoucanspotthissyndromeifWgetretriesgettingthesamedocumentagainandagain,eachtimeclaimingthatthe(otherwisenormal)connectionhasclosedontheverysamebyte.Withthisoption,WgetwillignoretheContent-Lengthheader|asifitneverex-isted.`--header=header-line'Sendheader-linealongwiththerestoftheheadersineachhttprequest.Thesuppliedheaderissentas-is,whichmeansitmustcontainnameandvalueseparatedbycolon,andmustnotcontainnewlines. Chapter2:Invoking18 Youmaydenemorethanoneadditionalheaderbyspecifying`--header'morethanonce.wget--header='Accept-Charset:iso-8859-2'\--header='Accept-Language:hr'\http://fly.srk.fer.hr/Specicationofanemptystringastheheadervaluewillclearallprevioususer-denedheaders.AsofWget1.10,thisoptioncanbeusedtooverrideheadersotherwisegeneratedautomatically.ThisexampleinstructsWgettoconnecttolocalhost,buttospecify`foo.bar'intheHostheader:wget--header="Host:foo.bar"http://localhost/InversionsofWgetpriorto1.10suchuseof`--header'causedsendingofduplicateheaders.`--compression=type'Choosethetypeofcompressiontobeused.Legalvaluesare`auto',`gzip'and`none'.If`auto'or`gzip'arespecied,Wgetaskstheservertocompresstheleusingthegzipcompressionformat.IftheservercompressestheleandrespondswiththeContent-Encodingheadereldsetappropriately,thelewillbedecompressedautomatically.If`none'isspecied,wgetwillnotasktheservertocompresstheleandwillnotdecompressanyserverresponses.Thisisthedefault.Compressionsupportiscurrentlyexperimental.Incaseitisturnedon,pleasereportanybugstobug-wget@gnu.org.`--max-redirect=number'Speciesthemaximumnumberofredirectionstofollowforaresource.Thedefaultis20,whichisusuallyfarmorethannecessary.However,onthoseoccasionswhereyouwanttoallowmore(orfewer),thisistheoptiontouse.`--proxy-user=user'`--proxy-password=password'Specifytheusernameuserandpasswordpasswordforauthenticationonaproxyserver.Wgetwillencodethemusingthebasicauthenticationscheme.Securityconsiderationssimilartothosewith`--http-password'pertainhereaswell.`--referer=url'Include`Referer:url'headerinHTTPrequest.Usefulforretrievingdocumentswithserver-sideprocessingthatassumetheyarealwaysbeingretrievedbyinteractivewebbrowsersandonlycomeoutproperlywhenRefererissettooneofthepagesthatpointtothem.`--save-headers'Savetheheaderssentbythehttpservertothele,precedingtheactualcontents,withanemptylineastheseparator.`-Uagent-string'`--user-agent=agent-string'Identifyasagent-stringtothehttpserver.ThehttpprotocolallowstheclientstoidentifythemselvesusingaUser-Agentheadereld.Thisenablesdistinguishingthewwwsoftware,usuallyforstatis- Chapter2:Invoking19 ticalpurposesorfortracingofprotocolviolations.Wgetnormallyidentiesas`Wget/version',versionbeingthecurrentversionnumberofWget.However,somesiteshavebeenknowntoimposethepolicyoftailoringtheoutputaccordingtotheUser-Agent-suppliedinformation.Whilethisisnotsuchabadideaintheory,ithasbeenabusedbyserversdenyinginformationtoclientsotherthan(historically)Netscapeor,morefrequently,MicrosoftInternetExplorer.ThisoptionallowsyoutochangetheUser-AgentlineissuedbyWget.Useofthisoptionisdiscouraged,unlessyoureallyknowwhatyouaredoing.Specifyingemptyuseragentwith`--user-agent=""'instructsWgetnottosendtheUser-Agentheaderinhttprequests.`--post-data=string'`--post-file=file'UsePOSTasthemethodforallHTTPrequestsandsendthespecieddataintherequestbody.`--post-data'sendsstringasdata,whereas`--post-file'sendsthecontentsofle.Otherthanthat,theyworkinexactlythesameway.Inparticular,theybothexpectcontentoftheformkey1=value1&key2=value2,withpercent-encodingforspecialcharacters;theonlydierenceisthatoneexpectsitscontentasacommand-lineparameterandtheotheracceptsitscontentfromale.Inparticu-lar,`--post-file'isnotfortransmittinglesasformattachments:thosemustap-pearaskey=valuedata(withappropriatepercent-coding)justlikeeverythingelse.Wgetdoesnotcurrentlysupportmultipart/form-datafortransmittingPOSTdata;onlyapplication/x-www-form-urlencoded.Onlyoneof`--post-data'and`--post-file'shouldbespecied.Pleasenotethatwgetdoesnotrequirethecontenttobeoftheformkey1=value1&key2=value2,andneitherdoesittestforit.Wgetwillsimplytransmitwhateverdataisprovidedtoit.MostservershoweverexpectthePOSTdatatobeintheaboveformatwhenprocessingHTMLForms.WhensendingaPOSTrequestusingthe`--post-file'option,WgettreatstheleasabinaryleandwillsendeverycharacterinthePOSTrequestwithoutstrippingtrailingnewlineorformfeedcharacters.Anyothercontrolcharactersinthetextwillalsobesentas-isinthePOSTrequest.PleasebeawarethatWgetneedstoknowthesizeofthePOSTdatainadvance.Thereforetheargumentto--post-filemustbearegularle;specifyingaFIFOorsomethinglike/dev/stdinwon'twork.It'snotquiteclearhowtoworkaroundthislimitationinherentinHTTP/1.0.AlthoughHTTP/1.1introduceschunkedtransferthatdoesn'trequireknowingtherequestlengthinadvance,aclientcan'tusechunkedunlessitknowsit'stalkingtoanHTTP/1.1server.Anditcan'tknowthatuntilitreceivesaresponse,whichinturnrequirestherequesttohavebeencompleted{achicken-and-eggproblem.Note:Asofversion1.15ifWgetisredirectedafterthePOSTrequestiscompleted,itsbehaviourwilldependontheresponsecodereturnedbytheserver.Incaseofa301MovedPermanently,302MovedTemporarilyor307TemporaryRedirect,Wgetwill,inaccordancewithRFC2616,continuetosendaPOSTrequest.IncaseaserverwantstheclienttochangetheRequestmethoduponredirection,itshouldsenda303SeeOtherresponsecode.ThisexampleshowshowtologintoaserverusingPOSTandthenproceedtodownloadthedesiredpages,presumablyonlyaccessibletoauthorizedusers: Chapter2:Invoking20 #Logintotheserver.Thiscanbedoneonlyonce.wget--save-cookiescookies.txt\--post-data'user=foo&password=bar'\http://example.com/auth.php#Nowgrabthepageorpageswecareabout.wget--load-cookiescookies.txt\-phttp://example.com/interesting/article.phpIftheserverisusingsessioncookiestotrackuserauthentication,theabovewillnotworkbecause`--save-cookies'willnotsavethem(andneitherwillbrowsers)andthecookies.txtlewillbeempty.Inthatcaseuse`--keep-session-cookies'alongwith`--save-cookies'toforcesavingofsessioncookies.`--method=HTTP-Method'ForthepurposeofRESTfulscripting,WgetallowssendingofotherHTTPMethodswithouttheneedtoexplicitlysetthemusing`--header=Header-Line'.Wgetwillusewhateverstringispassedtoitafter`--method'astheHTTPMethodtotheserver.`--body-data=Data-String'`--body-file=Data-File'MustbesetwhenadditionaldataneedstobesenttotheserveralongwiththeMethodspeciedusing`--method'.`--body-data'sendsstringasdata,whereas`--body-file'sendsthecontentsofle.Otherthanthat,theyworkinexactlythesameway.Currently,`--body-file'isnotfortransmittinglesasawhole.Wgetdoesnotcurrentlysupportmultipart/form-datafortransmittingdata;onlyapplication/x-www-form-urlencoded.Inthefuture,thismaybechangedsothatwgetsendsthe`--body-file'asacompleteleinsteadofsendingitscontentstotheserver.PleasebeawarethatWgetneedstoknowthecontentsofBODYDatainadvance,andhencetheargumentto`--body-file'shouldbearegularle.See`--post-file'foramoredetailedexplanation.Onlyoneof`--body-data'and`--body-file'shouldbespecied.IfWgetisredirectedaftertherequestiscompleted,WgetwillsuspendthecurrentmethodandsendaGETrequesttilltheredirectioniscompleted.Thisistrueforallredirectionresponsecodesexcept307TemporaryRedirectwhichisusedtoexplicitlyspecifythattherequestmethodshouldnotchange.AnotherexceptioniswhenthemethodissettoPOST,inwhichcasetheredirectionrulesspeciedunder`--post-data'arefollowed.`--content-disposition'Ifthisissettoon,experimental(notfully-functional)supportforContent-Dispositionheadersisenabled.Thiscancurrentlyresultinextraround-tripstotheserverforaHEADrequest,andisknowntosuerfromafewbugs,whichiswhyitisnotcurrentlyenabledbydefault.Thisoptionisusefulforsomele-downloadingCGIprogramsthatuseContent-Dispositionheaderstodescribewhatthenameofadownloadedleshouldbe.Whencombinedwith`--metalink-over-http'and`--trust-server-names',a`Content-Type:application/metalink4+xml'leisnamedusingtheContent-Dispositionlenameeld,ifavailable. Chapter2:Invoking21 `--content-on-error'Ifthisissettoon,wgetwillnotskipthecontentwhentheserverrespondswithahttpstatuscodethatindicateserror.`--trust-server-names'Ifthisisset,onaredirect,thelocallenamewillbebasedontheredirectionURL.BydefaultthelocallenameisbasedontheoriginalURL.WhendoingrecursiveretrievingthiscanbehelpfulbecauseinmanywebsitesredirectedURLscorrespondtoanunderlyinglestructure,whilelinkURLsdonot.`--auth-no-challenge'Ifthisoptionisgiven,WgetwillsendBasicHTTPauthenticationinformation(plaintextusernameandpassword)forallrequests,justlikeWget1.10.2andpriordidbydefault.Useofthisoptionisnotrecommended,andisintendedonlytosupportsomefewobscureservers,whichneversendHTTPauthenticationchallenges,butacceptun-solicitedauthinfo,say,inadditiontoform-basedauthentication.`--retry-on-host-error'Considerhosterrors,suchas\Temporaryfailureinnameresolution",asnon-fatal,transienterrors.`--retry-on-http-error=code[,code,...]'ConsidergivenHTTPresponsecodesasnon-fatal,transienterrors.Supplyacomma-separatedlistof3-digitHTTPresponsecodesasargument.Usefultoworkaroundspecialcircumstanceswhereretriesarerequired,buttheserverrespondswithanerrorcodenormallynotretriedbyWget.Sucherrorsmightbe503(Ser-viceUnavailable)and429(TooManyRequests).RetriesenabledbythisoptionareperformedsubjecttothenormalretrytimingandretrycountlimitationsofWget.Usingthisoptionisintendedtosupportspecialusecasesonlyandisgenerallynotrecommended,asitcanforceretriesevenincaseswheretheserverisactuallytryingtodecreaseitsload.Pleaseusewiselyandonlyifyouknowwhatyouaredoing.2.8HTTPS(SSL/TLS)OptionsTosupportencryptedHTTP(HTTPS)downloads,WgetmustbecompiledwithanexternalSSLlibrary.ThecurrentdefaultisGnuTLS.Inaddition,WgetalsosupportsHSTS(HTTPStrictTransportSecurity).IfWgetiscompiledwithoutSSLsupport,noneoftheseoptionsareavailable.`--secure-protocol=protocol'Choosethesecureprotocoltobeused.Legalvaluesare`auto',`SSLv2',`SSLv3',`TLSv1',`TLSv1_1',`TLSv1_2',`TLSv1_3'and`PFS'.If`auto'isused,theSSLlibraryisgiventhelibertyofchoosingtheappropriateprotocolautomatically,whichisachievedbysendingaTLSv1greeting.Thisisthedefault.Specifying`SSLv2',`SSLv3',`TLSv1',`TLSv1_1',`TLSv1_2'or`TLSv1_3'forcestheuseofthecorrespondingprotocol.ThisisusefulwhentalkingtooldandbuggySSLserverimplementationsthatmakeithardfortheunderlyingSSLlibrarytochoosethecorrectprotocolversion.Fortunately,suchserversarequiterare.Specifying`PFS'enforcestheuseoftheso-calledPerfectForwardSecurityciphersuites.Inshort,PFSaddssecuritybycreatingaone-timekeyforeachSSLconnec-tion.IthasabitmoreCPUimpactonclientandserver.Weuseknowntobesecureciphers(e.g.noMD4)andtheTLSprotocol.Thismodealsoexplicitlyexcludesnon-PFSkeyexchangemethods,suchasRSA. Chapter2:Invoking22 `--https-only'Wheninrecursivemode,onlyHTTPSlinksarefollowed.`--ciphers'Setthecipherliststring.TypicallythisstringsetstheciphersuitesandotherSSL/TLSoptionsthattheuserwishshouldbeused,inasetorderofpreference(GnuTLScallsit'prioritystring').ThisstringwillbefedverbatimtotheSSL/TLSengine(OpenSSLorGnuTLS)andhenceitsformatandsyntaxisdependantonthat.Wgetwillnotprocessormanipulateitinanyway.RefertotheOpenSSLorGnuTLSdocumentationformoreinformation.`--no-check-certificate'Don'tchecktheservercerticateagainsttheavailablecerticateauthorities.Alsodon'trequiretheURLhostnametomatchthecommonnamepresentedbythecerticate.AsofWget1.10,thedefaultistoverifytheserver'scerticateagainsttherecog-nizedcerticateauthorities,breakingtheSSLhandshakeandabortingthedownloadifthevericationfails.Althoughthisprovidesmoresecuredownloads,itdoesbreakinteroperabilitywithsomesitesthatworkedwithpreviousWgetversions,particu-larlythoseusingself-signed,expired,orotherwiseinvalidcerticates.Thisoptionforcesan\insecure"modeofoperationthatturnsthecerticatevericationerrorsintowarningsandallowsyoutoproceed.Ifyouencounter\certicateverication"errorsoronessayingthat\commonnamedoesn'tmatchrequestedhostname",youcanusethisoptiontobypasstheveri-cationandproceedwiththedownload.Onlyusethisoptionifyouareotherwiseconvincedofthesite'sauthenticity,orifyoureallydon'tcareaboutthevalidityofitscerticate.Itisalmostalwaysabadideanottocheckthecerticateswhentransmittingcondentialorimportantdata.Forself-signed/internalcerticates,youshoulddownloadthecerticateandverifyagainstthatinsteadofforcingthisinsecuremode.Ifyouarereallysureofnotdesiringanycerticateverication,youcanspecify{check-certicate=quiettotellwgettonotprintanywarningaboutinvalidcerticates,albeitinmostcasesthisisthewrongthingtodo.`--certificate=file'Usetheclientcerticatestoredinle.Thisisneededforserversthatareconguredtorequirecerticatesfromtheclientsthatconnecttothem.Normallyacerticateisnotrequiredandthisswitchisoptional.`--certificate-type=type'Specifythetypeoftheclientcerticate.Legalvaluesare`PEM'(assumedbydefault)and`DER',alsoknownas`ASN1'.`--private-key=file'Readtheprivatekeyfromle.Thisallowsyoutoprovidetheprivatekeyinaleseparatefromthecerticate.`--private-key-type=type'Specifythetypeoftheprivatekey.Acceptedvaluesare`PEM'(thedefault)and`DER'.`--ca-certificate=file'Useleasthelewiththebundleofcerticateauthorities(\CA")toverifythepeers.ThecerticatesmustbeinPEMformat.WithoutthisoptionWgetlooksforCAcerticatesatthesystem-speciedlocations,chosenatOpenSSLinstallationtime. Chapter2:Invoking23 `--ca-directory=directory'SpeciesdirectorycontainingCAcerticatesinPEMformat.EachlecontainsoneCAcerticate,andthelenameisbasedonahashvaluederivedfromthecer-ticate.Thisisachievedbyprocessingacerticatedirectorywiththec_rehashutilitysuppliedwithOpenSSL.Using`--ca-directory'ismoreecientthan`--ca-certificate'whenmanycerticatesareinstalledbecauseitallowsWgettofetchcerticatesondemand.WithoutthisoptionWgetlooksforCAcerticatesatthesystem-speciedlocations,chosenatOpenSSLinstallationtime.`--crl-file=file'SpeciesaCRLleinle.ThisisneededforcerticatesthathavebeenrevocatedbytheCAs.`--pinnedpubkey=file/hashes'Tellswgettousethespeciedpublickeyle(orhashes)toverifythepeer.ThiscanbeapathtoalewhichcontainsasinglepublickeyinPEMorDERformat,oranynumberofbase64encodedsha256hashesprecededby\sha256//"andseparatedby\;"WhennegotiatingaTLSorSSLconnection,theserversendsacerticateindicatingitsidentity.Apublickeyisextractedfromthiscerticateandifitdoesnotexactlymatchthepublickey(s)providedtothisoption,wgetwillaborttheconnectionbeforesendingorreceivinganydata.`--random-file=file'[OpenSSLandLibreSSLonly]Useleasthesourceofrandomdataforseedingthepseudo-randomnumbergeneratoronsystemswithout/dev/urandom.OnsuchsystemstheSSLlibraryneedsanexternalsourceofrandomnesstoinitialize.RandomnessmaybeprovidedbyEGD(see`--egd-file'below)orreadfromanexternalsourcespeciedbytheuser.Ifthisoptionisnotspecied,Wgetlooksforrandomdatain$RANDFILEor,ifthatisunset,in$HOME/.rnd.Ifyou'regettingthe\CouldnotseedOpenSSLPRNG;disablingSSL."error,youshouldproviderandomdatausingsomeofthemethodsdescribedabove.`--egd-file=file'[OpenSSLonly]UseleastheEGDsocket.EGDstandsforEntropyGatheringDaemon,auser-spaceprogramthatcollectsdatafromvariousunpredictablesystemsourcesandmakesitavailabletootherprogramsthatmightneedit.Encryptionsoftware,suchastheSSLlibrary,needssourcesofnon-repeatingrandomnesstoseedtherandomnumbergeneratorusedtoproducecryptographicallystrongkeys.OpenSSLallowstheusertospecifyhisownsourceofentropyusingtheRAND_FILEenvironmentvariable.Ifthisvariableisunset,orifthespeciedledoesnotproduceenoughrandomness,OpenSSLwillreadrandomdatafromEGDsocketspeciedusingthisoption.Ifthisoptionisnotspecied(andtheequivalentstartupcommandisnotused),EGDisnevercontacted.EGDisnotneededonmodernUnixsystemsthatsupport/dev/urandom.`--no-hsts'WgetsupportsHSTS(HTTPStrictTransportSecurity,RFC6797)bydefault.Use`--no-hsts'tomakeWgetactasanon-HSTS-compliantUA.Asaconsequence,WgetwouldignorealltheStrict-Transport-Securityheaders,andwouldnotenforceanyexistingHSTSpolicy. Chapter2:Invoking24 `--hsts-file=file'Bydefault,WgetstoresitsHSTSdatabasein~/.wget-hsts.Youcanuse`--hsts-file'tooverridethis.WgetwillusethesuppliedleastheHSTSdata-base.SuchlemustconformtothecorrectHSTSdatabaseformatusedbyWget.IfWgetcannotparsetheprovidedle,thebehaviourisunspecied.TheWget'sHSTSdatabaseisaplaintextle.EachlinecontainsanHSTSentry(ie.asitethathasissuedaStrict-Transport-SecurityheaderandthatthereforehasspeciedaconcreteHSTSpolicytobeapplied).Linesstartingwithadash(#)areignoredbyWget.Pleasenotethatinspiteofthisconvenienthuman-readabilityhand-hackingtheHSTSdatabaseisgenerallynotagoodidea.AnHSTSentrylineconsistsofseveraleldsseparatedbyoneormorewhitespace:&#xhost;&#xname;SP&#xport;[SP&#xincl;&#xude-;㍀subdomainsSPrea;&#xted0;SP&#xmax-; ge0;ThehostnameandporteldsindicatethehostnameandporttowhichthegivenHSTSpolicyapplies.Theporteldmaybezero,anditwill,inmostofthecases.ThatmeansthattheportnumberwillnotbetakenintoaccountwhendecidingwhethersuchHSTSpolicyshouldbeappliedonagivenrequest(onlythehost-namewillbeevaluated).Whenportisdierenttozero,boththetargethostnameandtheportwillbeevaluatedandtheHSTSpolicywillonlybeappliedifbothofthemmatch.Thisfeaturehasbeenincludedfortesting/developmentpurposesonly.TheWgettestsuite(intestenv/)createsHSTSdatabaseswithexplicitportswiththepurposeofensuringWget'scorrectbehaviour.ApplyingHSTSpoliciestoportsotherthanthedefaultonesisdiscouragedbyRFC6797(seeAppendixB"DierencesbetweenHSTSPolicyandSame-OriginPolicy").Thus,thisfunction-alityshouldnotbeusedinproductionenvironmentsandportwilltypicallybezero.Thelastthreeeldsdowhattheyareexpectedto.Theeldinclude subdomainscaneitherbe1or0anditsignalswhetherthesubdomainsofthetargetdomainshouldbepartofthegivenHSTSpolicyaswell.Thecreatedandmax-ageeldsholdthetimestampvaluesofwhensuchentrywascreated(rstseenbyWget)andtheHSTS-denedvalue'max-age',whichstateshowlongshouldthatHSTSpolicyremainactive,measuredinsecondselapsedsincethetimestampstoredincreated.Oncethattimehaspassed,thatHSTSpolicywillnolongerbevalidandwilleventuallyberemovedfromthedatabase.IfyousupplyyourownHSTSdatabasevia`--hsts-file',beawarethatWgetmaymodifytheprovidedleifanychangeoccursbetweentheHSTSpoliciesrequestedbytheremoteserversandthoseinthele.WhenWgetexists,iteectivelyupdatestheHSTSdatabasebyrewritingthedatabaselewiththenewentries.Ifthesuppliedledoesnotexist,Wgetwillcreateone.ThislewillcontainthenewHSTSentries.IfnoHSTSentriesweregenerated(noStrict-Transport-Securityheadersweresentbyanyoftheservers)thennolewillbecreated,notevenanemptyone.Thisbehaviourappliestothedefaultdatabasele(~/.wget-hsts)aswell:itwillnotbecreateduntilsomeserverenforcesanHSTSpolicy.CareistakennottooverridepossiblechangesmadebyotherWgetprocessesatthesametimeovertheHSTSdatabase.BeforedumpingtheupdatedHSTSentriesonthele,Wgetwillre-readitandmergethechanges.UsingacustomHSTSdatabaseand/ormodifyinganexistingoneisdiscouraged.Formoreinformationaboutthepotentialsecuritythreatsarisedfromsuchpractice,seesection14"SecurityConsiderations"ofRFC6797,speciallysection14.9"CreativeManipulationofHSTSPolicyStore".`--warc-file=file'UseleasthedestinationWARCle. Chapter2:Invoking25 `--warc-header=string'Usestringintoasthewarcinforecord.`--warc-max-size=size'SetthemaximumsizeoftheWARClestosize.`--warc-cdx'WriteCDXindexles.`--warc-dedup=file'DonotstorerecordslistedinthisCDXle.`--no-warc-compression'DonotcompressWARCleswithGZIP.`--no-warc-digests'DonotcalculateSHA1digests.`--no-warc-keep-log'DonotstorethelogleinaWARCrecord.`--warc-tempdir=dir'SpecifythelocationfortemporarylescreatedbytheWARCwriter.2.9FTPOptions`--ftp-user=user'`--ftp-password=password'Specifytheusernameuserandpasswordpasswordonanftpserver.Withoutthis,orthecorrespondingstartupoption,thepassworddefaultsto`-wget@',normallyusedforanonymousFTP.Anotherwaytospecifyusernameandpasswordisintheurlitself(see Section2.1[URLFormat],page2 ).Eithermethodrevealsyourpasswordtoanyonewhobotherstorunps.Topreventthepasswordsfrombeingseen,storethemin.wgetrcor.netrc,andmakesuretoprotectthoselesfromotheruserswithchmod.Ifthepasswordsarereallyimportant,donotleavethemlyinginthoseleseither|editthelesanddeletethemafterWgethasstartedthedownload.See Section9.2[SecurityConsiderations],page59 ,formoreinformationaboutse-curityissueswithWget.`--no-remove-listing'Don'tremovethetemporary.listinglesgeneratedbyftpretrievals.Normally,theselescontaintherawdirectorylistingsreceivedfromftpservers.Notremovingthemcanbeusefulfordebuggingpurposes,orwhenyouwanttobeabletoeasilycheckonthecontentsofremoteserverdirectories(e.g.toverifythatamirroryou'rerunningiscomplete).NotethateventhoughWgetwritestoaknownlenameforthisle,thisisnotasecurityholeinthescenarioofausermaking.listingasymboliclinkto/etc/passwdorsomethingandaskingroottorunWgetinhisorherdirectory.Dependingontheoptionsused,eitherWgetwillrefusetowriteto.listing,mak-ingtheglobbing/recursion/time-stampingoperationfail,orthesymboliclinkwillbedeletedandreplacedwiththeactual.listingle,orthelistingwillbewrittentoa.listing.numberle.Eventhoughthissituationisn'taproblem,though,rootshouldneverrunWgetinanon-trusteduser'sdirectory.Ausercoulddosomethingassimpleaslinking Chapter2:Invoking26 index.htmlto/etc/passwdandaskingroottorunWgetwith`-N'or`-r'sothelewillbeoverwritten.`--no-glob'Turnoftpglobbing.Globbingreferstotheuseofshell-likespecialcharacters(wildcards),like`*',`?',`['and`]'toretrievemorethanonelefromthesamedirectoryatonce,like:wgetftp://gnjilux.srk.fer.hr/*.msgBydefault,globbingwillbeturnedoniftheurlcontainsaglobbingcharacter.Thisoptionmaybeusedtoturnglobbingonoropermanently.Youmayhavetoquotetheurltoprotectitfrombeingexpandedbyyourshell.GlobbingmakesWgetlookforadirectorylisting,whichissystem-specic.ThisiswhyitcurrentlyworksonlywithUnixftpservers(andtheonesemulatingUnixlsoutput).`--no-passive-ftp'DisabletheuseofthepassiveFTPtransfermode.PassiveFTPmandatesthattheclientconnecttotheservertoestablishthedataconnectionratherthantheotherwayaround.IfthemachineisconnectedtotheInternetdirectly,bothpassiveandactiveFTPshouldworkequallywell.BehindmostrewallandNATcongurationspassiveFTPhasabetterchanceofworking.However,insomerarerewallcongurations,activeFTPactuallyworkswhenpassiveFTPdoesn't.Ifyoususpectthistobethecase,usethisoption,orsetpassive_ftp=offinyourinitle.`--preserve-permissions'Preserveremotelepermissionsinsteadofpermissionssetbyumask.`--retr-symlinks'Bydefault,whenretrievingftpdirectoriesrecursivelyandasymboliclinkisencoun-tered,thesymboliclinkistraversedandthepointed-tolesareretrieved.Currently,Wgetdoesnottraversesymboliclinkstodirectoriestodownloadthemrecursively,thoughthisfeaturemaybeaddedinthefuture.When`--retr-symlinks=no'isspecied,thelinked-toleisnotdownloaded.In-stead,amatchingsymboliclinkiscreatedonthelocallesystem.Thepointed-tolewillnotberetrievedunlessthisrecursiveretrievalwouldhaveencountereditseparatelyanddownloadeditanyway.ThisoptionposesasecurityriskwhereamaliciousFTPServermaycauseWgettowritetolesoutsideoftheintendeddirectoriesthroughaspeciallycrafted.listingle.Notethatwhenretrievingale(notadirectory)becauseitwasspeciedonthecommand-line,ratherthanbecauseitwasrecursedto,thisoptionhasnoeect.Symboliclinksarealwaystraversedinthiscase.2.10FTPSOptions`--ftps-implicit'ThisoptiontellsWgettouseFTPSimplicitly.ImplicitFTPSconsistsofinitializingSSL/TLSfromtheverybeginningofthecontrolconnection.ThisoptiondoesnotsendanAUTHTLScommand:itassumestheserverspeaksFTPSanddirectlystartsanSSL/TLSconnection.Iftheattemptissuccessful,thesessioncontinuesjustlikeregularFTPS(PBSZandPROTaresent,etc.).ImplicitFTPSisnolongerarequirementforFTPSimplementations,andthusmanyserversmaynotsupportit.If`--ftps-implicit'ispassedandnoexplicitportnumberspecied,thedefault Chapter2:Invoking27 portforimplicitFTPS,990,willbeused,insteadofthedefaultportforthe"normal"(explicit)FTPSwhichisthesameasthatofFTP,21.`--no-ftps-resume-ssl'DonotresumetheSSL/TLSsessioninthedatachannel.Whenstartingadataconnection,WgettriestoresumetheSSL/TLSsessionpreviouslystartedinthecontrolconnection.SSL/TLSsessionresumptionavoidsperforminganentirelynewhandshakebyreusingtheSSL/TLSparametersofaprevioussession.Typically,theFTPSserverswantitthatway,soWgetdoesthisbydefault.Underrarecircumstanceshowever,onemightwanttostartanentirelynewSSL/TLSsessionineverydataconnection.Thisiswhat`--no-ftps-resume-ssl'isfor.`--ftps-clear-data-connection'Allthedataconnectionswillbeinplaintext.OnlythecontrolconnectionwillbeunderSSL/TLS.WgetwillsendaPROTCcommandtoachievethis,whichmustbeapprovedbytheserver.`--ftps-fallback-to-ftp'FallbacktoFTPifFTPSisnotsupportedbythetargetserver.Forsecurityreasons,thisoptionisnotassertedbydefault.Thedefaultbehaviouristoexitwithanerror.IfaserverdoesnotsuccessfullyreplytotheinitialAUTHTLScommand,orinthecaseofimplicitFTPS,iftheinitialSSL/TLSconnectionattemptisrejected,itisconsideredthatsuchserverdoesnotsupportFTPS.2.11RecursiveRetrievalOptions`-r'`--recursive'Turnonrecursiveretrieving.See Chapter3[RecursiveDownload],page33 ,formoredetails.Thedefaultmaximumdepthis5.`-ldepth'`--level=depth'Specifyrecursionmaximumdepthleveldepth(see Chapter3[RecursiveDownload],page33 ).`--delete-after'ThisoptiontellsWgettodeleteeverysingleleitdownloads,afterhavingdoneso.Itisusefulforpre-fetchingpopularpagesthroughaproxy,e.g.:wget-r-nd--delete-afterhttp://whatever.com/~popular/page/The`-r'optionistoretrieverecursively,and`-nd'tonotcreatedirectories.Notethat`--delete-after'deleteslesonthelocalmachine.Itdoesnotis-suethe`DELE'commandtoremoteFTPsites,forinstance.Alsonotethatwhen`--delete-after'isspecied,`--convert-links'isignored,so`.orig'lesaresimplynotcreatedintherstplace.`-k'`--convert-links'Afterthedownloadiscomplete,convertthelinksinthedocumenttomakethemsuitableforlocalviewing.Thisaectsnotonlythevisiblehyperlinks,butanypartofthedocumentthatlinkstoexternalcontent,suchasembeddedimages,linkstostylesheets,hyperlinkstonon-htmlcontent,etc.Eachlinkwillbechangedinoneofthetwoways:ThelinkstolesthathavebeendownloadedbyWgetwillbechangedtorefertotheletheypointtoasarelativelink. Chapter2:Invoking28 Example:ifthedownloadedle/foo/doc.htmllinksto/bar/img.gif,alsodownloaded,thenthelinkindoc.htmlwillbemodiedtopointto`../bar/img.gif'.Thiskindoftransformationworksreliablyforarbitrarycombinationsofdirectories.ThelinkstolesthathavenotbeendownloadedbyWgetwillbechangedtoincludehostnameandabsolutepathofthelocationtheypointto.Example:ifthedownloadedle/foo/doc.htmllinksto/bar/img.gif(orto../bar/img.gif),thenthelinkindoc.htmlwillbemodiedtopointtohttp://hostname/bar/img.gif.Becauseofthis,localbrowsingworksreliably:ifalinkedlewasdownloaded,thelinkwillrefertoitslocalname;ifitwasnotdownloaded,thelinkwillrefertoitsfullInternetaddressratherthanpresentingabrokenlink.Thefactthattheformerlinksareconvertedtorelativelinksensuresthatyoucanmovethedownloadedhierarchytoanotherdirectory.NotethatonlyattheendofthedownloadcanWgetknowwhichlinkshavebeendownloaded.Becauseofthat,theworkdoneby`-k'willbeperformedattheendofallthedownloads.`--convert-file-only'ThisoptionconvertsonlythelenamepartoftheURLs,leavingtherestoftheURLsuntouched.Thislenamepartissometimesreferredtoasthe"basename",althoughweavoidthattermhereinordernottocauseconfusion.Itworksparticularlywellinconjunctionwith`--adjust-extension',althoughthiscouplingisnotenforced.ItprovesusefultopopulateInternetcacheswithlesdownloadedfromdierenthosts.Example:ifsomelinkpointsto//foo.com/bar.cgi?xyzwith`--adjust-extension'assertedanditslocaldestinationisintendedtobe./foo.com/bar.cgi?xyz.css,thenthelinkwouldbeconvertedto//foo.com/bar.cgi?xyz.css.Notethatonlythelenameparthasbeenmodied.TherestoftheURLhasbeenleftuntouched,includingthenetpath(//)whichwouldotherwisebeprocessedbyWgetandconvertedtotheeectivescheme(ie.http://).`-K'`--backup-converted'Whenconvertingale,backuptheoriginalversionwitha`.orig'sux.Aectsthebehaviorof`-N'(see Section5.2[HTTPTime-StampingInternals],page39 ).`-m'`--mirror'Turnonoptionssuitableformirroring.Thisoptionturnsonrecursionandtime-stamping,setsinniterecursiondepthandkeepsftpdirectorylistings.Itiscur-rentlyequivalentto`-r-N-linf--no-remove-listing'.`-p'`--page-requisites'ThisoptioncausesWgettodownloadallthelesthatarenecessarytoproperlydisplayagivenhtmlpage.Thisincludessuchthingsasinlinedimages,sounds,andreferencedstylesheets.Ordinarily,whendownloadingasinglehtmlpage,anyrequisitedocumentsthatmaybeneededtodisplayitproperlyarenotdownloaded.Using`-r'togetherwith`-l'canhelp,butsinceWgetdoesnotordinarilydistinguishbetweenexternaland Chapter2:Invoking29 inlineddocuments,oneisgenerallyleftwith\leafdocuments"thataremissingtheirrequisites.Forinstance,saydocument1.htmlcontainsan&#xIMG0;tagreferencing1.gifandanꀀtagpointingtoexternaldocument2.html.Saythat2.htmlissimilarbutthatitsimageis2.gifanditlinksto3.html.Saythiscontinuesuptosomearbitrarilyhighnumber.Ifoneexecutesthecommand:wget-r-l2http://site/1.htmlthen1.html,1.gif,2.html,2.gif,and3.htmlwillbedownloaded.Asyoucansee,3.htmliswithoutitsrequisite3.gifbecauseWgetissimplycountingthenumberofhops(upto2)awayfrom1.htmlinordertodeterminewheretostoptherecursion.However,withthiscommand:wget-r-l2-phttp://site/1.htmlalltheabovelesand3.html'srequisite3.gifwillbedownloaded.Similarly,wget-r-l1-phttp://site/1.htmlwillcause1.html,1.gif,2.html,and2.giftobedownloaded.Onemightthinkthat:wget-r-l0-phttp://site/1.htmlwoulddownloadjust1.htmland1.gif,butunfortunatelythisisnotthecase,because`-l0'isequivalentto`-linf'|thatis,inniterecursion.Todownloadasinglehtmlpage(orahandfulofthem,allspeciedonthecommand-lineorina`-i'urlinputle)andits(ortheir)requisites,simplyleaveo`-r'and`-l':wget-phttp://site/1.htmlNotethatWgetwillbehaveasif`-r'hadbeenspecied,butonlythatsinglepageanditsrequisiteswillbedownloaded.Linksfromthatpagetoexternaldocumentswillnotbefollowed.Actually,todownloadasinglepageandallitsrequisites(eveniftheyexistonseparatewebsites),andmakesurethelotdisplaysproperlylocally,thisauthorlikestouseafewoptionsinadditionto`-p':wget-E-H-k-K-phttp://site/documentTonishothistopic,it'sworthknowingthatWget'sideaofanexternaldocumentlinkisanyURLspeciedinanꀀtag,an REA;tag,ora&#xLINK;tagotherthan&#xLINK;&#x-333;REL="stylesheet".`--strict-comments'Turnonstrictparsingofhtmlcomments.Thedefaultistoterminatecommentsattherstoccurrenceof`&#xLINK;&#x-333;--'.Accordingtospecications,htmlcommentsareexpressedassgmldeclarations.Declarationisspecialmarkupthatbeginswith`'andendswith`&#x!]TJ;&#x/F55;&#x 10.;邑&#x Tf ;.4;U 0;&#x Td ;&#x[000;',suchas`&#x!DOC;&#xTYPE;&#x-333;...',thatmaycontaincommentsbetweenapairof`--'delimiters.htmlcommentsare\emptydeclarations",sgmldeclarationswithoutanynon-commenttext.Therefore,`&#x!--f;&#xoo--;'isavalidcomment,andsois`&#x!--o;&#xne--;&#x]TJ ;&#x-344;&#x.398;&#x -13;&#x.151;&#x Td ;&#x[000;--two--',but`&#x!--1;&#x--2-;&#x-000;'isnot.Ontheotherhand,mosthtmlwritersdon'tperceivecommentsasanythingotherthantextdelimitedwith`'and`&#x!--];&#xTJ/F;U 1;�.90;‘ T; 22;&#x.909;&#x 0 T; [0;--',whichisnotquitethesame.Forexample,somethinglike`&#x!---;&#x----;&#x----;&#x-000;'worksasavalidcommentaslongasthenumberofdashesisamultipleoffour(!).Ifnot,thecommenttechnicallylastsuntilthenext`--',whichmaybeattheotherendofthedocument.Becauseofthis,manypopularbrowserscompletelyignorethespecicationandimplementwhatusershavecometoexpect:commentsdelimitedwith`'and`&#x!--];&#xTJ/F;U 1;�.90;‘ T; 22;&#x.908;&#x 0 T; [0;--'. Chapter2:Invoking30 Untilversion1.9,Wgetinterpretedcommentsstrictly,whichresultedinmissinglinksinmanywebpagesthatdisplayedneinbrowsers,buthadthemisfortuneofcontainingnon-compliantcomments.Beginningwithversion1.9,Wgethasjoinedtheranksofclientsthatimplements\naive"comments,terminatingeachcommentattherstoccurrenceof`�--'.If,forwhateverreason,youwantstrictcommentparsing,usethisoptiontoturniton.2.12RecursiveAccept/RejectOptions`-Aacclist--acceptacclist'`-Rrejlist--rejectrejlist'Specifycomma-separatedlistsoflenamesuxesorpatternstoacceptorreject(see Section4.2[TypesofFiles],page34 ).Notethatifanyofthewildcardcharacters,`*',`?',`['or`]',appearinanelementofacclistorrejlist,itwillbetreatedasapattern,ratherthanasux.Inthiscase,youhavetoenclosethepatternintoquotestopreventyourshellfromexpandingit,likein`-A"*.mp3"'or`-A'*.mp3''.`--accept-regexurlregex'`--reject-regexurlregex'SpecifyaregularexpressiontoacceptorrejectthecompleteURL.`--regex-typeregextype'Specifytheregularexpressiontype.Possibletypesare`posix'or`pcre'.Notethattobeabletouse`pcre'type,wgethastobecompiledwithlibpcresupport.`-Ddomain-list'`--domains=domain-list'Setdomainstobefollowed.domain-listisacomma-separatedlistofdomains.Notethatitdoesnotturnon`-H'.`--exclude-domainsdomain-list'Specifythedomainsthatarenottobefollowed(see Section4.1[SpanningHosts],page34 ).`--follow-ftp'Followftplinksfromhtmldocuments.Withoutthisoption,Wgetwillignorealltheftplinks.`--follow-tags=list'Wgethasaninternaltableofhtmltag/attributepairsthatitconsiderswhenlookingforlinkeddocumentsduringarecursiveretrieval.Ifauserwantsonlyasubsetofthosetagstobeconsidered,however,heorsheshouldbespecifysuchtagsinacomma-separatedlistwiththisoption.`--ignore-tags=list'Thisistheoppositeofthe`--follow-tags'option.Toskipcertainhtmltagswhenrecursivelylookingfordocumentstodownload,specifytheminacomma-separatedlist.Inthepast,thisoptionwasthebestbetfordownloadingasinglepageanditsrequisites,usingacommand-linelike:wget--ignore-tags=a,area-H-k-K-rhttp://site/documentHowever,theauthorofthisoptioncameacrossapagewithtagslikeREL="home"&#xLINK;&#x]TJ ;&#x-367;&#x.307;&#x -13;&#x.151;&#x Td ;&#x[000;HREF="/"andcametotherealizationthatspecifyingtagstoignorewasnotenough.Onecan'tjusttellWgettoignore&#xLINK;,becausethenstylesheets Chapter2:Invoking31 willnotbedownloaded.Nowthebestbetfordownloadingasinglepageanditsrequisitesisthededicated`--page-requisites'option.`--ignore-case'Ignorecasewhenmatchinglesanddirectories.Thisin uencesthebehaviorof-R,-A,-I,and-Xoptions,aswellasglobbingimplementedwhendownloadingfromFTPsites.Forexample,withthisoption,`-A"*.txt"'willmatch`file1.txt',butalso`file2.TXT',`file3.TxT',andsoon.Thequotesintheexamplearetopreventtheshellfromexpandingthepattern.`-H'`--span-hosts'Enablespanningacrosshostswhendoingrecursiveretrieving(see Section4.1[Span-ningHosts],page34 ).`-L'`--relative'Followrelativelinksonly.Usefulforretrievingaspecichomepagewithoutanydistractions,noteventhosefromthesamehosts(see Section4.4[RelativeLinks],page37 ).`-Ilist'`--include-directories=list'Specifyacomma-separatedlistofdirectoriesyouwishtofollowwhendownloading(see Section4.3[Directory-BasedLimits],page36 ).Elementsoflistmaycontainwildcards.`-Xlist'`--exclude-directories=list'Specifyacomma-separatedlistofdirectoriesyouwishtoexcludefromdownload(see Section4.3[Directory-BasedLimits],page36 ).Elementsoflistmaycontainwildcards.`-np'`--no-parent'Donoteverascendtotheparentdirectorywhenretrievingrecursively.Thisisausefuloption,sinceitguaranteesthatonlythelesbelowacertainhierarchywillbedownloaded.See Section4.3[Directory-BasedLimits],page36 ,formoredetails.2.13ExitStatusWgetmayreturnoneofseveralerrorcodesifitencountersproblems.0Noproblemsoccurred.1Genericerrorcode.2Parseerror|forinstance,whenparsingcommand-lineoptions,the`.wgetrc'or`.netrc'...3FileI/Oerror.4Networkfailure.5SSLvericationfailure.6Username/passwordauthenticationfailure.7Protocolerrors. 32 8Serverissuedanerrorresponse.Withtheexceptionsof0and1,thelower-numberedexitcodestakeprecedenceoverhigher-numberedones,whenmultipletypesoferrorsareencountered.InversionsofWgetpriorto1.12,Wget'sexitstatustendedtobeunhelpfulandincon-sistent.Recursivedownloadswouldvirtuallyalwaysreturn0(success),regardlessofanyis-suesencountered,andnon-recursivefetchesonlyreturnedthestatuscorrespondingtothemostrecently-attempteddownload. 33 3RecursiveDownloadGNUWgetiscapableoftraversingpartsoftheWeb(orasinglehttporftpserver),followinglinksanddirectorystructure.Werefertothisastorecursiveretrieval,orrecursion.Withhttpurls,Wgetretrievesandparsesthehtmlorcssfromthegivenurl,retrievingthelesthedocumentrefersto,throughmarkuplikehreforsrc,orcssurivaluesspeciedusingthe`url()'functionalnotation.Ifthefreshlydownloadedleisalsooftypetext/html,application/xhtml+xml,ortext/css,itwillbeparsedandfollowedfurther.Recursiveretrievalofhttpandhtml/csscontentisbreadth-rst.ThismeansthatWgetrstdownloadstherequesteddocument,thenthedocumentslinkedfromthatdocument,thenthedocumentslinkedbythem,andsoon.Inotherwords,Wgetrstdownloadsthedocumentsatdepth1,thenthoseatdepth2,andsoonuntilthespeciedmaximumdepth.Themaximumdepthtowhichtheretrievalmaydescendisspeciedwiththe`-l'option.Thedefaultmaximumdepthisvelayers.Whenretrievinganftpurlrecursively,Wgetwillretrieveallthedatafromthegivendirectorytree(includingthesubdirectoriesuptothespecieddepth)ontheremoteserver,creatingitsmirrorimagelocally.ftpretrievalisalsolimitedbythedepthparameter.Unlikehttprecursion,ftprecursionisperformeddepth-rst.Bydefault,Wgetwillcreatealocaldirectorytree,correspondingtotheonefoundontheremoteserver.Recursiveretrievingcanndanumberofapplications,themostimportantofwhichismir-roring.Itisalsousefulforwwwpresentations,andanyotheropportunitieswhereslownetworkconnectionsshouldbebypassedbystoringtheleslocally.Youshouldbewarnedthatrecursivedownloadscanoverloadtheremoteservers.Becauseofthat,manyadministratorsfrownuponthemandmaybanaccessfromyoursiteiftheydetectveryfastdownloadsofbigamountsofcontent.WhendownloadingfromInternetservers,considerusingthe`-w'optiontointroduceadelaybetweenaccessestotheserver.Thedownloadwilltakeawhilelonger,buttheserveradministratorwillnotbealarmedbyyourrudeness.Ofcourse,recursivedownloadmaycauseproblemsonyourmachine.Iflefttorununchecked,itcaneasilyllupthedisk.Ifdownloadingfromlocalnetwork,itcanalsotakebandwidthonthesystem,aswellasconsumememoryandCPU.Trytospecifythecriteriathatmatchthekindofdownloadyouaretryingtoachieve.Ifyouwanttodownloadonlyonepage,use`--page-requisites'withoutanyadditionalrecursion.Ifyouwanttodownloadthingsunderonedirectory,use`-np'toavoiddownloadingthingsfromotherdirectories.Ifyouwanttodownloadallthelesfromonedirectory,use`-l1'tomakesuretherecursiondepthneverexceedsone.See Chapter4[FollowingLinks],page34 ,formoreinformationaboutthis.Recursiveretrievalshouldbeusedwithcare.Don'tsayyouwerenotwarned. 34 4FollowingLinksWhenretrievingrecursively,onedoesnotwishtoretrieveloadsofunnecessarydata.Mostofthetimetheusersbearinmindexactlywhattheywanttodownload,andwantWgettofollowonlyspeciclinks.Forexample,ifyouwishtodownloadthemusicarchivefrom`fly.srk.fer.hr',youwillnotwanttodownloadallthehomepagesthathappentobereferencedbyanobscurepartofthearchive.Wgetpossessesseveralmechanismsthatallowsyoutone-tunewhichlinksitwillfollow.4.1SpanningHostsWget'srecursiveretrievalnormallyrefusestovisithostsdierentthantheoneyouspeciedonthecommandline.Thisisareasonabledefault;withoutit,everyretrievalwouldhavethepotentialtoturnyourWgetintoasmallversionofgoogle.However,visitingdierenthosts,orhostspanning,issometimesausefuloption.Maybetheimagesareservedfromadierentserver.Maybeyou'remirroringasitethatconsistsofpagesinterlinkedbetweenthreeservers.Maybetheserverhastwoequivalentnames,andthehtmlpagesrefertobothinterchangeably.Spantoanyhost|`-H'The`-H'optionturnsonhostspanning,thusallowingWget'srecursiveruntovisitanyhostreferencedbyalink.Unlesssucientrecursion-limitingcriteriaareapplieddepth,theseforeignhostswilltypicallylinktoyetmorehosts,andsoonuntilWgetendsupsuckingupmuchmoredatathanyouhaveintended.Limitspanningtocertaindomains|`-D'The`-D'optionallowsyoutospecifythedomainsthatwillbefollowed,thuslimitingtherecursiononlytothehoststhatbelongtothesedomains.Obvi-ously,thismakessenseonlyinconjunctionwith`-H'.Atypicalexamplewouldbedownloadingthecontentsof`www.example.com',butallowingdownloadsfrom`images.example.com',etc.:wget-rH-Dexample.comhttp://www.example.com/Youcanspecifymorethanoneaddressbyseparatingthemwithacomma,e.g.`-Ddomain1.com,domain2.com'.Keepdownloadocertaindomains|`--exclude-domains'Iftherearedomainsyouwanttoexcludespecically,youcandoitwith`--exclude-domains',whichacceptsthesametypeofargumentsof`-D',butwillexcludeallthelisteddomains.Forexample,ifyouwanttodownloadallthehostsfrom`foo.edu'domain,withtheexceptionof`sunsite.foo.edu',youcandoitlikethis:wget-rH-Dfoo.edu--exclude-domainssunsite.foo.edu\http://www.foo.edu/4.2TypesofFilesWhendownloadingmaterialfromtheweb,youwilloftenwanttorestricttheretrievaltoonlycertainletypes.Forexample,ifyouareinterestedindownloadinggifs,youwillnotbeoverjoyedtogetloadsofPostScriptdocuments,andviceversa.Wgetoerstwooptionstodealwiththisproblem.Eachoptiondescriptionlistsashortname,alongname,andtheequivalentcommandin.wgetrc. Chapter4:FollowingLinks35 `-Aacclist'`--acceptacclist'`accept=acclist'`--accept-regexurlregex'`accept-regex=urlregex'Theargumentto`--accept'optionisalistoflesuxesorpatternsthatWgetwilldownloadduringrecursiveretrieval.Asuxistheendingpartofale,andconsistsof\normal"letters,e.g.`gif'or`.jpg'.Amatchingpatterncontainsshell-likewildcards,e.g.`books*'or`zelazny*196[0-9]*'.So,specifying`wget-Agif,jpg'willmakeWgetdownloadonlythelesend-ingwith`gif'or`jpg',i.e.gifsandjpegs.Ontheotherhand,`wget-A"zelazny*196[0-9]*"'willdownloadonlylesbeginningwith`zelazny'andcon-tainingnumbersfrom1960to1969anywherewithin.Lookupthemanualofyourshellforadescriptionofhowpatternmatchingworks.Ofcourse,anynumberofsuxesandpatternscanbecombinedintoacomma-separatedlist,andgivenasanargumentto`-A'.Theargumentto`--accept-regex'optionisaregularexpressionwhichismatchedagainstthecompleteURL.`-Rrejlist'`--rejectrejlist'`reject=rejlist'`--reject-regexurlregex'`reject-regex=urlregex'The`--reject'optionworksthesamewayas`--accept',onlyitslogicisthere-verse;Wgetwilldownloadalllesexcepttheonesmatchingthesuxes(orpatterns)inthelist.So,ifyouwanttodownloadawholepageexceptforthecumbersomempegsand.aules,youcanuse`wget-Rmpg,mpeg,au'.Analogously,todownloadalllesexcepttheonesbeginningwith`bjork',use`wget-R"bjork*"'.Thequotesaretopreventexpansionbytheshell.Theargumentto`--accept-regex'optionisaregularexpressionwhichismatchedagainstthecompleteURL.The`-A'and`-R'optionsmaybecombinedtoachieveevenbetterne-tuningofwhichlestoretrieve.E.g.`wget-A"*zelazny*"-R.ps'willdownloadalltheleshaving`zelazny'asapartoftheirname,butnotthePostScriptles.Notethatthesetwooptionsdonotaectthedownloadingofhtmlles(asdeterminedbya`.htm'or`.html'lenameprex).Thisbehaviormaynotbedesirableforallusers,andmaybechangedforfutureversionsofWget.Note,too,thatquerystrings(stringsattheendofaURLbeginningwithaquestionmark(`?')arenotincludedaspartofthelenameforaccept/rejectrules,eventhoughthesewillactuallycontributetothenamechosenforthelocalle.ItisexpectedthatafutureversionofWgetwillprovideanoptiontoallowmatchingagainstquerystrings.Finally,it'sworthnotingthattheaccept/rejectlistsarematchedtwiceagainstdownloadedles:onceagainsttheURL'slenameportion,todetermineiftheleshouldbedownloadedintherstplace;then,afterithasbeenacceptedandsuccessfullydownloaded,thelocalle'snameisalsocheckedagainsttheaccept/rejectliststoseeifitshouldberemoved.Therationalewasthat,since`.htm'and`.html'lesarealwaysdownloadedregardlessofaccept/rejectrules,theyshouldberemovedafterbeingdownloadedandscannedforlinks,iftheydidmatchtheaccept/rejectlists.However,thiscanleadtounexpectedresults,sincethelocallenamescan Chapter4:FollowingLinks36 dierfromtheoriginalURLlenamesinthefollowingways,allofwhichcanchangewhetheranaccept/rejectrulematches:Ifthelocallealreadyexistsand`--no-directories'wasspecied,anumericsuxwillbeappendedtotheoriginalname.If`--adjust-extension'wasspecied,thelocallenamemighthave`.html'appendedtoit.IfWgetisinvokedwith`-E-A.php',alenamesuchas`index.php'willmatchbeaccepted,butupondownloadwillbenamed`index.php.html',whichnolongermatches,andsothelewillbedeleted.QuerystringsdonotcontributetoURLmatching,butareincludedinlocallenames,andsodocontributetolenamematching.Thisbehavior,too,isconsideredless-than-desirable,andmaychangeinafutureversionofWget.4.3Directory-BasedLimitsRegardlessofotherlink-followingfacilities,itisoftenusefultoplacetherestrictionofwhatlestoretrievebasedonthedirectoriesthoselesareplacedin.Therecanbemanyreasonsforthis|thehomepagesmaybeorganizedinareasonabledirectorystructure;orsomedirectoriesmaycontainuselessinformation,e.g./cgi-binor/devdirectories.Wgetoersthreedierentoptionstodealwiththisrequirement.Eachoptiondescriptionlistsashortname,alongname,andtheequivalentcommandin.wgetrc.`-Ilist'`--includelist'`include_directories=list'`-I'optionacceptsacomma-separatedlistofdirectoriesincludedintheretrieval.Anyotherdirectorieswillsimplybeignored.Thedirectoriesareabsolutepaths.So,ifyouwishtodownloadfrom`http://host/people/bozo/'followingonlylinkstobozo'scolleaguesinthe/peopledirectoryandthebogusscriptsin/cgi-bin,youcanspecify:wget-I/people,/cgi-binhttp://host/people/bozo/`-Xlist'`--excludelist'`exclude_directories=list'`-X'optionisexactlythereverseof`-I'|thisisalistofdirectoriesexcludedfromthedownload.E.g.ifyoudonotwantWgettodownloadthingsfrom/cgi-bindirectory,specify`-X/cgi-bin'onthecommandline.Thesameaswith`-A'/`-R',thesetwooptionscanbecombinedtogetabetterne-tuningofdownloadingsubdirectories.E.g.ifyouwanttoloadallthelesfrom/pubhierarchyexceptfor/pub/worthless,specify`-I/pub-X/pub/worthless'.`-np'`--no-parent'`no_parent=on'Thesimplest,andoftenveryusefulwayoflimitingdirectoriesisdisallowingretrievalofthelinksthatrefertothehierarchyabovethanthebeginningdirectory,i.e.disallowingascenttotheparentdirectory/directories.The`--no-parent'option(short`-np')isusefulinthiscase.Usingitguaranteesthatyouwillneverleavetheexistinghierarchy.SupposingyouissueWgetwith:wget-r--no-parenthttp://somehost/~luzer/my-archive/ Chapter4:FollowingLinks37 Youmayrestassuredthatnoneofthereferencesto/~his-girls-homepage/or/~luzer/all-my-mpegs/willbefollowed.Onlythearchiveyouareinterestedinwillbedownloaded.Essentially,`--no-parent'issimilarto`-I/~luzer/my-archive',onlyithandlesredirectionsinamoreintelligentfashion.Notethat,forHTTP(andHTTPS),thetrailingslashisveryimportantto`--no-parent'.HTTPhasnoconceptofa\directory"|Wgetreliesonyoutoindicatewhat'sadirectoryandwhatisn't.In`http://foo/bar/',Wgetwillcon-sider`bar'tobeadirectory,whilein`http://foo/bar'(notrailingslash),`bar'willbeconsideredalename(so`--no-parent'wouldbemeaningless,asitsparentis`/').4.4RelativeLinksWhen`-L'isturnedon,onlytherelativelinksareeverfollowed.Relativelinksareheredenedthosethatdonotrefertothewebserverroot.Forexample,theselinksarerelative: -52;倀href="foo.gif" -52;倀href="foo/bar.gif" -52;倀href="../foo/bar.gif"Theselinksarenotrelative: -52;倀href="/foo.gif" -52;倀href="/foo/bar.gif" -52;倀href="http://www.example.com/foo/bar.gif"Usingthisoptionguaranteesthatrecursiveretrievalwillnotspanhosts,evenwithout`-H'.Insimplecasesitalsoallowsdownloadsto\justwork"withouthavingtoconvertlinks.Thisoptionisprobablynotveryusefulandmightberemovedinafuturerelease.4.5FollowingFTPLinksTherulesforftparesomewhatspecic,asitisnecessaryforthemtobe.ftplinksinhtmldocumentsareoftenincludedforpurposesofreference,anditisofteninconvenienttodownloadthembydefault.Tohaveftplinksfollowedfromhtmldocuments,youneedtospecifythe`--follow-ftp'option.Havingdonethat,ftplinkswillspanhostsregardlessof`-H'setting.Thisislogical,asftplinksrarelypointtothesamehostwherethehttpserverresides.Forsimilarreasons,the`-L'optionshasnoeectonsuchdownloads.Ontheotherhand,domainacceptance(`-D')andsuxrules(`-A'and`-R')applynormally.Alsonotethatfollowedlinkstoftpdirectorieswillnotberetrievedrecursivelyfurther. 38 5Time-StampingOneofthemostimportantaspectsofmirroringinformationfromtheInternetisupdatingyourarchives.Downloadingthewholearchiveagainandagain,justtoreplaceafewchangedlesisexpen-sive,bothintermsofwastedbandwidthandmoney,andthetimetodotheupdate.Thisiswhyallthemirroringtoolsoertheoptionofincrementalupdating.Suchanupdatingmechanismmeansthattheremoteserverisscannedinsearchofnewles.Onlythosenewleswillbedownloadedintheplaceoftheoldones.Aleisconsiderednewifoneofthesetwoconditionsaremet:1.Aleofthatnamedoesnotalreadyexistlocally.2.Aleofthatnamedoesexist,buttheremotelewasmodiedmorerecentlythanthelocalle.Toimplementthis,theprogramneedstobeawareofthetimeoflastmodicationofbothlocalandremoteles.Wecallthisinformationthetime-stampofale.Thetime-stampinginGNUWgetisturnedonusing`--timestamping'(`-N')option,orthroughtimestamping=ondirectivein.wgetrc.Withthisoption,foreachleitintendstodownload,Wgetwillcheckwhetheralocalleofthesamenameexists.Ifitdoes,andtheremoteleisnotnewer,Wgetwillnotdownloadit.Ifthelocalledoesnotexist,orthesizesofthelesdonotmatch,Wgetwilldownloadtheremotelenomatterwhatthetime-stampssay.5.1Time-StampingUsageTheusageoftime-stampingissimple.Sayyouwouldliketodownloadalesothatitkeepsitsdateofmodication.wget-Shttp://www.gnu.ai.mit.edu/Asimplels-lshowsthatthetimestamponthelocalleequalsthestateoftheLast-Modifiedheader,asreturnedbytheserver.Asyoucansee,thetime-stampinginfoispreservedlocally,evenwithout`-N'(atleastforhttp).Severaldayslater,youwouldlikeWgettocheckiftheremotelehaschanged,anddownloaditifithas.wget-Nhttp://www.gnu.ai.mit.edu/Wgetwillasktheserverforthelast-modieddate.Ifthelocallehasthesametimestampastheserver,oranewerone,theremotelewillnotbere-fetched.However,iftheremoteleismorerecent,Wgetwillproceedtofetchit.Thesamegoesforftp.Forexample:wget"ftp://ftp.ifi.uio.no/pub/emacs/gnus/*"(ThequotesaroundthatURLaretopreventtheshellfromtryingtointerpretthe`*'.)Afterdownload,alocaldirectorylistingwillshowthatthetimestampsmatchthoseontheremoteserver.Reissuingthecommandwith`-N'willmakeWgetre-fetchonlythelesthathavebeenmodiedsincethelastdownload.IfyouwishedtomirrortheGNUarchiveeveryweek,youwoulduseacommandlikethefollowing,weekly:wget--timestamping-rftp://ftp.gnu.org/pub/gnu/Notethattime-stampingwillonlyworkforlesforwhichtheservergivesatimestamp.Forhttp,thisdependsongettingaLast-Modifiedheader.Forftp,thisdependsongettingadirectorylistingwithdatesinaformatthatWgetcanparse(see Section5.3[FTPTime-StampingInternals],page39 ). Chapter5:Time-Stamping39 5.2HTTPTime-StampingInternalsTime-stampinginhttpisimplementedbycheckingoftheLast-Modifiedheader.Ifyouwishtoretrievethelefoo.htmlthroughhttp,Wgetwillcheckwhetherfoo.htmlexistslocally.Ifitdoesn't,foo.htmlwillberetrievedunconditionally.Iftheledoesexistlocally,Wgetwillrstcheckitslocaltime-stamp(similartothewayls-lchecksit),andthensendaHEADrequesttotheremoteserver,demandingtheinformationontheremotele.TheLast-Modifiedheaderisexaminedtondwhichlewasmodiedmorerecently(whichmakesit\newer").Iftheremoteleisnewer,itwillbedownloaded;ifitisolder,Wgetwillgiveup.1When`--backup-converted'(`-K')isspeciedinconjunctionwith`-N',serverle`X'iscomparedtolocalle`X.orig',ifextant,ratherthanbeingcomparedtolocalle`X',whichwillalwaysdierifit'sbeenconvertedby`--convert-links'(`-k').Arguably,httptime-stampingshouldbeimplementedusingtheIf-Modified-Sincere-quest.5.3FTPTime-StampingInternalsIntheory,ftptime-stampingworksmuchthesameashttp,onlyftphasnoheaders|time-stampsmustbeferretedoutofdirectorylistings.Ifanftpdownloadisrecursiveorusesglobbing,WgetwillusetheftpLISTcommandtogetalelistingforthedirectorycontainingthedesiredle(s).Itwilltrytoanalyzethelisting,treatingitlikeUnixls-loutput,extractingthetime-stamps.Therestisexactlythesameasforhttp.Notethatwhenretrievingindividuallesfromanftpserverwithoutusingglobbingorrecursion,listingleswillnotbedownloaded(andthusleswillnotbetime-stamped)unless`-N'isspecied.AssumptionthateverydirectorylistingisaUnix-stylelistingmaysoundextremelycon-straining,butinpracticeitisnot,asmanynon-UnixftpserversusetheUnixoidlistingformatbecausemost(all?)oftheclientsunderstandit.Bearinmindthatrfc959denesnostandardwaytogetalelist,letalonethetime-stamps.Wecanonlyhopethatafuturestandardwilldenethis.Anothernon-standardsolutionincludestheuseofMDTMcommandthatissupportedbysomeftpservers(includingthepopularwu-ftpd),whichreturnstheexacttimeofthespeciedle.Wgetmaysupportthiscommandinthefuture. 1Asanadditionalcheck,WgetwilllookattheContent-Lengthheader,andcomparethesizes;iftheyarenotthesame,theremotelewillbedownloadednomatterwhatthetime-stampsays. 40 6StartupFileOnceyouknowhowtochangedefaultsettingsofWgetthroughcommandlinearguments,youmaywishtomakesomeofthosesettingspermanent.YoucandothatinaconvenientwaybycreatingtheWgetstartuple|.wgetrc.Besides.wgetrcisthe\main"initializationle,itisconvenienttohaveaspecialfacilityforstoringpasswords.ThusWgetreadsandinterpretsthecontentsof$HOME/.netrc,ifitndsit.Youcannd.netrcformatinyoursystemmanuals.Wgetreads.wgetrcuponstartup,recognizingalimitedsetofcommands.6.1WgetrcLocationWheninitializing,Wgetwilllookforaglobalstartuple,/usr/local/etc/wgetrcbydefault(orsomeprexotherthan/usr/local,ifWgetwasnotinstalledthere)andreadcommandsfromthere,ifitexists.Thenitwilllookfortheuser'sle.IftheenvironmentalvariableWGETRCisset,Wgetwilltrytoloadthatle.Failingthat,nofurtherattemptswillbemade.IfWGETRCisnotset,Wgetwilltrytoload$HOME/.wgetrc.Thefactthatuser'ssettingsareloadedafterthesystem-wideonesmeansthatincaseofcol-lisionuser'swgetrcoverridesthesystem-widewgetrc(in/usr/local/etc/wgetrcbydefault).Fascistadmins,away!6.2WgetrcSyntaxThesyntaxofawgetrccommandissimple:variable=valueThevariablewillalsobecalledcommand.Validvaluesaredierentfordierentcommands.Thecommandsarecase-,underscore-andminus-insensitive.Thus`DIr__PrefiX',`DIr-PrefiX'and`dirprefix'arethesame.Emptylines,linesbeginningwith`#'andlinescontainingwhite-spaceonlyarediscarded.Commandsthatexpectacomma-separatedlistwillclearthelistonanemptycommand.So,ifyouwishtoresettherejectionlistspeciedinglobalwgetrc,youcandoitwith:reject=6.3WgetrcCommandsThecompletesetofcommandsislistedbelow.Legalvaluesarelistedafterthe`='.SimpleBooleanvaluescanbesetorunsetusing`on'and`off'or`1'and`0'.Somecommandstakepseudo-arbitraryvalues.addressvaluescanbehostnamesordotted-quadIPaddresses.ncanbeanypositiveinteger,or`inf'forinnity,whereappropriate.stringvaluescanbeanynon-emptystring.Mostofthesecommandshavedirectcommand-lineequivalents.Also,anywgetrccommandcanbespeciedonthecommandlineusingthe`--execute'switch(see Section2.3[BasicStartupOptions],page3 .)accept/reject=stringSameas`-A'/`-R'(see Section4.2[TypesofFiles],page34 ).add hostdir=on/oEnable/disablehost-prexedlenames.`-nH'disablesit. Chapter6:StartupFile41 ask password=on/oPromptforapasswordforeachconnectionestablished.Cannotbespeciedwhen`--password'isbeingused,becausetheyaremutuallyexclusive.Equivalentto`--ask-password'.auth no challenge=on/oIfthisoptionisgiven,WgetwillsendBasicHTTPauthenticationinformation(plaintextusernameandpassword)forallrequests.See`--auth-no-challenge'.background=on/oEnable/disablegoingtobackground|thesameas`-b'(whichenablesit).backup converted=on/oEnable/disablesavingpre-convertedleswiththesux`.orig'|thesameas`-K'(whichenablesit).backups=numberUseuptonumberbackupsforale.Backupsarerotatedbyaddinganincrementalcounterthatstartsat`1'.Thedefaultis`0'.base=stringConsiderrelativeurlsininputles(speciedviathe`input'commandorthe`--input-file'/`-i'option,togetherwith`force_html'or`--force-html')asbe-ingrelativetostring|thesameas`--base=string'.bind address=addressBindtoaddress,likethe`--bind-address=address'.ca certicate=leSetthecerticateauthoritybundleletole.Thesameas`--ca-certificate=file'.ca directory=directorySetthedirectoryusedforcerticateauthorities.Thesameas`--ca-directory=directory'.cache=on/oWhensettoo,disallowserver-caching.Seethe`--no-cache'option.certicate=leSettheclientcerticatelenametole.Thesameas`--certificate=file'.certicate type=stringSpecifythetypeoftheclientcerticate,legalvaluesbeing`PEM'(thedefault)and`DER'(akaASN1).Thesameas`--certificate-type=string'.check certicate=on/oIfthisissettoo,theservercerticateisnotcheckedagainstthespeciedclientauthorities.Thedefaultis\on".Thesameas`--check-certificate'.connect timeout=nSettheconnecttimeout|thesameas`--connect-timeout'.content disposition=on/oTurnonrecognitionofthe(non-standard)`Content-Disposition'HTTPheader|ifsetto`on',thesameas`--content-disposition'.trust server names=on/oIfsettoon,constructthelocallenamefromredirectionURLsratherthanoriginalURLs. Chapter6:StartupFile42 continue=on/oIfsettoon,forcecontinuationofpreexistentpartiallyretrievedles.See`-c'beforesettingit.convert links=on/oConvertnon-relativelinkslocally.Thesameas`-k'.cookies=on/oWhensettoo,disallowcookies.Seethe`--cookies'option.cut dirs=nIgnorenremotedirectorycomponents.Equivalentto`--cut-dirs=n'.debug=on/oDebugmode,sameas`-d'.default page=stringDefaultpagename|thesameas`--default-page=string'.delete after=on/oDeleteafterdownload|thesameas`--delete-after'.dir prex=stringTopofdirectorytree|thesameas`-Pstring'.dirstruct=on/oTurningdirstructonoro|thesameas`-x'or`-nd',respectively.dns cache=on/oTurnDNScachingon/o.SinceDNScachingisonbydefault,thisoptionisnor-mallyusedtoturnitoandisequivalentto`--no-dns-cache'.dns timeout=nSettheDNStimeout|thesameas`--dns-timeout'.domains=stringSameas`-D'(see Section4.1[SpanningHosts],page34 ).dot bytes=nSpecifythenumberofbytes\contained"inadot,asseenthroughouttheretrieval(1024bydefault).Youcanpostxthevaluewith`k'or`m',representingkilobytesandmegabytes,respectively.Withdotsettingsyoucantailorthedotretrievaltosuityourneeds,oryoucanusethepredenedstyles(see Section2.5[DownloadOptions],page6 ).dot spacing=nSpecifythenumberofdotsinasinglecluster(10bydefault).dots in line=nSpecifythenumberofdotsthatwillbeprintedineachlinethroughouttheretrieval(50bydefault).egd le=leUsestringastheEGDsocketlename.Thesameas`--egd-file=file'.exclude directories=stringSpecifyacomma-separatedlistofdirectoriesyouwishtoexcludefromdownload|thesameas`-Xstring'(see Section4.3[Directory-BasedLimits],page36 ).exclude domains=stringSameas`--exclude-domains=string'(see Section4.1[SpanningHosts],page34 ). Chapter6:StartupFile43 follow ftp=on/oFollowftplinksfromhtmldocuments|thesameas`--follow-ftp'.follow tags=stringOnlyfollowcertainhtmltagswhendoingarecursiveretrieval,justlike`--follow-tags=string'.force html=on/oIfsettoon,forcetheinputlenametoberegardedasanhtmldocument|thesameas`-F'.ftp password=stringSetyourftppasswordtostring.Withoutthissetting,thepassworddefaultsto`-wget@',whichisausefuldefaultforanonymousftpaccess.ThiscommandusedtobenamedpasswdpriortoWget1.10.ftp proxy=stringUsestringasftpproxy,insteadoftheonespeciedinenvironment.ftp user=stringSetftpusertostring.ThiscommandusedtobenamedloginpriortoWget1.10.glob=on/oTurnglobbingon/o|thesameas`--glob'and`--no-glob'.header=stringDeneaheaderforHTTPdownloads,likeusing`--header=string'.compression=stringChoosethecompressiontypetobeused.Legalvaluesare`auto'(thedefault),`gzip',and`none'.Thesameas`--compression=string'.adjust extension=on/oAdda`.html'extensionto`text/html'or`application/xhtml+xml'lesthatlackone,a`.css'extensionto`text/css'lesthatlackone,anda`.br',`.Z',`.zlib'or`.gz'tocompressedleslike`-E'.Previouslynamed`html_extension'(stillacceptable,butdeprecated).http keep alive=on/oTurnthekeep-alivefeatureonoro(defaultstoon).Turningitoisequivalentto`--no-http-keep-alive'.http password=stringSethttppassword,equivalentto`--http-password=string'.http proxy=stringUsestringashttpproxy,insteadoftheonespeciedinenvironment.http user=stringSethttpusertostring,equivalentto`--http-user=string'.https only=on/oWheninrecursivemode,onlyHTTPSlinksarefollowed(defaultstoo).https proxy=stringUsestringashttpsproxy,insteadoftheonespeciedinenvironment.ignore case=on/oWhensettoon,matchlesanddirectoriescaseinsensitively;thesameas`--ignore-case'. Chapter6:StartupFile44 ignore length=on/oWhensettoon,ignoreContent-Lengthheader;thesameas`--ignore-length'.ignore tags=stringIgnorecertainhtmltagswhendoingarecursiveretrieval,like`--ignore-tags=string'.include directories=stringSpecifyacomma-separatedlistofdirectoriesyouwishtofollowwhendownloading|thesameas`-Istring'.iri=on/oWhensettoon,enableinternationalizedURI(IRI)support;thesameas`--iri'.inet4 only=on/oForceconnectingtoIPv4addresses,obydefault.YoucanputthisintheglobalinitletodisableWget'sattemptstoresolveandconnecttoIPv6hosts.AvailableonlyifWgetwascompiledwithIPv6support.Thesameas`--inet4-only'or`-4'.inet6 only=on/oForceconnectingtoIPv6addresses,obydefault.AvailableonlyifWgetwascompiledwithIPv6support.Thesameas`--inet6-only'or`-6'.input=leReadtheurlsfromstring,like`-ifile'.keep session cookies=on/oWhenspecied,causes`save_cookies=on'toalsosavesessioncookies.See`--keep-session-cookies'.limit rate=rateLimitthedownloadspeedtonomorethanratebytespersecond.Thesameas`--limit-rate=rate'.load cookies=leLoadcookiesfromle.See`--load-cookiesfile'.local encoding=encodingForceWgettouseencodingasthedefaultsystemencoding.See`--local-encoding'.logle=leSetlogletole,thesameas`-ofile'.max redirect=numberSpeciesthemaximumnumberofredirectionstofollowforaresource.See`--max-redirect=number'.mirror=on/oTurnmirroringon/o.Thesameas`-m'.netrc=on/oTurnreadingnetrconoro.no clobber=on/oSameas`-nc'.no parent=on/oDisallowretrievingoutsidethedirectoryhierarchy,like`--no-parent'(see Section4.3[Directory-BasedLimits],page36 ). Chapter6:StartupFile45 no proxy=stringUsestringasthecomma-separatedlistofdomainstoavoidinproxyloading,insteadoftheonespeciedinenvironment.output document=leSettheoutputlename|thesameas`-Ofile'.page requisites=on/oDownloadallancillarydocumentsnecessaryforasinglehtmlpagetodisplayproperly|thesameas`-p'.passive ftp=on/oChangesettingofpassiveftp,equivalenttothe`--passive-ftp'option.password=stringSpecifypasswordstringforbothftpandhttpleretrieval.Thiscommandcanbeoverriddenusingthe`ftp_password'and`http_password'commandforftpandhttprespectively.post data=stringUsePOSTasthemethodforallHTTPrequestsandsendstringintherequestbody.Thesameas`--post-data=string'.post le=leUsePOSTasthemethodforallHTTPrequestsandsendthecontentsofleintherequestbody.Thesameas`--post-file=file'.prefer family=none/IPv4/IPv6Whengivenachoiceofseveraladdresses,connecttotheaddresseswithspeciedaddressfamilyrst.TheaddressorderreturnedbyDNSisusedwithoutchangebydefault.Thesameas`--prefer-family',whichseeforadetaileddiscussionofwhythisisuseful.private key=leSettheprivatekeyletole.Thesameas`--private-key=file'.private key type=stringSpecifythetypeoftheprivatekey,legalvaluesbeing`PEM'(thedefault)and`DER'(akaASN1).Thesameas`--private-type=string'.progress=stringSetthetypeoftheprogressindicator.Legaltypesare`dot'and`bar'.Equivalentto`--progress=string'.protocol directories=on/oWhenset,usetheprotocolnameasadirectorycomponentoflocallenames.Thesameas`--protocol-directories'.proxy password=stringSetproxyauthenticationpasswordtostring,like`--proxy-password=string'.proxy user=stringSetproxyauthenticationusernametostring,like`--proxy-user=string'.quiet=on/oQuietmode|thesameas`-q'.quota=quotaSpecifythedownloadquota,whichisusefultoputintheglobalwgetrc.Whendownloadquotaisspecied,Wgetwillstopretrievingafterthedownloadsumhas Chapter6:StartupFile46 becomegreaterthanquota.Thequotacanbespeciedinbytes(default),kbytes`k'appended)ormbytes(`m'appended).Thus`quota=5m'willsetthequotato5megabytes.Notethattheuser'sstartupleoverridessystemsettings.random le=leUseleasasourceofrandomnessonsystemslacking/dev/random.random wait=on/oTurnrandombetween-requestwaittimesonoro.Thesameas`--random-wait'.read timeout=nSettheread(andwrite)timeout|thesameas`--read-timeout=n'.reclevel=nRecursionlevel(depth)|thesameas`-ln'.recursive=on/oRecursiveon/o|thesameas`-r'.referer=stringSetHTTP`Referer:'headerjustlike`--referer=string'.(Notethatitwasthefolkswhowrotethehttpspecwhogotthespellingof\referrer"wrong.)relative only=on/oFollowonlyrelativelinks|thesameas`-L'(see Section4.4[RelativeLinks],page37 ).remote encoding=encodingForceWgettouseencodingasthedefaultremoteserverencoding.See`--remote-encoding'.remove listing=on/oIfsettoon,removeftplistingsdownloadedbyWget.Settingittooisthesameas`--no-remove-listing'.restrict le names=unix/windowsRestrictthelenamesgeneratedbyWgetfromURLs.See`--restrict-file-names'foramoredetaileddescription.retr symlinks=on/oWhensettoon,retrievesymboliclinksasiftheywereplainles;thesameas`--retr-symlinks'.retry connrefused=on/oWhensettoon,consider\connectionrefused"atransienterror|thesameas`--retry-connrefused'.robots=on/oSpecifywhetherthenorobotsconventionisrespectedbyWget,\on"bydefault.Thisswitchcontrolsboththe/robots.txtandthe`nofollow'aspectofthespec.See Section9.1[RobotExclusion],page58 ,formoredetailsaboutthis.Besureyouknowwhatyouaredoingbeforeturningthiso.save cookies=leSavecookiestole.Thesameas`--save-cookiesfile'.save headers=on/oSameas`--save-headers'.secure protocol=stringChoosethesecureprotocoltobeused.Legalvaluesare`auto'(thedefault),`SSLv2',`SSLv3',and`TLSv1'.Thesameas`--secure-protocol=string'. Chapter6:StartupFile47 server response=on/oChoosewhetherornottoprintthehttpandftpserverresponses|thesameas`-S'.show all dns entries=on/oWhenaDNSnameisresolved,showalltheIPaddresses,notjusttherstthree.span hosts=on/oSameas`-H'.spider=on/oSameas`--spider'.strict comments=on/oSameas`--strict-comments'.timeout=nSetallapplicabletimeoutvalueston,thesameas`-Tn'.timestamping=on/oTurntimestampingon/o.Thesameas`-N'(see Chapter5[Time-Stamping],page38 ).use server timestamps=on/oIfsetto`off',Wgetwon'tsetthelocalle'stimestampbytheoneontheserver(sameas`--no-use-server-timestamps').tries=nSetnumberofretriesperurl|thesameas`-tn'.use proxy=on/oWhensettoo,don'tuseproxyevenwhenproxy-relatedenvironmentvariablesareset.Inthatcaseitisthesameasusing`--no-proxy'.user=stringSpecifyusernamestringforbothftpandhttpleretrieval.Thiscommandcanbeoverriddenusingthe`ftp_user'and`http_user'commandforftpandhttprespectively.user agent=stringUseragentidenticationsenttotheHTTPServer|thesameas`--user-agent=string'.verbose=on/oTurnverboseon/o|thesameas`-v'/`-nv'.wait=nWaitnsecondsbetweenretrievals|thesameas`-wn'.wait retry=nWaituptonsecondsbetweenretriesoffailedretrievalsonly|thesameas`--waitretry=n'.Notethatthisisturnedonbydefaultintheglobalwgetrc.6.4SampleWgetrcThisisthesampleinitializationle,asgiveninthedistribution.Itisdividedintwosection|oneforglobalusage(suitableforglobalstartuple),andoneforlocalusage(suitablefor$HOME/.wgetrc).Becarefulaboutthethingsyouchange.Notethatalmostallthelinesarecommentedout.Foracommandtohaveanyeect,youmustremovethe`#'characteratthebeginningofitsline.######SampleWgetinitializationfile.wgetrc Chapter6:StartupFile48 #####Youcanusethisfiletochangethedefaultbehaviourofwgetorto##avoidhavingtotypemanymanycommand-lineoptions.Thisfiledoes##notcontainacomprehensivelistofcommands--lookatthemanual##tofindoutwhatyoucanputintothisfile.Youcanfindthishere:##$infowget.info'StartupFile'##Oronlinehere:##https://www.gnu.org/software/wget/manual/wget.html#Startup-File####Wgetinitializationfilecanresidein/usr/local/etc/wgetrc##(global,forallusers)or$HOME/.wgetrc(forasingleuser).####Tousethesettingsinthisfile,youwillhavetouncommentthem,##aswellaschangethem,inmostcases,asthevaluesonthe##commented-outlinesarethedefaultvalues(e.g."off").####Commandarecase-,underscore-andminus-insensitive.##Forexampleftp_proxy,ftp-proxyandftpproxyarethesame.####Globalsettings(usefulforsettingupin/usr/local/etc/wgetrc).##Thinkwellbeforeyouchangethem,sincetheymayreducewget's##functionality,andmakeitbehavecontrarytothedocumentation:###Youcansetretrievequotaforbeginnersbyspecifyingavalue#optionallyfollowedby'K'(kilobytes)or'M'(megabytes).The#defaultquotaisunlimited.#quota=inf#Youcanlower(orraise)thedefaultnumberofretrieswhen#downloadingafile(defaultis20).#tries=20#Loweringthemaximumdepthoftherecursiveretrievalishandyto#preventnewbiesfromgoingtoo"deep"whentheyunwittinglystart#therecursiveretrieval.Thedefaultis5.#reclevel=5#BydefaultWgetuses"passiveFTP"transferwheretheclient#initiatesthedataconnectiontotheserverratherthantheother#wayaround.ThatisrequiredonsystemsbehindNATwheretheclient#computercannotbeeasilyreachedfromtheInternet.However,some#firewallssoftwareexplicitlysupportsactiveFTPandinfacthas#problemssupportingpassivetransfer.Ifyouareinsuch#environment,use"passive_ftp=off"toreverttoactiveFTP.#passive_ftp=off#The"wait"commandbelowmakesWgetwaitbetweeneveryconnection.#If,instead,youwantWgettowaitonlybetweenretriesoffailed Chapter6:StartupFile49 #downloads,setwaitretrytomaximumnumberofsecondstowait(Wget#willuse"linearbackoff",waiting1secondafterthefirstfailure#onafile,2secondsafterthesecondfailure,etc.uptothismax).#waitretry=10####Localsettings(forausertosetinhis$HOME/.wgetrc).Itis##*highly*undesirabletoputthesesettingsintheglobalfile,since##theyarepotentiallydangerousto"normal"users.####Evenwhensettingupyourown~/.wgetrc,youshouldknowwhatyou##aredoingbeforedoingso.###Setthistoontousetimestampingbydefault:#timestamping=off#ItisagoodideatomakeWgetsendyouremailaddressina`From:'#headerwithyourrequest(sothatserveradministratorscancontact#youincaseoferrors).Wgetdoes*not*send`From:'bydefault.#header=From:YourName&#xuser;&#xname;&#x@sit;.do;&#xmain;#Youcansetupotherheaders,likeAccept-Language.Accept-Language#is*not*sentbydefault.#header=Accept-Language:en#YoucansetthedefaultproxiesforWgettouseforhttp,https,andftp.#Theywilloverridethevalueintheenvironment.#https_proxy=http://proxy.yoyodyne.com:18023/#http_proxy=http://proxy.yoyodyne.com:18023/#ftp_proxy=http://proxy.yoyodyne.com:18023/#Ifyoudonotwanttouseproxyatall,setthistooff.#use_proxy=on#Youcancustomizetheretrievaloutlook.Validoptionsaredefault,#binary,megaandmicro.#dot_style=default#SettingthistooffmakesWgetnotdownload/robots.txt.Besureto#know*exactly*what/robots.txtisandhowitisusedbeforechanging#thedefault!#robots=on#ItcanbeusefultomakeWgetwaitbetweenconnections.Setthisto#thenumberofsecondsyouwantWgettowait.#wait=0#Youcanforcecreatingdirectorystructure,evenifasingleisbeing#retrieved,bysettingthistoon.#dirstruct=off 50 #Youcanturnonrecursiveretrievingbydefault(don'tdothisif#youarenotsureyouknowwhatitmeans)bysettingthistoon.#recursive=off#ToalwaysbackupfileXasX.origbeforeconvertingitslinks(due#to-k/--convert-links/convert_links=onhavingbeenspecified),#setthisvariabletoon:#backup_converted=off#TohaveWgetfollowFTPlinksfromHTMLfilesbydefault,setthis#toon:#follow_ftp=off#Totryipv6addressesfirst:#prefer-family=IPv6#SetdefaultIRIsupportstate#iri=off#Forcethedefaultsystemencoding#localencoding=UTF-8#Forcethedefaultremoteserverencoding#remoteencoding=UTF-8#Turnontopreventfollowingnon-HTTPSlinkswheninrecursivemode#httpsonly=off#TuneHTTPSsecurity(auto,SSLv2,SSLv3,TLSv1,PFS)#secureprotocol=auto 51 7ExamplesTheexamplesaredividedintothreesectionslooselybasedontheircomplexity.7.1SimpleUsageSayyouwanttodownloadaurl.Justtype:wgethttp://fly.srk.fer.hr/Butwhatwillhappeniftheconnectionisslow,andtheleislengthy?Theconnectionwillprobablyfailbeforethewholeleisretrieved,morethanonce.Inthiscase,Wgetwilltrygettingtheleuntiliteithergetsthewholeofit,orexceedsthedefaultnumberofretries(thisbeing20).Itiseasytochangethenumberoftriesto45,toinsurethatthewholelewillarrivesafely:wget--tries=45http://fly.srk.fer.hr/jpg/flyweb.jpgNowlet'sleaveWgettoworkinthebackground,andwriteitsprogresstologlelog.Itistiringtotype`--tries',soweshalluse`-t'.wget-t45-ologhttp://fly.srk.fer.hr/jpg/flyweb.jpg&TheampersandattheendofthelinemakessurethatWgetworksinthebackground.Tounlimitthenumberofretries,use`-tinf'.Theusageofftpisassimple.Wgetwilltakecareofloginandpassword.wgetftp://gnjilux.srk.fer.hr/welcome.msgIfyouspecifyadirectory,Wgetwillretrievethedirectorylisting,parseitandconvertittohtml.Try:wgetftp://ftp.gnu.org/pub/gnu/linksindex.html7.2AdvancedUsageYouhavealethatcontainstheURLsyouwanttodownload?Usethe`-i'switch:wget-ifileIfyouspecify`-'aslename,theurlswillbereadfromstandardinput.CreateavelevelsdeepmirrorimageoftheGNUwebsite,withthesamedirectorystructuretheoriginalhas,withonlyonetryperdocument,savingthelogoftheactivitiestognulog:wget-rhttps://www.gnu.org/-ognulogThesameastheabove,butconvertthelinksinthedownloadedlestopointtolocalles,soyoucanviewthedocumentso-line:wget--convert-links-rhttps://www.gnu.org/-ognulogRetrieveonlyonehtmlpage,butmakesurethatalltheelementsneededforthepagetobedisplayed,suchasinlineimagesandexternalstylesheets,arealsodownloaded.Alsomakesurethedownloadedpagereferencesthedownloadedlinks.wget-p--convert-linkshttp://www.example.com/dir/page.htmlThehtmlpagewillbesavedtowww.example.com/dir/page.html,andtheimages,stylesheets,etc.,somewhereunderwww.example.com/,dependingonwheretheywereontheremoteserver.Thesameastheabove,butwithoutthewww.example.com/directory.Infact,Idon'twanttohaveallthoserandomserverdirectoriesanyway|justsaveallthoselesunderadownload/subdirectoryofthecurrentdirectory.wget-p--convert-links-nH-nd-Pdownload\http://www.example.com/dir/page.html Chapter7:Examples52 Retrievetheindex.htmlof`www.lycos.com',showingtheoriginalserverheaders:wget-Shttp://www.lycos.com/Savetheserverheaderswiththele,perhapsforpost-processing.wget--save-headershttp://www.lycos.com/moreindex.htmlRetrievethersttwolevelsof`wuarchive.wustl.edu',savingthemto/tmp.wget-r-l2-P/tmpftp://wuarchive.wustl.edu/Youwanttodownloadallthegifsfromadirectoryonanhttpserver.Youtried`wgethttp://www.example.com/dir/*.gif',butthatdidn'tworkbecausehttpretrievaldoesnotsupportglobbing.Inthatcase,use:wget-r-l1--no-parent-A.gifhttp://www.example.com/dir/Moreverbose,buttheeectisthesame.`-r-l1'meanstoretrieverecursively(see Chapter3[RecursiveDownload],page33 ),withmaximumdepthof1.`--no-parent'meansthatreferencestotheparentdirectoryareignored(see Section4.3[Directory-BasedLimits],page36 ),and`-A.gif'meanstodownloadonlythegifles.`-A"*.gif"'wouldhaveworkedtoo.Supposeyouwereinthemiddleofdownloading,whenWgetwasinterrupted.Nowyoudonotwanttoclobberthelesalreadypresent.Itwouldbe:wget-nc-rhttps://www.gnu.org/Ifyouwanttoencodeyourownusernameandpasswordtohttporftp,usetheappropriateurlsyntax(see Section2.1[URLFormat],page2 ).wgetftp://hniksic:mypassword@unix.example.com/.emacsNote,however,thatthisusageisnotadvisableonmulti-usersystemsbecauseitrevealsyourpasswordtoanyonewholooksattheoutputofps.Youwouldliketheoutputdocumentstogotostandardoutputinsteadoftoles?wget-O-http://jagor.srce.hr/http://www.srce.hr/Youcanalsocombinethetwooptionsandmakepipelinestoretrievethedocumentsfromremotehotlists:wget-O-http://cool.list.com/|wget--force-html-i-7.3VeryAdvancedUsageIfyouwishWgettokeepamirrorofapage(orftpsubdirectories),use`--mirror'(`-m'),whichistheshorthandfor`-r-linf-N'.YoucanputWgetinthecrontableaskingittorecheckasiteeachSunday:crontab00**0wget--mirrorhttps://www.gnu.org/-o/home/me/weeklogInadditiontotheabove,youwantthelinkstobeconvertedforlocalviewing.But,afterhavingreadthismanual,youknowthatlinkconversiondoesn'tplaywellwithtimestamping,soyoualsowantWgettobackuptheoriginalhtmllesbeforetheconversion.Wgetinvocationwouldlooklikethis:wget--mirror--convert-links--backup-converted\https://www.gnu.org/-o/home/me/weeklogButyou'vealsonoticedthatlocalviewingdoesn'tworkallthatwellwhenhtmllesaresavedunderextensionsotherthan`.html',perhapsbecausetheywereservedasindex.cgi.Soyou'dlikeWgettorenameallthelesservedwithcontent-type`text/html'or`application/xhtml+xml'toname.html.wget--mirror--convert-links--backup-converted\ 53 --html-extension-o/home/me/weeklog\https://www.gnu.org/Or,withlesstyping:wget-m-k-K-Ehttps://www.gnu.org/-o/home/me/weeklog 54 8VariousThischaptercontainsallthestuthatcouldnottanywhereelse.8.1ProxiesProxiesarespecial-purposehttpserversdesignedtotransferdatafromremoteserverstolocalclients.Onetypicaluseofproxiesislighteningnetworkloadforusersbehindaslowconnection.Thisisachievedbychannelingallhttpandftprequeststhroughtheproxywhichcachesthetransferreddata.Whenacachedresourceisrequestedagain,proxywillreturnthedatafromcache.Anotheruseforproxiesisforcompaniesthatseparate(forsecurityreasons)theirinternalnetworksfromtherestofInternet.InordertoobtaininformationfromtheWeb,theirusersconnectandretrieveremotedatausinganauthorizedproxy.Wgetsupportsproxiesforbothhttpandftpretrievals.Thestandardwaytospecifyproxylocation,whichWgetrecognizes,isusingthefollowingenvironmentvariables:http_proxyhttps_proxyIfset,thehttp_proxyandhttps_proxyvariablesshouldcontaintheurlsoftheproxiesforhttpandhttpsconnectionsrespectively.ftp_proxyThisvariableshouldcontaintheurloftheproxyforftpconnections.Itisquitecommonthathttp_proxyandftp_proxyaresettothesameurl.no_proxyThisvariableshouldcontainacomma-separatedlistofdomainextensionsproxyshouldnotbeusedfor.Forinstance,ifthevalueofno_proxyis`.mit.edu',proxywillnotbeusedtoretrievedocumentsfromMIT.Inadditiontotheenvironmentvariables,proxylocationandsettingsmaybespeciedfromwithinWgetitself.`--no-proxy'`proxy=on/off'Thisoptionandthecorrespondingcommandmaybeusedtosuppresstheuseofproxy,eveniftheappropriateenvironmentvariablesareset.`http_proxy=URL'`https_proxy=URL'`ftp_proxy=URL'`no_proxy=string'Thesestartuplevariablesallowyoutooverridetheproxysettingsspeciedbytheenvironment.Someproxyserversrequireauthorizationtoenableyoutousethem.Theauthorizationconsistsofusernameandpassword,whichmustbesentbyWget.Aswithhttpauthorization,severalauthenticationschemesexist.ForproxyauthorizationonlytheBasicauthenticationschemeiscurrentlyimplemented.Youmayspecifyyourusernameandpasswordeitherthroughtheproxyurlorthroughthecommand-lineoptions.Assumingthatthecompany'sproxyislocatedat`proxy.company.com'atport8001,aproxyurllocationcontainingauthorizationdatamightlooklikethis:http://hniksic:mypassword@proxy.company.com:8001/Alternatively,youmayusethe`proxy-user'and`proxy-password'options,andtheequiva-lent.wgetrcsettingsproxy_userandproxy_passwordtosettheproxyusernameandpassword. Chapter8:Various55 8.2DistributionLikeallGNUutilities,thelatestversionofWgetcanbefoundatthemasterGNUarchivesiteftp.gnu.org,anditsmirrors.Forexample,Wget1.20canbefoundat https://ftp.gnu.org/pub/gnu/wget/wget-1.20.tar.gz 8.3WebSiteTheocialwebsiteforGNUWgetisat https//www.gnu.org/software/wget/ .However,mostusefulinformationresidesat\TheWgetWgiki", http://wget.addictivecode.org/ .8.4MailingListsPrimaryListTheprimarymailinglistfordiscussion,bug-reports,orquestionsaboutGNUWgetisat bug-wget@gnu.org .Tosubscribe,sendanemailto bug-wget-join@gnu.org ,orvisit https://lists.gnu.org/mailman/listinfo/bug-wget .Youdonotneedtosubscribetosendamessagetothelist;however,pleasenotethatunsub-scribedmessagesaremoderated,andmaytakeawhilebeforetheyhitthelist|usuallyaroundaday.Ifyouwantyourmessagetoshowupimmediately,pleasesubscribetothelistbeforepost-ing.Archivesforthelistmaybefoundat https://lists.gnu.org/archive/html/bug-wget/ .AnNNTP/Usenettishgatewayisalsoavailablevia Gmane(http://gmane.org/about.php) .YoucanseetheGmanearchivesat http://news.gmane.org/gmane.comp.web.wget.general .NotethattheGmanearchivesconvenientlyincludemessagesfromboththecurrentlist,andthepreviousone.MessagesalsoshowupintheGmanearchivessoonerthantheydoat https://lists.gnu.org .ObsoleteListsPreviously,themailinglist wget@sunsite.dk wasusedasthemaindiscussionlist,andanotherlist, wget-patches@sunsite.dk wasusedforsubmittinganddiscussingpatchestoGNUWget.Messagesfrom wget@sunsite.dk arearchivedat https://www.mail-archive.com/wget%40sunsite.dk/ andat http://news.gmane.org/gmane.comp.web.wget.general (whichalsocontinuestoarchivethecurrentlist, bug-wget@gnu.org ).Messagesfrom wget-patches@sunsite.dk arearchivedat http://news.gmane.org/gmane.comp.web.wget.patches .8.5InternetRelayChatInadditiontothemailinglists,wealsohaveasupportchannelsetupviaIRCatirc.freenode.org,#wget.Comecheckitout!8.6ReportingBugsYouarewelcometosubmitbugreportsviatheGNUWgetbugtracker(see https://savannah.gnu.org/bugs/?func=additem&group=wget )ortoourmailinglist bug-wget@gnu.org .Visit https://lists.gnu.org/mailman/listinfo/bug-wget togetmoreinfo(howtosubscribe,listarchives,...). Chapter8:Various56 Beforeactuallysubmittingabugreport,pleasetrytofollowafewsimpleguidelines.1.Pleasetrytoascertainthatthebehavioryouseereallyisabug.IfWgetcrashes,it'sabug.IfWgetdoesnotbehaveasdocumented,it'sabug.Ifthingsworkstrange,butyouarenotsureaboutthewaytheyaresupposedtowork,itmightwellbeabug,butyoumightwanttodouble-checkthedocumentationandthemailinglists(see Section8.4[MailingLists],page55 ).2.Trytorepeatthebuginassimplecircumstancesaspossible.E.g.ifWgetcrasheswhiledownloading`wget-rl0-kKE-t5--no-proxyhttp://example.com-o/tmp/log',youshouldtrytoseeifthecrashisrepeatable,andifwilloccurwithasimplersetofoptions.Youmighteventrytostartthedownloadatthepagewherethecrashoccurredtoseeifthatpagesomehowtriggeredthecrash.Also,whileIwillprobablybeinterestedtoknowthecontentsofyour.wgetrcle,justdumpingitintothedebugmessageisprobablyabadidea.Instead,youshouldrsttrytoseeifthebugrepeatswith.wgetrcmovedoutoftheway.Onlyifitturnsoutthat.wgetrcsettingsaectthebug,mailmetherelevantpartsofthele.3.PleasestartWgetwith`-d'optionandsendustheresultingoutput(orrelevantpartsthereof).IfWgetwascompiledwithoutdebugsupport,recompileit|itismucheasiertotracebugswithdebugsupporton.Note:pleasemakesuretoremoveanypotentiallysensitiveinformationfromthedebuglogbeforesendingittothebugaddress.The-dwon'tgooutofitswaytocollectsensitiveinformation,butthelogwillcontainafairlycompletetranscriptofWget'scommunicationwiththeserver,whichmayincludepasswordsandpiecesofdownloadeddata.Sincethebugaddressispublicallyarchived,youmayassumethatallbugreportsarevisibletothepublic.4.IfWgethascrashed,trytorunitinadebugger,e.g.gdb`whichwget`coreandtypewheretogetthebacktrace.Thismaynotworkifthesystemadministratorhasdisabledcoreles,butitissafetotry.8.7PortabilityLikeallGNUsoftware,WgetworksontheGNUsystem.However,sinceitusesGNUAutoconfforbuildingandconguring,andmostlyavoidsusing\special"featuresofanyparticularUnix,itshouldcompile(andwork)onallcommonUnix avors.VariousWgetversionshavebeencompiledandtestedundermanykindsofUnixsystems,includingGNU/Linux,Solaris,SunOS4.x,MacOSX,OSF(akaDigitalUnixorTru64),Ultrix,*BSD,IRIX,AIX,andothers.SomeofthosesystemsarenolongerinwidespreaduseandmaynotbeabletosupportrecentversionsofWget.IfWgetfailstocompileonyoursystem,wewouldliketoknowaboutit.Thankstokindcontributors,thisversionofWgetcompilesandworkson32-bitMicrosoftWindowsplatforms.IthasbeencompiledsuccessfullyusingMSVisualC++6.0,Watcom,BorlandC,andGCCcompilers.Naturally,itiscrippledofsomefeaturesavailableonUnix,butitshouldworkasasubstituteforpeoplestuckwithWindows.NotethatWindows-specicportionsofWgetarenotguaranteedtobesupportedinthefuture,althoughthishasbeenthecaseinpracticeformanyyearsnow.AllquestionsandproblemsinWindowsusageshouldbereportedtoWgetmailinglistat wget@sunsite.dk wherethevolunteerswhomaintaintheWindows-relatedfeaturesmightlookatthem.SupportforbuildingonMS-DOSviaDJGPPhasbeencontributedbyGisleVanem;aporttoVMSismaintainedbyStevenSchweda,andisavailableat https://antinode.info/dec/sw/wget.html . Chapter8:Various57 8.8SignalsSincethepurposeofWgetisbackgroundwork,itcatchesthehangupsignal(SIGHUP)andignoresit.Iftheoutputwasonstandardoutput,itwillberedirectedtoalenamedwget-log.Otherwise,SIGHUPisignored.ThisisconvenientwhenyouwishtoredirecttheoutputofWgetafterhavingstartedit.$wgethttp://www.gnus.org/dist/gnus.tar.gz&...$kill-HUP%%SIGHUPreceived,redirectingoutputto`wget-log'.Otherthanthat,Wgetwillnottrytointerferewithsignalsinanyway.C-c,kill-TERMandkill-KILLshouldkillitalike. 58 9AppendicesThischaptercontainssomereferencesIconsideruseful.9.1RobotExclusionItisextremelyeasytomakeWgetwanderaimlesslyaroundawebsite,suckingalltheavailabledatainprogress.`wget-rsite',andyou'reset.Great?Notfortheserveradmin.AslongasWgetisonlyretrievingstaticpages,anddoingitatareasonablerate(seethe`--wait'option),there'snotmuchofaproblem.ThetroubleisthatWgetcan'ttellthedierencebetweenthesmalleststaticpageandthemostdemandingCGI.AsiteIknowhasasectionhandledbyaCGIPerlscriptthatconvertsInfolestohtmlonthe y.Thescriptisslow,butworkswellenoughforhumanusersviewinganoccasionalInfole.However,whensomeone'srecursiveWgetdownloadstumblesupontheindexpagethatlinkstoalltheInfolesthroughthescript,thesystemisbroughttoitskneeswithoutprovidinganythingusefultotheuser(ThistaskofconvertingInfolescouldbedonelocallyandaccesstoInfodocumentationforallinstalledGNUsoftwareonasystemisavailablefromtheinfocommand).Toavoidthiskindofaccident,aswellastopreserveprivacyfordocumentsthatneedtobeprotectedfromwell-behavedrobots,theconceptofrobotexclusionwasinvented.Theideaisthattheserveradministratorsanddocumentauthorscanspecifywhichportionsofthesitetheywishtoprotectfromrobotsandthosetheywillpermitaccess.Themostpopularmechanism,andthedefactostandardsupportedbyallthemajorrobots,isthe\RobotsExclusionStandard"(RES)writtenbyMartijnKosteretal.in1994.ItspeciestheformatofatextlecontainingdirectivesthatinstructtherobotswhichURLpathstoavoid.Tobefoundbytherobots,thespecicationsmustbeplacedin/robots.txtintheserverroot,whichtherobotsareexpectedtodownloadandparse.AlthoughWgetisnotawebrobotinthestrictestsenseoftheword,itcandownloadlargepartsofthesitewithouttheuser'sinterventiontodownloadanindividualpage.Becauseofthat,WgethonorsRESwhendownloadingrecursively.Forinstance,whenyouissue:wget-rhttp://www.example.com/Firsttheindexof`www.example.com'willbedownloaded.IfWgetndsthatitwantstodownloadmoredocumentsfromthatserver,itwillrequest`http://www.example.com/robots.txt'and,iffound,useitforfurtherdownloads.robots.txtisloadedonlyoncepereachserver.Untilversion1.8,Wgetsupportedtherstversionofthestandard,writtenbyMar-tijnKosterin1994andavailableat http://www.robotstxt.org/orig.html .Asofversion1.8,Wgethassupportedtheadditionaldirectivesspeciedintheinternetdraft` raf;&#xt-ko;&#xster;&#x-rob;&#xots-;�.t;&#xxt00;'titled\AMethodforWebRobotsControl".Thedraft,whichhasasfarasIknownevermadetoanrfc,isavailableat http://www.robotstxt.org/norobots-rfc.txt .ThismanualnolongerincludesthetextoftheRobotExclusionStandard.Thesecond,lessknownmechanism,enablestheauthorofanindividualdocumenttospecifywhethertheywantthelinksfromtheletobefollowedbyarobot.ThisisachievedusingtheMETAtag,likethis:name="robots"&#xmeta;&#x-525;content="nofollow"Thisisexplainedinsomedetailat http://www.robotstxt.org/meta.html .Wgetsupportsthismethodofrobotexclusioninadditiontotheusual/robots.txtexclusion.Ifyouknowwhatyouaredoingandreallyreallywishtoturnotherobotexclusion,settherobotsvariableto`off'inyour.wgetrc.Youcanachievethesameeectfromthecommandlineusingthe-eswitch,e.g.`wget-erobots=offurl...'. Chapter9:Appendices59 9.2SecurityConsiderationsWhenusingWget,youmustbeawarethatitsendsunencryptedpasswordsthroughthenetwork,whichmaypresentasecurityproblem.Herearethemainissues,andsomesolutions.1.Thepasswordsonthecommandlinearevisibleusingps.Thebestwayarounditistousewget-i-andfeedtheurlstoWget'sstandardinput,eachonaseparateline,terminatedbyC-d.Anotherworkaroundistouse.netrctostorepasswords;however,storingunencryptedpasswordsisalsoconsideredasecurityrisk.2.Usingtheinsecurebasicauthenticationscheme,unencryptedpasswordsaretransmittedthroughthenetworkroutersandgateways.3.Theftppasswordsarealsoinnowayencrypted.Thereisnogoodsolutionforthisatthemoment.4.Althoughthe\normal"outputofWgettriestohidethepasswords,debugginglogsshowthem,inallforms.Thisproblemisavoidedbybeingcarefulwhenyousenddebuglogs(yes,evenwhenyousendthemtome).9.3ContributorsGNUWgetwaswrittenbyHrvojeNiksic hniksic@xemacs.org ,However,thedevelopmentofWgetcouldneverhavegoneasfarasithas,wereitnotforthehelpofmanypeople,eitherwithbugreports,featureproposals,patches,orletterssaying\Thanks!".Specialthanksgoestothefollowingpeople(noparticularorder):DanHarkless|contributedalotofcodeanddocumentationofextremelyhighquality,aswellasthe--page-requisitesandrelatedoptions.HewastheprincipalmaintainerforsometimeandreleasedWget1.6.IanAbbott|contributedbugxes,Windows-relatedxes,andprovidedaprototypeim-plementationofthebreadth-rstrecursivedownload.Co-maintainedWgetduringthe1.8releasecycle.Thedotsrc.orgcrew,inparticularKarstenThygesen|donatedsystemresourcessuchasthemailinglist,webspace,ftpspace,andversioncontrolrepositories,alongwithalotoftimetomaketheseactuallywork.ChristianReinigerwasofinvaluablehelpwithsettingupSubversion.HeikoHerold|providedhigh-qualityWindowsbuildsandcontributedbugandbuildreportsformanyyears.ShawnMcHorse|bugreportsandpatches.KavehR.Ghazi|on-the- yansi2knr-ization.Lotsofportabilityxes.GordonMatzigkeit|.netrcsupport.ZlatkoCalusic,TomislavVujecandDrazenKacar|featuresuggestionsand\philosophical"discussions.DarkoBudor|initialporttoWindows.AntonioRosella|helpandsuggestions,plustheinitialItaliantranslation.TomislavPetrovic,MarioMikocevic|manybugreportsandsuggestions.FrancoisPinard|manythoroughbugreportsanddiscussions.KarlEichwalder|lotsofhelpwithinternationalization,Makelelayoutandmanyotherthings.JunioHamano|donatedsupportforOpieandhttpDigestauthentication.MauroTortonesi|improvedIPv6support,addingsupportfordualfamilysystems.Refac-toredandenhancedFTPIPv6code.MaintainedGNUWgetfrom2004{2007. 60 ChristopherG.Lewis|maintenanceoftheWindowsversionofGNUWGet.GisleVanem|manyhelpfulpatchesandimprovements,especiallyforWindowsandMS-DOSsupport.RalfWildenhues|contributedpatchestoconvertWgettouseAutomakeaspartofitsbuildprocess,andvariousbugxes.StevenSchubiger|Manyhelpfulpatches,bugxesandimprovements.Notably,conversionofWgettousetheGnulibquotesandquoteargsmodules,andtheadditionofpasswordpromptsattheconsole,viatheGnulibgetpasswd-gnumodule.TedMielczarek|donatedsupportforCSS.SaintXavier|SupportforIRIs(RFC3987).Peoplewhoprovideddonationsfordevelopment|includingBrianGough.Thefollowingpeoplehaveprovidedpatches,bug/buildreports,usefulsuggestions,betatest-ingservices,fanmailandalltheotherthingsthatmakemaintenancesomuchfun:TimAdam,AdrianAichner,MartinBaehr,DieterBaron,RogerBeeman,DanBerger,T.Bharath,ChristianBiere,PaulBludov,DanielBodea,MarkBoyns,JohnBurden,JulienButy,WanderleiCavassin,GillesCedoc,TimCharron,NoelCragg,KristijanConkas,JohnDaily,An-dreasDamm,AhmonDancy,AndrewDavison,BertrandDemiddelaer,AlexanderDergachev,AndrewDeryabin,UlrichDrepper,MarcDuponcheel,DamirDzeko,AlanEldridge,Hans-AndreasEngel,AleksandarErkalovic,AndyEskilsson,Jo~aoFerreira,ChristianFraenkel,DavidFritz,MikeFrysinger,CharlesC.Fu,FUJISHIMASatsuki,MasashiFujita,HowardGayle,Mar-celGerrits,LembleGregory,HansGrobler,AlainGuibert,MathieuGuillaume,AaronHawley,JochenHein,KarlHeuer,MadhusudanHosaagrahara,HIROSEMasaaki,UlfHarnhammar,Gre-gorHoeit,ErikMagnusHulthen,RichardHuveneers,JonasJensen,LarryJones,SimonJosef-sson,MarioJuric,HackKampbjrn,ConstKaplinsky,GoranKezunovic,IgorKhristophorov,RobertKleine,KOJIMAHaime,FilaKolodny,AlexanderKourakos,MartinKraemer,SamiKrank,JayKrell,o&&(SimosKSenitellis),ChristianLackas,HrvojeLacko,DanielS.Lewart,NicolasLichtmeier,DaveLove,AlexanderV.Lukyanov,ThomasLunig,AndreMajorel,AurelienMarchand,MatthewJ.Mellon,JordanMendelson,TedMielczarek,RobertMillan,LinZheMin,JanMinar,TimMooney,KeithMoore,AdamD.Moss,SimonMunton,CharlieNegyesi,R.K.Owen,JimParis,KennyParnell,LeonidPetrov,SimonePi-unno,AndrewPollock,StevePothier,JanPrikryl,MarinPurgar,CsabaRaduly,KeithRefson,BillRichardson,TylerRiddle,TobiasRingstrom,JochenRoderburg,JuanJoseRodrguez,Ma-ciejW.Rozycki,EdwardJ.Sabol,HeinzSalzmann,RobertSchmidt,NicolasSchodet,BennoSchulenberg,AndreasSchwab,StevenM.Schweda,ChrisSeawood,PranabShenoy,DennisSmit,ToomasSoome,TageStabell-Kulo,PhilipStadermann,DanielStenberg,SvenSternberger,MarkusStrasser,JohnSummereld,SzakacsitsSzabolcs,MikeThomas,PhilippThomas,MauroTortonesi,DaveTurner,GisleVanem,RabinVincent,RussellVincent,ZeljkoVrba,CharlesGWaldman,DouglasE.Wegscheid,RalfWildenhues,JoshuaDavidWilliams,BenjaminWolsey,SaintXavier,YAMAZAKIMakoto,JasminZainul,BojanZdrnja,KristijanZimmer,XinZou.ApologiestoallwhoIaccidentallyleftout,andmanythankstoallthesubscribersoftheWgetmailinglist. 61 AppendixACopyingthismanualA.1GNUFreeDocumentationLicenseVersion1.3,3November2008Copyrightc 2000-2002,2007-2008,2015,2018FreeSoftwareFoundation,Inc. http://fsf.org/ Everyoneispermittedtocopyanddistributeverbatimcopiesofthislicensedocument,butchangingitisnotallowed.0.PREAMBLEThepurposeofthisLicenseistomakeamanual,textbook,orotherfunctionalandusefuldocumentfreeinthesenseoffreedom:toassureeveryonetheeectivefreedomtocopyandredistributeit,withorwithoutmodifyingit,eithercommerciallyornoncommercially.Secondarily,thisLicensepreservesfortheauthorandpublisherawaytogetcreditfortheirwork,whilenotbeingconsideredresponsibleformodicationsmadebyothers.ThisLicenseisakindof\copyleft",whichmeansthatderivativeworksofthedocumentmustthemselvesbefreeinthesamesense.ItcomplementstheGNUGeneralPublicLicense,whichisacopyleftlicensedesignedforfreesoftware.WehavedesignedthisLicenseinordertouseitformanualsforfreesoftware,becausefreesoftwareneedsfreedocumentation:afreeprogramshouldcomewithmanualsprovidingthesamefreedomsthatthesoftwaredoes.ButthisLicenseisnotlimitedtosoftwaremanuals;itcanbeusedforanytextualwork,regardlessofsubjectmatterorwhetheritispublishedasaprintedbook.WerecommendthisLicenseprincipallyforworkswhosepurposeisinstructionorreference.1.APPLICABILITYANDDEFINITIONSThisLicenseappliestoanymanualorotherwork,inanymedium,thatcontainsanoticeplacedbythecopyrightholdersayingitcanbedistributedunderthetermsofthisLicense.Suchanoticegrantsaworld-wide,royalty-freelicense,unlimitedinduration,tousethatworkundertheconditionsstatedherein.The\Document",below,referstoanysuchmanualorwork.Anymemberofthepublicisalicensee,andisaddressedas\you".Youacceptthelicenseifyoucopy,modifyordistributetheworkinawayrequiringpermissionundercopyrightlaw.A\ModiedVersion"oftheDocumentmeansanyworkcontainingtheDocumentoraportionofit,eithercopiedverbatim,orwithmodicationsand/ortranslatedintoanotherlanguage.A\SecondarySection"isanamedappendixorafront-mattersectionoftheDocumentthatdealsexclusivelywiththerelationshipofthepublishersorauthorsoftheDocumenttotheDocument'soverallsubject(ortorelatedmatters)andcontainsnothingthatcouldfalldirectlywithinthatoverallsubject.(Thus,iftheDocumentisinpartatextbookofmathematics,aSecondarySectionmaynotexplainanymathematics.)Therelationshipcouldbeamatterofhistoricalconnectionwiththesubjectorwithrelatedmatters,oroflegal,commercial,philosophical,ethicalorpoliticalpositionregardingthem.The\InvariantSections"arecertainSecondarySectionswhosetitlesaredesignated,asbeingthoseofInvariantSections,inthenoticethatsaysthattheDocumentisreleasedunderthisLicense.IfasectiondoesnotttheabovedenitionofSecondarythenitisnotallowedtobedesignatedasInvariant.TheDocumentmaycontainzeroInvariantSections.IftheDocumentdoesnotidentifyanyInvariantSectionsthentherearenone. AppendixA:Copyingthismanual62 The\CoverTexts"arecertainshortpassagesoftextthatarelisted,asFront-CoverTextsorBack-CoverTexts,inthenoticethatsaysthattheDocumentisreleasedunderthisLicense.AFront-CoverTextmaybeatmost5words,andaBack-CoverTextmaybeatmost25words.A\Transparent"copyoftheDocumentmeansamachine-readablecopy,representedinaformatwhosespecicationisavailabletothegeneralpublic,thatissuitableforrevisingthedocumentstraightforwardlywithgenerictexteditorsor(forimagescomposedofpixels)genericpaintprogramsor(fordrawings)somewidelyavailabledrawingeditor,andthatissuitableforinputtotextformattersorforautomatictranslationtoavarietyofformatssuitableforinputtotextformatters.AcopymadeinanotherwiseTransparentleformatwhosemarkup,orabsenceofmarkup,hasbeenarrangedtothwartordiscouragesubsequentmodicationbyreadersisnotTransparent.AnimageformatisnotTransparentifusedforanysubstantialamountoftext.Acopythatisnot\Transparent"iscalled\Opaque".ExamplesofsuitableformatsforTransparentcopiesincludeplainASCIIwithoutmarkup,Texinfoinputformat,LaTEXinputformat,SGMLorXMLusingapubliclyavailableDTD,andstandard-conformingsimpleHTML,PostScriptorPDFdesignedforhumanmodi-cation.ExamplesoftransparentimageformatsincludePNG,XCFandJPG.Opaqueformatsincludeproprietaryformatsthatcanbereadandeditedonlybyproprietarywordprocessors,SGMLorXMLforwhichtheDTDand/orprocessingtoolsarenotgenerallyavailable,andthemachine-generatedHTML,PostScriptorPDFproducedbysomewordprocessorsforoutputpurposesonly.The\TitlePage"means,foraprintedbook,thetitlepageitself,plussuchfollowingpagesasareneededtohold,legibly,thematerialthisLicenserequirestoappearinthetitlepage.Forworksinformatswhichdonothaveanytitlepageassuch,\TitlePage"meansthetextnearthemostprominentappearanceofthework'stitle,precedingthebeginningofthebodyofthetext.The\publisher"meansanypersonorentitythatdistributescopiesoftheDocumenttothepublic.Asection\EntitledXYZ"meansanamedsubunitoftheDocumentwhosetitleeitherispreciselyXYZorcontainsXYZinparenthesesfollowingtextthattranslatesXYZinanotherlanguage.(HereXYZstandsforaspecicsectionnamementionedbelow,suchas\Acknowledgements",\Dedications",\Endorsements",or\History".)To\PreservetheTitle"ofsuchasectionwhenyoumodifytheDocumentmeansthatitremainsasection\EntitledXYZ"accordingtothisdenition.TheDocumentmayincludeWarrantyDisclaimersnexttothenoticewhichstatesthatthisLicenseappliestotheDocument.TheseWarrantyDisclaimersareconsideredtobeincludedbyreferenceinthisLicense,butonlyasregardsdisclaimingwarranties:anyotherimplicationthattheseWarrantyDisclaimersmayhaveisvoidandhasnoeectonthemeaningofthisLicense.2.VERBATIMCOPYINGYoumaycopyanddistributetheDocumentinanymedium,eithercommerciallyornoncom-mercially,providedthatthisLicense,thecopyrightnotices,andthelicensenoticesayingthisLicenseappliestotheDocumentarereproducedinallcopies,andthatyouaddnootherconditionswhatsoevertothoseofthisLicense.Youmaynotusetechnicalmeasurestoobstructorcontrolthereadingorfurthercopyingofthecopiesyoumakeordistribute.However,youmayacceptcompensationinexchangeforcopies.Ifyoudistributealargeenoughnumberofcopiesyoumustalsofollowtheconditionsinsection3.Youmayalsolendcopies,underthesameconditionsstatedabove,andyoumaypubliclydisplaycopies. AppendixA:Copyingthismanual63 3.COPYINGINQUANTITYIfyoupublishprintedcopies(orcopiesinmediathatcommonlyhaveprintedcovers)oftheDocument,numberingmorethan100,andtheDocument'slicensenoticerequiresCoverTexts,youmustenclosethecopiesincoversthatcarry,clearlyandlegibly,alltheseCoverTexts:Front-CoverTextsonthefrontcover,andBack-CoverTextsonthebackcover.Bothcoversmustalsoclearlyandlegiblyidentifyyouasthepublisherofthesecopies.Thefrontcovermustpresentthefulltitlewithallwordsofthetitleequallyprominentandvisible.Youmayaddothermaterialonthecoversinaddition.Copyingwithchangeslimitedtothecovers,aslongastheypreservethetitleoftheDocumentandsatisfytheseconditions,canbetreatedasverbatimcopyinginotherrespects.Iftherequiredtextsforeithercoveraretoovoluminoustotlegibly,youshouldputtherstoneslisted(asmanyastreasonably)ontheactualcover,andcontinuetherestontoadjacentpages.IfyoupublishordistributeOpaquecopiesoftheDocumentnumberingmorethan100,youmusteitherincludeamachine-readableTransparentcopyalongwitheachOpaquecopy,orstateinorwitheachOpaquecopyacomputer-networklocationfromwhichthegeneralnetwork-usingpublichasaccesstodownloadusingpublic-standardnetworkprotocolsacompleteTransparentcopyoftheDocument,freeofaddedmaterial.Ifyouusethelatteroption,youmusttakereasonablyprudentsteps,whenyoubegindistributionofOpaquecopiesinquantity,toensurethatthisTransparentcopywillremainthusaccessibleatthestatedlocationuntilatleastoneyearafterthelasttimeyoudistributeanOpaquecopy(directlyorthroughyouragentsorretailers)ofthateditiontothepublic.Itisrequested,butnotrequired,thatyoucontacttheauthorsoftheDocumentwellbeforeredistributinganylargenumberofcopies,togivethemachancetoprovideyouwithanupdatedversionoftheDocument.4.MODIFICATIONSYoumaycopyanddistributeaModiedVersionoftheDocumentundertheconditionsofsections2and3above,providedthatyoureleasetheModiedVersionunderpreciselythisLicense,withtheModiedVersionllingtheroleoftheDocument,thuslicensingdistributionandmodicationoftheModiedVersiontowhoeverpossessesacopyofit.Inaddition,youmustdothesethingsintheModiedVersion:A.UseintheTitlePage(andonthecovers,ifany)atitledistinctfromthatoftheDocument,andfromthoseofpreviousversions(whichshould,iftherewereany,belistedintheHistorysectionoftheDocument).Youmayusethesametitleasapreviousversioniftheoriginalpublisherofthatversiongivespermission.B.ListontheTitlePage,asauthors,oneormorepersonsorentitiesresponsibleforauthorshipofthemodicationsintheModiedVersion,togetherwithatleastveoftheprincipalauthorsoftheDocument(allofitsprincipalauthors,ifithasfewerthanve),unlesstheyreleaseyoufromthisrequirement.C.StateontheTitlepagethenameofthepublisheroftheModiedVersion,asthepublisher.D.PreserveallthecopyrightnoticesoftheDocument.E.Addanappropriatecopyrightnoticeforyourmodicationsadjacenttotheothercopy-rightnotices.F.Include,immediatelyafterthecopyrightnotices,alicensenoticegivingthepublicpermissiontousetheModiedVersionunderthetermsofthisLicense,intheformshownintheAddendumbelow.G.PreserveinthatlicensenoticethefulllistsofInvariantSectionsandrequiredCoverTextsgivenintheDocument'slicensenotice. AppendixA:Copyingthismanual64 H.IncludeanunalteredcopyofthisLicense.I.PreservethesectionEntitled\History",PreserveitsTitle,andaddtoitanitemstatingatleastthetitle,year,newauthors,andpublisheroftheModiedVersionasgivenontheTitlePage.IfthereisnosectionEntitled\History"intheDocument,createonestatingthetitle,year,authors,andpublisheroftheDocumentasgivenonitsTitlePage,thenaddanitemdescribingtheModiedVersionasstatedintheprevioussentence.J.Preservethenetworklocation,ifany,givenintheDocumentforpublicaccesstoaTransparentcopyoftheDocument,andlikewisethenetworklocationsgivenintheDocumentforpreviousversionsitwasbasedon.Thesemaybeplacedinthe\History"section.YoumayomitanetworklocationforaworkthatwaspublishedatleastfouryearsbeforetheDocumentitself,oriftheoriginalpublisheroftheversionitreferstogivespermission.K.ForanysectionEntitled\Acknowledgements"or\Dedications",PreservetheTitleofthesection,andpreserveinthesectionallthesubstanceandtoneofeachofthecontributoracknowledgementsand/ordedicationsgiventherein.L.PreservealltheInvariantSectionsoftheDocument,unalteredintheirtextandintheirtitles.Sectionnumbersortheequivalentarenotconsideredpartofthesectiontitles.M.DeleteanysectionEntitled\Endorsements".SuchasectionmaynotbeincludedintheModiedVersion.N.DonotretitleanyexistingsectiontobeEntitled\Endorsements"ortocon ictintitlewithanyInvariantSection.O.PreserveanyWarrantyDisclaimers.IftheModiedVersionincludesnewfront-mattersectionsorappendicesthatqualifyasSecondarySectionsandcontainnomaterialcopiedfromtheDocument,youmayatyouroptiondesignatesomeorallofthesesectionsasinvariant.Todothis,addtheirtitlestothelistofInvariantSectionsintheModiedVersion'slicensenotice.Thesetitlesmustbedistinctfromanyothersectiontitles.YoumayaddasectionEntitled\Endorsements",provideditcontainsnothingbutendorse-mentsofyourModiedVersionbyvariousparties|forexample,statementsofpeerrevieworthatthetexthasbeenapprovedbyanorganizationastheauthoritativedenitionofastandard.YoumayaddapassageofuptovewordsasaFront-CoverText,andapassageofupto25wordsasaBack-CoverText,totheendofthelistofCoverTextsintheModiedVersion.OnlyonepassageofFront-CoverTextandoneofBack-CoverTextmaybeaddedby(orthrougharrangementsmadeby)anyoneentity.IftheDocumentalreadyincludesacovertextforthesamecover,previouslyaddedbyyouorbyarrangementmadebythesameentityyouareactingonbehalfof,youmaynotaddanother;butyoumayreplacetheoldone,onexplicitpermissionfromthepreviouspublisherthataddedtheoldone.Theauthor(s)andpublisher(s)oftheDocumentdonotbythisLicensegivepermissiontousetheirnamesforpublicityforortoassertorimplyendorsementofanyModiedVersion.5.COMBININGDOCUMENTSYoumaycombinetheDocumentwithotherdocumentsreleasedunderthisLicense,underthetermsdenedinsection4aboveformodiedversions,providedthatyouincludeinthecombinationalloftheInvariantSectionsofalloftheoriginaldocuments,unmodied,andlistthemallasInvariantSectionsofyourcombinedworkinitslicensenotice,andthatyoupreservealltheirWarrantyDisclaimers.ThecombinedworkneedonlycontainonecopyofthisLicense,andmultipleidenticalInvariantSectionsmaybereplacedwithasinglecopy.IftherearemultipleInvariant AppendixA:Copyingthismanual65 Sectionswiththesamenamebutdierentcontents,makethetitleofeachsuchsectionuniquebyaddingattheendofit,inparentheses,thenameoftheoriginalauthororpublisherofthatsectionifknown,orelseauniquenumber.MakethesameadjustmenttothesectiontitlesinthelistofInvariantSectionsinthelicensenoticeofthecombinedwork.Inthecombination,youmustcombineanysectionsEntitled\History"inthevariousoriginaldocuments,formingonesectionEntitled\History";likewisecombineanysectionsEntitled\Acknowledgements",andanysectionsEntitled\Dedications".YoumustdeleteallsectionsEntitled\Endorsements."6.COLLECTIONSOFDOCUMENTSYoumaymakeacollectionconsistingoftheDocumentandotherdocumentsreleasedunderthisLicense,andreplacetheindividualcopiesofthisLicenseinthevariousdocumentswithasinglecopythatisincludedinthecollection,providedthatyoufollowtherulesofthisLicenseforverbatimcopyingofeachofthedocumentsinallotherrespects.Youmayextractasingledocumentfromsuchacollection,anddistributeitindividuallyunderthisLicense,providedyouinsertacopyofthisLicenseintotheextracteddocument,andfollowthisLicenseinallotherrespectsregardingverbatimcopyingofthatdocument.7.AGGREGATIONWITHINDEPENDENTWORKSAcompilationoftheDocumentoritsderivativeswithotherseparateandindependentdocumentsorworks,inoronavolumeofastorageordistributionmedium,iscalledan\aggregate"ifthecopyrightresultingfromthecompilationisnotusedtolimitthelegalrightsofthecompilation'susersbeyondwhattheindividualworkspermit.WhentheDocumentisincludedinanaggregate,thisLicensedoesnotapplytotheotherworksintheaggregatewhicharenotthemselvesderivativeworksoftheDocument.IftheCoverTextrequirementofsection3isapplicabletothesecopiesoftheDocument,theniftheDocumentislessthanonehalfoftheentireaggregate,theDocument'sCoverTextsmaybeplacedoncoversthatbrackettheDocumentwithintheaggregate,ortheelectronicequivalentofcoversiftheDocumentisinelectronicform.Otherwisetheymustappearonprintedcoversthatbracketthewholeaggregate.8.TRANSLATIONTranslationisconsideredakindofmodication,soyoumaydistributetranslationsoftheDocumentunderthetermsofsection4.ReplacingInvariantSectionswithtranslationsrequiresspecialpermissionfromtheircopyrightholders,butyoumayincludetranslationsofsomeorallInvariantSectionsinadditiontotheoriginalversionsoftheseInvariantSections.YoumayincludeatranslationofthisLicense,andallthelicensenoticesintheDocument,andanyWarrantyDisclaimers,providedthatyoualsoincludetheoriginalEnglishversionofthisLicenseandtheoriginalversionsofthosenoticesanddisclaimers.IncaseofadisagreementbetweenthetranslationandtheoriginalversionofthisLicenseoranoticeordisclaimer,theoriginalversionwillprevail.IfasectionintheDocumentisEntitled\Acknowledgements",\Dedications",or\History",therequirement(section4)toPreserveitsTitle(section1)willtypicallyrequirechangingtheactualtitle.9.TERMINATIONYoumaynotcopy,modify,sublicense,ordistributetheDocumentexceptasexpresslypro-videdunderthisLicense.Anyattemptotherwisetocopy,modify,sublicense,ordistributeitisvoid,andwillautomaticallyterminateyourrightsunderthisLicense.However,ifyouceaseallviolationofthisLicense,thenyourlicensefromaparticularcopy-rightholderisreinstated(a)provisionally,unlessanduntilthecopyrightholderexplicitlyandnallyterminatesyourlicense,and(b)permanently,ifthecopyrightholderfailstonotifyyouoftheviolationbysomereasonablemeanspriorto60daysafterthecessation. AppendixA:Copyingthismanual66 Moreover,yourlicensefromaparticularcopyrightholderisreinstatedpermanentlyifthecopyrightholdernotiesyouoftheviolationbysomereasonablemeans,thisisthersttimeyouhavereceivednoticeofviolationofthisLicense(foranywork)fromthatcopyrightholder,andyoucuretheviolationpriorto30daysafteryourreceiptofthenotice.TerminationofyourrightsunderthissectiondoesnotterminatethelicensesofpartieswhohavereceivedcopiesorrightsfromyouunderthisLicense.Ifyourrightshavebeenterminatedandnotpermanentlyreinstated,receiptofacopyofsomeorallofthesamematerialdoesnotgiveyouanyrightstouseit.10.FUTUREREVISIONSOFTHISLICENSETheFreeSoftwareFoundationmaypublishnew,revisedversionsoftheGNUFreeDoc-umentationLicensefromtimetotime.Suchnewversionswillbesimilarinspirittothepresentversion,butmaydierindetailtoaddressnewproblemsorconcerns.See http://www.gnu.org/copyleft/ .EachversionoftheLicenseisgivenadistinguishingversionnumber.IftheDocumentspeciesthataparticularnumberedversionofthisLicense\oranylaterversion"appliestoit,youhavetheoptionoffollowingthetermsandconditionseitherofthatspeciedversionorofanylaterversionthathasbeenpublished(notasadraft)bytheFreeSoftwareFoundation.IftheDocumentdoesnotspecifyaversionnumberofthisLicense,youmaychooseanyversioneverpublished(notasadraft)bytheFreeSoftwareFoundation.IftheDocumentspeciesthataproxycandecidewhichfutureversionsofthisLicensecanbeused,thatproxy'spublicstatementofacceptanceofaversionpermanentlyauthorizesyoutochoosethatversionfortheDocument.11.RELICENSING\MassiveMultiauthorCollaborationSite"(or\MMCSite")meansanyWorldWideWebserverthatpublishescopyrightableworksandalsoprovidesprominentfacilitiesforanybodytoeditthoseworks.Apublicwikithatanybodycaneditisanexampleofsuchaserver.A\MassiveMultiauthorCollaboration"(or\MMC")containedinthesitemeansanysetofcopyrightableworksthuspublishedontheMMCsite.\CC-BY-SA"meanstheCreativeCommonsAttribution-ShareAlike3.0licensepublishedbyCreativeCommonsCorporation,anot-for-protcorporationwithaprincipalplaceofbusinessinSanFrancisco,California,aswellasfuturecopyleftversionsofthatlicensepublishedbythatsameorganization.\Incorporate"meanstopublishorrepublishaDocument,inwholeorinpart,aspartofanotherDocument.AnMMCis\eligibleforrelicensing"ifitislicensedunderthisLicense,andifallworksthatwererstpublishedunderthisLicensesomewhereotherthanthisMMC,andsubsequentlyincorporatedinwholeorinpartintotheMMC,(1)hadnocovertextsorinvariantsections,and(2)werethusincorporatedpriortoNovember1,2008.TheoperatorofanMMCSitemayrepublishanMMCcontainedinthesiteunderCC-BY-SAonthesamesiteatanytimebeforeAugust1,2009,providedtheMMCiseligibleforrelicensing. AppendixA:Copyingthismanual67 ADDENDUM:HowtousethisLicenseforyourdocumentsTousethisLicenseinadocumentyouhavewritten,includeacopyoftheLicenseinthedocumentandputthefollowingcopyrightandlicensenoticesjustafterthetitlepage:Copyright(C)yearyourname.Permissionisgrantedtocopy,distributeand/ormodifythisdocumentunderthetermsoftheGNUFreeDocumentationLicense,Version1.3oranylaterversionpublishedbytheFreeSoftwareFoundation;withnoInvariantSections,noFront-CoverTexts,andnoBack-CoverTexts.Acopyofthelicenseisincludedinthesectionentitled``GNUFreeDocumentationLicense''.IfyouhaveInvariantSections,Front-CoverTextsandBack-CoverTexts,replacethe\with...Texts."linewiththis:withtheInvariantSectionsbeinglisttheirtitles,withtheFront-CoverTextsbeinglist,andwiththeBack-CoverTextsbeinglist.IfyouhaveInvariantSectionswithoutCoverTexts,orsomeothercombinationofthethree,mergethosetwoalternativestosuitthesituation.Ifyourdocumentcontainsnontrivialexamplesofprogramcode,werecommendreleasingtheseexamplesinparallelunderyourchoiceoffreesoftwarelicense,suchastheGNUGeneralPublicLicense,topermittheiruseinfreesoftware. 68 ConceptIndex##wget::::::::::::::::::::::::::::::::::::::::::: 55 ..cssextension::::::::::::::::::::::::::::::::::::: 15 .htmlextension::::::::::::::::::::::::::::::::::: 15 .listingles,removing:::::::::::::::::::::::::::: 25 .netrc:::::::::::::::::::::::::::::::::::::::::::: 40 .wgetrc::::::::::::::::::::::::::::::::::::::::::: 40 Aacceptdirectories::::::::::::::::::::::::::::::::: 36 acceptsuxes:::::::::::::::::::::::::::::::::::: 34 acceptwildcards:::::::::::::::::::::::::::::::::: 34 appendtolog::::::::::::::::::::::::::::::::::::: 4 arguments::::::::::::::::::::::::::::::::::::::::: 2 authentication:::::::::::::::::::::::::::: 13 , 15 , 21 authenticationcredentials:::::::::::::::::::::::::: 7 Bbackingupconvertedles:::::::::::::::::::::::: 28 backinguples:::::::::::::::::::::::::::::::::::: 7 bandwidth,limit::::::::::::::::::::::::::::::::: 10 baseforrelativelinksininputle:::::::::::::::::: 5 bindaddress::::::::::::::::::::::::::::::::::::::: 6 bindDNSaddress::::::::::::::::::::::::::::::::: 6 bugreports::::::::::::::::::::::::::::::::::::::: 55 bugs::::::::::::::::::::::::::::::::::::::::::::: 55 Ccache::::::::::::::::::::::::::::::::::::::::::::: 16 cachingofDNSlookups:::::::::::::::::::::::::: 11 casefold::::::::::::::::::::::::::::::::::::::::: 31 clientDNSaddress:::::::::::::::::::::::::::::::: 6 clientIPaddress::::::::::::::::::::::::::::::::::: 6 clobbering,le::::::::::::::::::::::::::::::::::::: 6 commandline::::::::::::::::::::::::::::::::::::: 2 comments,html::::::::::::::::::::::::::::::::: 29 connecttimeout:::::::::::::::::::::::::::::::::: 10 ContentOnError:::::::::::::::::::::::::::::::: 20 Content-Disposition:::::::::::::::::::::::::::::: 20 Content-Encoding,choose:::::::::::::::::::::::: 18 Content-Length,ignore::::::::::::::::::::::::::: 17 continueretrieval::::::::::::::::::::::::::::::: 7 , 8 contributors:::::::::::::::::::::::::::::::::::::: 59 conversionoflinks:::::::::::::::::::::::::::::::: 27 cookies::::::::::::::::::::::::::::::::::::::::::: 16 cookies,loading:::::::::::::::::::::::::::::::::: 16 cookies,saving::::::::::::::::::::::::::::::::::: 17 cookies,session::::::::::::::::::::::::::::::::::: 17 cutdirectories:::::::::::::::::::::::::::::::::::: 14 Ddebug::::::::::::::::::::::::::::::::::::::::::::: 4 defaultpagename:::::::::::::::::::::::::::::::: 15 deleteafterretrieval:::::::::::::::::::::::::::::: 27 directories:::::::::::::::::::::::::::::::::::::::: 36 directories,exclude::::::::::::::::::::::::::::::: 36 directories,include::::::::::::::::::::::::::::::: 36 directorylimits::::::::::::::::::::::::::::::::::: 36 directoryprex::::::::::::::::::::::::::::::::::: 15 DNScache::::::::::::::::::::::::::::::::::::::: 11 DNSIPaddress,client,DNS::::::::::::::::::::::: 6 DNSserver:::::::::::::::::::::::::::::::::::::::: 6 DNStimeout::::::::::::::::::::::::::::::::::::: 10 dotstyle::::::::::::::::::::::::::::::::::::::::::: 8 downloadingmultipletimes:::::::::::::::::::::::: 6 EEGD::::::::::::::::::::::::::::::::::::::::::::: 23 entropy,specifyingsourceof:::::::::::::::::::::: 23 examples::::::::::::::::::::::::::::::::::::::::: 51 excludedirectories:::::::::::::::::::::::::::::::: 36 executewgetrccommand:::::::::::::::::::::::::: 3 FFDL,GNUFreeDocumentationLicense:::::::::: 61 features::::::::::::::::::::::::::::::::::::::::::: 1 lenames,restrict:::::::::::::::::::::::::::::::: 12 lepermissions::::::::::::::::::::::::::::::::::: 26 llingproxycache:::::::::::::::::::::::::::::::: 27 followFTPlinks::::::::::::::::::::::::::::::::: 30 followingftplinks:::::::::::::::::::::::::::::::: 37 followinglinks:::::::::::::::::::::::::::::::::::: 34 forcehtml::::::::::::::::::::::::::::::::::::::::: 5 ftpauthentication:::::::::::::::::::::::::::::::: 25 ftppassword::::::::::::::::::::::::::::::::::::: 25 ftptime-stamping:::::::::::::::::::::::::::::::: 39 ftpuser:::::::::::::::::::::::::::::::::::::::::: 25 Gglobbing,toggle:::::::::::::::::::::::::::::::::: 26 Hhangup::::::::::::::::::::::::::::::::::::::::::: 57 header,add:::::::::::::::::::::::::::::::::::::: 17 hosts,spanning::::::::::::::::::::::::::::::::::: 34 HSTS:::::::::::::::::::::::::::::::::::::::::::: 23 htmlcomments:::::::::::::::::::::::::::::::::: 29 httppassword:::::::::::::::::::::::::::::::::::: 15 httpreferer::::::::::::::::::::::::::::::::::::::: 18 httptime-stamping::::::::::::::::::::::::::::::: 39 httpuser::::::::::::::::::::::::::::::::::::::::: 15 ConceptIndex69 Iidnsupport:::::::::::::::::::::::::::::::::::::: 13 ignorecase::::::::::::::::::::::::::::::::::::::: 31 ignorelength::::::::::::::::::::::::::::::::::::: 17 includedirectories:::::::::::::::::::::::::::::::: 36 incompletedownloads::::::::::::::::::::::::::: 7 , 8 incrementalupdating::::::::::::::::::::::::::::: 38 index.html::::::::::::::::::::::::::::::::::::::: 15 input-le:::::::::::::::::::::::::::::::::::::::::: 4 input-metalink:::::::::::::::::::::::::::::::::::: 5 InternetRelayChat:::::::::::::::::::::::::::::: 55 invoking::::::::::::::::::::::::::::::::::::::::::: 2 IPaddress,client:::::::::::::::::::::::::::::::::: 6 IPv6::::::::::::::::::::::::::::::::::::::::::::: 12 IRC:::::::::::::::::::::::::::::::::::::::::::::: 55 irisupport::::::::::::::::::::::::::::::::::::::: 13 KKeep-Alive,turningo::::::::::::::::::::::::::: 16 keep-badhash:::::::::::::::::::::::::::::::::::::: 5 Llatestversion::::::::::::::::::::::::::::::::::::: 55 limitbandwidth:::::::::::::::::::::::::::::::::: 10 linkconversion::::::::::::::::::::::::::::::::::: 27 links::::::::::::::::::::::::::::::::::::::::::::: 34 list::::::::::::::::::::::::::::::::::::::::::::::: 55 loadingcookies::::::::::::::::::::::::::::::::::: 16 localencoding:::::::::::::::::::::::::::::::::::: 13 locationofwgetrc:::::::::::::::::::::::::::::::: 40 logle::::::::::::::::::::::::::::::::::::::::::::: 4 Mmailinglist::::::::::::::::::::::::::::::::::::::: 55 metalink-index:::::::::::::::::::::::::::::::::::: 5 metalink-over-http::::::::::::::::::::::::::::::::: 5 mirroring::::::::::::::::::::::::::::::::::::::::: 52 Nnoparent:::::::::::::::::::::::::::::::::::::::: 36 no-clobber::::::::::::::::::::::::::::::::::::::::: 6 nohup::::::::::::::::::::::::::::::::::::::::::::: 2 numberoftries:::::::::::::::::::::::::::::::::::: 6 Ooset:::::::::::::::::::::::::::::::::::::::::::::: 8 operatingsystems:::::::::::::::::::::::::::::::: 56 optionsyntax:::::::::::::::::::::::::::::::::::::: 3 OtherHTTPMethods:::::::::::::::::::::::::::: 20 outputle::::::::::::::::::::::::::::::::::::::::: 4 overview::::::::::::::::::::::::::::::::::::::::::: 1 Ppagerequisites::::::::::::::::::::::::::::::::::: 28 passiveftp::::::::::::::::::::::::::::::::::::::: 26 password::::::::::::::::::::::::::::::::::::::::: 13 pause:::::::::::::::::::::::::::::::::::::::::::: 10 PersistentConnections,disabling::::::::::::::::: 16 portability::::::::::::::::::::::::::::::::::::::: 56 POST:::::::::::::::::::::::::::::::::::::::::::: 19 preferred-location:::::::::::::::::::::::::::::::::: 5 progressindicator:::::::::::::::::::::::::::::::::: 8 proxies::::::::::::::::::::::::::::::::::::::::::: 54 proxy::::::::::::::::::::::::::::::::::::::::: 11 , 16 proxyauthentication::::::::::::::::::::::::::::: 18 proxylling:::::::::::::::::::::::::::::::::::::: 27 proxypassword::::::::::::::::::::::::::::::::::: 18 proxyuser:::::::::::::::::::::::::::::::::::::::: 18 Qquiet:::::::::::::::::::::::::::::::::::::::::::::: 4 quota:::::::::::::::::::::::::::::::::::::::::::: 11 Rrandomwait::::::::::::::::::::::::::::::::::::: 11 randomness,specifyingsourceof:::::::::::::::::: 23 rate,limit:::::::::::::::::::::::::::::::::::::::: 10 readtimeout::::::::::::::::::::::::::::::::::::: 10 recursion::::::::::::::::::::::::::::::::::::::::: 33 recursivedownload::::::::::::::::::::::::::::::: 33 redirect:::::::::::::::::::::::::::::::::::::::::: 18 redirectingoutput:::::::::::::::::::::::::::::::: 52 referer,http:::::::::::::::::::::::::::::::::::::: 18 rejectdirectories:::::::::::::::::::::::::::::::::: 36 rejectsuxes::::::::::::::::::::::::::::::::::::: 35 rejectwildcards:::::::::::::::::::::::::::::::::: 35 relativelinks::::::::::::::::::::::::::::::::::::: 37 remoteencoding:::::::::::::::::::::::::::::::::: 14 reportingbugs:::::::::::::::::::::::::::::::::::: 55 requiredimages,downloading::::::::::::::::::::: 28 resumedownload:::::::::::::::::::::::::::::::: 7 , 8 retries::::::::::::::::::::::::::::::::::::::::::::: 6 retries,waitingbetween::::::::::::::::::::::::::: 11 retrieving:::::::::::::::::::::::::::::::::::::::: 33 robotexclusion::::::::::::::::::::::::::::::::::: 58 robots.txt:::::::::::::::::::::::::::::::::::::::: 58 Ssamplewgetrc:::::::::::::::::::::::::::::::::::: 47 savingcookies:::::::::::::::::::::::::::::::::::: 17 security:::::::::::::::::::::::::::::::::::::::::: 59 servermaintenance::::::::::::::::::::::::::::::: 58 serverresponse,print:::::::::::::::::::::::::::::: 9 serverresponse,save:::::::::::::::::::::::::::::: 18 sessioncookies::::::::::::::::::::::::::::::::::: 17 signalhandling::::::::::::::::::::::::::::::::::: 57 spanninghosts::::::::::::::::::::::::::::::::::: 34 specifycong:::::::::::::::::::::::::::::::::::::: 5 spider::::::::::::::::::::::::::::::::::::::::::::: 9 SSL:::::::::::::::::::::::::::::::::::::::::::::: 21 SSLcerticate:::::::::::::::::::::::::::::::::::: 22 SSLcerticateauthority:::::::::::::::::::::::::: 22 SSLcerticatetype,specify:::::::::::::::::::::: 22 ConceptIndex70 SSLcerticate,check::::::::::::::::::::::::::::: 22 SSLCRL,certicaterevocationlist::::::::::::::: 23 SSLprotocol,choose::::::::::::::::::::::::::::: 21 SSLPublicKeyPin:::::::::::::::::::::::::::::: 23 startposition:::::::::::::::::::::::::::::::::::::: 8 startup::::::::::::::::::::::::::::::::::::::::::: 40 startuple::::::::::::::::::::::::::::::::::::::: 40 suxes,accept::::::::::::::::::::::::::::::::::: 34 suxes,reject:::::::::::::::::::::::::::::::::::: 35 symboliclinks,retrieving::::::::::::::::::::::::: 26 syntaxofoptions:::::::::::::::::::::::::::::::::: 3 syntaxofwgetrc:::::::::::::::::::::::::::::::::: 40 Ttag-basedrecursivepruning::::::::::::::::::::::: 30 time-stamping:::::::::::::::::::::::::::::::::::: 38 time-stampingusage:::::::::::::::::::::::::::::: 38 timeout:::::::::::::::::::::::::::::::::::::::::: 10 timeout,connect::::::::::::::::::::::::::::::::: 10 timeout,DNS:::::::::::::::::::::::::::::::::::: 10 timeout,read::::::::::::::::::::::::::::::::::::: 10 timestamping::::::::::::::::::::::::::::::::::::: 38 tries::::::::::::::::::::::::::::::::::::::::::::::: 6 Trustservernames::::::::::::::::::::::::::::::: 21 typesofles:::::::::::::::::::::::::::::::::::::: 34 Uunlink:::::::::::::::::::::::::::::::::::::::::::: 14 updatingthearchives::::::::::::::::::::::::::::: 38 URL:::::::::::::::::::::::::::::::::::::::::::::: 2 URLsyntax::::::::::::::::::::::::::::::::::::::: 2 usage,time-stamping::::::::::::::::::::::::::::: 38 user:::::::::::::::::::::::::::::::::::::::::::::: 13 user-agent:::::::::::::::::::::::::::::::::::::::: 18 Vvarious::::::::::::::::::::::::::::::::::::::::::: 54 verbose:::::::::::::::::::::::::::::::::::::::::::: 4 Wwait:::::::::::::::::::::::::::::::::::::::::::::: 10 wait,random::::::::::::::::::::::::::::::::::::: 11 waitingbetweenretries::::::::::::::::::::::::::: 11 WARC::::::::::::::::::::::::::::::::::::::::::: 24 website:::::::::::::::::::::::::::::::::::::::::: 55 Wgetasspider:::::::::::::::::::::::::::::::::::: 9 wgetrc::::::::::::::::::::::::::::::::::::::::::: 40 wgetrccommands:::::::::::::::::::::::::::::::: 40 wgetrclocation::::::::::::::::::::::::::::::::::: 40 wgetrcsyntax:::::::::::::::::::::::::::::::::::: 40 wildcards,accept::::::::::::::::::::::::::::::::: 34 wildcards,reject:::::::::::::::::::::::::::::::::: 35 Windowslenames:::::::::::::::::::::::::::::: 12 iii TableofContents1Overview:::::::::::::::::::::::::::::::::::::::::::::::::::::::: 1 2Invoking::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 2 2.1URLFormat::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 2 2.2OptionSyntax::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 3 2.3BasicStartupOptions::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 3 2.4LoggingandInputFileOptions:::::::::::::::::::::::::::::::::::::::::::::::::::: 4 2.5DownloadOptions::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 6 2.6DirectoryOptions:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 14 2.7HTTPOptions::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 15 2.8HTTPS(SSL/TLS)Options:::::::::::::::::::::::::::::::::::::::::::::::::::::: 21 2.9FTPOptions::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 25 2.10FTPSOptions::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 26 2.11RecursiveRetrievalOptions:::::::::::::::::::::::::::::::::::::::::::::::::::::: 27 2.12RecursiveAccept/RejectOptions:::::::::::::::::::::::::::::::::::::::::::::::: 30 2.13ExitStatus:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 31 3RecursiveDownload:::::::::::::::::::::::::::::::::::::::::: 33 4FollowingLinks::::::::::::::::::::::::::::::::::::::::::::::: 34 4.1SpanningHosts::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 34 4.2TypesofFiles:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 34 4.3Directory-BasedLimits::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 36 4.4RelativeLinks:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 37 4.5FollowingFTPLinks::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 37 5Time-Stamping:::::::::::::::::::::::::::::::::::::::::::::::: 38 5.1Time-StampingUsage:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 38 5.2HTTPTime-StampingInternals:::::::::::::::::::::::::::::::::::::::::::::::::: 39 5.3FTPTime-StampingInternals:::::::::::::::::::::::::::::::::::::::::::::::::::: 39 6StartupFile::::::::::::::::::::::::::::::::::::::::::::::::::: 40 6.1WgetrcLocation:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 40 6.2WgetrcSyntax::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 40 6.3WgetrcCommands::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 40 6.4SampleWgetrc::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 47 7Examples:::::::::::::::::::::::::::::::::::::::::::::::::::::: 51 7.1SimpleUsage::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 51 7.2AdvancedUsage:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 51 7.3VeryAdvancedUsage::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 52 iv 8Various::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 54 8.1Proxies::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 54 8.2Distribution:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 55 8.3WebSite::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 55 8.4MailingLists::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 55 PrimaryList:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 55 ObsoleteLists:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 55 8.5InternetRelayChat:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 55 8.6ReportingBugs::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 55 8.7Portability:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 56 8.8Signals::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 57 9Appendices:::::::::::::::::::::::::::::::::::::::::::::::::::: 58 9.1RobotExclusion:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 58 9.2SecurityConsiderations::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 59 9.3Contributors:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 59 AppendixACopyingthismanual:::::::::::::::::::::::::::::: 61 A.1GNUFreeDocumentationLicense:::::::::::::::::::::::::::::::::::::::::::::::: 61 ConceptIndex:::::::::::::::::::::::::::::::::::::::::::::::::::: 68