Joseph Randy H Katz Andrew Konwinski Gunho Lee David A Patterson Ariel Rabkin Ion Stoica Matei Zaharia Electrical Engineering and Computer Sciences University of California at Berkeley Technical Report No UCBEECS200928 httpwwweecsberkeleyeduPubsTech ID: 1754
Download Pdf The PPT/PDF document "Above the Clouds A Berkeley View of Clou..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Table1:QuickPreviewofTop10ObstaclestoandOpportunitiesforGrowthofCloudComputing. Obstacle Opportunity 1 AvailabilityofService UseMultipleCloudProviders;UseElasticitytoPreventDDOS 2 DataLock-In StandardizeAPIs;CompatibleSWtoenableSurgeComputing 3 DataCondentialityandAuditability DeployEncryption,VLANs,Firewalls;GeographicalDataStorage 4 DataTransferBottlenecks FedExingDisks;DataBackup/Archival;HigherBWSwitches 5 PerformanceUnpredictability ImprovedVMSupport;FlashMemory;GangScheduleVMs 6 ScalableStorage InventScalableStore 7 BugsinLargeDistributedSystems InventDebuggerthatreliesonDistributedVMs 8 ScalingQuickly InventAuto-ScalerthatreliesonML;SnapshotsforConservation 9 ReputationFateSharing Offerreputation-guardingserviceslikethoseforemail 10 SoftwareLicensing Pay-for-uselicenses;Bulkusesales hardwareindustry.Atonetime,leadinghardwarecompaniesrequiredacaptivesemiconductorfabricationfacility,andcompanieshadtobelargeenoughtoaffordtobuildandoperateiteconomically.However,processingequipmentdoubledinpriceeverytechnologygeneration.Asemiconductorfabricationlinecostsover$3Btoday,soonlyahandfulofmajor“merchant”companieswithveryhighchipvolumes,suchasIntelandSamsung,canstilljustifyowningandoperatingtheirownfabricationlines.Thismotivatedtheriseofsemiconductorfoundriesthatbuildchipsforothers,suchasTaiwanSemiconductorManufacturingCompany(TSMC).Foundriesenable“fab-less”semiconductorchipcompanieswhosevalueisininnovativechipdesign:AcompanysuchasnVidiacannowbesuccessfulinthechipbusinesswithoutthecapital,operationalexpenses,andrisksassociatedwithowningastate-of-the-artfabricationline.Conversely,companieswithfabricationlinescantime-multiplextheiruseamongtheproductsofmanyfab-lesscompanies,tolowertheriskofnothavingenoughsuccessfulproductstoamortizeoperationalcosts.Similarly,theadvantagesoftheeconomyofscaleandstatisticalmultiplexingmayultimatelyleadtoahandfulofCloudComputingproviderswhocanamortizethecostoftheirlargedatacentersovertheproductsofmany“datacenter-less”companies.CloudComputinghasbeentalkedabout[10],bloggedabout[13,25],writtenabout[15,37,38]andbeenfeaturedinthetitleofworkshops,conferences,andevenmagazines.Nevertheless,confusionremainsaboutexactlywhatitisandwhenit'suseful,causingOracle'sCEOtoventhisfrustration:TheinterestingthingaboutCloudComputingisthatwe'veredenedCloudComputingtoincludeev-erythingthatwealreadydo....Idon'tunderstandwhatwewoulddodifferentlyinthelightofCloudComputingotherthanchangethewordingofsomeofourads.LarryEllison,quotedintheWallStreetJournal,September26,2008TheseremarksareechoedmoremildlybyHewlett-Packard'sVicePresidentofEuropeanSoftwareSales:Alotofpeoplearejumpingonthe[cloud]bandwagon,butIhavenotheardtwopeoplesaythesamethingaboutit.Therearemultipledenitionsoutthereof“thecloud.”AndyIsherwood,quotedinZDnetNews,December11,2008RichardStallman,knownforhisadvocacyof“freesoftware”,thinksCloudComputingisatrapforusers—ifapplicationsanddataaremanaged“inthecloud”,usersmightbecomedependentonproprietarysystemswhosecostswillescalateorwhosetermsofservicemightbechangedunilaterallyandadversely:It'sstupidity.It'sworsethanstupidity:it'samarketinghypecampaign.Somebodyissayingthisisinevitable—andwheneveryouhearsomebodysayingthat,it'sverylikelytobeasetofbusinessescampaigningtomakeittrue.RichardStallman,quotedinTheGuardian,September29,2008Ourgoalinthispapertoclarifyterms,providesimpleformulastoquantifycomparisonsbetweenofcloudandconventionalComputing,andidentifythetoptechnicalandnon-technicalobstaclesandopportunitiesofCloudCom-puting.Ourviewisshapedinpartbyworkingsince2005intheUCBerkeleyRADLabandinpartasusersofAmazonWebServicessinceJanuary2008inconductingourresearchandourteaching.TheRADLab'sresearchagendaistoinventtechnologythatleveragesmachinelearningtohelpautomatetheoperationofdatacentersforscalableInternetservices.WespentsixmonthsbrainstormingaboutCloudComputing,leadingtothispaperthattriestoanswerthefollowingquestions:3 Figure1:UsersandProvidersofCloudComputing.ThebenetsofSaaStobothSaaSusersandSaaSprovidersarewelldocumented,sowefocusonCloudComputing'seffectsonCloudProvidersandSaaSProviders/Cloudusers.Thetoplevelcanberecursive,inthatSaaSproviderscanalsobeaSaaSusers.Forexample,amashupproviderofrentalmapsmightbeauseroftheCraigslistandGooglemapsservices.WewillarguethatallthreeareimportanttothetechnicalandeconomicchangesmadepossiblebyCloudCom-puting.Indeed,pasteffortsatutilitycomputingfailed,andwenotethatineachcaseoneortwoofthesethreecriticalcharacteristicsweremissing.Forexample,IntelComputingServicesin2000-2001requirednegotiatingacontractandlonger-termusethanperhour.Asasuccessfulexample,ElasticComputeCloud(EC2)fromAmazonWebServices(AWS)sells1.0-GHzx86ISA“slices”for10centsperhour,andanew“slice”,orinstance,canbeaddedin2to5minutes.Amazon'sScalableStorageService(S3)charges$0.12to$0.15pergigabyte-month,withadditionalbandwidthchargesof$0.10to$0.15pergigabytetomovedataintoandoutofAWSovertheInternet.Amazon'sbetisthatbystatisticallymultiplexingmultipleinstancesontoasinglephysicalbox,thatboxcanbesimultaneouslyrentedtomanycustomerswhowillnotingeneralinterferewitheachothers'usage(seeSection7).WhiletheattractiontoCloudComputingusers(SaaSproviders)isclear,whowouldbecomeaCloudComputingprovider,andwhy?Tobeginwith,realizingtheeconomiesofscaleaffordedbystatisticalmultiplexingandbulkpurchasingrequirestheconstructionofextremelylargedatacenters.Building,provisioning,andlaunchingsuchafacilityisahundred-million-dollarundertaking.However,becauseofthephenomenalgrowthofWebservicesthroughtheearly2000's,manylargeInternetcompanies,includingAmazon,eBay,Google,Microsoftandothers,werealreadydoingso.Equallyimportant,thesecompaniesalsohadtodevelopscalablesoftwareinfrastructure(suchasMapReduce,theGoogleFileSystem,BigTable,andDynamo[16,20,14,17])andtheoperationalexpertisetoarmortheirdatacentersagainstpotentialphysicalandelectronicattacks.Therefore,anecessarybutnotsufcientconditionforacompanytobecomeaCloudComputingprovideristhatitmusthaveexistinginvestmentsnotonlyinverylargedatacenters,butalsoinlarge-scalesoftwareinfrastructureandoperationalexpertiserequiredtorunthem.Giventheseconditions,avarietyoffactorsmightinuencethesecompaniestobecomeCloudComputingproviders:1.Makealotofmoney.Although10centsperserver-hourseemslow,Table2summarizesJamesHamilton'sestimates[23]thatverylargedatacenters(tensofthousandsofcomputers)canpurchasehardware,networkbandwidth,andpowerfor1=5to1=7thepricesofferedtoamedium-sized(hundredsorthousandsofcomputers)datacenter.Further,thexedcostsofsoftwaredevelopmentanddeploymentcanbeamortizedovermanymoremachines.Othersestimatethepriceadvantageasafactorof3to5[37,10].Thus,asufcientlylargecompanycouldleveragetheseeconomiesofscaletoofferaservicewellbelowthecostsofamedium-sizedcompanyandstillmakeatidyprot.2.Leverageexistinginvestment.AddingCloudComputingservicesontopofexistinginfrastructureprovidesanewrevenuestreamat(ideally)lowincrementalcost,helpingtoamortizethelargeinvestmentsofdatacenters.Indeed,accordingtoWernerVogels,Amazon'sCTO,manyAmazonWebServicestechnologieswereinitiallydevelopedforAmazon'sinternaloperations[42].3.Defendafranchise.AsconventionalserverandenterpriseapplicationsembraceCloudComputing,vendorswithanestablishedfranchiseinthoseapplicationswouldbemotivatedtoprovideacloudoptionoftheirown.Forexample,MicrosoftAzureprovidesanimmediatepathformigratingexistingcustomersofMicrosoftenter-priseapplicationstoacloudenvironment.5 Table2:Economiesofscalein2006formedium-sizeddatacenter(1000servers)vs.verylargedatacenter(50,000servers).[24] Technology CostinMedium-sizedDC CostinVeryLargeDC Ratio Network $95perMbit/sec/month $13perMbit/sec/month 7.1 Storage $2.20perGByte/month $0.40perGByte/month 5.7 Administration 140Servers/Administrator �1000Servers/Administrator 7.1 Table3:Priceofkilowatt-hoursofelectricitybyregion[7]. PriceperKWH Where PossibleReasonsWhy 3.6¢ Idaho Hydroelectricpower;notsentlongdistance 10.0¢ California Electricitytransmittedlongdistanceoverthegrid;limitedtransmissionlinesinBayArea;nocoalredelectricityallowedinCalifornia. 18.0¢ Hawaii Mustshipfueltogenerateelectricity 4.Attackanincumbent.Acompanywiththerequisitedatacenterandsoftwareresourcesmightwanttoestablishabeachheadinthisspacebeforeasingle“800poundgorilla”emerges.GoogleAppEngineprovidesanalternativepathtoclouddeploymentwhoseappealliesinitsautomationofmanyofthescalabilityandloadbalancingfeaturesthatdevelopersmightotherwisehavetobuildforthemselves.5.Leveragecustomerrelationships.ITserviceorganizationssuchasIBMGlobalServiceshaveextensivecus-tomerrelationshipsthroughtheirserviceofferings.ProvidingabrandedCloudComputingofferinggivesthosecustomersananxiety-freemigrationpaththatpreservesbothparties'investmentsinthecustomerrelationship.6.Becomeaplatform.Facebook'sinitiativetoenableplug-inapplicationsisagreattforcloudcomputing,aswewillsee,andindeedoneinfrastructureproviderforFacebookplug-inapplicationsisJoyent,acloudprovider.YetFacebook'smotivationwastomaketheirsocial-networkingapplicationanewdevelopmentplatform.SeveralCloudComputing(andconventionalcomputing)datacentersarebeingbuiltinseeminglysurprisingloca-tions,suchasQuincy,Washington(Google,Microsoft,Yahoo!,andothers)andSanAntonio,Texas(Microsoft,USNationalSecurityAgency,others).Themotivationbehindchoosingtheselocalesisthatthecostsforelectricity,cool-ing,labor,propertypurchasecosts,andtaxesaregeographicallyvariable,andofthesecosts,electricityandcoolingalonecanaccountforathirdofthecostsofthedatacenter.Table3showsthecostofelectricityindifferentlocales[10].Physicstellsusit'seasiertoshipphotonsthanelectrons;thatis,it'scheapertoshipdataoverberopticcablesthantoshipelectricityoverhigh-voltagetransmissionlines.4CloudsinaPerfectStorm:WhyNow,NotThen?Althoughwearguethattheconstructionandoperationofextremelylargescalecommodity-computerdatacenterswasthekeynecessaryenablerofCloudComputing,additionaltechnologytrendsandnewbusinessmodelsalsoplayedakeyroleinmakingitarealitythistimearound.OnceCloudComputingwas“offtheground,”newapplicationopportunitiesandusagemodelswerediscoveredthatwouldnothavemadesensepreviously.4.1NewTechnologyTrendsandBusinessModelsAccompanyingtheemergenceofWeb2.0wasashiftfrom“high-touch,high-margin,high-commitment”provisioningofservice“low-touch,low-margin,low-commitment”self-service.Forexample,inWeb1.0,acceptingcreditcardpaymentsfromstrangersrequiredacontractualarrangementwithapaymentprocessingservicesuchasVeriSignorAuthorize.net;thearrangementwaspartofalargerbusinessrelationship,makingitonerousforanindividualoraverysmallbusinesstoacceptcreditcardsonline.WiththeemergenceofPayPal,however,anyindividualcanacceptcreditcardpaymentswithnocontract,nolong-termcommitment,andonlymodestpay-as-you-gotransactionfees.Thelevelof“touch”(customersupportandrelationshipmanagement)providedbytheseservicesisminimaltononexistent,but6 fortheCloud,stocktradingthatrequiresmicrosecondprecisionisnot.Untilthecost(andpossiblylatency)ofwide-areadatatransferdecrease(seeSection7),suchapplicationsmaybelessobviouscandidatesforthecloud.5ClassesofUtilityComputingAnyapplicationneedsamodelofcomputation,amodelofstorageand,assumingtheapplicationiseventriviallydistributed,amodelofcommunication.Thestatisticalmultiplexingnecessarytoachieveelasticityandtheillusionofinnitecapacityrequiresresourcestobevirtualized,sothattheimplementationofhowtheyaremultiplexedandsharedcanbehiddenfromtheprogrammer.Ourviewisthatdifferentutilitycomputingofferingswillbedistinguishedbasedonthelevelofabstractionpresentedtotheprogrammerandthelevelofmanagementoftheresources.AmazonEC2isatoneendofthespectrum.AnEC2instancelooksmuchlikephysicalhardware,anduserscancontrolnearlytheentiresoftwarestack,fromthekernelupwards.TheAPIexposedis“thin”:afewdozenAPIcallstorequestandcongurethevirtualizedhardware.Thereisnoapriorilimitonthekindsofapplicationsthatcanbehosted;thelowlevelofvirtualization—rawCPUcycles,block-devicestorage,IP-levelconnectivity—allowdeveloperstocodewhatevertheywant.Ontheotherhand,thismakesitinherentlydifcultforAmazontoofferautomaticscalabilityandfailover,becausethesemanticsassociatedwithreplicationandotherstatemanagementissuesarehighlyapplication-dependent.AWSdoesofferanumberofhigher-levelmanagedservices,includingseveraldifferentmanagedstorageservicesforuseinconjunctionwithEC2,suchasSimpleDB.However,theseofferingshavehigherlatencyandnonstandardAPI's,andourunderstandingisthattheyarenotaswidelyusedasotherpartsofAWS.Attheotherextremeofthespectrumareapplicationdomain-specicplatformssuchasGoogleAppEngineandForce.com,theSalesForcebusinesssoftwaredevelopmentplatform.AppEngineistargetedexclusivelyattraditionalwebapplications,enforcinganapplicationstructureofcleanseparationbetweenastatelesscomputationtierandastatefulstoragetier.Furthermore,AppEngineapplicationsareexpectedtoberequest-replybased,andassuchtheyareseverelyrationedinhowmuchCPUtimetheycanuseinservicingaparticularrequest.AppEngine'simpressiveautomaticscalingandhigh-availabilitymechanisms,andtheproprietaryMegaStore(basedonBigTable)datastorageavailabletoAppEngineapplications,allrelyontheseconstraints.Thus,AppEngineisnotsuitableforgeneral-purposecomputing.Similarly,Force.comisdesignedtosupportbusinessapplicationsthatrunagainstthesalesforce.comdatabase,andnothingelse.Microsoft'sAzureisanintermediatepointonthisspectrumofexibilityvs.programmerconvenience.Azureapplicationsarewrittenusingthe.NETlibraries,andcompiledtotheCommonLanguageRuntime,alanguage-independentmanagedenvironment.Thesystemsupportsgeneral-purposecomputing,ratherthanasinglecategoryofapplication.Usersgetachoiceoflanguage,butcannotcontroltheunderlyingoperatingsystemorruntime.Thelibrariesprovideadegreeofautomaticnetworkcongurationandfailover/scalability,butrequirethedevelopertodeclarativelyspecifysomeapplicationpropertiesinordertodoso.Thus,AzureisintermediatebetweencompleteapplicationframeworkslikeAppEngineontheonehand,andhardwarevirtualmachineslikeEC2ontheother.Table4summarizeshowthesethreeclassesvirtualizecomputation,storage,andnetworking.ThescattershotofferingsofscalablestoragesuggestthatscalablestoragewithanAPIcomparableinrichnesstoSQLremainsanopenresearchproblem(seeSection7).AmazonhasbegunofferingOracledatabaseshostedonAWS,buttheeconomicsandlicensingmodelofthisproductmakesitalessnaturaltforCloudComputing.WillonemodelbeatouttheothersintheCloudComputingspace?Wecandrawananalogywithprogramminglanguagesandframeworks.Low-levellanguagessuchasCandassemblylanguageallownecontrolandclosecommunicationwiththebaremetal,butifthedeveloperiswritingaWebapplication,themechanicsofmanagingsockets,dispatchingrequests,andsoonarecumbersomeandtedioustocode,evenwithgoodlibraries.Ontheotherhand,high-levelframeworkssuchasRubyonRailsmakethesemechanicsinvisibletotheprogrammer,butareonlyusefuliftheapplicationreadilytstherequest/replystructureandtheabstractionsprovidedbyRails;anydeviationrequiresdivingintotheframeworkatbest,andmaybeawkwardtocode.NoreasonableRubydeveloperwouldargueagainstthesuperiorityofCforcertaintasks,andviceversa.Correspondingly,webelievedifferenttaskswillresultindemandfordifferentclassesofutilitycomputing.Continuingthelanguageanalogy,justashigh-levellanguagescanbeimplementedinlower-levelones,highly-managedcloudplatformscanbehostedontopofless-managedones.Forexample,AppEnginecouldbehostedontopofAzureorEC2;AzurecouldbehostedontopofEC2.Ofcourse,AppEngineandAzureeachofferproprietaryfeatures(AppEngine'sscaling,failoverandMegaStoredatastorage)orlarge,complexAPI's(Azure's.NETlibraries)thathavenofreeimplementation,soanyattemptto“clone”AppEngineorAzurewouldrequirere-implementingthosefeaturesorAPI's—aformidablechallenge.8 6CloudComputingEconomicsInthissectionwemakesomeobservationsaboutCloudComputingeconomicmodels:•Indecidingwhetherhostingaserviceinthecloudmakessenseoverthelongterm,wearguethatthene-grainedeconomicmodelsenabledbyCloudComputingmaketradeoffdecisionsmoreuid,andinparticulartheelasticityofferedbycloudsservestotransferrisk.•Aswell,althoughhardwareresourcecostscontinuetodecline,theydosoatvariablerates;forexample,com-putingandstoragecostsarefallingfasterthanWANcosts.CloudComputingcantrackthesechanges—andpotentiallypassthemthroughtothecustomer—moreeffectivelythanbuildingone'sowndatacenter,resultinginaclosermatchofexpendituretoactualresourceusage.•Inmakingthedecisionaboutwhethertomoveanexistingservicetothecloud,onemustadditionallyexaminetheexpectedaverageandpeakresourceutilization,especiallyiftheapplicationmayhavehighlyvariablespikesinresourcedemand;thepracticallimitsonreal-worldutilizationofpurchasedequipment;andvariousoperationalcoststhatvarydependingonthetypeofcloudenvironmentbeingconsidered.6.1Elasticity:ShiftingtheRiskAlthoughtheeconomicappealofCloudComputingisoftendescribedas“convertingcapitalexpensestooperatingexpenses”(CapExtoOpEx),webelievethephrase“payasyougo”moredirectlycapturestheeconomicbenettothebuyer.HourspurchasedviaCloudComputingcanbedistributednon-uniformlyintime(e.g.,use100server-hourstodayandnoserver-hourstomorrow,andstillpayonlyforwhatyouuse);inthenetworkingcommunity,thiswayofsellingbandwidthisalreadyknownasusage-basedpricing.3Inaddition,theabsenceofup-frontcapitalexpenseallowscapitaltoberedirectedtocorebusinessinvestment.Therefore,eventhoughAmazon'spay-as-you-gopricing(forexample)couldbemoreexpensivethanbuyinganddepreciatingacomparableserveroverthesameperiod,wearguethatthecostisoutweighedbytheextremelyimportantCloudComputingeconomicbenetsofelasticityandtransferenceofrisk,especiallytherisksofoverprovisioning(underutilization)andunderprovisioning(saturation).Westartwithelasticity.ThekeyobservationisthatCloudComputing'sabilitytoaddorremoveresourcesatanegrain(oneserveratatimewithEC2)andwithaleadtimeofminutesratherthanweeksallowsmatchingresourcestoworkloadmuchmoreclosely.Realworldestimatesofserverutilizationindatacentersrangefrom5%to20%[37,38].Thismaysoundshockinglylow,butitisconsistentwiththeobservationthatformanyservicesthepeakworkloadexceedstheaveragebyfactorsof2to10.Fewusersdeliberatelyprovisionforlessthantheexpectedpeak,andthereforetheymustprovisionforthepeakandallowtheresourcestoremainidleatnonpeaktimes.Themorepronouncedthevariation,themorethewaste.Asimpleexampledemonstrateshowelasticityallowsreducingthiswasteandcanthereforemorethancompensateforthepotentiallyhighercostperserver-hourofpaying-as-you-govs.buying. Example:Elasticity.Assumeourservicehasapredictabledailydemandwherethepeakrequires500serversatnoonbutthetroughrequiresonly100serversatmidnight,asshowninFigure2(a).Aslongastheaverageutilizationoverawholedayis300servers,theactualutilizationoverthewholeday(shadedareaunderthecurve)is30024=7200server-hours;butsincewemustprovisiontothepeakof500servers,wepayfor50024=12000server-hours,afactorof1.7morethanwhatisneeded.Therefore,aslongasthepay-as-you-gocostperserver-hourover3years4islessthan1.7timesthecostofbuyingtheserver,wecansavemoneyusingutilitycomputing. Infact,theaboveexampleunderestimatesthebenetsofelasticity,becauseinadditiontosimplediurnalpatterns,mostnontrivialservicesalsoexperienceseasonalorotherperiodicdemandvariation(e.g.,e-commercepeaksinDe-cemberandphotosharingsitespeakafterholidays)aswellassomeunexpecteddemandburstsduetoexternalevents(e.g.,newsevents).Sinceitcantakeweekstoacquireandracknewequipment,theonlywaytohandlesuchspikesistoprovisionfortheminadvance.Wealreadysawthatevenifserviceoperatorspredictthespikesizescorrectly,capacityiswasted,andiftheyoverestimatethespiketheyprovisionfor,it'sevenworse.Theymayalsounderestimatethespike(Figure2(b)),however,accidentallyturningawayexcessusers.Whilethemonetaryeffectsofoverprovisioningareeasilymeasured,thoseofunderprovisioningarehardertomeasureyetpotentiallyequallyserious:notonlydorejectedusersgeneratezerorevenue,theymaynevercomebackduetopoorservice.Figure2(c)aimstocapturethisbehavior:userswilldesertanunderprovisionedserviceuntilthepeakuser10 Table5:WeupdateGray'scostsofcomputingresourcesfrom2003to2008,normalizetowhat$1couldbuyin2003vs.2008,andcomparetothecostofpayingperuseof$1worthofresourcesonAWSat2008prices. WANbandwidth/mo. CPUhours(allcores) diskstorage Itemin2003 1MbpsWANlink 2GHzCPU,2GBDRAM 200GBdisk,50Mb/stransferrate Costin2003 $100/mo. $2000 $200 $1buysin2003 1GB 8CPUhours 1GB Itemin2008 100MbpsWANlink 2GHz,2sockets,4cores/socket,4GBDRAM 1TBdisk,115MB/ssus-tainedtransfer Costin2008 $3600/mo. $1000 $100 $1buysin2008 2.7GB 128CPUhours 10GB cost/performanceimprovement 2.7x 16x 10x Costtorent$1 $0.27–$0.40 $2.56 $1.20–$1.50 worthonAWSin2008 ($0.10–$0.15/GB3GB) (1282VM's@$0.10each) ($0.12–$0.15/GB-month10GB) ingothers.Pay-as-you-goCloudComputingcanchargetheapplicationseparatelyforeachtypeofresource,reducingthewasteofunderutilization.Whiletheexactsavingsdependsontheapplication,supposetheCPUisonly50%utilizedwhilethenetworkisatcapacity;theninadatacenteryouareeffectivelypayingfordoublethenumberofCPUcyclesactuallybeingused.Soratherthansayingitcosts$2.56torentonly$1worthofCPU,itwouldbemoreaccuratetosayitcosts$2.56torent$2worthofCPU.Asasidenote,AWS'spricesforwide-areanetworkingareactuallymorecompetitivethanwhatamedium-sizedcompanywouldpayforthesamebandwidth.Power,coolingandphysicalplantcosts.Thecostsofpower,cooling,andtheamortizedcostofthebuildingaremissingfromoursimpleanalysessofar.HamiltonestimatesthatthecostsofCPU,storageandbandwidthroughlydoublewhenthosecostsareamortizedoverthebuilding'slifetime[23,26].Usingthisestimate,buying128hoursofCPUin2008reallycosts$2ratherthan$1,comparedto$2.56onEC2.Similarly,10GBofdiskspacecosts$2ratherthan$1,comparedto$1.20–$1.50permonthonS3.Lastly,S3actuallyreplicatesthedataatleast3timesfordurabilityandperformance,ensuredurability,andwillreplicateitfurtherforperformanceisthereishighdemandforthedata.Thatmeansthecostsare$6.00whenpurchasingvs.$1.20to$1.50permonthonS3.Operationscosts.Today,hardwareoperationscostsareverylow—rebootingserversiseasy(e.g.,IPaddressablepowerstrips,separateoutofbandcontrollers,andsoon)andminimallytrainedstaffcanreplacebrokencomponentsattherackorserverlevel.Ononehand,sinceUtilityComputingusesvirtualmachinesinsteadofphysicalmachines,fromtheclouduser'spointofviewthesetasksareshiftedtothecloudprovider.Ontheotherhand,dependingonthelevelofvirtualization,muchofthesoftwaremanagementcostsmayremain—upgrades,applyingpatches,andsoon.Returningtothe“managedvs.unmanaged”discussionofSection5,webelievethesecostswillbelowerformanagedenvironments(e.g.MicrosoftAzure,GoogleAppEngine,Force.com)thanforhardware-levelutilitycomputing(e.g.AmazonEC2),butitseemshardtoquantifythesebenetsinawaythatmanywouldagreewith.Withtheabovecaveatsinmind,hereisasimpleexampleofdecidingwhethertomoveaserviceintothecloud. Example:Movingtocloud.Supposeabiologylabcreates500GBofnewdataforeverywetlabexperi-ment.AcomputerthespeedofoneEC2instancetakes2hoursperGBtoprocessthenewdata.Thelabhastheequivalent20instanceslocally,sothetimetoevaluatetheexperimentis5002=20or50hours.Theycouldprocessitinasinglehouron1000instancesatAWS.Thecosttoprocessoneexperimentwouldbejust1000$0:10or$100incomputationandanother500$0:10or$50innetworktransferfees.Sofar,sogood.TheymeasurethetransferratefromthelabtoAWSat20Mbits/second.[19]Thetransfertimeis(500GB1000MB=GB8bits=Byte)=20Mbits=sec=4;000;000=20=200;000secondsormorethan55hours.Thus,ittakes50hourslocallyvs.55+1or56hoursonAWS,sotheydon'tmovetothecloud.(Thenextsectionoffersanopportunityonhowtoovercomethetransferdelayobstacle.) Arelatedissueisthesoftwarecomplexityandcostsof(partialorfull)migratingdatafromalegacyenterpriseapplicationintotheCloud.Whilemigrationisaone-timetask,theamountofeffortcanbesignicantanditneedstobeconsideredasafactorindecidingtouseCloudComputing.ThistaskisalreadyspawningnewbusinessopportunitiesforcompaniesthatprovidedataintegrationacrosspublicandprivateClouds.13 bot(simulatedbogususer)perweek[36].UtilityComputingoffersSaaSproviderstheopportunitytodefendagainstDDoSattacksbyusingquickscale-up.SupposeanEC2instancecanhandle500bots,andanattackislaunchedthatgeneratesanextra1GB/secondofbogusnetworkbandwidthand500,000bots.At$0.03perbot,suchanattackwouldcosttheattacker$15,000investedupfront.AtAWS'scurrentprices,theattackwouldcostthevictimanextra$360perhourinnetworkbandwidthandanextra$100perhour(1,000instances)ofcomputation.Theattackwouldthereforehavetolast32hoursinordertocostthepotentialvictimmorethanitwouldtheblackmailer.Abotnetattackthislongmaybedifculttosustain,sincethelongeranattacklaststheeasieritistouncoveranddefendagainst,andtheattackingbotscouldnotbeimmediatelyre-usedforotherattacksonthesameprovider.Aswithelasticity,CloudComputingshiftstheattacktargetfromtheSaaSprovidertotheUtilityComputingprovider,whocanmorereadilyabsorbitand(aswearguedinSection3)isalsolikelytohavealreadyDDoSprotectionasacorecompetency.Number2Obstacle:DataLock-InSoftwarestackshaveimprovedinteroperabilityamongplatforms,buttheAPIsforCloudComputingitselfarestillessentiallyproprietary,oratleasthavenotbeenthesubjectofactivestandardization.Thus,customerscannoteasilyextracttheirdataandprogramsfromonesitetorunonanother.ConcernaboutthedifcultofextractingdatafromthecloudispreventingsomeorganizationsfromadoptingCloudComputing.Customerlock-inmaybeattractivetoCloudComputingproviders,butCloudComputingusersarevulnerabletopriceincreases(asStallmanwarned),toreliabilityproblems,oreventoprovidersgoingoutofbusiness.Forexample,anonlinestorageservicecalledTheLinkupshutdownonAugust8,2008afterlosingaccessasmuchas45%ofcustomerdata[12].TheLinkup,inturn,hadreliedontheonlinestorageserviceNirvanixtostorecustomerdata,andnowthereisngerpointingbetweenthetwoorganizationsastowhycustomerdatawaslost.Meanwhile,TheLinkup's20,000usersweretoldtheservicewasnolongeravailableandwereurgedtotryoutanotherstoragesite.TheobvioussolutionistostandardizetheAPIssothataSaaSdevelopercoulddeployservicesanddataacrossmultipleCloudComputingproviderssothatthefailureofasinglecompanywouldnottakeallcopiesofcustomerdatawithit.Theobviousfearisthatthiswouldleadtoa“race-to-the-bottom”ofcloudpricingandattentheprotsofCloudComputingproviders.Weoffertwoargumentstoallaythisfear.First,thequalityofaservicemattersaswellastheprice,socustomerswillnotnecessarilyjumptothelowestcostservice.SomeInternetServiceProviderstodaycostafactoroftenmorethanothersbecausetheyaremoredependableandofferextraservicestoimproveusability.Second,inadditiontomitigatingdatalock-inconcerns,standardizationofAPIsenablesanewusagemodelinwhichthesamesoftwareinfrastructurecanbeusedinaPrivateCloudandinaPublicCloud.9Suchanoptioncouldenable“SurgeComputing,”inwhichthepublicCloudisusedtocapturetheextratasksthatcannotbeeasilyruninthedatacenter(orprivatecloud)duetotemporarilyheavyworkloads.10Number3Obstacle:DataCondentialityandAuditability“Mysensitivecorporatedatawillneverbeinthecloud.”Anecdotallywehaveheardthisrepeatedmultipletimes.Currentcloudofferingsareessentiallypublic(ratherthanprivate)networks,exposingthesystemtomoreattacks.Therearealsorequirementsforauditability,inthesenseofSarbanes-OxleyandHealthandHumanServicesHealthInsurancePortabilityandAccountabilityAct(HIPAA)regulationsthatmustbeprovidedforcorporatedatatobemovedtothecloud.Webelievethattherearenofundamentalobstaclestomakingacloud-computingenvironmentassecureasthevastmajorityofin-houseITenvironments,andthatmanyoftheobstaclescanbeovercomeimmediatelywithwell-understoodtechnologiessuchasencryptedstorage,VirtualLocalAreaNetworks,andnetworkmiddleboxes(e.g.rewalls,packetlters).Forexample,encryptingdatabeforeplacingitinaCloudmaybeevenmoresecurethanunencrypteddatainalocaldatacenter;thisapproachwassuccessfullyusedbyTC3,ahealthcarecompanywithaccesstosensitivepatientrecordsandhealthcareclaims,whenmovingtheirHIPAA-compliantapplicationtoAWS[2].Similarly,auditabilitycouldbeaddedasanadditionallayerbeyondthereachofthevirtualizedguestOS(orvirtualizedapplicationenvironment),providingfacilitiesarguablymoresecurethanthosebuiltintotheapplicationsthemselvesandcentralizingthesoftwareresponsibilitiesrelatedtocondentialityandauditabilityintoasinglelogicallayer.SuchanewfeaturereinforcestheCloudComputingperspectiveofchangingourfocusfromspecichardwaretothevirtualizedcapabilitiesbeingprovided.ArelatedconcernisthatmanynationshavelawsrequiringSaaSproviderstokeepcustomerdataandcopyrightedmaterialwithinnationalboundaries.Similarly,somebusinessesmaynotliketheabilityofacountrytogetaccesstotheirdataviathecourtsystem;forexample,aEuropeancustomermightbeconcernedaboutusingSaaSintheUnitedStatesgiventheUSAPATRIOTAct.15 Figure3:(a)Memorybenchmarkperformanceon75VirtualMachinesrunningtheSTREAMbenchmarkonleftand(b)Diskperformancewriting1GBleson75VirtualMachinesonright.widespreaddeploymentinsidethecloudsinceithasthehighlydesirableeffectofreducingdatatransferlatenciesandnetworkcontention.Thisinturnenablesmorecoresandvirtualmachinesperphysicalservernodebyscalingupthenetwork.Alsoin2010,40GbEand100GbEwillappearforthehigheraggregationlayers[10].Number5Obstacle:PerformanceUnpredictabilityOurexperienceisthatmultipleVirtualMachinescanshareCPUsandmainmemorysurprisinglywellinCloudCom-puting,butthatI/Osharingismoreproblematic.Figure3(a)showstheaveragememorybandwidthfor75EC2instancesrunningtheSTREAMmemorybenchmark[32].Themeanbandwidthis1355MBytespersecond,withastandarddeviationofjust52MBytes/sec,lessthan4%ofthemean.Figure3(b)showstheaveragediskbandwidthfor75EC2instanceseachwriting1GBlestolocaldisk.Themeandiskwritebandwidthisnearly55MBytespersecondwithastandarddeviationofalittleover9MBytes/sec,morethan16%ofthemean.ThisdemonstratestheproblemofI/Ointerferencebetweenvirtualmachines.OneopportunityistoimprovearchitecturesandoperatingsystemstoefcientlyvirtualizeinterruptsandI/Ochan-nels.TechnologiessuchasPCIexpressaredifculttovirtualize,buttheyarecriticaltothecloud.OnereasontobehopefulisthatIBMmainframesandoperatingsystemslargelyovercametheseproblemsinthe1980s,sowehavesuccessfulexamplesfromwhichtolearn.AnotherpossibilityisthatashmemorywilldecreaseI/Ointerference.Flashissemiconductormemorythatpreservesinformationwhenpoweredofflikemechanicalharddisks,butsinceithasnomovingparts,itismuchfastertoaccess(microsecondsvs.milliseconds)anduseslessenergy.FlashmemorycansustainmanymoreI/Ospersecondpergigabyteofstoragethandisks,somultiplevirtualmachineswithconictingrandomI/Oworkloadscouldcoexistbetteronthesamephysicalcomputerwithouttheinterferenceweseewithmechanicaldisks.ThelackofinterferencethatweseewithsemiconductormainmemoryinFigure3(a)mightextendtosemiconductorstorageaswell,therebyincreasingthenumberofapplicationsthatcanrunwellonVMsandthusshareasinglecomputer.ThisadvancecouldlowercoststoCloudComputingproviders,andeventuallytoCloudComputingconsumers.Anotherunpredictabilityobstacleconcernstheschedulingofvirtualmachinesforsomeclassesofbatchprocessingprograms,specicallyforhighperformancecomputing.Giventhathigh-performancecomputingisusedtojustifyGovernmentpurchasesof$100Msupercomputercenterswith10,000to1,000,000processors,therecertainlyaremanytaskswithparallelismthatcanbenetfromelasticcomputing.Costassociativitymeansthatthereisnocostpenaltyforusing20timesasmuchcomputingfor1=20ththetime.Potentialapplicationsthatcouldbenetincludethosewithveryhighpotentialnancialreturns—nancialanalysis,petroleumexploration,movieanimation—andcouldeasilyjustifypayingamodestpremiumfora20xspeedup.Oneestimateisthatathirdoftoday'sservermarketishigh-performancecomputing[10].TheobstacletoattractingHPCisnottheuseofclusters;mostparallelcomputingtodayisdoneinlargeclustersusingthemessage-passinginterfaceMPI.TheproblemisthatmanyHPCapplicationsneedtoensurethatallthethreadsofaprogramarerunningsimultaneously,andtoday'svirtualmachinesandoperatingsystemsdonotprovide17 Number10Obstacle:SoftwareLicensingCurrentsoftwarelicensescommonlyrestrictthecomputersonwhichthesoftwarecanrun.Userspayforthesoftwareandthenpayanannualmaintenancefee.Indeed,SAPannouncedthatitwouldincreaseitsannualmaintenancefeetoatleast22%ofthepurchasepriceofthesoftware,whichiscomparabletoOracle'spricing[38].Hence,manycloudcomputingprovidersoriginallyreliedonopensourcesoftwareinpartbecausethelicensingmodelforcommercialsoftwareisnotagoodmatchtoUtilityComputing.TheprimaryopportunityiseitherforopensourcetoremainpopularorsimplyforcommercialsoftwarecompaniestochangetheirlicensingstructuretobettertCloudComputing.Forexample,MicrosoftandAmazonnowofferpay-as-you-gosoftwarelicensingforWindowsServerandWindowsSQLServeronEC2.AnEC2instancerunningMicrosoftWindowscosts$0.15perhourinsteadofthetraditional$0.10perhouroftheopensourceversion.15ArelatedobstacleisencouragingsalesforcesofsoftwarecompaniestosellproductsintoCloudComputing.Pay-as-you-goseemsincompatiblewiththequarterlysalestrackingusedtomeasureeffectiveness,whichisbasedonone-timepurchases.Theopportunityforcloudprovidersissimplytoofferprepaidplansforbulkusethatcanbesoldatdiscount.Forexample,Oraclesalespeoplemightsell100,000instancehoursusingOraclethatcanbeusedoverthenexttwoyearsatacostlessthanisthecustomerweretopurchase100,000hoursontheirown.Theycouldthenmeettheirquarterlyquotasandmaketheircommissionsfromcloudsalesaswellasfromtraditionalsoftwaresales,potentiallyconvertingthiscustomer-facingpartofacompanyfromnaysayersintoadvocatesofcloudcomputing.8ConclusionandQuestionsabouttheCloudsofTomorrowThelongdreamedvisionofcomputingasautilityisnallyemerging.TheelasticityofautilitymatchestheneedofbusinessesprovidingservicesdirectlytocustomersovertheInternet,asworkloadscangrow(andshrink)farfasterthan20yearsago.Itusedtotakeyearstogrowabusinesstoseveralmillioncustomers–nowitcanhappeninmonths.Fromthecloudprovider'sview,theconstructionofverylargedatacentersatlowcostsitesusingcommoditycomputing,storage,andnetworkinguncoveredthepossibilityofsellingthoseresourcesonapay-as-you-gomodelbelowthecostsofmanymedium-sizeddatacenters,whilemakingaprotbystatisticallymultiplexingamongalargegroupofcustomers.Fromtheclouduser'sview,itwouldbeasstartlingforanewsoftwarestartuptobuilditsowndatacenterasitwouldforahardwarestartuptobuilditsownfabricationline.Inadditiontostartups,manyotherestablishedorganizationstakeadvantageoftheelasticityofCloudComputingregularly,includingnewspapersliketheWashingtonPost,moviecompanieslikePixar,anduniversitieslikeours.Ourlabhasbenetedsubstantiallyfromtheabilitytocompleteresearchbyconferencedeadlinesandadjustresourcesoverthesemestertoaccommodatecoursedeadlines.AsCloudComputingusers,wewererelievedofdealingwiththetwindangersofover-provisioningandunder-provisioningourinternaldatacenters.Somequestionwhethercompaniesaccustomedtohigh-marginbusinesses,suchasadrevenuefromsearchenginesandtraditionalpackagedsoftware,cancompeteinCloudComputing.First,thequestionpresumesthatCloudCom-putingisasmallmarginbusinessbasedonitslowcost.Giventhetypicalutilizationofmedium-sizeddatacenters,thepotentialfactorsof5to7ineconomiesofscale,andthefurthersavingsinselectionofclouddatacenterlocations,theapparentlylowcostsofferedtocloudusersmaystillbehighlyprotabletocloudproviders.Second,thesecompaniesmayalreadyhavethedatacenter,networking,andsoftwareinfrastructureinplacefortheirmainlinebusinesses,soCloudComputingrepresentstheopportunityformoreincomeatlittleextracost.AlthoughCloudComputingprovidersmayrunafouloftheobstaclessummarizedinTable6,webelievethatoverthelongrunproviderswillsuccessfullynavigatethesechallengesandsetanexampleforotherstofollow,perhapsbysuccessfullyexploitingtheopportunitiesthatcorrespondtothoseobstacles.Hence,developerswouldbewisetodesigntheirnextgenerationofsystemstobedeployedintoCloudComput-ing.Ingeneral,theemphasisshouldbehorizontalscalabilitytohundredsorthousandsofvirtualmachinesovertheefciencyofthesystemonasinglevirtualmachine.Therearespecicimplicationsaswell:•ApplicationsSoftwareofthefuturewilllikelyhaveapiecethatrunsonclientsandapiecethatrunsintheCloud.Thecloudpieceneedstobothscaledownrapidlyaswellasscaleup,whichisanewrequirementforsoftwaresystems.TheclientpieceneedstobeusefulwhendisconnectedfromtheCloud,whichisnotthecaseformanyWeb2.0applicationstoday.Suchsoftwarealsoneedsapay-for-uselicensingmodeltomatchneedsofCloudComputing.•InfrastructureSoftwareofthefutureneedstobecognizantthatitisnolongerrunningonbaremetalbutonvirtualmachines.Moreover,itneedstohavebillingbuiltinfromthebeginning,asitisverydifculttoretrotanaccountingsystem.19 •Diskarmseeking60%•DiskIOpersecondorMB/s80%peakHence,60%to80%isasafeupperbound.7Table8showschangesinpricesforAWSstorageandnetworkingover2.5years.Table8:ChangesinpriceofAWSS3storageandnetworkingovertime. Storage CostofDataStoredperGB-Month Date 50TB50-100TB100-500TB&#x]TJ/;༳ ;.97; T; 8.;չ ;� Td;&#x [00;500TB 3/13/06 $0.15$0.15$0.15$0.15 10/9/08 $0.15$0.14$0.13$0.12 %OriginalPrice 100%93%87%80% Networking CostperGBofWide-AreaNetworkingTrafc Date InOut:10TBOut:10-50TBOut:50-150TBOut:&#x]TJ/;༳ ;.97; T; 8.;չ ;� Td;&#x [00;150TB 3/13/06 $0.20$0.20$0.20$0.20$0.20 10/31/07 $0.10$0.18$0.16$0.13$0.13 5/1/08 $0.10$0.17$0.13$0.11$0.10 %OriginalPrice 50%85%65%55%50% 8Table9showsthenewservicesandsupportoptionsAWSaddedduring2008,andthedateofeachintroduction.Table10showsthedifferenttypesofAWScomputeinstancesandthedateeachtypewasintroduced.Table9:NewAWSServices. Date NewService 3-Dec-08 PublicDataSetsonAWSNowAvailable 18-Nov-08 AnnouncingAmazonCloudFront(ContentDistributionNetwork) 23-Oct-08 AmazonEC2RunningWindowsServerNowAvailable 23-Oct-08 AmazonEC2ExitsBetaandNowOffersaServiceLevelAgreement 22-Sep-08 OracleProductsLicensedforAmazonWebServices 20-Aug-08 AmazonElasticBlockStoreNowAvailable 5-May-08 OpenSolarisandMySQLEnterpriseonAmazonEC2 16-Apr-08 AnnouncingAWSPremiumSupport 26-Mar-08 AnnouncingElasticIPAddressesandAvailabilityZonesforAmazonEC2 Table10:DiversityofEC2instancesovertime. Date Type Cost/Hour ComputeUnitsDRAM(GB)Disk(GB) Compute/$GBDRAM/$GBDisk/$ 8/24/06 Small $0.10 11.7160 1017.01600 10/22/07 Large $0.40 47.5850 1018.82130 10/22/07 ExtraLarge $0.80 815.01690 1018.82110 5/29/08 High-CPUMedium $0.20 51.7350 258.51750 5/29/08 High-CPUExtraLarge $0.80 207.01690 258.82110 9Whilesuchstandardizationcanoccurforthefullspectrumofutilitycomputing,theabilityoftheleadingcloudproviderstodistributesoftwaretomatchstandardizedAPIsvaries.Microsoftisinthesoftwaredistributionbusiness,soitwouldseemtobeasmallstepforAzuretopublishalltheAPIsandoffersoftwaretoruninthedatacenter.InterestinglyforAWSandGoogleAppEngine,thebestexamplesofstandardizingAPIscomefromopensourceseffortsfromoutsidethesecompanies.HadoopandHypertableareeffortstorecreatetheGoogleinfrastructure[11],andEucalyptusrecreatesimportantaspectsoftheEC2API[34].10Indeed,harkingbacktoSection2,“surgechipfabrication”isoneofthecommonusesof“chip-les”fabricationcompanieslikeTSMC.11A1TB3.5”diskweighs1.4pounds.Ifweassumethatpackagingmaterialaddsabout20%totheweight,theshippingweightof10disksis17pounds.FedExchargesabout$100todeliversuchapackageby10:30AMthenextdayandabout$50todeliveritin2days.SimilartoNetix,Amazonmightletyouhaveone“diskboat”onloantousewhenyouneedit.Thus,theround-tripshippingcostforAmazontoshipyouasetofdisksandforyoutoshipitbackis$150,assuming2-daydeliveryfromAmazonandovernightdeliverytosendittoAmazon.ItwouldthentakeAmazonabout2.4hoursto“dump”thediskcontentsintotheirdatacenter(a1TBdiskcantransferat115Mbytes/sec).Ifeachdiskcontainswholeles(e.g.aLinuxext3orWindowsNTFSlesystem),alldiskscouldbereadorwritteninparallel.Whileit'shardtoputacostofinternaldatacenterLANbandwidth,itissurelyatleast100xlessexpensivethanWANbandwidth.Let'sassumethelaborcoststounpackdisks,loadthemsothattheycanberead,repackagethem,andsoonis$20perdisk.Thetotallatencyisthenlessthanaday(2.4hourstowrite,14-18hoursforovernightshipping,2.4hourstoread)atacostabout$400($50toreceivefromAmazon,$100tosendtoAmazon,$200forlaborcosts,and$40chargeforinternalAmazonLANbandwidthandlaborcharges).Ratherthanshipdisks,anotheroptionwouldbetoshipawholediskarrayincludingsheetmetal,fans,powersuppliers,andnetworkinterfaces.Theextracomponentswouldincreasetheshippingweight,butitwouldsimplifyconnectionofstoragetotheCloudandtothelocaldeviceandreducelabor.Notethatyouwouldwantalotmorenetworkbandwidththanistypicallyprovidedinconventionaldiskarrays,sinceyoudon'twanttostretchthetimeloadorunloadthedata.12TherelativelynewcompanyDataDomainusesspecializedcompressionalgorithmstailoredtoincrementalbackups,theycanreducethesizeofthesebackupsbyafactorof20.Notethatcompressioncanalsoreducethecosttoutilitycomputingprovidersoftheirstandardstorageproducts.Losslesscompressioncanreducedatademandsbyfactorsfortwotothreeformanydatatypes,andmuchhigherforsome.TheCloudComputing21 [19]GARFINKEL,S.AnEvaluationofAmazon'sGridComputingServices:EC2,S3andSQS.Tech.Rep.TR-08-07,HarvardUniversity,August2007.[20]GHEMAWAT,S.,GOBIOFF,H.,ANDLEUNG,S.-T.Thegooglelesystem.InSOSP'03:ProceedingsofthenineteenthACMsymposiumonOperatingsystemsprinciples(NewYork,NY,USA,2003),ACM,pp.29–43.Availablefrom:http://portal.acm.org/ft_gateway.cfm?id=945450&type=pdf&coll=Portal&dl=GUIDE&CFID=19219697&CFTOKEN=50259492.[21]GRAY,J.DistributedComputingEconomics.Queue6,3(2008),63–68.Availablefrom:http://portal.acm.org/ft_gateway.cfm?id=1394131&type=digital%20edition&coll=Portal&dl=GUIDE&CFID=19219697&CFTOKEN=50259492.[22]GRAY,J.,ANDPATTERSON,D.AconversationwithJimGray.ACMQueue1,4(2003),8–17.[23]HAMILTON,J.CostofPowerinLarge-ScaleDataCenters[online].November2008.Availablefrom:http://perspectives.mvdirona.com/2008/11/28/CostOfPowerInLargeScaleDataCenters.aspx.[24]HAMILTON,J.Internet-ScaleServiceEfciency.InLarge-ScaleDistributedSystemsandMiddleware(LADIS)Workshop(September2008).[25]HAMILTON,J.Perspectives[online].2008.Availablefrom:http://perspectives.mvdirona.com.[26]HAMILTON,J.CooperativeExpendableMicro-SliceServers(CEMS):LowCost,LowPowerServersforInternet-ScaleServices.InConferenceonInnovativeDataSystemsResearch(CIDR'09)(January2009).[27]H¨OLZLE,U.Privatecommunication,January2009.[28]HOSANAGAR,K.,KRISHNAN,R.,SMITH,M.,ANDCHUANG,J.Optimalpricingofcontentdeliverynetwork(CDN)services.InThe37thAnnualHawaiiInternationalConferenceonSystemSciences(2004),pp.205–214.[29]JACKSON,T.Wefeelyourpain,andwe'resorry[online].August2008.Availablefrom:http://gmailblog.blogspot.com/2008/08/we-feel-your-pain-and-were-sorry.html.[30]KISTLER,J.J.,ANDSATYANARAYANAN,M.Disconnectedoperationinthecodalesystem.InThirteenthACMSymposiumonOperatingSystemsPrinciples(AsilomarConferenceCenter,PacicGrove,U.S.,1991),vol.25,ACMPress,pp.213–225.[31]KREBS,B.Amazon:HeySpammers,GetOffMyCloud!WashingtonPost(July2008).[32]MCCALPIN,J.Memorybandwidthandmachinebalanceincurrenthighperformancecomputers.IEEETechnicalCommitteeonComputerArchitectureNewsletter(1995),19–25.[33]MCKEOWN,N.,ANDERSON,T.,BALAKRISHNAN,H.,PARULKAR,G.,PETERSON,L.,REXFORD,J.,SHENKER,S.,,ANDTURNER,J.OpenFlow:Enablinginnovationincampusnetworks.ACMSIGCOMMComputerCommunicationReview38,2(April2008).[34]NURMI,D.,WOLSKI,R.,GRZEGORCZYK,C.,OBERTELLI,G.,SOMAN,S.,YOUSEFF,L.,ANDZAGORODNOV,D.Eucalyptus:ATechnicalReportonanElasticUtilityComputingArchietctureLinkingYourProgramstoUsefulSystems.Tech.Rep.2008-10,UniversityofCalifornia,SantaBarbara,October2008.[35]PARKHILL,D.TheChallengeoftheComputerUtility.Addison-WesleyEducationalPublishersInc.,US,1966.[36]PAXSON,V.privatecommunication,December2008.[37]RANGAN,K.TheCloudWars:$100+billionatstake.Tech.rep.,MerrillLynch,May2008.[38]SIEGELE,L.LetItRise:ASpecialReportonCorporateIT.TheEconomist(October2008).[39]STERN,A.UpdateFromAmazonRegardingFriday'sS3Downtime.CenterNetworks(February2008).Availablefrom:http://www.centernetworks.com/amazon-s3-downtime-update.[40]STUER,G.,VANMECHELEN,K.,ANDBROECKHOVE,J.AcommoditymarketalgorithmforpricingsubstitutableGridresources.FutureGenerationComputerSystems23,5(2007),688–701.[41]THEAMAZONS3TEAM.AmazonS3AvailabilityEvent:July20,2008[online].July2008.Availablefrom:http://status.aws.amazon.com/s3-20080720.html.[42]VOGELS,W.AHeadintheClouds—ThePowerofInfrastructureasaService.InFirstworkshoponCloudComputingandinApplications(CCA'08)(October2008).[43]WILSON,S.AppEngineOutage.CIOWeblog(June2008).Availablefrom:http://www.cio-weblog.com/50226711/appengine\_outage.php.23