White PaperIP Telephony: The Five NinesIP Telephony AvailabilityOvervi - PDF document

White PaperIP Telephony: The Five NinesIP Telephony AvailabilityOverviewAvailability can be deÞned as theavailability.AnetworkthatisnÕtavailableismore than annoying or frustrating, itÕsexpensive. With increased dependency oncosts everyone valuable time and money.voice services to IP telephony, a routineexpectations for voice service availability.Acommon goal is to consistently achieveallmission critical applications, includingbest-practices,CiscoIPtelephonysolutionscansupport99.999%uptime.ÒFiveninesÓdesiredgoal,itisanachievablegoalthatisavailabilityIPtelephonysolutiondiscussedin this paper.perceptionisthatlegacyPBXsolutionsareachievesÞveninesbydeÞningavailabilityinIPtelephonysolutioniscarefullyexaminedTelcordia (formerly Bellcore) MTBFhardware reliability section of this paper.negligibleduetoredundantdevicesateach http://www.apcc.com/support.ThislevelofpoweravailabilityrequiresUPSsystemswithgeneratorbackupandUPSmonitoringwithfourhourresponseUPSserviceandsupport.Theoreticalnetworkdesignavailabilityis99.99947%CallManager and PSTN gateways. The analysis is provided in the design section later in this paper. Using normaltheoreticalavailabilityanalysistechniques.Actualavailabilitymayvaryduetootherfactorsthatincludelink/carrieravailability, network management processes and expertise. By reading the following deÞnition of availability andreadershould also be aware that Cisco is committed to providing high availability and improving availability forLegacy PBX AvailabilityWhatdoes99.999%reliabilitymeantoacustomer?Itmeansabsoluteminimumdowntimeintermsofexistingcallsdroppedandinabilitytoplacecalls.Fiveninesequatestolessthat5.26minutesofdowntimeinayearofoperation.requirement.LegacyPBXvendorsaresometimesabletoachieveÞveninesofavailabilityusingastrictdeÞnitionofbroken phones, wiring or lost power can be a factor. The deÞnition, as you will see below, is primarily a factor ofItisimportanttounderstandhowÞveninesreliabilityismeasuredinthelegacyPBXworld.Inordertodothat,letÕstakealookatthebasiccomponentsofaPBXandhowtheyrelatetomeasuringÞveninesuptime.ThetwodiagramsRedundant and Non-redundant PBXs TDM BusLegacy Line andTrunk CardsCommonControl CardsSingle PowerSupply PowerSupply Non-redundant PBX TDM BusTDM BusLegacy Line andTrunk CardsRedundant CommonControl CardsRedundant PowerSupplies Redundant PBX Supply LineCardsÑAnaloglinecardssupportanalogdevices(phones,modems,fax,recorders,etc.).DigitallinecardsTrunkCardsÑProvideconnectivitytothePSTN.AnalogtrunkcardsforFXO,OPS,orTIEtrunks.DigitaltrunkCommonControlÑCPU,memory,andothercardsusedtoperformcallprocessing,diagnostics,administration,ÑTakes power from an external AC or DC source and distributes it to the shelves to power theÑThis is an internal UPS that is found in some PBXÕs. It provides power to operate thethroughanorderlyshutdown.FormorethanjustafewminutesofprotectionanexternalUPSsystemisrequired.Cards, the Power Supplies, and the TDM Bus. Sometimes redundancy only means CPU and memory redundancy.Powersupplyredundancyisoftenaseparatelyorderableredundancyoption.Note,however,thatthelineandtrunkNowthatweÕveexaminedthecomponentsofaPBX,whattypesoffailureswouldcountinacalculationofÞvenines¥Failure of a line card (typically 16 to 32 ports) putting phones out of service?¥Failure of a trunk card (typically 4 to 30 bearer channels) putting a group of trunks out of service?¥Loss of a couple of attendant consoles?Theanswer?Noneofthesewouldcountassystemdowntime.Ingeneral,accordingtoBellCoreGR512standards,downtime,whichislossofalllines,trunks,orallattendantconsoles.Inotherwords,inanon-redundantPBXyoubackup element. ThatÕs why redundant PBXs are so reliable.Ofcourse,ifyouthinkaboutitlogically,itwouldbedifÞculttoconsistentlyachieve99.999%uptimeifyoucountedminorfailures,suchasanon-redundantlinecard.MostPBXvendorsofferfour-hourresponsetominoroutagessuchaslossofalineortrunkcard(unlessspecialservicelevelagreementsareinplace).ForinstanceifaPBXhas48linecardsinaPBXandfourhourreplacementispossible,theMTBFwouldhavetobeabout400,000hoursor45years.ThisusesthebasiccalculationofavailabilityequalingMTBF/MTBF+MTTR.400,000hoursisachievable,butisInfact,legacyPBXvendorsarecurrentlyhardpressedtoshowÞveninesreliabilityfornonredundantsystems.Again, where redundancy, power backup and fast repair times are not possible. When end-to-end availability is closelyserviceavailabilityintherangeof99.95%.Thisisclosertofourhoursofdowntimeperyearandcouldbeattributedtolocalpower,brokenphonesorbrokenconnectionstothemainswitch.ThemainPBXmayverywellbeÞveninesIn summary, legacy PBX systems can maintain Þve nines of availability. However, this level of availability onlyprovidershavebudgetedend-to-endavailabilitymoreintherangeof99.95%availabilityor4+hoursofdowntimeper year.IP Telephony Availability DefinitionInamannersimilartolegacyPBXvendors,CiscocanachieveÞveninesofavailabilitywiththeIPTelephonysolutionbydeÞningavailabilityintermsofaredundantcoresystem.UsingthelegacyPBXavailabilitydeÞnition,CiscocandeÞneCampusIPtelephonyavailabilityastheabilitytomaintainandoriginatecallswithintheredundantcoreanddistribution layers using a redundant Cisco CallManager cluster.UsingthisdeÞnition,anycomponentorindividualsoftwarefailurewillforceafailovertoanalternatepathorCiscoCallManagerdevice.WithonelevelofredundancyandfourhourMTTR(meantimetorepair),theIPtelephonycorefromtheoverallavailabilitydeÞnitionandcalculation.OnewouldsupposethereasonisthatthelegacyPBXvendorcannotachieveÞveninesofreliabilitybyincludingtheaccesslayer.LikelegacyPBXvendors,CiscocannotachieveÞvenineswithoutredundancy.Accessswitchestypicallyhave60,000to150,000hoursMTBF(meantimebetweenfailure)ascomputedbytheTelcordia(formerlyBellcore)ÒPartsCountMethod.ÓUsingthismethodologyweÞndthatatypicalaccessswitchwitha100,000-hourMTBFanda4-hourMTTRhastheoreticalavailabilityof99.996%or21minutesofdowntimeperyear.Clearly,novendorcaneasilysupplyÞveninesofavailabilityend-to-endandinfactcurrentlegacyvoiceserviceproviderstargetavailabilitysigniÞcantlylower,around99.95%.TheCiscoIPphonehas less complicated hardware and does maintain MTBF over 400,000 hours with Þve nines plus reliability. 1.Cisco uses a 2X multiplier from the Telcordia computed value to provide theoretical results in supports full line with histor ÒCommon InfrastructureÓ or AVVIDThediagramaboveshowsacampusnetworkcoredesignoftenreferredtoasÒCommonInfrastructureÓorAVVIDusing the Telcordia parts count method. More information on the Cisco Þve nines solution is available in the nextThe IP Telephony Five Nines SolutionAvailability of the IP telephony solution can be theoretically determined by closely examining availabilitycharacteristicsinseveralareasandapplyingknownavailabilityanalysistechniques.Byunderstandingandapplyingthismethodology,organizationswillbetterunderstandtheavailabilitycharacteristicsofapotentialsolution.Thisis 2.ThebookÒHighAvailabilityNetworkFundamentalsÓbyChrisOggerinoprovidesdetailedmathematicaltechniquesforpredictingavailabilityof PSTNBoundaryIP WANAccess SwitchesAccessSwitchesDistributionSwitchesDistributionSwitchesDistributionSwitchesCoreSwitchesCiscoCallManagerClusterPSTN Gateway A meaningful analysis of availability includes the following areas of network availability.¥Hardware reliability¥Software reliability¥Link/Carrier availability¥Power/Environment availability¥Network Design reliability¥User-error & network support processesroot-causesasanyenterpriseenvironment.Networkdesignincludestopologyandfailovercapabilitiesofthesystem.times or spanning tree convergence times. User-error & network support processes includes network managementpractices that promote network consistency, supportability and rapid problem resolution times.redundancyisimplementedforthecore,distributionandCiscoCallManagerserverdevices.AccessdevicesarenotLANmodel.Figure3belowshowstheloadsharingredundanttopologyoftheÞveninessolutionwithdevicesoutside BoundaryAccess SwitchesDistributionSwitchesDistributionSwitchesDistributionSwitchesCoreSwitchesCiscoCallManagerClusterPSTN Gateways Hardware ReliabilityHardwarereliabilityisestimatedusingtheTelcordiaÒPartsCountMethodÓandapplyingtheCisco2Xfactorbasedonhistoricalfailuredata.ThisisaccomplishedbyunderstandingMTBFvaluesfordevices,devicecomponentsandpathsthroughthenetwork.AtheoreticalMTTRoffourhoursisusedforthecalculationandisconsideredtobethenorm for theoretical vendor-comparison availability prediction analysis.Usingthismethodology,hardwarereliabilitywascalculatedtobe99.99993%fortheIPtelephonysolutionusingthereliability analysis. Also keep in mind that many vendors based availability solely on hardware reliability.telephonyenvironments,twopathsarerequiredforoperation.OnefromphonetoCiscoCallManagerandonefromphone to phone or phone to PSTN gateway. To allow for the two paths, analysis of two paths will be doneindividuallyandthentogethertodeterminehardwarereliabilityinatypicalCIrecommendedinfrastructuremodel.Reliability Blocks for Path AnalysisThereliabilitycalculationbeginswiththecalculationforoneswitch.ThefollowingtablesshowMTBFinformationavailability calculations identiÞed in the book ÒHigh Availability Networking FundamentalsÓ. The followingfrom1ton(thenumberofcomponents),multiplybyavailabilityi.Inaddition,thesecondtableprovidesfailurerate SwitchesCore SwitchesDistribution SwitchesPSTNGatewaysDistribution SwitchesAccessSwitchesCisco CallManagerCluster ¹ = availability(i) Thenextcalculationcanbeappliedtoshowavailabilityofaredundantparallelsystem.Thismethodonlyaccountstheavailabilitylooksveryhigh.Inreality,thisalmostneverhappens.TherealproblemsareÞrst,thatevenwhenthereis a successful protocol switchover, all existing calls will be dropped and no new calls can be processed during theswitchover. The availability due to this problem is addressed in the design section of this paper. Other problemsincludetheinabilitytofailoverduetoanunavailableorbrokenstandbypathorthattheswitchoverdoesnotoccurswitches.ForthisequationNequalsthenumberofcomponentsinparallel(inthiscase2)andIequalsthecomponentnumber. In our case, this is equivalent to 1 Ð [ (1 Ð .99985988) * (1 Ð .99985988) ] or 99.99999802%. Table 1Distribution Switch Module Reliability Part # Required AvailabilityWS-650911369,8974hr99.99892%WS-CAC-1300W21316,4564hr99.99999%WS-X6K-SUP1A-MSFC21146,2354hr99.99135%WS-X6408A-BGIC1193,4574hr99.99572% Table 2Distribution Switch Reliability (Non-Redundant)Availability73.7 minutesAnnual Failure Rate Table 3Distribution Switch Reliability (Redundant)Availability101,880,673Annual Failure Rate.01% [ ¹ (1 Ð component availability(i))] redundant blocks for two distribution layers, one core layer, Cisco CallManager layer and gateway layer. Overallavailabilityisthendeterminedbyfactoringeachlayer.ButÞrstwewillneedtodeterminemoduleavailability,chassisavailability,andredundantchassisavailabilityforeachnetworklayer.Thefollowingtablesprovidethisinformation Table 4Core Switch Module Reliability Part # Required AvailabilityWS-650911369,8974hr99.99892%WS-CAC-1300W21316,4564hr99.99999%WS-X6K-SUP1A-MSFC21146,2354hr99.99135%WS-X6408A-BGIC2293,4574hr99.99572% Table 5Core Switch Reliability (Non-redundant)Availability99.981709%96.2 minutesAnnual Failure Rate Table 6Core Switch Reliability (Redundant)Availability59,787,111Annual Failure Rate.01% Table 7Access Switch Module Reliability Part # Required AvailabilityWS-650911369,8974hr99.99892%WS-CAC-1300W21316,4564hr99.99999%WS-X6K-SUP1A-2GE2182,6544hr99.99999%WS-G548421432,4704hr99.99999%WS-X6248A (10/100 card)1194,6844hr99.99577% Cisco Systems, Inc.All contents are Copyright © 1992Ð2002 Cisco Systems, Inc. All rights reserved. Important Notices and Privacy Statement.Page 10 of 17 Table 8Access Switch Reliability (Non-redundant)Availability27.9 minutesAnnual Failure Rate10.98% Table 9Access Switch Reliability (Redundant)Availability710,343,430Annual Failure Rate.01% Table 10Gateway/Access Switch Module Reliability Part # Required AvailabilityWS-650911369,8974hr99.99892%WS-CAC-1300W21316,4564hr99.99999%WS-X6K-SUP1A-2GE2182,6544hr99.99999%WS-G548421432,4704hr99.99999%WS-X6608-T11140,5124hr99.99577% Table 11Gateway/Access Switch Reliability (Non-redundant)Availability57.6 minutes36,511Annual Failure Rate21.34% Table 12Gateway/Access Switch Reliability (Redundant)AvailabilityAnnual Failure Rate.01% TheonlyremainingtheoreticalavailabilitynumberspertaintotheCiscoCallManagerservers.Hardwareiscurrentlymanufactured by outside vendors and the Telcordia MTBF information is not available. Cisco can only basehoursofcombinedoperationwithaMTBFofover1,000,000hrsbasedonthenumberofreturnsduringa3-monthperiod. Given the excellent current reliability of the MCS-7835 chassis and modules, the Cisco High AvailabilityServicesgroupoffersthefollowingestimatesofavailabilityforprimaryandredundantCiscoCallManagersystems.ThesecondtablebelowprovidesreliabilityforredundantCiscoCallManagerdevices,whichisacapabilitywiththeanalysiscalculation,whichisobtainedbymultiplicationofavailabilitynumbersforeachredundantmodule.GivenhighavailabilityateachlayeroverallhardwarereliabilityoftheIPtelephonysolutioninacampusenvironmentwithanMTTRoffourhoursis.Again,thisisdeterminedsimplybyusingthebasicserialcalculationmethod Table 13Cisco CallManager Chassis Reliability (Non-redundant)Availability57.6 minutes40,000Annual Failure Rate19.68% Table 14Cisco CallManager Chassis Reliability (Redundant)Availability200,040,000Annual Failure Rate.01% Table 15Redundant Layer Availability Availability Path Analysis Based on Redundant (N+1) ModulesCoreDistribution2ÑPSTNPSTN accessgateways99.9999999.9999999.9999999.9999999.9999999.9999999.99999 Software ReliabilityTo accurately predict the availability of a network or device, the software contribution to downtime must beconsidered. The problem however is that no industry standard exists for software reliability. Several methods areavailableforcalculatingsoftwarereliabilitybutthesemethodsareknowntobeinaccurateanddifÞculttoperform.ThebestmethodknownistoanalyzeÞeldmeasurementsforsoftwarereliability.ThissectionexaminesafewCiscoAvailability Networking FundamentalsÓ by Chris Oggerino.mediumsizedenterpriseenvironment.Thesemeasurementsdidnotaccountforsoftwaredefectsthatdidnotrestartthedevice,astheseweredifÞculttomeasure.Usingtheirmethodology,OggerinoandCherfconcludethatCiscoIOSGeneralDeployment(GD)softwareisÞveninesreliable.Cherf,forinstance,concludesthatCiscoIOSSoftwareGD99.9996%reliablewithanMTBFof25,484hours.Bothindividualsalsoshowedthatsoftwarematuritywasamajorfactor in software reliability.EveryCiscocustomershouldalreadyknowthatsoftwarereliabilityisprimarilyafactorofsoftwarematurity.Astheexpected. As IP telephony software increases in maturity, Cisco fully expects general deployment highly reliablesoftware. Unfortunately, some problems surface with customer trafÞc patterns in real world environments. Ciscothereforerecommendsthatallearlydeploymentsoftwarebetestedand/orpilotedinthereal-worldenvironmenttopapers at http://www.cisco.com/warp/public/126/index.shtml.BecausetheCiscoIPtelephonysolutionsutilizeCiscoIOSsoftwareclassiÞedasÒearlydeployment,Óaconservativeestimateisneededthatreßectsthisstatus.BasedonÞeldmeasurementsfornewsoftware,aconservativeestimateisÒextremely conservative number.Ó SwitchesCore SwitchesDistribution SwitchesPSTNGatewaysDistribution SwitchesAccessSwitchesCiscoCallManagerCluster 30,000 hours. These expected MTBF values will actually impact availability more due to failover. In the networkdesign section, a 10,000 hour MTBF is used for newer and more complicated layers including gateway, CiscoCallManageranddistributionlayer.A30,000-hourMTBFisusedformorereliablesoftwareinthecoreandaccesslayer. This analysis is provided in the design section of the paper later on.We can use the software reliability number in the hardware calculation by factoring in software reliability as aindependentactivitiesisolatedtoonehardwareplatform.FieldmeasurementsalsoconÞrmthatsoftwarefailuresdonotoccurtobothredundantdevicesinaparticularnetworklayeratthesametime.Thisisnottosaythatsoftwaredefects cannot affect both devices in the same network layer. Routing protocols, HSRP, spanning tree and otherprotocolscanaffectavailabilityofanetworklayerandshouldbedesignedcarefullyandtestedpriortodeployment.Fortunately, these protocols have been around for years and are not expected to cause non-availability underUsingthepreviousdeÞnitionofavailabilityandfullyredundantsystem,softwaredoesnotimpactavailabilityoftheswitchto99.98498%.ReliabilityofaredundantlayerdoesnotchangesigniÞcantlybutdropslessthanasecondperthis does not appear to impact availability. In the design section, we will use this MTBF however to look atconvergence times of devices and the number of expected failovers to understand how this impacts availability.Power/Environmental Availabilitytheoretical availability depending on the power protection strategy used for IP telephony.To understand theoretical availability for power and the environment, Cisco has turned to APC Corporation fortheoretical availability. APC can be contacted, http://www.apcc.com, for more precise availability information inhowever,thefollowingcharacteristicsapplytoNorthAmericanpower.Thedataisvariablefromsitetositeandsomeareas like Florida are expected to experience an order of magnitude higher of non-availability. The data is alsoexpected to be representative of Japan and Western Europe.¥The average number of outages sufÞcient to cause IT system malfunction per year at a typical site is¥90% of the outages are less than Þve minutes in duration¥99% of the outages are less than one hour in duration¥Total cumulative outage duration is approximately 100 minutes per year 3.Page 67 of ÒHigh Availability Networking FundamentalsÓ by Chris Oggerino4.Based on technical note #24 from APC Corporation located at http://sturgeon.apcc.com/kbasewb2.nsf/Tnotes+External/ levelsofÞveninesorhigherrequireaUPSsystemwithaminimumofonehourbatterylifeorageneratorwithaonsiteservicecontractorfourhourresponseforUPSsystemfailuresorproblems.TheCiscorecommendedhighavailabilityresponse to support the UPS/generator. Given the following recommended core infrastructure, APC corporationestimates that non-availability will be two minutes per year.¥UPS & generator backup¥UPS systems have auto-restart capability¥UPS system monitoring¥Four hour service response contract for UPS system problems¥Maintain recommended equipment operating temperatures 24X7Overallpoweravailabilityusingtheabovesolutionisestimatedtobe99.99962%.Thisimpactsoverallavailability,Network Design AvailabilityNetwork design availability includes network topology and network protocol design used to provide scalability,performance and convergence following a device or link failure. This is critical to availability. Without a properlydesignednetworkwithrecommendedprotocolsandconÞguration,convergenceorfailovertimescouldeasilyexceedfailures. To achieve the highest availability in network design, protocols need to determine when a failure hasoccurredandimmediatelyswitchtoaredundantcomponentordevice.Thesystemmustalsorecognizeandmonitorbackupcomponentstoensurethatthefailoverwillbesuccessful.Fortunately,CiscoistherecognizedleaderinthesenetworkingcapabilitiesandcanprovidehighavailabilityaslongasthedesignandimplementationmeetIPtelephonydesignrules.ThissectionhelpstoidentifythedesignrulesforthedeÞnedIPtelephonyavailabilitysolutionandalso¥Standard Common Infrastructure design with core/distribution/access layers¥EIGRP or OSPF routing in the campus with default timers¥HSRP for all IP access subnets including phones and Cisco CallManager servers¥Spanning tree environments eliminated or fast-converging spanning tree features (Use spanning tree only when¥Dual home access switches to two distribution switches¥Dual home distribution switches to two core switches¥Ingeneraldonottrunkdevicesatthesamelayer(exceptwheremultipleaccessswitchesareneededon1subnet/¥Quality of Service conÞgured across the LAN (or plenty of bandwidth) can investigate the failure rate at each network layer. This is accomplished by Þrst understanding the time requiredfor failover and convergence at each network layer. Fortunately, this has been previously tested within Cisco. Thefailure rate can then be multiplied by the expected failover time to understand the overall impact to availability. Amajorassumptioninthisanalysisisthatthenetworkhasadequateresources(bandwidth,CPU&processormemory)where failover will occur.IP Telephony PathOfthesevenlayersabove,fourseparatefailovermechanismswillbediscussedthatleadtoslightlydifferentfailoveroccur. Table 16Failover Analysis Failover Type AnalysisA failed distribution device or links may include three different failover mechanisms. Theseinclude HSRP, IP routing and possibly spanning tree. Thom Bryant documents testing results forfailover in the ÒRedundant Network Design Failure AnalysisÓ paper available internally at Cisco.Thom has applied tuning to improve network convergence at the distribution layer. For thepurposeofthisdocument,wearesimplytakingaconservativenumberbasedonhistestresults,Core layerDistribution devices should have equal cost routes through the core, making IP convergenceunnecessary to route trafÞc. Thom Bryant documents testing results for failover in theÒRedundantNetworkDesignFailureAnalysisÓwhitepaper.Thomhasappliedtuningtoimprovenetwork convergence at the distribution layer. For the purpose of this document, we are simplytaking a conservative number based on his test results, 10 seconds. The document is currentlyPSTN gatewayThe Cisco CallManager will attempt to initiate a PSTN call through the designated route group,device is selected. This takes approximately 10 seconds. SwitchesCore SwitchesDistribution SwitchesPSTNGatewaysDistribution SwitchesAccessSwitchesCisco CallManagerCluster Availabilityhasnowbeencreatedforhardware,software,power/environment&networkdesign.Keepinmindthattheoretical availability. This is (.9999962) * (.9999947) * (.9999993) which equate to 99.999% or approximately5.3 minutes of downtime per year.switchesRedundant Cisco CallManagers are not currently stateful. This capability is expected in futurereleases. Currently, a TCP keepalive is sent every 30 seconds. Three simultaneous negativeresponses cause failover to the standby, which can initiate new calls. Current calls are notdropped during failover however DTMF signaling info is lost and call may be reset. A failedaccessdevicewillcauseafailovertotheredundantCiscoCallManager.Becausethreekeepalivescan occur in 61 seconds, the mean time before the 1st keepalive is used bringing the averagedowntime to 75 seconds. Shortly following this, the redundant Cisco CallManager will have the Table 17Expected Failure Rate Per Year Access Gateway Software MTBF10,000 hrs30,0000 hrs30,000 hrs.10,000 hrs 10,000 hrsHardware MTBF (fromTable 2)28,545 hrs21,866 hrs75,380 hrs36,511 hrs40,000 hrsSoftware failure rate per.876 (365*24/10,000 = .876).292.292.876.876Hardware failure rate per.306.401.116.240.219Total failure rate1.182.693.4081.1161.095Failure impact30 seconds10 seconds75 seconds10 seconds75 seconds35 seconds7 seconds31 seconds11 seconds82 secondsTotal downtime166 seconds2.77 minutes 1.It is generally not a good idea to combine failure rates since software and hardware failure rates have different MTTR values Table 16Failover Analysis Failover Type Analysis 170 West Tasman Drivewww.cisco.comTel:408 526-4000Fax:408 526-4100Tel:33 1 58 04 60 00Fax:33 1 58 04 61 00170 West Tasman Drivewww.cisco.comTel:408 526-7660Fax:408 527-0883Capital Towerwww.cisco.comTel:65 317 7777Fax:65 317 7799CiscoWebsiteatwww.cisco.com/go/officesArgentina¥Australia¥Austria¥Belgium¥Brazil¥Bulgaria¥Canada¥Chile¥ChinaPRC¥Colombia¥CostaRica¥CroatiaCzechRepublic¥Denmark¥Dubai,UAE¥Finland¥France¥Germany¥Greece¥HongKongSAR¥Hungary¥India¥Indonesia¥IrelandIsrael¥Italy¥Japan¥Korea¥Luxembourg¥Malaysia¥Mexico¥TheNetherlands¥NewZealand¥Norway¥Peru¥Philippines¥PolandPortugal¥PuertoRico¥Romania¥Russia¥SaudiArabia¥Scotland¥Singapore¥Slovakia¥Slovenia¥SouthAfrica¥Spain¥SwedenSwitzerland¥Taiwan¥Thailand¥Turkey¥Ukraine¥UnitedKingdom¥UnitedStates¥Venezuela¥Vietnam¥ZimbabweAllcontentsareCopyright©1992Ð2002CiscoSystems,Inc.Allrightsreserved.CCIP,theCiscoNetworkmark,theCiscoSystemsVerifiedlogo,CiscoUnity,FastStep,FollowMeBrowsing,FormShare,InternetQuotient,iQBreakthrough,iQExpertise,iQFastTrack,theiQlogo,iQNetReadinessScorecard,NetworkingAcademy,ScriptShare,SMARTnet,TransPath,andVoiceLANaretrademarksofCiscoSystems,Inc.;ChangingtheWayWeWork,Live,Play,andLearn,DiscoverAllThatÕsPossible,TheFastestWaytoIncreaseYourInternetQuotient,andiQuickStudyareservicemarksofCiscoSystems,Inc.;andAironet,ASIST,BPX,Catalyst,CCDA,CCDP,CCIE,CCNA,CCNP,Cisco,theCiscoCertifiedInternetworkExpertlogo,CiscoIOS,theCiscoIOSlogo,CiscoPress,CiscoSystems,CiscoSystemsCapital,theCiscoSystemslogo,EmpoweringtheInternetGeneration,Enterprise/Solver,EtherChannel,EtherSwitch,GigaStack,IOS,IP/TV,LightStream,MGX,MICA,theNetworkerslogo,NetworkRegistrar,,PIX,Post-Routing,Pre-Routing,RateMUX,Registrar, SlideCast, StrataView Plus, Stratm, SwitchProbe, TeleRouter, and VCO are registered trademarks of Cisco Systems, IncAllothertrademarksmentionedinthisdocumentorWebsitearethepropertyoftheirrespectiveowners.TheuseofthewordpartnerdoesnotimplyapartnershiprelationshipbetweenCiscoandanyothercompany.201640/ETMG02/02 User Error and Network Support ProcessesUsererrorandnetworksupportprocesscanaccountforsigniÞcant However, the amount ofgapsthatmayimpactavailability.Unfortunately,thisiswellbeyondestimating availability, user-error and process have not beensimilar user-error and process issues and therefore should beRatherthanfocusontheimpacttoavailabilityinthisarea,itmakessense to at least identify some of the user-error and process issuesthatmayimpactavailabilityinanIPtelephonyenvironment.Thesesuccessfully, quickly identify and resolve faults, keep the networkproblems proactively. More information is available on thesewww.cisco.com/warp/customer/788/solution_guide/Operationalsupportplan(Servicerepairplans&problemTelephone inventory management 5.AGartnerGroupreportstatesthat40%ofnon-availabilityisduetouser-errorand