Kirk Kirkpatrick Managing Partner Gartner

slide1. Kirk Kirkpatrick
Managing PartnerGartner Consulting
Kirk.Kirkpatrick@gartner.com

Mehrdad Foroozesh
Senior Director Gartner Consulting
mehrdad,foroozesh@gartner.com Data Center Migration Project IV&V Report
November 25, 2015 Engagement: 330024312<br>
slide2. Executive Summary<br>
slide3. Gartner has conducted a review of the ADES DC Migration program. Areas Reviewed to Date Key Interviews to Date Bold names indicate those that were recently interviewed prior to this report revision. Work streams reviewed Risk rating for the 4 most recent reports 10/26 08/10 09/18 11/25<br>
slide4. 10/26 Key Activities in This Period Key activities in this period 11/19/2015 and 11/20/2015 Site Visit
Conducted the following on-site interview sessions with ADES:
Stephen Welsh and Allen Platt (Executive Sponsors and Steering Committee Members)
Linda Jewell ( DCS Perspective)
Michael Mills (DCS PMO Administrator)
Clay Sikes, Paul Wills (Systems and Network Ops)
Robert Mayer (Project Management)
Tom Fruciano (Assistant PM)
David Mathis (Project Governance)
Todd Templeton and Al Barbieri (Architecture Team)
Richard Glashan and Darryl Duke (Mainframe)
Wesley Fletcher (Finance)
Jeff Raynor (Security)
John Boland (Mainframe App Dev Governance)
Ryan Riach (Web Services App Dev Governance)
Chad Rohner (CenturyLink AE)
John Green (CenturyLink PM)
John Sadnick (CenturyLink SME)
Ghada Giliana, Daniel Emburey, and Bryan Hanes (Change and Release Management)
Patrick Cravens (ADOA Mainframe Ops)
Suzan Tasvibi-Tanha, Aaron Freyermouth (AZNET) DES Internal Calls:
Listened in DES status calls
Weekly Status Calls with:
Allen Platt
Robert Mayer
David Mathis
Todd Templeton and Al Barbieri
Weekly Executive Calls with:
Allen Platt
Stephen Welsh 08/10 Risk rating for the 4 most recent reports 09/18 11/25<br>
slide5. Major Documents Reviewed Major Documents Reviewed 7/10//2015 Report
CenturyLink Network Documents
Weekly status reports
Final network architecture diagrams
Final rack elevation
Final Rack Power Configuration
DES DCR weekly communications
Weekly Steering Committee Meeting Minutes and Decisions
Updated RACI
Inventory of non-virtualized physical servers and their disposition
Misc. Updated Project Artifacts 8/10//2015 Report
DES DCR weekly communications
Weekly Steering Committee Meeting Minutes and Decisions
Application MAD spreadsheet
DCR Roadshow
DCR Project Schedule
DCR PM Finance Workbook
Mainframe-move
Physical Server Migration Plan
CenturyLink proposed scope for application migration. 9/18//2015 Report
DES DCR weekly communications
Weekly Steering Committee Meeting Minutes and Decisions
Application MAD spreadsheet
Sample DCR Roadshow Call with PPMs
DCR PM Finance Workbook
CenturyLink Scope of Work
Physical Server Migration Plan Risk rating for the 4 most recent reports 10/26//2015 Report
DES DCR weekly communications
Weekly Steering Committee Meeting Minutes and Decisions 10/26 08/10 09/18 11/25 11/25//2015 Report
DES DCR weekly communications
Weekly Steering Committee Meeting Minutes and Decisions<br>
slide6. Key Findings: General Observations Finding: Project is Showing Positive Results and Incremental Wins
DCR project continues to create incremental wins. The Exchange migration has been completed successfully and production servers now run in the IO data center. No major issues have been reported that can be attributed to the transition.
On 11/18 upgrade of the 1720 vSphere environment to version 6.0 was completed successfully. This was a required upgrade to facilitate migration of the virtual machines to the IO data center.
Other progress for this period Includes:
DCS was invited to attend the DCR Steering Committee meetings to ensure alignment and coordination of major activities. DCS has also assigned Michael Mills as its new PMO Administrator to oversee its application portfolio. These positive changes can contribute to the success of the DCR project.
DR testing for mainframe VTL is schedule for 11/21.
C-Store has completed their scope of work for configuration of the Cisco UCS environment at the IO Data Center
VMWare team is now integrated into the project team. Configuration of the IO data center vSphere environment is scheduled for 12/11.
Project budget has been stable in this period. Project’s overall spending is expected to remain at $12.1M. Review with ITAC will now take place on December 16th for final approval.
PM team in collaboration with CenturyLink is addressing any additional staffing support that may be required due to recent DES staff attrition. CenturyLink is prepared to address additional staff augmentation requests. Change order may be required.
Project team continues to reach out to ITAB, Portfolio Managers, and Functional Managers with great success.
Change Management and CAB process is working well. No issues or concerns have been reported.
All dependencies to the mainframe tape library have been removed in preparation for decommissioning of the tape system.
All production dependencies to the old Cisco OSR core at 1720 have been removed, including mainframe and Active Directory dependencies (It is possible that some older systems that are not in the scope of DCR project may still be connected).<br>
slide7. Key Findings: General Observations Finding: Areas that need to be watched carefully.
Project Timeline – It is becoming more probable that the project timeline may need to be pushed out to Q3, 2016.
Application migration plan which was due for completion by November 9th has now been pushed to December 2nd. CenturyLink has missed two target completion dates prior to the new December 2nd proposed date. These delays have been escalated to the CenturyLink account team by the DES PM. Should the December 2nd target date be missed, further escalation will be required.
The swing of the AZNET network (inside the firewalls as well as outside of the firewalls) has been postponed to January 13th at the earliest from the prior early to mid December incremental dates. This may not directly impact the final project completion date, but could impact scheduling of application and server migration by limiting the number of concurrent changes that can take place in the environment at any given time.
Gartner is concerned that if the scheduled end date for DCR project continues to slip, at some point, it may conflict with other large system DES development projects or with DCS separation activities. An updated project schedule/work breakdown structure which is more detailed and which highlights key dependencies and the critical path to success is needed. Gartner recommends that a project schedule baseline be established as soon as the application migration plan is completed by CenturyLink to ensure that all major project milestones and completion dates are well communicated to the organization. Producing a draft detailed schedule should be a priority for the DCR project manager and steering committee.
Important Note: Gartner continues to believes that risks to project timeline are “significant” until the application migration plan is completed by CenturyLink. Also, application migration activities cannot begin until the scope of work assigned to VMWare such as configuration of the vSphere at the IO data center is completed as scheduled (Dec. 11).
Staff Attrition – While CenturyLink is being engaged to provide staff augmentation to minimize the impact, any future staff attrition could place the project at risk. Retirement of key personnel may also impact the order of application migration so that the institutional and technical knowledge of those team members could be leveraged prior to their departure (Scheduling of DDD migration will be expedited for this reason if possible)<br>
slide8. Key Findings: General Observations Finding: Areas that need to be watched carefully (continued)
Mainframe Migration – The Steering Committee has made a decision to transition the mainframe to the IO data center in April or May of 2016. DES may continue to explore longer term strategies for the mainframe in the future.
Next mainframe DR test is scheduled for November 21. This test will primarily focus on the failover capabilities of the new EMC VTL libraries, although some additional tests such as printing may also be performed. Our conversation with the Change Management team on November 25 indicates that this test may not have been completely successful (Gartner will review with the mainframe team during our next visit scheduled for December 10th and 11th).
DES will be scheduling at least one additional mainframe DR test prior to the mainframe migration. This test will be broader in scope to ensure complete failover capability should the physical move of the mainframe encounter major issues. The scope of this broader DR test is yet to be documented. For example it is not clear if this test will include end user connectivity and running applications in production mode at the ADOA data center. Gartner recommends that the detailed test plans and scope be developed as soon as possible.
Currently there are 10 mainframe related cabinets at 1720, four of which are the DES mainframe and storage, approximately 4 or 5 cabinets belong to ADOA (storage devices), and the remaining 1 or 2 storage cabinets belong to DPS. DES needs to reach an agreement with ADOA and DPS regarding responsibility for relocation of non DES equipment (this was also noted in the prior reports).
Most recent conversation between Gartner and ADOA mainframe administrator indicates that ADOA has not formally accepted the fact that by not relocating their mainframe storage backups at 1720 to the IO data center concurrently with the DES mainframe their DR capabilities will be lost. This is a significant decision that warrants additional confirmation to ensure that ADOA’s DR concerns do not interfere or postpose the DES mainframe migration.
Target Location for the Mainframe - CenturyLink recently provided a proposed space configuration and cost for the mainframe space at the IO data center. This solution is pending approval by the Steering Committee. This is not in the project critical path at this time but may impact long term operating costs. DES should pay close attention to shortening the “Term” of the contract for this space should other long term mainframe strategies be in play.<br>
slide9. Key Findings: General Observations Finding: Areas that need to be watched carefully (continued)
Participation of the DES Architecture Team – The architecture team continues to be actively engaged with the PM team and continues its participation in the CAB process. Communication between Todd Templeton and Robert Mayer (PM) is strong. However, over the past two months Gartner has made the observation that the level of participation from the Architecture Team in leading, reviewing, and signing off on key technical decisions and vendor deliverables has declined. While there is no evidence that this may have contributed to CenturyLink missing multiple deadlines, there is no doubt that CenturyLink can benefit from the institutional and technical contributions and the oversight that the Architecture Team can provide. We strongly recommend that the Architecture Team re-engage with CenturyLink, VMWare, C-Store and other participating vendors to provide guidance and review and sign-off on the deliverables.
Participation of the DES Application Governance Team – As the DCR project transitions from infrastructure preparation to application migration, the role that the Application Governance Team (John Boland and Ryan Riach) can play in bringing application environment knowhow and institutional knowledge to the project will increase. Up to this point participation of this team has been minimal at best. However, Gartner believes that this team can provide knowledge and guidance to address many of the application migration planning issues, such as application testing, and possibly other issues that may have proven to be a challenge as CenturyLink works to develop the application migration plan.
User Acceptance Testing – As the DCR application migration phase nears, attention should be paid to defining the scope of application acceptance testing. Most application acceptance tests at DES are manual and will require some level of coordination with the end user community. As it stands today, most application test windows are greater than the application maintenance windows. Using the application maintenance windows for migration may require some additional planning to either limit the testing activities or plan for a longer window. Furthermore, based on interviews with the Application Governance team, a minimum of one month notification may be required for most application teams to plan and coordinate the required test plans.<br>
slide10. Key Findings: General Observations Finding: Areas that need to be watched carefully (continued)
AZNET Swing to IO Data Center – After additional due diligence and planning, the project team has reached the conclusion that the level of documentation, team coordination, WBS, and planning for validation and testing of the AZNET swing from 1720 to the IO data center requires more work. As a result the planned AZNET swing for December has been postponed to January 13.
In our prior report Gartner recommended that at a minimum DES pay close attention to the following:
Clearly document all process steps and coordination requirements between CenturyLink teams, AZNET team, DES team, and other stakeholders. Clearly articulate what changes and in what order, including the redundant AZNET routes from 1510 and 206 ADOT data centers.
Conduct several table-top dry runs with stakeholder groups to identify potential concerns, what if scenarios, and ensure that the impact is fully understood by all stakeholders.
Clearly document all fall back or fall forward strategies and decision criteria for selecting one vs. another.
To the extent possible, test and validate all automated scripts and procedures.
Finalize and document testing requirements
CenturyLink/ADOA DC Hosting Contract – The amendment to the CenturyLink / ADOA contract need to be finalized and communicated to the project team. CenturyLink has confirmed that ADOA has not signed the CTL contract amendment. This amendment provides clarifications related to pricing for future engagements and does not directly impact the DCR project.<br>
slide11. Scorecard, Key Findings and Recommendations<br>
slide12. Project Governance, Organization, Scope, and Objectives Dashboard. Risk rating for the 4 most recent reports 10/26 08/10 09/18 11/25<br>
slide13. Project Management Summary Dashboard Risk rating for the 4 most recent reports 10/26 08/10 09/18 11/25<br>
slide14. Project Work Stream Summary Dashboard Risk rating for the 4 most recent reports 10/26 08/10 09/18 11/25<br>
slide15. Appendix: Historical Documents Reviewed<br>
slide16. Major Documents Reviewed Major Documents Reviewed (Historical) 11/17/2014 Report
Network Plan and BOM
Project Plan
Project Charter
AZNET Proposed DC Network Connectivity
B&D SOW and Scope
Project Communication Plan
Project Risk & Issues Registers 12/02/2014 Report
ENS Network Documents
Master Plan
Core Plan
BOM, Power and Space v2
DES Design and Timeline
DNS, DHCP, IPAM Plan
NetScalar Plan
Security Plan
Security Designs 2003 W NS High-Level
Security Designs 2003 W NS V2
ENS Project Statement of Work 9-16-14 FE
Aggregation Switch Plan 12/16/2014 Report
CenturyLink Network Documents
Design and Implementation Plan Executive Summary DES 12-15-2014 CMR
State of Arizona DES Core Diagram V2 12-16-2014
State of AZ DES Core Quote 12-15-14 1/22/2015 Report
CenturyLink Network Documents
State of Arizona Master Final v4-1-2015
DES DCR Recommendation
Executive Summary – DCR Recommendation
ADES DC BOM cSTOR 011615
DTS RAID_Data Center Relocation
DCR Project_Plan01092015 2/20/2015 Report
CenturyLink Network Documents
AZ DES Rack Elevation V1
State of Arizona DES Network Design and Configuration SOW (Clean)
DES DCR Recommended Approach Project Plan V2 4/10/2015 Report
Updated rack elevations
C-store ADES BOM option A 021815
C-store ADES DC 2015 Proposal V4
DCR Communication 3-27-15b
DES Data Center Relocation – Compute Implementation ITAC Presentation V5 rev A
DES F5 Quote
DES Raid Data Center Relocation 03_31_2015
Gartner Tracking Log
IBM VTL AZ DES PA – PROD & DR EMC
IBM Mainframe Move State of Arizona SOW
IO-Mainframe-layout-Caged
Revised Approach Project Plan V3
DR Slides for Gartner 04062015 5/22/2015 Report
CenturyLink Network Documents
Weekly status reports
Draft network architecture diagrams
Draft rack
DES DCR weekly communications Risk rating for the 4 most recent reports 10/26 08/10 09/18 11/25<br>
slide17. Kirk Kirkpatrick
Managing PartnerGartner Consulting
Telephone: +1 201 417 7824
Kirk.Kirkpatrick@gartner.com Contacts Mehrdad Foroozesh
Senior Director Gartner Consulting
Telephone: +1 504 782 5093
Mehrdad.Foroozesh@gartner.com<br>