1 Closeout Report on the - PowerPoint Presentation

Slide1

1

Closeout Report on the

DOE/SC CD-2 Review of the Dark Energy Spectroscopic Instrument (DESI) Project Lawrence Berkeley National LaboratoryJuly 28-30, 2015

Kurt Fisher

Committee Chair

Office of Science, U.S. Department of Energy

http://www.science.doe.gov/opa/

Slide2

Review Committee Participants

2

Kurt Fisher, DOE/SC, Chairperson

Slide3

3

SC Organization

Slide4

4

Charge Questions

Preliminary Design: Is the DESI preliminary design sound for establishing the Project Performance Baseline? Does the preliminary design support meeting the Key Performance Parameters (KPPs) to achieve DOE Dark Energy Stage IV Program objectives as described in the Mission Need Statement, and the technical flow-down requirements of the overall experimental program? 2. Project Scope: Are the Project’s scope and high-level specifications sufficiently defined to support establishing the cost and schedule baselines? Cost and Schedule: Are the cost and schedule estimates, and bases of estimate, credible to establish the Performance Baseline? Does the Project schedule and planned off-project operations funding support the overall experimental program, including access to the Mayall telescope?

Contingency

: Are there adequate scope, cost and schedule contingencies to address the remaining Project risks?

Management

: Is the Project being properly managed? Is there a capable team in place to effectively manage all the major interfaces and risks, and to achieve successful delivery of the Project?

ES&H

: Are ES&H systems and processes in place to mitigate of all the hazards identified and to ensure delivery of the Project in a safe and environmentally sound manner?

Has the Project responded appropriately to the CD-1 review, met all the CD-2 prerequisites, and is it ready for CD-2 approval?

Slide5

5

2.1 Science,

Computing, and Survey Planning I. Dell’Antonio, Brown / Subcommittee 1FindingsCommentsRecommendations

Preliminary Design

: Is the DESI preliminary design sound for establishing the Project Performance Baseline? Does the preliminary design support meeting the Key Performance Parameters (KPPs) to achieve DOE Dark Energy Stage IV Program objectives as described in the Mission Need Statement, and the technical flow-down requirements of the overall experimental program?

Project

Scope

: Are the Project’s scope and high-level specifications sufficiently defined to support establishing the cost and schedule baselines?

Contingency

: Are there adequate scope, cost and schedule contingencies to address the remaining Project risks

?

Has the Project responded appropriately to the CD-1 review, met all the CD-2 prerequisites, and is it ready for CD-2 approval

?

Slide6

6

2.1 Science, Computing, and Survey Planning

I. Dell’Antonio, Brown / Subcommittee 1Preliminary Design

: Is the DESI preliminary design sound for establishing the Project Performance Baseline? Does the preliminary design support meeting the Key Performance Parameters (KPPs) to achieve DOE Dark Energy Stage IV Program objectives as described in the Mission Need Statement, and the technical flow-down requirements of the overall experimental program?

Yes

, although the KPP(s) related to analysis software are very high-level.

Project Scope

: Are the Project’s scope and high-level specifications sufficiently defined to support establishing the cost and schedule baselines?

Yes

Contingency

: Are there adequate scope, cost and schedule contingencies to address the remaining Project risks

?

Yes

Has the Project responded appropriately to the CD-1 review, met all the CD-2 prerequisites, and is it ready for CD-2

approval?

Yes

Slide7

2.1.1: Findings

The DESI experiment is designed to significantly exceed the Stage IV dark energy standards from the BAO signal alone, and the Level 1 milestones are designed around the BAO experiment (although they do not impede cosmological measurements using redshift space distortions)

The main survey will involve measuring redshifts of three classes of objects that will most efficiently yield redshifts (luminous red galaxies, quasars, emission line galaxies), as well as Lyman-alpha absorption lines in high redshift quasars. A supplementary survey of nearby galaxies during bright time will provide a low-redshift BAO signature to higher accuracy than is currently been measured.The survey is planned to cover 14,000 square degrees (9000 is the threshold), essentially the entire low-extinction sky available to the Mayall telescope. Imaging deep enough for the source selection over the whole survey area does not exist at this time. The project has initiated two ground based surveys covering three telescopes to obtain the data. Two of the datasets are about 30% in-hand, the other observations are due to start in 2016. 7

Slide8

2.1.1: Findings

The projects expects to complete the ground-based surveys by late 2018.

Source selection also depends on WISE IR imaging. The WISE extended mission is expected to release its final data in 2017.The imaging surveys are not part of the project (although they are led by DESI scientists), but the data reduction and target selection are project responsibilities.DESI target selection tests are already underway with the first year of DECals data.The DESI collaboration has grown to 200 researchers at 45+ institutions in the US and multiple foreign partnersThe collaboration has set up 13 working groups to perform science analyses, survey planning and design, and oversee development of the data analysis algorithms. A preliminary version of the science readiness plan has been produced, listing tasks that need to be completed to be ready for cosmological analyses with DESI.8

Slide9

2.1.1: Findings

Survey

planning has begun, and trade studies of target prioritization are being done (with staged deliverables). Simulations of survey progress are being developed to inform survey strategy. At the moment these are integrated exposure times, but plans exist to create observation-scale simulations including the effects of weather. The WBS for Data Systems appears complete 10-12 FTEs are engaged in the subsystem, with 90% of the effort off-project. Effort levels have been estimated and staffing is at the appropriate level. Computing resources will reside at NERSC, with CPU cycles being free to the Project.Data Systems is $2.4M Federal funded (costed)55% in-kind off-project but DOE funded, 30% off-project MOU, 10% collaboration “tax”, 5% otherData

systems overall contingency is ~35%, software is 40% (but only on

costed

resources

)

9

Slide10

2.1.1: Findings

Have

risk register items if need to pay for in-kind additional resources, can buy out teaching or get more DOE time if need to exercise contingencyHave EVMS, but in-kind is managed only by schedule, not EVScope for Data Systems includes the target selection pipeline, survey planning and operations, spectroscopic pipeline, tracking the 3D survey efficiency for LSS catalogs, data transfer, archive and distribution, and collaboration tools.Continuous Integration with Travis + github, public git repository “desihub”, iterative development cycles, unit/functional/integration tests, git branches and pull requestsEnd to end pipeline running on simulated DESI and BOSS pre-cursor data by end of CY15, spectrograph comes in CY16, scaling tests in CY17.WBS 1.8 has some schedule variance on planning/documenting activities; core software development activities minimal varianceBOSS/SDSS pipelines were O(200k source lines of code),

DESI’s

is

comparable.

Pipeline

data model is files on disk, some data is also loaded into databases for processing (e.g. for Next Field Selection

)

10

Slide11

2.1.1: Findings

Project expects

to consume 200-800 TB of storage for data and 500-5000 TB for the collaboration, as well as 60-175M CPU-hrs.Data Systems is planning a 5-yr survey Data Challenge in 2017.CRD personnel are optimizing the Target Selection code, and with little effort, obtained a 3x improvement in speed.Simulations are used in many areas. PixSIm and QuickSim are the main tools. Both are written by DESI staff. Currently 20 people across 7 institutions involved (LBNL, NYU, U Utah, ETH, …), 10 – 12 FTE required in planManagement is challenging, must communicate often, use weekly telecons, mailing lists, github issue tracking, in-person code festsHave some pair effort planned with professional software engineers

Team

has many/most members from BOSS, assumed similar overall productivity when estimating software

effort

11

Slide12

2.1.1: Findings

Have

1 FTE on archiving, data transfer, distributionScience operations is close enough in time that the PM expects the core people to stay involved through commissioningComputing and storage was specified based on daily processing and 2 full reprocessings/yearCPUs are provided by DOE at NERSC, have to buy disk/tape spaceProcessing is like SDSS spectroscopy but fainterField Selector is only element that must execute every exposureData volume is 10 TB/year, much smaller than DESData transfer volume much less than DTS does now, can do in minutes12 hour data transfer requirement is not driven by need to keep up with observing, it is desire to keep up QA

checks

All

processing modules will be implemented in selected technologies, tested with BOSS and DESI

sim

data in end-to-end mode by July

2016

All

but one process are already in place now (not at scale

)

All

will be refined over several

years

12

Slide13

2.1.1: Findings

Processing

is mostly fiber-, spectrograph-parallel, decision to do smaller Focal Plate, fewer petals would have little effect on processing, mostly in final modulesThere is some state-dependent or historical information in spectroscopy pipeline, i.e. the Flat field model will be built up over timeThe Spectroscopy Pipeline confirms whether field was good enough or not back to the ICSScience Readiness Planning is well underway, most tasks identified, need to be scheduledFor target selection validationELGs: have DECam DEEP2 fields, have eBOSS pilot data, have MMT time to supplementQSOs: have BOSS/eBOSS, stripe82 and SpIES for variability, have MMT time to supplementLRGS: eBOSS and VIPERS, have AGES and GAMA for bright galaxies

13

Slide14

2.1.1: Findings

Individual cosmological analyses

are well-known, experienced. Now need to combine them, focus in SRP.Would like to more crisply define how to test Modified Gravity (everybody has this problem)Imaging survey ata reduction is led by Dustin Lang as DESI Imaging Survey Scientist, TRACTOR pipeline, computing resources provided by NERSC DESI Imaging Validation Team does data quality assessment on each releaseRisk register being maintained, all impacts are hits on DESI targets, smaller footprint, targeting efficiency for ELGsImaging Survey Data Release milestones are not in PMCS as milestonesUse Tractor photometry to process imaging survey data, improved to embed in OpenMPI, OpenMP environment for parallelizationThere is one Tractor codebase which is on github, two products now, core and

wrapper

Now

3 - 4 people are familiar with codebase enough to

modify

Model

-fitting and chi-square iterations, multi-instrument, multi-model, avoids

deblending

14

Slide15

2.1.1: Findings

Rewriting image processing code in C++ to optimize, goal is to be able reprocess everything in 2 weeks

At NERSC Cray O(130k cores, 2.57 pFLOPS, 357 TB RAM, 7.5 PB Disk) , Global Filesystem currently GPFS (may move to Luster), 240 PB tape archiveSoon Cori intel KNL systems will increase compute by 7x i/o by 4xHEP has 17% allocation, DESI needs only small fractionInput imaging data is 1 TB DECam, 0.5TB other, 60 TB WISECurrently have iPTF database 3.0B object catalog in Postgres database and 3M imagesDESI is much smaller, currently 10 sq deg, 138M objects, target queries currently take O(10 min)

q3c

plug-in for spatial queries

Survey

SQA interface is using

LINeA

DES Brazil quick reduce,

LINeA

is doing this for

DESI

15

Slide16

2.1.1: Findings

Can

already run early version of NFS with ICS, full version on sim data in CY17Spectrograph test stand data will validate many things by end of 2016Most milestones after this point are both functional and performanceThere are letters covering DOE funded people and those on MOUs that specify deliverables (did not discuss level of detail)Each 6 months there is a detailed planning session that generates a list of tasks, results in tickets, (but not in P6)Evaluated JIRA Agile, not planning to use, developers are mostly very part-time, not co-located3 risks associated with insufficient off-project labor (buying out teaching time is main approach)FTEs required varies only by small % year to year until FY19, have BCR in to further smooth curveSoftware estimates are based heavily on BOSS, eBOSS

experience (which is very relevant)

16

Slide17

2.1.2: Comments

General Comments:

Great thought and care went into reviewProject has extensive background and the experience to execute the DESI project successfullyScience case is mature and flows to survey requirementsTeam has answered all CD-1 recommendations and reviewer questions17

Slide18

2.1.2: Comments

Imaging Surveys

The northern imaging survey has some risk that hasn’t been retired, because Mosaic2 has not been commissioned and no backup plan to acquire the z-band data if 2016-2017 are poor weather. Spectroscopic and deep imaging datasets to perform validation of the source selection are mostly in hand. A plan for assigning tasks to individual science team members is still needed. Priority should be on imaging the overlap regions to know if either half-survey has issues in time to correct the imaging strategy.

Imaging Survey milestones are not in P6,

though

other parts of the project depend on them. They are off-project activities, but

given

dependencies and potential impact

on

project schedule

if

they slip it

is

advisable to track them as external milestones in

P6 (probably owned by WBS 1.1)

18

Slide19

2.1.2: comments

Simulations

The science working groups are not formally obligated to return any information on the data challenges. A formal process of reports back to the DM team at each data challenge would ensure against surprises.DESI performance is predicted using yet-to-be-validated simulations tools, PixSIm and QuickSim. The intention is to validate them against test bench data in the coming year. This should be high priority. 19

Slide20

2.1.2: Comments

Analysis software

The spectroscopy pipeline should be tested on incomplete and deficient data (only partial fiber coverage, odd illumination pattern) to determine what the failure modes are and identify how much data degradation affects the redshift extradition.Early availability of EM spectrograph data is essential to test out the analysis pipeline algorithms, particularly the PSF stability and variations. These tests should include light reimaged onto the fibers at the proper f-ratio to simulate the Mayall.The goal is to be able to do complete reprocessings of the full dataset in 2 weeks. Scaling tests are currently planned for

CY2017.

Earlier scaling

tests are

desirable and seem feasible given the imaging survey data release schedule.

20

Slide21

2.1.2: Comments

Software Development and Computing Resources

There are now 3 or 4 people with sufficient familiarity of the Tractor codebase to modify it. Nevertheless, it seems desirable to ensure that the Tractor author, Dustin Lang, is available to support the project at least through the end of imaging data reduction.The software is generally run on two OSes: linux and mac, but the build system runs only on linux. It is good practice to test code on at least two OSes to flush out any bugs specific to one. The project should run their integration tests on the mac platform as well.21

Slide22

2.1.2: Comments

Software Development and Computing Resources

The software team is using good software practices for distributed code development. The team mentioned using a ticketing system for assigning development work as well. A ticketing system should be broadly used for all software development.The Project and Collaboration are depending on NERSC for computing cycles. DOE has agreed to the requested allocations, but the priority of DESI at NERSC seems very informal. An SLA might be in order.22

Slide23

2.1.2: Comments

Management and Systems Engineering

Data systems team has done a good job of capturing the scope of WBS1.8 including both costed and uncosted components into P6. It would be prudent for key players to at least be at a 75-80% commitment level, especially during “high-intensity” periods of software releases and system integration.90% of the manpower is off-project. This should be recognized in the risk registry. The loss of key personnel risk has been retired prematurely.

23

Slide24

2.1.2: Comments

Management and Systems Engineering

Some key software development activities in P6 are long-duration. This makes status less transparent. Tracking would be facilitated by having finer grained activities (either in P6 or in another tool)Data Systems has only one ICD, with ICS. Tests of interfaces with ICS are planned in CY2016 and CY2017. This should not slip.The Objective and Threshold KPPs for Data Systems are very high level; a more quantitative list of metrics would make it easier to judge compliance.

24

Slide25

Recommendations

25

Consider making integration builds with both operating systems integral to the nightly testing by the time of CD-3Reword (by CD-2) the analysis pipeline KPP to more directly reflect the requirement to meet science goals detailed in the mission statement Prepare (by the time of CD-3) a verification document detailing the criteria and methods to ensure the analysis pipeline meets the flowed down science and system requirements.

Slide26

26

2.2

Optics and Focal Plane Instrumentation H. Cease, ANL / P. Thompson, GSFC Subcommittee 2Preliminary Design: Is the DESI preliminary design sound for establishing the Project Performance Baseline? Yes Does the preliminary design support meeting the Key Performance Parameters (KPPs) to achieve DOE Dark Energy Stage IV Program objectives as described in the Mission Need Statement, and the technical flow-down requirements of the overall experimental program? Yes

Project Scope

: Are the Project’s scope and high-level specifications sufficiently defined to support establishing the cost and schedule baselines?

Yes

Contingency

: Are there adequate scope, cost and schedule contingencies to address the remaining Project risks

?

Yes

Has the Project responded appropriately to the CD-1 review, met all the CD-2 prerequisites, and is it ready for CD-2 approval

?

Yes

Slide27

27

2.2

Optics and Focal Plane Instrumentation H. Cease, ANL / P. Thompson, GSFC Subcommittee 2Findings WBS 1.2, 1.3 and 1.4 have an experienced team that leverages heritage from previous projects such as DECam.Corrector and ADC Assembly

,

WBS

1.2

WBS total cost is $6,421K, of which $436K is funded by the project.

Lenses

are procured and polishing contracts

will be in place soon

Contingency for project funded activity is 14%

STFC holds 15% contingency for WBS 1.02.04-.09

A recent optics coating RFP

is included in the baseline cost and schedule

No overage on glass blanks were procured.

Highest risks identified are schedule related to delivery of the ADC blank and corrector polishing taking longer than planned.

Recommendations from CD-1 review are addressed

Stray light lessons learned from DES have been documented.

A

stray light

analysis

is in the schedule, added in WBS 1.9

. and is on track.

Slide28

28

2.2

Optics and Focal Plane Instrumentation H. Cease, ANL / P. Thompson, GSFC Subcommittee 2FindingsCage & Ring Assembly with Barrel Components, WBS 1.3WBS total cost is $2,944K , of which $2,046K is funded by the project

Contingency is between 20-50% depending on activity

Includes overages

for

one Hexapod arm, a partial barrel segment that will be used for ADC bearing testing and the

fiducialized

barrel shroud

Barrel design is developed and has been externally reviewed

ADC Rotator bearing is at preliminary design.

Recommendations from CD-1 review

have been addressed

An appropriately sized CMM is now available at FNAL for measurements associated with Barrel alignment. The availability of the new CMM retires three risks associated with barrel alignment.

Slide29

29

2.2

Optics and Focal Plane Instrumentation H. Cease, ANL / P. Thompson, GSFC Subcommittee 2FindingsFocal Plane System (FPS), WBS 1.4

WBS total cost is

$8,679K, of which $5,336K is funded by the project. Large cost activities include:

Fiber positioners: $3,823K, of which $2,141K is funded by the project

GFA: $1,390K which is completely funded with non-project funds

Focal plate integration: $1,430K, of which $1,385K is funded by the project

Average

c

ontingency across WBS 1.4 is 27%

4% procurement overage planned for fiber positioner units

Fiber positioners have been reviewed and prototypes

are well

developed

Focal plate petal assembly to be assembled and tested in summer

2016

Fibers from the focal plane positioners to the full fiber bundle are spliced at LBNL prior to shipping.

Slide30

30

2.2

Optics and Focal Plane Instrumentation H. Cease, ANL / P. Thompson, GSFC Subcommittee 2CommentsCorrector and ADC Assembly, WBS 1.2The lens cell analysis has a 2.5 safety factor on the cell yield stress. Document the cell analysis including the assumptions, boundary conditions, and temperature dependent material properties. Include this document in the backup materials for reviewers.

Cage & Ring Assembly with Barrel Components, WBS 1.3

Consider including in the cost and schedule a fit test for the top ring, fins, cage and hexapod assembly at FNAL.

Slide31

31

2.2

Optics and Focal Plane Instrumentation H. Cease, ANL / P. Thompson, GSFC Subcommittee 2CommentsFocal Plane System (FPS), WBS 1.4

Design requirements flow from the KPPs, however, the 10 microns fiber positioner accuracy KPP is

rms

, not absolute. This requirement flows through several DESI documents (DESI-579, DESI-810, DESI-455, DESI-347). The DESI Systems Engineering throughput budget includes lateral offset of the fiber positioner actuator. The best effort allocation is 2 microns

rms

lateral offset for the actuator. A single summary document with hyperlinks to the referenced requirements document would be helpful in following the requirements flow.

Consider including environmental testing with the fiber positioner over the full operations temperature range with positioners mounted in a plate with the final material, mounting interface and hole spacing prior to starting fiber positioner production.

GFA electronics are cooled by ambient air flow which could be dryer than the humidity requirements for ESD.

Evaluate how the

general grounding and humidity

requirements will be met on the

GFA

system to address ESD concerns.

Slide32

32

2.2

Optics and Focal Plane Instrumentation H. Cease, ANL / P. Thompson, GSFC Subcommittee 2Recommendations WBS 1.2 Corrector and ADC Assembly, WBS 1.3

Cage

& Ring Assembly with Barrel Components,

WBS 1.4 Focal

Plane System (FPS

)

Capture the flow of requirements from KPPs to sub-system technical requirement documents prior to CD-3. This could be accomplished in the DESI Requirements Tracking

S

preadsheet by adding hyperlinks to referenced requirement documents.

Include

rms

in the KPP prior to CD-2:

5,000 fiber positioners in 10 petal

assemblies capable

of <10

μm

rms

accuracy

.

Instrumented Field

of View not less than 7.5

deg2

Prior to CD-3 A

single

document is needed that

tracks Signal to Noise

requirements from top-level (science) “signal to noise > 1 per mode (

nP

~ 1)”, to mid-level “Signal-to-noise ratio ≥ 7”, to lowest levels (e.g. stray-light analyses, detector dark/read noise, realistic atmospheric effects, bright planets/stars, . . . etc.). This document should not only contain “Throughput” (

opto

-electronic efficiency), but also much more comprehensive sources of Noise than is currently held in DESI-0922

.

Slide33

33

2.3

Spectrograph Instrumentation, Instrument Control P. O’Connor, BNL / Subcommittee 3Preliminary Design: Is the DESI preliminary design sound for establishing the Project Performance Baseline? Does the preliminary design support meeting the Key Performance Parameters (KPPs) to achieve DOE Dark Energy Stage IV Program objectives as described in the Mission Need Statement, and the technical flow-down requirements of the overall experimental program? YesProject

Scope

: Are the Project’s scope and high-level specifications sufficiently defined to support establishing the cost and schedule baselines?

Yes

Contingency

: Are there adequate scope, cost and schedule contingencies to address the remaining Project risks?

Yes

Has the Project responded appropriately to the CD-1 review, met all the CD-2 prerequisites, and is it ready for CD-2 approval?

Yes

Slide34

FindingsThe instrumentation reviewed by this subcommittee consists of a set of 10 three-arm spectrographs (WBS 1.6), a fiber system (WBS 1.5), and the instrument control software system (WBS 1.7).

Non-project funds were used to procure demonstrator components and begin fabricating an Engineering Model spectrograph and some elements of the fiber system.Advanced procurement authority was obtained to procure VPH grating components early as risk mitigation; and additional authority to procure CCDs has been requested for FY15.There are three risks rated HIGH and three rated MEDIUM in the reviewed subsystems:

HIGH: WBS 1.5 R5-06 PFA rate of productionHIGH: WBS1.6 R6-03 Grating efficiency/wavefront does not meet specsHIGH: WBS 1.6 R6-05 LBNL CCDs late due to MSL process equipmentMEDIUM: WBS 1.6 R6-08 Yield model for LBNL CCDs too optimisticMEDIUM: WBS 1.6 R6-09 Vibration of multiple cryocoolersMEDIUM: WBS 1.7 R7-01 ICS development is lateMitigation plans are in place for all risks. 34

Slide35

Findings (continued)Foreign partners contribute significantly to the reviewed WBS elements:STFC (UK) provides the fiber cables and

slithead assembliesCEA (FR) provides the cryostatsAix-Marseilles University provides integration and test of the spectrographsCRADA agreements with the foreign agencies have been drafted A $2.5M sole-source procurement to a French opto

-mechanics company is planned in mid-2016Process issues following a power failure in the MSL resulted in a several-month delay to the pre-production run of Red/NIR CCDs.There is a proposed change to the design of the LBNL CCD antireflection coating to meet the quantum efficiency requirement in the Red band.Early EVMS performance data has been collected for April-June 201535

Slide36

Comments

We congratulate the DESI team on good progress in all subsystems since CD-1.The fiber system and instrument control software teams have significant heritage from previous projects and their technical risks are low. The fiber system has done a thorough time-and-motion study of PFA production rate, which significantly addresses this risk issue. They have also identified pools of technical labor from which to add resources during production, if needed.

The front-end electronics specification document had no requirements for dynamic range and linearity, which are related to noise performance. Recovery from large, saturating signal levels should be investigated.Multiple institutions are involved in WBS1.6, particularly in the assembly and test phase. Managing and tracking their activities will be critically important.The noise budget prepared in response to a CD-1 recommendation identifies the important contributions, but does not show how they roll up to contribute to the S/N>7 KPP.36

Slide37

Comments

The new Red/NIR CCD coating introduces a new material (TiO2). Evaluation should verify robustness of physical characteristics such as adhesion as well as optical performance. Further fabrication and testing is needed to verify CCD performance for both the Blue and Red/NIR channels.Unexpected delays may be encountered in the ramp-up of CCD production at LBNL and ITL.

37

Slide38

Recommendations

The front-end electronics specification document should be updated to include dynamic range and linearity.The TiO2 AR coating should be verified as soon as possible.In collaboration with Systems Engineering, the noise and downtime budgets should be further developed, detailing individual contributions

and their interplay. Continue to evaluate the approach to managing and coordinating the multiple partners involved in spectrograph production.38

Slide39

39

2.4

Assembly

,

Integration

, Test and

Commissioning

WBS 1.9

&

ProtoDESI

D. Kieda, U of Utah (lead)

George

Ginther

, FNAL

2.4

Assembly

,

Integration

, Test and

Commissioning

D. Kieda, G .

Ginther

/ Subcommittee 4

Slide40

40

Preliminary Design

: Is the DESI preliminary design sound for establishing the Project Performance Baseline? Does the preliminary design support meeting the Key Performance Parameters (KPPs) to achieve DOE Dark Energy Stage IV Program objectives as described in the Mission Need Statement, and the technical flow-down requirements of the overall experimental program? YesProject Scope: Are the Project’s scope and high-level specifications sufficiently defined to support establishing the cost and schedule baselines? Adequate, needs clerical scrub for self-consistency/reduced uncertainty Contingency

: Are there adequate scope, cost and schedule contingencies to address the remaining Project risks

?

Appears to be adequate. AITC occurs at end of project giving sufficient contingency for

remaining risks.

4. Has

the Project responded appropriately to the CD-1 review, met all the CD-

2

prerequisites

, and is it ready for CD-2

approval?

Yes. Project is ready for CD-2 after

addressing outstanding concerns.

2.4

Assembly

,

Integration

, Test and

Commissioning

Response to Committee Charge

Slide41

41

2.4 Assembly

, Integration, Test and Commissioning Findings AITCDESI will be mounted in the 4m Mayall telescope, which is operated by the NSF.An MOA is

now in

place for the transition of

Mayall

operations support

from NSF to DOE

.

The baseline

costs for the installation and commissioning WBS are 8775k$, including a contingency of 1258k$ (17% contingency).

The

escalation from

the CD

-

1 review (2147k$) is due to the inclusion of the MOA and commissioning activities.

This

WBS represents 16% of the DOE funded

TPC. Total

non project funds 81k

$

WBS

1.9.1 AIT

Management 206k

$ project funds and 33k$ non project funds for the stray light studies)

WBS

1.9.2

Mayall

Prep,

NOAO 2930k

$ project funds (includes 1600k$ of the MOA funds for transition of the

Mayall

)

WBS

1.9.3 Installation and functional verification,

NOAO 35k

$ project funds

WBS

1.9.4 Installation and functional verification, non-

NOAO 2769k

$ project funds and 48k$ non project funds for UCL corrector re-integration

WBS

1.9.5 Commissioning, non-

NOAO 2835k

$ project funds

NOAO

effort is in WBS

1.1

This

WBS is dominated by LOE tasks combined with the

Mayall

passthrough

MOA.

WBS 1.9

is on the critical path from the time equipment becomes available through

KPP

Slide42

42

ProtoDESI is primarily an integration exercise to advance the advance the development of the software associated with the DESI concept which will test approximately 10 fibers, as well as the guidance and control systems, but will not include the spectrographs.

protoDESI will provide means for early retirement of interface risks between GFA, fiber positioner, Focus Sensor, Fiber View camera, TCS, ICS, & Skymapping. Incremental ProtoDESI cost is 363k$ WBS 1.4 FPS 249k$ (161k$ Labor, 88k$ non-labor) WBS 1.9 AITC 114k$ (Labor)2.4 Assembly

,

Integration

, Test and

Commissioning

Findings

protoDESI

Slide43

43

The project has made substantial progress

in defining the material, labor and schedule for Assembly, Integration, Test and Commissioning since the CD-1 review. The CD-1 review recommended expansion of WBS to explicitly include the Integration and Commissioning tasks. This task has been accomplished. ICDs have all progressed to Level 2 or higher, and a detailed resources loaded schedule and cost estimate has been developed. The project has appropriately reviewed risks associated with this subproject and has has retired 3 risks and mitigated the remainder.Project team members are experienced and bring valuable experiences from commissioning of DECam, Mayall facilities, and other projects.Discussions with subproject leaders indicated strong confidence in the reliability of the existing design. It was discovered that failure rates of these elements were assumed to be negligible, and have not been included in calculations of camera throughput. Discussions also revealed that field repair of individual broken fibers/fiber positioners is generally not considered.

The project appears

to be responsive to all recommendations of

Integration and Test IPR.

2.4

Assembly

,

Integration

, Test and

Commissioning

Comments-General

Slide44

44

This

subcommittee explored risks associated with several key production items which have demonstrated low risk during small quantity testing during prototyping phase. These items (fibers and fiber positioners) are large quantity items and changes in reliability, increases in cost, or increases in production time could have a multiplicative effect on science, cost, and/or schedule. There is an unquantified risk associated with potential breakage of fiber splices during shipping from LBNL to KPNO and installation. There are some important low risks in fiber positioner reliability that should be better quantified.

There are some time/cost uncertainties associated with camera petal assembly production which need better quantification to reduce risk.

WBS 1.9.2 has

very short float

between end of planning and start of

implementation. The start date for

WBS 1.9.2 does

not appear to be constrained. It may be possible to begin these activities earlier.

2.4

Assembly

,

Integration

, Test and

Commissioning

Comments: Schedule Risks

Slide45

45

This review panel drilled down on WBS 1.9 BOE, ICDs and schedule. These were very detailed and based upon solid costing assumptions. Several clerical errors were found:

Mayall ICD had not been updated to reflected revised focal plane chiller power requirements in the TDR.FP weight for maintaining telescope balance had not been designed/included in WBS Mayall facilities cost for protoDESI was double booked on two independent WBS numbers Schedule had several unrealistic deadlines (1 day spectroscope shipping from France to KPNO)A $200k estimate for operator training (booked as a large training event for ~180 collaboration members) is not well matched to cost/needs of the project.

1.0 FTE in WBS were inconsistently booked at either 8 hours/day or 12 hours/day.

ICD between fiber optic cables and telescope was lacking some important details regarding strain relief requirements (TBD).

The EVMS results exhibit some typical signs of start-up complications

.

These discrepancies may be due to relatively short period for increasing WBS scope since CD-

1

Such

discrepancies currently add

a level of uncertainty to establishing an accurate baseline.

.

2.4

Assembly

,

Integration

, Test and

Commissioning

Comments: Budget, Schedule, ICD

Slide46

46

The

ProtoDESI schedule is tight, with delivery of the GFA scheduled 10 June 2016, with installation scheduled for 20 July 2016ProtoDESI results expected to be available Q3 CY2016ProtoDESI results will not be available to inform proposed CD-3 (Q1 CY2016)The CD-2 review committee confirmed the vital importance of the ensuring the successful completion of ProtoDESI tests in Q3 CY2016

The CD-2 review committee stressed the priority of completing the full scope

of

the 2016

ProtoDESI

tests in Q3

over the planned resumption of the previously scheduled

imaging

survey.

2.4

Assembly

,

Integration

, Test and

Commissioning

Comments:

ProtoDESI

Slide47

47

Develop plan to quantify/reduce risk in high quantity items

Run several fiber positioners continually over the next year under realistic orientation and environmental conditions to search for possible failure modes and better quantify reliabilityPeriodically perform extensive sample test of randomly selected fiber positioners during production phase on an ongoing basis to firmly establish reliability of production-line quality.Develop and test suitable camera/petal transportation box which minimizes fiber optic breakage. Quantify reliability of fiber splices/strain relief using expected shipping vibrations, proposed strain relief system, proposed shipping protection methods.

2.4

Assembly

,

Integration

, Test and

Commissioning

Recommendations-Risk reduction

Slide48

48

2

) Clarify Baseline Project Scope of WBS 1.9Define formal commissioning policy for focal plane camera, detailing response to onsite testing at different levels of performance. Include schedule/cost contingency as appropriate. [Complete prior to CD-2 baseline.]Improve throughput calculation to include estimated number of broken/marginal channels assuming adherence to above acceptance criteria. Propagate calculation to estimate potential impact on baseline. [Complete prior to

CD-

2 baseline.]

Generate WBS placeholder cost estimate, schedule, ICD for FP dummy weight

.

Replace with design parameters when complete.

[Recommend completion prior to CD-3.]

Specify

fiber

optic

strain

relief requirements in ICD between fiber optic cabling and

Mayall

telescope.

The

ICD should specify cable weight as well as strain relief to avoid excess drape weight at FP end of cable and reduce cable sway during

slew.

[ Recommend completion prior to CD-3.]

2.4

Assembly

,

Integration

, Test and

Commissioning

Recommendation:

B

aseline Scope

Slide49

49

3

) Review and update documentation [Complete prior to CD-2 baseline] Update the TDR to address the residual inconsistencies and include missing references• Review the revised schedule, BOEs, contingencies and risk registry to ensure that they

are

consistent and fully reflect the proposed baseline

AITC plan

• Ensure that task durations/lags and resource assignments are appropriated assigned

(e.g. shipping

times

)

• Verify that the relevant

costed

resources are properly represented in the

schedule

• Verify that the risks registry entries properly reflect the current understanding of

risks

2.4

Assembly

,

Integration

, Test and

Commissioning

Recommendation: CD-2 documentation

Slide50

50

4) Provide resources and commitment to ensure success of

ProtoDESI. [Complete prior to end of CY2015]Identify personnel for rapid analysis of ProtoDESI resultsEstablish dialog with NOAO regarding priority for successful completion of ProtoDESI

tests

in Q3 CY2016

Develop NOAO planning/agreement for possible delay in resumption of

ongoing

Mayall

imaging survey.

2.4

Assembly

,

Integration

, Test and

Commissioning

Recommendation:

ProtoDESI

Slide51

51

3. Environment, Safety and Health

F. O’Neill, retired SLAC / Subcommittee 5ES&H: Are ES&H systems and processes in place to mitigate of all the hazards identified and to ensure delivery of the Project in a safe and environmentally sound manner?Yes.Has the Project responded appropriately to the CD-1 review, met all the CD-2 prerequisites, and is it ready for CD-2 approval?Yes.Findings

DESI Equipment Protection Plan developed since CD-1

DESI Machine Protection Hazard Analysis developed since CD-1

PHAR updated and expanded

LBNL ISMS Plan in place

NEPA Categorical Exclusion Determination complete

Two Pre-Installation Reviews planned in 2017, Safety of AI&T will be part of these reviews

Slide52

52

Comments

Very good progress since CD-1Hazard Analysis documents comprehensive and thoroughExperienced and competent Safety Officers and Engineers in placeDESI Safety OfficerKPNO Safety OfficerEquipment Protection Safety

Engineer

L2 Managers covered safety in their presentations and have ownership of the safety issues associated with their subsystems

DESI is utilizing the experience gained on

DECam

AI&T Scientist and Manager recognize that the most significant mishap risk is associated with the work on the telescope at Kit Peak

Significant work ahead in preparation for work at KP – project is on track

Recommendations

The Project is ready to proceed to CD-2

3. Environment, Safety and Health

F. O’Neill, retired SLAC / Subcommittee 5

Slide53

53

4. Cost and Schedule

A. Bampton, PNNL / P. Utley, SLAC / R. Won, DOE-OPA Subcommittee 62. Project Scope: Are the Project’s scope and high-level specifications sufficiently defined to support establishing the cost and schedule baselines? YesCost and Schedule: Are the cost and schedule estimates, and bases of estimate, credible to establish the Performance Baseline? Yes Does the Project schedule and planned off-project operations funding support the overall experimental program, including access to the Mayall telescope

?

Yes

Contingency

: Are there adequate scope, cost and schedule contingencies to address the remaining Project risks

?

Yes

Has the Project responded appropriately to the CD-1 review, met all the CD-2 prerequisites, and is it ready for CD-2 approval

?

Yes

Slide54

54

4. Cost and Schedule

A. Bampton, PNNL / P. Utley, SLAC / R. Won, DOE-OPA Subcommittee 6FindingsWBS includes work needed to accomplish Objective KPP’s as documented in the PEP.System engineering function tracked requirements from KPPs down through to systems and subsystems that became WBS elements. DESI-0579 provides tracking of KPPs/technical requirements to individual systems and subsystems/WBS. DESI Project scope cost and schedule estimates are aided by prior experience on DECam

, DES and other similar

project

.

Drill

downs of selected WBS elements found cost estimates to be in

a common format including consistent content and level of detail including vendor quotes or equivalent backup

.

Overheads and burdens are applied. Partner institutions provided burdened rates and some were either escalated by the home org or in LBNLs system

.

Monthly reporting includes EV status as well as schedule performance reporting for off-project work. Weekly status, schedule and cost meetings are attended by the L2 managers.

Level 2 schedule milestones have been appropriately and evenly distributed across the project life.

Slide55

55

4. Cost and Schedule

A. Bampton, PNNL / P. Utley, SLAC / R. Won, DOE-OPA Subcommittee 6FindingsOff project work scope is integrated with the baseline via schedule task/ milestones indicating performance and interface (delivery/supply) points and can be identified by the DESI_Fund and DESI_Inst codes within P6.The project baseline includes ~$12M (36%) of cost contingency on TEC work and includes ~$5M of scope contingency.

Cost contingency

included

to offset risk at the 90% confidence level - Monte Carlo.

Funding

availability would limit access to contingency until late in the project. FY16 & FY17 are funding constrained budget years

.

Schedule

contingency of 18 months (90% confidence level) is derived from Monte Carlo simulations of the Risk Register data and an additional 6 months added based on management judgment.

The project is in 4

th

month of reporting. 3 reports have been produced with CAM developed variance analysis. Variances

reports noted mechanical

issues existed.

Most CAMs were new to EVM and have been trained to LBNL EVMS standard prior to the establishment of the preliminary baseline and then retrained after two months of practice (in June and July). 3 training sessions have been conducted. Mandatory participation. Additional training is intended.

Slide56

56

4. Cost and Schedule

A. Bampton, PNNL / P. Utley, SLAC / R. Won, DOE-OPA Subcommittee 6CommentsCost contingency of 36% on remaining TEC work and scope contingency already included in the baseline provides good margin for completing the project.Practicing variance reporting using EVM is providing good experience for CAMs and Project Leadership. Setting the cost and schedule baseline at CD-2 and updating documentation will allow the team to implement lessons learned during this practice period. Variance reporting can be improved.Managing to the schedule will be critical as many off project activities have no cost data

.

Control

accounts

may

be too high to uncover significant emerging issues at lower

levels. Consider breaking large control accounts into more manageable and visible components (e.g. WBS level, labor, material,

etc

).

Slide57

57

4. Cost and Schedule

A. Bampton, PNNL / P. Utley, SLAC / R. Won, DOE-OPA Subcommittee 6RecommendationsUpdate preliminary baseline and documentation for CD-2. Then proceed to CD-2.Conduct more CAM training emphasizing schedule and early management awareness of issues after CD-2.

Slide58

58

4. Cost and Schedule

A. Bampton, PNNL / Subcommittee 6PROJECT STATUS as of June 2015

Project Type

MIE

CD-1

Planned:

Q4FY14

Actual:

3/19/15

CD-2

Planned:

Q4FY15

Actual:

TBD

CD-3

Planned:

Q4FY16

Actual:

TBD

CD-4

Planned:

Q4FY21

Actual

:

TBD

TPC Percent Complete

Planned:

_3.3_

%

Actual:

_3.3__

%

TPC Cost to Date

 

$1,453K

 

 

 

 

TPC Committed to Date

 

$1,453K

TPC

 

$54,

628

K

TEC

 

$32,077K

Contingency Cost (w/Mgmt Reserve)

$12,124K

_29.5_% - on

ETC

Contingency Schedule

on CD-4

__24__months

_50_% - on

ETC

CPI

Cumulative

 

N/A

– prelim 0.96

 

 

SPI Cumulative

 

N/A prelim 0.87

Slide59

59

5. Management H. Lee, D. Green, G. Rameika Subcommittee 7

Preliminary Design

: Is the DESI preliminary design sound for establishing the Project Performance Baseline? Does the preliminary design support meeting the Key Performance Parameters (KPPs) to achieve DOE Dark Energy Stage IV Program objectives as described in the Mission Need Statement, and the technical flow-down requirements of the overall experimental program?

YES

Contingency

: Are there adequate scope, cost and schedule contingencies to address the remaining Project risks?

YES

Management

: Is the Project being properly managed? Is there a capable team in place to effectively manage all the major interfaces and risks, and to achieve successful delivery of the Project?

YES

Has the Project responded appropriately to the CD-1 review, met all the CD-2 prerequisites, and is it ready for CD-2 approval?

YES

Slide60

60

5. Management H. Lee, D. Green, G. Rameika Subcommittee 7

Findings

The Project builds on experience from BOSS, DES,

DECam

and SDSS.

The required CD-2 documentation was posted by the project and is appropriately developed.

DESI is a Major Item of Equipment project with DOE TPC of $54.6M and non-DOE contributions (in-kind and funding) of $16.7M.

MOU or other agreements are in place with non-DOE US and foreign partners except for four CRADAs which are nearing completion.

A MOA is in place between DOE and NSF for support of the

Mayall

telescope until transition to DESI operations. Project OPC funding supports this MOA as a pass through to NOAO.

Project management processes are in place.

­Risks are identified by L2 managers; Risk Management Board monitors risks bimonthly and mitigation costs are included in the baseline. Risk registry contains residual risks after mitigation. There are 64 active risks and 28 retired.

­EVMS reporting has started and CAMs are in place. Schedule status will be used for non-DOE contributions to DESI.

­Change control process has been exercised.

Slide61

61

5. Management H. Lee, D. Green, G. Rameika Subcommittee 7

Findings – continued

A notional steady state operations budget of $3.4M for NOAO operations and support and $3M for DESI instrument operations was presented to the Committee.

The project threshold KPPs are the minimum performance to meet the mission need but the baseline will support meeting the objective KPPs. The difference between threshold and objective KPPs represents scope contingency which is about $5M in cost savings or ~10% of TEC.

An objective KPP is to provide 100% of the intended focal plane and spectrograph components. There is a level 2 milestone in April 2017 to decide whether to reduce the deliverable by 40%.

The critical path runs through the focal plane and fiber optics.

The project is holding 36% cost contingency on TEC and OPC contingency is 23% without the NOAO pass through funds. There is 24 months of schedule contingency.

DOE funding in FY16 and FY17 is tight. As a result, annual contingency amounts are not optimal for those years.

An end to end test with prototypes is no longer in the baseline.

Slide62

62

5. Management H. Lee, D. Green, G. Rameika Subcommittee 7

Comments

The project team is experienced with seasoned managers from other similar projects.

Non DOE groups will provide $5.6M before CY16 which allows the project to advance the schedule with early procurements. The Common Fund concept will give the project added flexibility.

Major technical risks are being mitigated through development of prototypes and early procurement of components.

However, there are other risks that should be evaluated and included in the risk registry, e.g. failure modes of spliced optical fibers

The project team using the ICD (N^2) should make the project I&C run smoothly. ICD and N^2 controlled documents are important to connect elements of DESI since it is a complex project.

WBS 1.9 is inclusive and encompasses integration, testing, installation and commissioning which is a good development and responds to the CD-1 recommendation.

Slide63

63

5. Management H. Lee, D. Green, G. Rameika Subcommittee 7

Comments – continued

The DESI Spokespersons are involved in the change control process which will preserve the science goals of DESI to the extent possible.

Additionally, communications with non-DOE entities should be formalized by the project and the program office to convey project progress, issues and significant technical changes.

The limited cost contingency in FY16 and FY17 needs to be managed. FY16 may not be a problem as the project effort is to continue design towards achieving CD-3. However, FY17 will entail procurements that may require additional contingency should risks arise.

DESI has been assigned a dedicated procurement officer which will aid in the FY17 procurements.

In lieu of end to end testing, the project has a plan to conduct system level tests of a complete single petal and to fully characterize the first spectrograph.

Data challenges are being set up and they will advance the goals of the DESI Collaboration.

Slide64

64

5. Management H. Lee, D. Green, G. Rameika Subcommittee 7

Recommendations

Assess risks discussed by the other subcommittees (failure

modes of spliced optical

fibers, camera petal assembly, data system off-project resources, etc.) that could impact the project and include them in the risk register by CD-2

Proceed with CD-2 approval after updating the pre-requisite documentation.

After CD-2, evaluate project activities to identify additional contingency to handle potential risks in FY17. Discuss options and flexibility of FY17 funding allotments with the program office.