Aggressive hardware & beam commissioning in - PowerPoint Presentation

1. Aggressive hardware & beam commissioning in the injectorsGian Piero Di GiovanniPiotr SkowronskiDecember 8, 2022JAP WS @CERN, Week #351

2. IntroductionDecember 8, 2022JAP WS @CERN, Week #352

3. The electricity crisis, shortage during winter and its price increase,led to shortening of 2023 run by 6 weeks314th IEFC issued action for BE-OP: Make startup planning in 2023 (IST, HW and BC commissioning) with a most aggressive scheduleDecember 8, 2022JAP WS @CERN, Week #353SPS: 38 WeeksYETS’23-242022 Run17 WeeksYETS’22-232023 RunWas 40 weeks  -2 WeeksSPS: 32 Weeks19 WeeksWas 38 weeks  -6 WeeksYETS is defined beam to beam

4. PreambleIn this talk We discuss what are the options to speed up commissioningWhat are their pros and consWhich ones we try to endeavour should come from this workshopDecember 8, 2022JAP WS @CERN, Week #354

5. Commissioning planningDecember 8, 2022JAP WS @CERN, Week #355

6. Accelerator commissioningEvery year, each accelerator, is setup basically from scratch ensuring Its safety The best beam performanceIt involves many actors:BE/OP TeamOperatorsPhysicists/Engineers in ChargeBE/ABP TeamSY Experts responsible for all various subsystemsSY/BI, SY/EPC, SY/RF, SY/ABT, SY/STI, …December 8, 2022JAP WS @CERN, Week #356

7. Accelerator commissioning prerequisitesHardware readiness of all systems All systems fully integrated and tested in the controls environment FESA classesSettings: parameters, value generators, make rulesTools and applications ready and tested during dry-runs and checkout All systems successfully commissioned and tested without beam Operational scenarios tested during dry-runs and checkout December 8, 2022JAP WS @CERN, Week #357

8. Accelerator commissioningBefore enabling the beam all the hardware elements and interlocks are checked Individual System Tests (IST) by the ExpertsHardware Commissioning (HC) by OperationBeam is started with the lowest reasonable intensityAt each step it is sent only to the next safe dump locationBeam parameters are measured with all possible instrumentsOften, the instruments first need to be checked and calibrated with beamThe results are verified against the expectations and, if needed, the settings are correctedIntensities are ramped to reach the design / maximum performanceDecember 8, 2022JAP WS @CERN, Week #358

9. Commissioning planningCommissioning is a more or less a sequential processTesting the hardware before starting the beamInterlocksBeam Instrumentation setup and calibration with beamInjection  Steering  RF Setup  Ejection  DumpingOptics  Collimation  VerificationDecember 8, 2022JAP WS @CERN, Week #359

10. Commissioning planningSometimes, the tasks of different experts are densely interleaved as the beam is transported further down the machineCircular dependencies For example: beam setup <--> BI setup & calibration The planning is started with a list of steps to be completedTime estimate for each step is defined by the ExpertsBy the EiC for the operational stepsThe plan is constructed such that The required sequence is respectedThe Expert steps are scheduled during working hoursNights are populated with OP stepsFor HC the operations work on morning & afternoon shiftsDecember 8, 2022JAP WS @CERN, Week #3510

11. 2022 Commissioning in LINAC47 + 12 days (HC+BC)The longest items:Power ConvertersPhasingLLRF setup with beamDecember 8, 2022JAP WS @CERN, Week #3511

12. 2022 Commissioning in the PSB14 + 7 daysThe longest items:Power convertersInterlocksDecember 8, 2022JAP WS @CERN, Week #3512

13. 2022 Commissioning in the PS14 + 7 daysIn 2023 it will be 7 + 7 daysThe longest items:Power convertersRF SetupDecember 8, 2022JAP WS @CERN, Week #3513

14. 2022 Commissioning in the SPS 3 + 2 weeks (LHC) and additional 2-3 weeks for NAThe longest items:ScrubbingSlow extraction steering to targetPower convertersDecember 8, 2022JAP WS @CERN, Week #3514

15. No gain in some casesAggressive commissioning doesn’t always make senseFor each machine minimum YETS duration is differentDownstream machine may not be ready yet to take the beam if the upstream accelerator provides the beam earlierFor example, for LINAC4 YETS, normally is 2 weeks shorter than for the PSB, which is exactly the time needed for beam commissioning of LN4December 8, 2022JAP WS @CERN, Week #3515

16. Automation & OptimizationDecember 8, 2022JAP WS @CERN, Week #3516

17. Automation is not aggressive!We don’t consider automation as an aggressive approach:In other words: “automation” is not a synonym of “aggressive”It’s our obligation to improve performance and minimize the required time by automation of all possible tasks Every year, in each machine, there are many examples of new automatizationsPhasing and beam intensity change in LINAC4FGC power converter testing in SPS & PSBLM channel testing in SPS…December 8, 2022JAP WS @CERN, Week #3517

18. Automation: space for improvementHowever, there is still space for further improvements in the field of automationBut with the limited work force we are not able to implement everything within one or two yearsA good program takes time to developAlso, testing automatics for commissioning is not always straight forward during beam operation or YETSAutomation of regular beam operation, improving machine operability, reducing fault recovery time, is usually given higher priority than the tasks done once per year during commissioningDowntime recovery vs. commissioning speedDecember 8, 2022JAP WS @CERN, Week #3518

19. Automation prospectsPotential to reduce Hardware Commissioning by 1 week in PSB and PSAutomatic testing won’t bring all its fruits in the first time it’s implementedTeething problems: bugs, corner cases, real vs. test environmentSoftware obsolescenceEven if the tests can be run very quickly,we need time to solve the discovered problemsDecember 8, 2022JAP WS @CERN, Week #3519

20. OptimizationThink what really needs to be tested: can we reduce the list of checks?New hardware technologies contain sufficient intelligence to immediately show problems on the alarm consoleThe experts normally do test their hardware in Individual System Tests  Full testing by OP is likely redundant It depends on the particular system technology For example, it could be reduced for FGC 62&63 of LN4 & PSB,but it cannot be done in AD (old technology)Interlocks are still critical and this cannot be shrunkDecember 8, 2022JAP WS @CERN, Week #3520

21. Aggressive OptionsDecember 8, 2022JAP WS @CERN, Week #3521

22. Option 1: Use the past year setupIf hardware did not change too much over YETS thenit might be possible to run the machine with the past year settingsIf necessary, correct the steering and tunes to the old referenceVerify that the bunch sizes, losses and other key parameters are still fineBefore LS1 it was often the way to startThe post LS1 troubles (see Chamonix 2014 slides&paper by K. Hanke)lead to creation of a Injectors Recommissioning WG (119th IEFC )IRWG IndicoIt imposed more rigorous approach that we have nowDecember 8, 2022JAP WS @CERN, Week #3522

23. Use the past year setup: consequencesIt’s likely that the beam parameters won’t have the best performance  brightness, losses (activation), intensityMachine parameters usually drift away in course of a runA set of good enough parameter values to be defined for each beamThe parameters will likely continue drifting away: count-in safety marginsCertain items will be left uncheckedMore difficult beam setup for the downstream machines The imperfect incoming beam will be the reason for all their troublesRisk of full re-commissioning  uncertainty for the downstream machines planningDecember 8, 2022JAP WS @CERN, Week #3523

24. Option 2: Reduce contingenciesEvery team counts in safety margins in the required duration of their activitiesObservation from the past time: Most of the activities are completed much faster than declared The downstream facility usually can start commissioning a few days earlierDeclared time is longer than the expectation valueTime needed for standard amount of problemsGives flexibility in execution of parallel activitiesNobody wants to be responsible for delays and rescheduling of subsequent activitiesDecember 8, 2022JAP WS @CERN, Week #3524expectation valuedeclared value

25. Reduction of contingencies: consequencesEveryone needs to accept delays, that would become much more likelyEnvironment needs to be prepared for delaysThis includes the downstream machines and the experimentsThey could receive no beam or poor quality beam to start withAs more downstream facility as bigger the uncertainty is!The planning of each team would have to be very agile Consequences for all other activities of the teamDecember 8, 2022JAP WS @CERN, Week #3525expectation valuedeclared value

26. Agile planningEnsure that machine commissioning has the top priority Dynamic rescheduling of other activitiesIncluding YETS installations & ISTs in other machinesCan all the teams afford it?Be ready to start an activity earlier then scheduledEffective problem communication and handlingDecember 8, 2022JAP WS @CERN, Week #3526

27. Option 3: Extend Expert working hoursNaturally, if the expert activities could be scheduled also beyond working hours then the total commissioning time could be reducedConsequences:Longer support requires larger pool of expertsMaximum working time per day must be respectedAn Expert working a late evening cannot be available on the following morningHomogenous compensation schema across the sectorDecember 8, 2022JAP WS @CERN, Week #3527

28. Expert availabilityEnsure that the Experts can focus on commissioning and avoid conflicts with other activitiesMERITsMeetings: reduce to minimumRegular meetings (FOM, IEFC, IPP)Morning meeting only?Zoom?Avoid conferences, workshops & events (FCC week, IPAC)All the experts, also the backup ones, need to be available leave policy during the commissioning periodFor some machines the commissioning systematically falls into winter or Easter vacation periodsDecember 8, 2022JAP WS @CERN, Week #3528

29. Option 4: Transfer expert activities to OPIf Operators are trained to perform currently expert tasks then they could be scheduled outside working hoursStudy what can and what definitively cannot be given to OPFor example, RF setup clearly must be done by the expertsIt requiresExpert agreementDocumentationSoftware tooling and adjustment of operational control interfaceTrainingDecember 8, 2022JAP WS @CERN, Week #3529

30. RemarksCommissioning time is the best moment to train new staff membersAbility to react to unexpected problems comes from the experience gained during the setup December 8, 2022JAP WS @CERN, Week #3530

31. ConclusionsThere are possibilities to speed up machine restart, that all the teams already actively work onAutomatizationOptimizationRationalizationFour aggressive optionsRunning with past year settingsReducing contingenciesScheduling expert activities also outside working hoursTransfer selected Expert activities to OPThese are not exclusive, each of them can be applied independentlyDecember 8, 2022JAP WS @CERN, Week #3531

32. ConclusionsEach of the aggressive options comes with a price to pay: important consequencesNot the best beam performanceUncertainty in time of the activities and the commissioning completionCompensations for outside working hours activitiesWork on documentation and trainingAgile planning of the commissioning, YETS and other adjacent activitiesIncreased entropy requires energy to keep the temperature cool, well below boiling pointDecember 8, 2022JAP WS @CERN, Week #3532

33. ConclusionsThere are opportunities to improve the whole processof accelerator restartIt requires global discussion and a shared vision across whole ATS sector December 8, 2022JAP WS @CERN, Week #3533