Outline Machine Controls - PowerPoint Presentation

Slide1

Outline

Machine Controls Coordinators (MCC): team and roleOverview of renovations during LS1Proposal for after-LS1 Commissioning organization

ACCOR PROJECT – IEFC, June 8th, 2012 – M.Vanden Eynden

1

MCC team

2

LS1 Plans and Organization – TC October 18th, 2012

MACHINE

MCC

SPECIFICITIES

Linac3 + LEIR (ions)

Marine

Mainly renovated in 2012Linac4 Ioan New machineLinac2 + PSB SteenCO system mainly developed + maintained by COREX + ISOLDEFrankCPS MarineADMatthieuSPSMarineCO system maintained by EQP GPsCTFMickNo shutdown - Live test bed

Role of MCC

Scope of responsibilityIn scope: all renovation + upgrade activities – any sizeOut of scope: bug / issues on operational systems. Handled by Exploitation team

MCC = focal point for all requests from EQP / OPRole: coordinate the work over the whole process from request to OP commissioning

Next slide

Continuous proactive

approach

EQP groups

sometimes reluctant

to renovate or take over responsibilitySome conservative behavior from OP3LS1 Plans and Organization – TC October 18th, 2012

Renovation process

4

LS1 Plans and Organization – TC October 18th, 2012

New Request

Solution agreed

Functional specification

Breakdown in work packages

MCC

Gather renovation or upgrade request Organize technical meetingsProvide EDMS DocumentHW FEWhat when Who

SW FE

What when

Who

AP

What when

Who

DM

What when

Who

HW tests

Commissioning

Plan

Renovation Request

Installation

Team

CO

CO

LS1 Planning officer

Development of Work packages

Follow and ensure planning is met

Deployment - HW Tests

OP Commissioning

Follow deployment

Participate to commissioning,

led by OP

Ensure it is complete + successful

Transfer to Exploitation team

Organize taking over by EXPL

Commissioning Plan

OP

OP

EXPL

EQP

OP

CO

EQP

EXPL

LS1 Challenges

Significant number of renovations, many inter-dependenciesNext slideRenovations with large scale & critical impactLarge scale: 1900 devices (TIMING) 1500 (Power converters)

400 (function generators)

Critical domains: TIMING, POWER, RF , KICKERS

Most renovations driven by BE/CO

70 % of total

Systems deployed for the

1

st time – never validated in operationSome examplesCGDAO Class for RF FMC Fine delays HW modules for Kickers Transfer of Front-End responsibility from CO to EQP GroupsGeneralize the LHC model : FEC under EQP GP responsibilityThis implies FEC re-organization, knowledge transfer and support at taking-overSynchronized renovations between CO and EQP GroupsEnsure our commissioning time is properly estimated + integrated in Accelerator schedule 2014LS1 Plans and Organization – TC October 18th, 2012

LS1 objectives Large scale renovations driven by BE/CO

Legacy eradication campaignTowards a ‘Lynx-OS free zone’ ± 250 FE to be migrated to LinuxCAMAC 22 Crates - Lin2/PSB/CPS/CTFSL-EQUIP [SPS]

Progressive phase-outObsolete FE ‘GM’ Classes & driversPriority : classes under CO responsibility ‘

GFAS’, ‘POW-V’, ‘DIGCTL’,…

Obsolete FE HW modules

MPV908, STR55/71…

LS1 Plans and Organization – TC October 18th, 2012

LS1 objectives Large scale renovations driven by BE/CO

-2UpgradesTIMING UPGRADE & TG8 ERADICATION -1900 devicesHW module: TG8 -> CTR , FE SW: PTIM-V -> LTIM Deployment of new Timing implies not a single TG8 left in the whole chain : LIN2-PSB-CPS-AD

NEW CENTRAL TIMING for ADX/Motif AP not compliant anymore, to be upgraded

Retuning of all

inj

/ejection timings (CTR delays # TG8 delays) at commissioning

NEW Controls for PC

- 1500 devicesBC1553 -> CBMIA card + new FESA class (TE/EPC) FGC Integration NEW controls for Function Generator (clients: RF + PC) – 400 devicesOASIS: upgrade of obsolete OASIS + migration of Samplers to OASISOASIS: 25 VXI FE ~1500 signalsSAMPLERS : 30 FE – 550 signals Integration with evolving servicesLSA / INCA , CMW , FESA 3.0

LS1 objectives Large scale renovations by EQP Groups

TE-EPCPSB/ CPS: new FESA class replacing POW-V + FGC @ PSBABTAll SPS KICKERS : new electronics + controls PS CT extraction Kickers: CAMAC eradication EN/STI CPS Motor Controls: CAMAC eradication

All Beam Stoppers BE/RF

PSB Low Level RF: new electronics + controls

SPS RF: removal of BC1553 /SL-EQUIP

BE/BI

PSComplex

: many systems renovated

LS1 Plans and Organization – TC October 18th, 2012

LS1 Objectives in figures

9

MACHINE

FEC

Already Renovated

Renovation in LS1

FE NOT renovated after LS1

LINAC 3

14

8

5

1 ?

LEIR

30

17

11

2

LINAC 2

9

1

5 (3*)

3

PSB

46

8

32

6 (+7?*)

CPS

68

28

35

5

AD

31

10

15

2 (+3 ?*)

SPS

250

±90

±90 *

±70 *

Isolde-REX

22

13

8

1

CTF

42

26

8?

10 ?

LINAC 4

Continuous installation of new FE

* Partial CO renovations

* current RF crates, hot spares

* Under discussion

Leftovers

Huge effort in LS1

By end LS1, 70 - 90% of FECs will be renovated

* Uncertainty from RF & BI

Initial

Known constraints on planned renovations

NEW TIMING : green light for the full deployment Lynx-OS & CAMAC Eradication About 40 FE (out of 250) not renovated after LS1 (Lynx-OS)Uncertainty on SPS (140 LynxOS FE ->renovation driven by EQP GPs)3 CAMAC

crates (out of 10) kept after LS1Strong constraints

EQP GPs are not ready for LS1 or not willing

Linac2: ’No change on Lin2 because of Lin4‘

CTF : running machine (no stop) + renovation at low priority in EQP GPs

OP/EQP require the current system to run in // with the new

EN/EL refused cabling requests -> no renovation by EQP GPs

-> BE/CO support is requested on some obsolete systemsIntermediate compromises by BE/CO Migration to Linux of simple ‘GM’ class, waiting for FESA replacement by EQPNegotiation between CO and EQP to anticipate FE responsibility transfer OASIS / Samplers No budget allocated for new fast digitizers Ongoing review of each installation + cost estimate. Priority will be assigned depending on available budget

Outline

Machine Controls CoordinatorsOverview of renovation packages for LS1

After LS1 Commissioning Preparation

A

snapshot of weak points

at commissioning

of new controls @ LEIR

/ Linac3 in

2012Based on this experience, proposed organization for LS1 commissioning11LS1 Plans and Organization – TC October 18th, 2012

Migration process : BE/CO observed issues

Device class migration : long, complex, error proneUse case: ‘GFAS’ GM class -> ‘CGAFG’ FESA classA N-week notice period requested to process the class mappingMany steps required , error -proneproperty + new/old device mapping,, settings migration, meta-properties, backup,…Lack of reliable LSA DB data configurators assigned by OP (Leir/linac3)

Missing description of proceduresGM -> FESA: some doc exists but is specific & not up to date anyway

BC1553 >

CBMIA@Kontron

: No doc. Knowledge by Nicolas only.

Renamed FECs not recognized in DIAMON and CMW

Manual intervention by DIAMON expert

Manual restart of INCA server to reinitialize CMW subscriptionsIMPACT Too manual steps in the processDifficult coordination task for MCCs Delays in delivery 12LS1 Plans and Organization – TC October 18th, 2012

Enhancements in pipeline

Tool for MCCs to ease coordination work1st version expected

by end 2012Automated data migration

workflow

Aim

:

smooth

propagation of device data thru CCDB, LSADB, DIAMON, CMWStatus and planningOngoing activityBy end October 2012: formal description + publication of current procedures Early LS1: integrated environment with automated workflow in placeEarly LS1: qualified LSA (+CONFIG) data configurators are assigned by OPLS1 Plans and Organization – TC October 18th, 2012Record the issueBreakdown in WorkPackages

WP1

HW install

Jan 13

Manu

WP2

Driver

Mar 13

Toto

WP3

FESA

Apr 13

Titi

Automatic Notification when action is completed

Follow-up Progress

Detect

bottenecks

EDMS Doc

Prepare deployment

HW tests + Commissioning: BE/CO issuesUse case: CBMIA deployment at Lin3/LEIR

Too short CO commissioning time requested to OP We wrongly believed that systems validated on other machines would work right away on LEIR/LN3 Insufficient involvement asked to OP + EQP Commissioning mainly done by CO service providers on their own, with expert toolsCloser collaboration with TE-EPC would have sped up diagnostics

Missing collaboration from OP for validation and acceptance checkFeedback from experience

New controls delivered to OP were not fully debugged

Abnormally high number of EPC + CO Piquet interventions.

± 60 % of CO issues could have been detected at commissioning

, if time were properly allocated.

R

emaining issues not easily detectable before OP deployment

Projection on LS1 Commissioning : WARNING

Large uncertainty margin on commissioning time If we extrapolate from LEIR/Linac3 2012-experience, our request for LS1 commissioning time would be infinite !Renovations from EQP / CO groups to be synchronizedElectrical / cabling issues expected

Schedule a 3-phase commissioning (LHC model)HW TESTs: integrity check of CO component

DRY RUNs (no beam)

:

eqp

switched ON, timing ON

BEAM TESTs: nominal conditions available

Recommendation to all CO service providersApply safety margin to your commissioning time requests to MCCsEnsure backup solutions wherever the cost is reasonable Slots will appear in the official accelerator schedule (decision at OP commissioning meetings)Time for fix / repair

Time for final tuning

LS1 Plans and Organization – TC October 18th, 2012

OP-CO-EQP collaboration

Tests by OP