Process status lessons learnt and forecast Bertrand Bauvir IODGSCOPSCODCDCCI Contents Objectives Process Status Lessons learnt Forecast References Contents Context Context ITER Instrumentation and Control System ID: 1044025
Download Presentation The PPT/PDF document "ITER control system integration" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
1. ITER control system integrationProcess, status, lessons learnt and forecastBertrand Bauvir IO/DG/SCOP/SCOD/CD/CCI
2. ContentsObjectivesProcessStatusLessons learntForecastReferencesContents
3. ContextContextITER Instrumentation and Control System is segregated in three vertical tiers and two horizontal layers
4. SAFETY CONTROL SYSTEMSContextContextITER Instrumentation and Control System is segregated in three vertical tiers and two horizontal layersCONVENTIONAL CONTROL SYSTEMINTERLOCK CONTROL SYST.
5. PLANT I&C SYSTEMSIn kindCENTRAL I&C SYSTEMSIn fundContextContextITER Instrumentation and Control System is segregated in three vertical tiers and two horizontal layers
6. ObjectivesAchieve the Integrated and Automated operation of ITERDelivered systems are made to co-operate in a functionally integrated control systemScope if all tiers (except SCS-N by reason of PIA) and layersEnsures effective transfer of responsibility for Integrated Operation and Maintenance to IO CT / SCODEarly preparation of the I&C integration is mandatoryTo reduce the risk of fragmentation of Instrumentation and Control (I&C) systemsTo anticipate problems and account for solutionsTo reduce the risk of project delays squeezing Control System activities ahead of First Plasma operation Objectives
7. ProcessIn-kind procurement model induces risks mitigated throughSupport with early involvement with Plant System I&C designStandardisation with setting I&C standard process and toolsRefer to e.g. ITER Controls Design Status & Progress (cern.ch)Process
8. Horizontal IntegrationStrategy is horizontal integration through the central I&C systemsTo be able to cope with changing requirements with manageable effort and contain complexityChallenge since the procurement model implies a vertical integration (silo model) concentrated in each PBS And cases of network of privately negotiated direct I&C interfaces (star model)Horizontal integration meansPhysical integration by means of CNP, central I&C networks .. well established and almost universally followedFunctional integration (for conventional) by means of CODAC Supervision and Automation (SUP) systemManage Plant System dependencies through coordinated configuration and automated operationProcess
9. I&C Integration Process Process7 independent packages of integration and verification activities and verification have been defined and have to function as a single entity to be ready on time for the Plant process commissioningObjective is to verify conformance to PCDH and prepare for Plant System process commissioning, whilst also ensuring that all I&C software and configuration data is available under version control.
10. I&C Integration Process ProcessPackage 1:Verification of I&C documentation, deliverables, configuration data and software
11. I&C Integration Process ProcessPackage 2:Integration of Plant System I&C configuration in the Temporary Control Room services, for all tiers(*)(*) Detailed procedure exists today for conventional, work in progress for CSS-OS and CIS
12. I&C Integration Process ProcessPackage 3:Verify I&C cubicles are ready for power and network interconnection, incl. provisions under French legal framework
13. I&C Integration Process ProcessPackages 4 and 5:Configuration and inter-connection of network elements
14. I&C Integration Process ProcessPackage 6:Configuration and verification of all Plant System I&C controllers from version controlled sources(*)(*) Slow, fast controllers, Commercial-Off-The-Shelf (COTS) or bespoke configurable components
15. I&C Integration Process ProcessPackage 7:Verification of Plant System I&C function and performance using central I&C services
16. Configuration Control ProcessAim of having robust configuration control for conventional controllers/servers/services is achieved by using RPMRPM relate to uniquely identified tags in ITER Version Control System through naming conventionsSingle configuration item for each configurable componentI&C applications which are (expected to have been) tested on Mini-CODAC need to be adapted to the integrated production environment (integrated alarm, archive, OPI navigation system)Server and services roles, applications, configuration, any post installation patch required is managed in single host-specific meta-package through RPM deployment, RPM dependency management and scriptlets.
17. Configuration ControlProcess
18. Configuration ControlProcess
19. Configuration ControlProcess
20. Temporary Control Room (TCR)Status
21. Temporary Control Room (TCR)StatusHuman Machine InterfaceData handling including archiving, storage and accessInter plant communicationRole-based and condition-based access controlAlarm handling including SMS/email notificationTime synchronizationElectronic logbookAccess to central software repository and issue tracking (configuration control)Development stations for software updates (fast turn-around)Central supervision and monitoringArchive data access from officeThese functions are not all provided by mini-CODAC making plant system commissioning much smoother with TCR
22. First milestone – 17-19 Sept. 2018Status
23. I&C Integration Status as Q2-2021Integrated units as of April 2021SSEN Batch 1 completeHV, MV and LV load centersBuilding services and utilitiesB3x, B61, LGA in B33 and B61Secondary cooling waterHRS, H2, 2D in progressFunctional integration consisted in providing integrated operation (role-based access control) from TCR and missing domain-specific monitoring functionSynthesis of operationally relevant alarmsConsolidated temp. fire monitoring, plant state monitoring (COS per LC, SSEN overall), etc.We estimate we have achieved 5% of the scopeStatus
24. Electrical supply and distribution
25. Secondary Cooling WaterSCW I&C was conceived in a monolithic way that did not match with the staged integration needsInvolved since 2019 with the control system re-design to prepare for early operation13 cubicles redesigned, rewired and documentation updated5 cooling loops under development (~2500 hardwired signals)3 cooling loops commissioning since last year and 1 more forecasted6 I&C engineers engaged for more than 1.5 years alreadyHRS water sampling and release I&COrphaned scope to be re-born from hashesStatus
26. I&C Integration status as Q2-2021Status
27. EPICS gatewaysGateways between CTRL and Plant System networks.. providing role-based access controlAnybody has read access, restricted writeUAG members are retrieved from CODAC OpenLDAP during bootstrap.. and condition-based access controlConditions in the machine further restricting write accessE.g. equipment maintenance mode, use of local panels, etc.And between control zone and external worldPairs of gateways with UNICAST links across firewallBoth Channel Access and pvAccessEarly adoption of pvAccess gateway (no ACL) Will be migrated when pvAccess is used for more than monitoring and archiving
28. EPICS gateways
29. Temporary Main Control Room (T-MCR)Status
30. SW for central I&C functionsCentral I&C control and monitoring software functions are primarily built using pvAccessExcept soft IOC with soft records database and link plumbing are deployed for operationally relevant alarm synthesisLow processing complexity and current limitations with CS-Studio BeastTwo application software frameworks are currently in useAdoption of MARTe2 for monitoring and data processing functionsEPICS Collaboration Meeting June 2019 (3-June 7, 2019): Interfacing MARTe2 to the EPICS Channel Access and pvAccess protocolsIn-house development of sequencer for configuration and automationProviding Behavior Tree semantics and EPICS7 integration through plug-insICALEPCS 2021 (18-22 October 2021): VanHerck et al... supported with ITER HDF5 archiver extended to pvAccessSpring 2021 EPICS Collaboration Meeting: Archiving and accessing PVA data at ITER
31. Lessons learntCompliance assessmentIncomplete functional scope deliveredCWS architecture and HW changes, SW adaptations/developments requiredUtilities which are only partially operable through (at times) inaccessible field touch panels Missing deliverablesTo a point where we are not in position to effectively perform corrective maintenanceSingle-source situation by reason of high reliance on non-standard components or non delivery of sources/toolsNo spares for non-standard componentsNo as-built documentationLessons learnt
32. Lessons learntAlarm handlingNoise, repetition, avalanche, little operational relevance, wrong logic (at times hard-coded), no synthesis, etc.Being iteratively disabled/dis-integrated/re-builtConfiguration control process is not as agile as necessary given the large number of iterations during SAT and the poor quality of the deliverablesIterations, regression, roll-back, etc.This is the nature of these project phases and we should not try and apply the final process already nowLessons learnt
33. ForecastForecast
34. ForecastForecast
35. Forecast 2021Perfective and corrective maintenance of building and utilities I&CAddressing current NCR on operability and maintainabilityAlarm handling re-designContinuous integration and configuration control process improvement Inhomogeneity across deployed CCS versions means we need to have more loosely coupled build/packaging/deployment to be able to cope with the planned work with our current resourcesB13/17, B52 building services Which are a concern as to amount of I&C integration resources required to support SAT and transition to early operationCC2B, CC2C, start-up of CWSForecast
36. Forecast 2021PPEN, RPC&HF (and CPS eventually)Real-time control function (requires TCN) with feed-forward from predicted client loads (requires SDN)First Plant System requiring as well PIS-CIS integrationNeed to manage cross-dependencies between CPS and RPC&HF as to operating modes, configuration to make sure they collaborate seamlesslyUsing CODAC Supervision and Automation system (SUP) sequencer and Configuration Verification and Validation Framework (CVVF)Start-up of ECH components commissioningData archive and visualizationHigh number of procedures, operation modes, components and plant configurations to be verified and automated using SUPForecast
37. Forecast 2021CryogenicsExpecting high level of design maturity and design executionProspective challenges areLarge scale, process complexity, multiple cross Plant System I&C interfaces that all have to seamlessly interplay for overall operation, function and performanceForecast
38. ConclusionsProcess established, demonstrated, and continuous improvement initiative is in placeGood successes in the past and foundation for the futureAddressed already identified scalability issues with central servicesWe are much more heavily involved than originally foreseenCases of low quality or maturity, incomplete functional scopeConcernsWe are already challenged (quality/schedule/manpower) with facing the low-criticality and low-complexity I&C systems deliveryI am personally worried about our ability to cope in the next phases with higher-criticality and higher-complexity I&C systems if I extrapolate from lessons learnt over the last 2.5 yearsConclusions
39. Thank you for your attention
40. Supporting Plant System I&C designDiagnostics I&C methodology, workshops and reference systemsMethodology based on principles established by the INCOSE and supported by Enterprise Architect (EA) add-insAnnual workshops involving all DA and large set of I&C teamsProcess
41. Diagnostics I&C methodology, workshops and reference systemsReference systems provided and used to demonstrate best practice and integration with CODAC Operational ApplicationsNeutron diagnostics (PXIe and MTCA.4)Imaging diagnostics (PXIe, MTCA.4 under development)Thompson Scattering (MTCA.4, Remote IO)Supporting Plant System I&C designProcess
42. S7-1500 components, OPC UA, standard PLC control library, design methodology and code generation toolBuilding upon the ACL (F4E) and ICL (PBS34) initiativesHomogeneous architecture, re-usable components, PLC code and documentation generation, CI, etc.Setting (and developing new) I&C standardsProcess
43. Electrical supply and distribution
44. Electrical supply and distribution