technical overview 26th June 201 3 Wojciech Sliwinski BECOIN for the BECO Middleware team Felix Ehm Kris Kostro Joel Lauener Radoslaw Orecki Ilia Yastrebov ID: 785889
Download The PPT/PDF document "Controls Middleware renovation" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Controls Middleware renovation –technical overview26th June 2013
Wojciech Sliwinski
BE-CO-IN
for the
BE-CO
Middleware
team:
Felix Ehm, Kris Kostro, Joel Lauener,
Radoslaw Orecki, Ilia Yastrebov
, [Andrzej Dworak]
Special thanks to
: Vito Baggiolini and Pierre Charrue
Slide2AgendaContext & Motivation for RenovationMiddleware Review processTechnical evaluation of the transport layer
Changes
in the MW Architecture in LS1
MW Upgrade milestones in 2013Conclusions
2
Slide3AgendaContext & Motivation for Renovation3
Slide4Motivations for MW RenovationCurrent CORBA-based CMW-RDAIntegrated in the Control systemUsed to operate all CERN acceleratorsProvides widely accepted Device/Property model > 10 years oldWhy to review & upgrade MW ?
CORBA was
choosen
15 years ago
Technical limitations
of CORBA-based transport
Functional limitations
of the current CMW-RDA
Codebase with
long history difficult to maintain, needs architecture reviewMajor issue of long-term support & future evolutionEvolution of technology over last 10 years: HW, OS, middleware, 3rd party libraries Human factor less & less CORBA expertise on the market
4
Slide5Technical limitations of CORBA transportBecame legacy, not actively supported maintenance issueShrinking community,
slow
response time
omniORB (C++) – 1 developer/
maintainer
,
last
release mid-2011JacORB (Java) – few developers, small communityMajor technical limitationsLack of
fully
asynchronous processing channelBlocking communication infamous JacORB blocking issueLack of low-level control of IO resources (sockets, request queues)Development issuesDifficult to extend the wire protocol Backward compatibility issueComplex, error prone APIHeavy in memory usage
5
Slide6Summary: Why change CORBA?CORBA was choosen 15 years agoNot actively
maintained
big risk for the MW project
Better
solutions
exist
on the marketInvest in future solution rather than maintaining old oneWith current CORBA-based middleware we
can’t
solve the pending operational issuesWe can’t provide better scalability & reliabilityCMW-RDA is difficult to evolve & extend6
Slide7AgendaMiddleware Review process7
Slide8Middleware Renovation process MW Renovation = MW Review + MW UpgradeMW Review aims to provide the most appropriate
technical
s
olution satisfying the user requirementsMW Upgrade
establishes
the
plan &
strategy
for
introduction of the new MWObjective: LS1 the unique opportunity for the major MW upgradeMiddleware Review ProcessGathering of users feedback and requirements (2010-11)
Review of communication and serialization libraries (2011-12)
Prototyping using selected communication products (2012)
Design & impl. of new RDA3: Data, Client & Server (2012-13)Testing & validation of core MW infrastructure (summer’13)Upgrade of all dependent MW libraries & services (2013-14)JAPC, Directory Service, Proxy, DIP Gateway8
Slide9Review of users requirements2010-11 – series of interviews with major users
Lars
Jensen
, Stephen Jackson (BI)Andy Butterworth, Frode Weierud, Roman Sorokoletov (RF)Brice Copy, Clara Gaspar
(DIP, DIM)
Frederic
Bernard,
Herve
Milcent, Alexander Egorov (PVSS)Alexey Dubrovskiy (CTF), Kris Kostro (DIP gateways)Marine Gourber-Pace, Nicolas Hoibian (Logging) Nicolas De Metz-Noblat (Front-Ends), Alastair Bland (Infrastructure)Michel Arruat (FESA), Stephen Page (FGC)Niall Stapley, Mark Buttner, Marek Misiowiec (LASER & DIAMON) Nicolas Magnin, Christophe Chanavat (ABT)
Stephane
Deghaye, Jakub Wozniak (InCA, SIS)
Vito Baggiolini, Roman Gorbonosov (JAPC & DA systems)+ regular feedback from OP+ internal team inputhttp://wikis/display/MW/Interviews+with+Experts 9
Slide10New RDA3: Accepted requirementsGeneralJava & C++ API, Win (64-bit) & Linux (SLC5 32-bit & SLC6 64-bit)Accelerator Device Model (i.e. Device/Property)Get, Set,
Async
-Get,
Async-Set, SubscribeEarly detection of communication failuresImprove error reporting in all the layers: client, server, gatewaysAdmin interface & runtime diagnostics & statistics
Data support
Data
object: primitives,
n-dim arrays, data structures
Subscription mechanism
Subscription behaviour the same regardless condition of the server (active, down)Several client subscription policies (default: continuous)Provide subscription notification orderingFirst-Update
enforced via CMW
on
server-sideProvide callback to front-end framework for the server-side GetDrop support for on-change flagStandardise use of subscription filters and update flags (e.g. immediate update)Add header
for
acquired
Data
common
metadata
(
e.g
.
acq
. stamp, cycle name)All loss of data (dropped updates) must be notified to clients
10
New
requirement
Slide11New RDA3: Accepted requirementsClient sideRDA3 client API connects with both: RDA2 (old) & RDA3 (
new
)
serversEfficient mechanism
for:
connection
,
disconnection
&
reconnectionMust be able to recover from any interruption of communication with the serverServer restarts, IP address change,
rename
/
move of a device to another serverImproved semantics of Array Calls, i.e. handling of individual parametersEnhanced diagnostics & collection of statistics Server sidePolicies for discarding notifications, i.e. deal with overflows and ’bad clients
’
Instrument with
counters
&
timings
allowing
to
diagnose
the
notifications
delivery
Prioritisation
of Get/Set requests for high-priority clientsServer-side subscription tree fully managed by CMWServer does not need to manage client
subscriptions any more
Manage the client
connections
,
e.g. forced disconnect of a clientClient lifetime callbacks (i.e. connected, disconnected)
11
New
requirement
Slide12New RDA3: Accepted requirementsServer side (cont.)
Client
discovery
for the diagnostics purposes (i.e. connected
clients
with
payload
)
Enhanced diagnostics & collection of statisticsOngoing discussions (not accepted yet)
Prioritisation
of
subscription notifications for high-priority clientsTechnical notesInvest in asynchronous & non-blocking communicationPrefer 0-copy & lock-free data structures, message queueshttp://wikis/display/MW/Design+of+New+RDA12New requirement
Slide13New RDA3: Summary of requirementsUnchangedDevice/Property modelSet of basic operations (Get, Set, Subscribe)
Fixes
&
improvementsSubscription mechanism
Connection management
Diagnostics
&
statistics
New
functionality Policies for subscription management (client & server)Client prioritiesServer-side subscription treeExtended Data supportStandardise
First-Update
concept 13
Slide14AgendaTechnical evaluation of thetransport layer
14
Slide15Middleware transport requirements15Desirable
Mandatory
Fundamental
Lightweight
Friendly API, documentation
Request/reply & pub/sub patterns
Open source license
Asynchronous
Active community
Stability, Maturity & Longevity
Performance & Scalability
C++/Java
Linux/Windows
Over TCP/IP LAN
Slide16Evaluated middleware products16Ice
Thrift
omniORB
YAMI
OpenSpliceDDS
OpenAMQ
CoreDX
RTI DDS
ZeroMQ
QPid
MQtt
RSMB
JacORB
Mosquito
All
opinions
are based only on
our knowledge
and
evaluation
. Each of the products, depending on the requirements, may constitute a good solution.
RabbitMQ
Andrzej Dworak, ICALEPCS 2011
Slide17Products comparison (according to the criteria)17
Sync,
async
&
msg
patterns
QoS
Dependencies
& memory f-p
Performance
Look & feel,
API, docs
Community & maturity
Score
ZeroMQ
6
Ice
5
YAMI4
4
RTI
3
Qpid
3
CORBA
2
Thrift
2
Andrzej Dworak, ICALEPCS 2011
Slide18ConclusionsSeveral good middleware solutions availableThe choice is dictated by the most critical requirementsNot easy performance matters but also ease of use, community, …
Prototyp
ing
was done with the most promising candidates:
ZeroMQ
,
Ice
& YAMI
Finally we decided to choose ZeroMQ (http://www.zeromq.org/)Asynchronous & non-blocking communication0-copy & lock-free data structures
,
message
queuesNice API, good documentation & active community18
Slide19New RDA3 Java – Sync Get round-trip time19Test setup: 1kB message
payload
,
cs-ccr-* machines
, 1
server
host & 10
client
hosts
Slide20New RDA3 Java – subscription notification latency20Test setup: 1kB message
payload
,
cs-ccr-* machines, 1
server
host & 10
client
hosts
Slide21New RDA3 Java – subscription notification latency21Test setup: 1kB message payload
,
cs-ccr
-* machines, 1 server
host & 10
client
hosts
Slide22AgendaChanges in the MW Architecture in LS122
Slide23Current MW Architecture
User written
Middleware
Central services
Physical Devices (
BI, BT, CRYO, COLL, QPS, PC,
RF
, VAC, …
)
Java Control
Programs
RDA Client
API (C++/Java)Device/Property Model
Directory
Service
Configuration
Database
CCDB
VB, Excel,
LabView
Servers
Clients
Virtual Devices
(
Java)
PS-GM
Server
FESA
Server
FGC
Server
PVSS
Gateway
C++ Programs
More
Servers
Administration
console
Passerelle
C++
CMW
Infrastructure
CORBA-IIOP
RDA
Server
API
(C++/Java)
Device/Property Model
RBAC A1
Service
Directory
Service
RBAC
Service
JAPC API
CMW
integr.CMW int.CMW int.CMW int.CMW int.
CMW int.23
Slide24Changes in MW Architecture in LS1
User written
Middleware
Central services
Physical Devices
(
BI, BT, CRYO, COLL, QPS, PC,
RF
, VAC, …
)
Java Control
Programs
RDA Client API (C++/Java)
Device/Property Model
Directory
Service
Configuration
Database
CCDB
VB, Excel,
LabView
Servers
Clients
Virtual Devices
(
Java)
PS-GM
Server
FESA
Server
FGC
Server
PVSS
Gateway
C++ Programs
More
Servers
Administration
console
Passerelle
C++
CMW
Infrastructure
ZeroMQ
RDA
Server
API
(C++/Java)
Device/Property Model
RBAC A1
Service
Directory
Service
RBAC
Service
JAPC API
CMW integr.CMW int.CMW int.CMW int.CMW int.
CMW int.Upgrade in LS124
Slide25AgendaMW Upgrade milestones in 2013
25
Slide26MW Upgrade Milestones in 2013MilestoneCompleted by ?
RDA3 Java (
client
/server)
(
alpha
)
June’13
RDA3 C++
server (alpha)July’13
RDA3
integration
with: FESA, FGC, PVSSJuly-Oct’13RDA3 C++/Java (client/server) validatedSeptember’13New JAPC release with RDA3 JavaSeptember’13New FESA3.2 release with RDA3December’13
26
July’13
July-Oct’13
September’13
Winter’13/14
August’14
December’13
End-of-Life for RDA2: LS2
Slide27MW Upgrade strategy in LS1 and towards LS2No BIG-BANG migration but gradualBackward compatible (connection-wise)
new
RDA3 client libraryNew RDA3 clients can communicate
with RDA2 & RDA3
servers
FESA3
will
exist with both: old RDA2 (FESA3.1) and new RDA3 (FESA3.2) 27
Old
JAPCOld RDA2serverFESA2.10FESA3.1Old RDA
2
server
New RDA
3
server
FESA3
.2
Old RDA
2
client
New
JAPC
New RDA
3
client
R
DA
2
RDA3
Gateway
Client
apps
will
migrate
during
LS1
Only
for
justified
,
exceptional
cases
FEC
developers
should
migrate to FESA3.2 ASAP
Slide28ConclusionsWe have to replace CORBA with a new solution We collected updated
users
requirementsMW upgrade will be performed
during
LS1 (2013-2014)
Interoperability
between RDA2 RDA3Gradual control system migration until LS2 (end-2017)End-of-Life for RDA2: LS228