on behalf of the CMS DAQ group Overview Overview of CMS DAQ for LHC run II Eventbuilding protocol Performance me asurements 2 CMS DAQ for LHC Run II Requirements 100 kHz level 1 trigger rate unchanged ID: 787413
Download The PPT/PDF document "A Ne w Event Builder for CMS Run II" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
A New Event Builderfor CMS Run II
on behalf of the CMS DAQ group
Slide2OverviewOverview of CMS DAQ for LHC run II
Event-building protocol
Performance me
asurements
2
Slide3CMS DAQ for LHC Run IIRequirements
100 kHz level 1 trigger rate (unchanged)
Event size might double to 2 MB
Increase in
pileup
New detectors
Accommodate legacy and new uTCA-based detector readouts1-2 kB or 8 kB fragments Aging hardwareMost components reachedend-of-life cycleNew technologiesMyrinet widely used whenDAQ-1 was designedEthernet and Infiniband dominate the top-500 supercomputers today
3
Top500.org share by Interconnect family
Infiniband
DAQ1 TDR (2002)
Myrinet
1 Gb/s
Ethernet
10 Gb/s
Ethernet
2014
Slide4CMS Event BuilderDetector front-end (custom electronics)
Front-End Readout Optical Link (FEROL)
Optical 10 GbE TCP/IP
Data Concentrator switches
Data to Surface
Aggregate into 40 GbE links
72 Readout Units (RUs)Combine FEROL fragments into super-fragmentEvent Builder switchInfiniband FDR 56 Gbps CLOS network62 Builder Units (BUs)Event buildingWrite events to transient files on RAM diskFilter Units (FUs)Run HLT selection using files from RAM disk4
576 x 10 GbE
200 m
72 x 40 GbE
72 x 64 IB 56 Gbps
Slide5EvB Protocol
EVM
RU 1
RU 2
BU 1
BU 2
FEROLs
Fragment
Fragment
Assign event to BU1
Assign event to BU1
5
Event
Request
Event RequestEvent RequestFragmentSuper-fragment
Super-fragmentSuper-fragmentEventFragmentFragment
FragmentFragmentFragment
Slide6Achieving PerformanceAvoid high rate of small messages
Request multiple events at the same time
Pack data of multiple events into one message
Avoid
copying data
Operate on pointers to data in receiving buffers
Copy data directly into RDMA buffers of Infiniband NICsStay in kernel space when writing dataParallelize the workUse multiple threads for data transmission and event handlingWrite events concurrently into multiple filesBind everything to CPU cores and memory (NUMA)Each thread bound to a coreMemory structures allocated on pre-defined CPU
Interrupts from NICs restricted to certain coresTune Linux TCP stack for maximum performance
6
Slide7ComputersReadout Unit (RU)Dell PowerEdge R620Dual 8 core Xeon CPUE5-2670 0 @ 2.60GHz32 GB of
memory
Builder Unit (BU)
Dell PowerEdge R720Dual 8 core Xeon CPUE5-2670 0 @ 2.60GHz32+256GB of memory
(240 GB for
Ramdisk
on CPU 1)
Slide8Data Network40/56Gb NICs (Infiniband or Ethernet)Mellanox TechnologiesMT27500 Family [ConnectX-3]
10
/
40 GbE switchesMellanoxSX1024 & SX1036
Infiniband
switches
Mellanox SX6036Infiniband CLOS network
Slide9Performance Measurements
9
Slide10Data Concentrator
10
4
–
16
FEROLsBU 1
1 kB
1 RU
2 BUs
EVM
RU
BU 2
256B -
16kB
1 kB1 – 256 kB
Slide11Data Concentrator
11
Legacy
FEDs
uTCA
FEDs
Slide121 kB
1 kB
256B -
1
6
kB
EVM
Builder Unit
12
8 FEROLs per RU
44 RUs
2 - 128 kB1 BU
89 - 5633
kB
Slide13Builder Unit
13
Slide141 kB
2 - 128 kB
1 kB
256B -
1
6
kB
Scalability
14
8 FEROLs per RU
1 - 44 RUs
1 – 44 BUsEVM
3 - 5633kB
Slide15Scalability
15
Slide16SummaryCMS has a complete new DAQ system for LHC run II
State-of-the-art technology
Order of magnitude smaller than
previous DAQ
system
Optimal use of high-end hardware
New event-building protocolNew software to exploit hardware capabilitiesA lot of fine-tuning to get full performanceEvB scaling behavior not understood Study of Infiniband CLOS network under EvB traffic ongoingDAQ is ready for first physics dataMore work is needed to achieve the ultimate performance16
Slide17⬅⬆
Slide18BU Performance vs Threads
Slide19Scalability19