Example of DAQ Trigger issues for the SoLID experiment - PowerPoint Presentation

Slide1

Example of DAQ Trigger issues for the SoLID experiment

Alexandre Camsonne

Workshop on

Future

Trends in

Nuclear Physics Computing

March 17

th

2016

JLab DAQ specifics

Almost

continuous

machine : 499 MHz repetition rate usually prevents to trigger on beam crossing like at collider usually trigger on detectorHigh luminosity up to 1039 cm-2s-1Typical scale of experiments if of order of 100 M$ ( different from HEP and CERN LHC )Several different setup run for a few years

Workshop on future trends in Nuclear Physics Computing DAQ Trigger

2

3/17/2016

Trigger goals

Reduce rate so it can be handled by the electronics

Front end dead time

Data transferReduce data sizeOnly record event of interestImprove signal to backgroundWorkshop on future trends in Nuclear Physics Computing DAQ Trigger

3

3/17/2016

L3

Drive

Silo

VME

100 MB/s

Ethernet

100 MB/s

SAS

250 MB/s

DLO6

Tape

250 MB/s

Calo

Cerenkov

Scintillator

GEM

APV

MPD

FADC

CPU

Event

Builder

APV transfer

8

0 MB/s

VME

100 MB/s

CPU

DAQ bottle necks

CPU

Data readout

Workshop on future trends in Nuclear Physics Computing DAQ Trigger

4

3/17/2016

Example SoLID PVDIS experiment

Inclusive electron scattering

Calorimeter and Gas Cerenkov

200 to 500 KHz of electrons

Maximum

1.8 MHz total

rate

30 individual sectors

to reduce rate per sector to 60

KHz maximum

Workshop on future trends in Nuclear Physics Computing DAQ Trigger

5

3/17/2016

FADC readout full waveform 10 samples Only want to readout FADC channel in the cluster to reduce number of channels readout because of background in case zero suppression does not work

CTP generates a 64 bit pattern

Send pattern to TI or FADC directly to trigger FADC

Only channels from pattern are put in bufferFADC readoutWorkshop on future trends in Nuclear Physics Computing DAQ Trigger

6

3/17/2016

ECAL trigger

Workshop on future trends in Nuclear Physics Computing DAQ Trigger

7

3/17/2016

Detector segmented in 30 sectorsOne crate per sector

Calorimeter Geometry

CTP

CTP

Workshop on future trends in Nuclear Physics Computing DAQ Trigger

8

3/17/2016

CTP connections

To neighbor CTP

To neighbor CTP

Workshop on future trends in Nuclear Physics Computing DAQ Trigger

9

3/17/2016

Neighboring sectors

CTP

CTP

CTP

New CTP : has two

additional

optical links

Can send Cerenkov and

c

alorimeter edges to other sectors.

8 Gbp/s optical link

8 Gbp/s optical link

8 Gbp/s optical link

36 calorimeters

9 Cherenkov

=

150ns + 5 ns per m+ 300 ns ( data ) = 500 ns

overhead

Trigger decision = 500 ns (Transfer ) + 1us ( clustering ) < 4 us (APV)

Workshop on future trends in Nuclear Physics Computing DAQ Trigger

10

3/17/2016

PVDIS electron trigger

Coincidence ECAL and Gas Cerenkov

Workshop on future trends in Nuclear Physics Computing DAQ Trigger

Singles ECAL

286 KHz

Singles rates Cerenkov

2 MHz

Accidental 30

ns

16.6 KHz

DIS electron

10.4 KHz

Total rate

27 KHz

11

3/17/2016

Event size FADC PVDIS with waveform

Detector

Total number of channels

Number of channels

firing

Number

of samples

Max size detector

bytes

Minimal size detector

bytes

Typical

size

Shower

58

7

10

2784

336

772

Preshower

58

7

102784336

772Gas Cerenkov9

310432144

432

Max total size

46KB0.816KB

1.544KBMax rate Assuming

100 MB/s per crateOne crate2.1 KHz

121 KHz60 KHz

FADC data rate for 30 KHz = 60 MB/s

Workshop on future trends in Nuclear Physics Computing DAQ Trigger

123/17/2016

GEM readout

APV25 Front GEM ASICs

Up to 164 000 channels

APV 25 : 128 channelsReadout VME based readout : 16 APV25 = 2048 channels( ~ 10 $ / channels )SRS readout : ethernet /PC based = 2048 channels( ~ 3 $ / channels )1 crate per sectors for FADC and GEM

13

Workshop on future trends in Nuclear Physics Computing DAQ Trigger

3/17/2016

APV25 readout

Switch Capacitor Array ASICS with buffer length 192 samples at 40 MHz :

4.8 us Look back 160 samples : 4 us

APV readout time : t_APV = 141 x number_of_sample / 40 MHz

t_APV(1 sample) = 3.7 us.Max rate APV front end :

270 KHz in 1 sample mode90 KHz in 3 samples modeWill be triggered at max 60 KHz in 3 samples

100KHz Max in 1 sample

Optional on chip deconvolution

Front-end FPGA deconvolution being implemented

Deadtimeless pipelined electronics / parallel read and write

Workshop on future trends in Nuclear Physics Computing DAQ Trigger

14

3/17/2016

GEM event occupancy and size

Sector

Rate

XY

Bytes

3

samples

( bytes)

0

199

398

1592

4776

1

147

294

1176

3528

2

107

214

856

2568

3

102

204

816

2448

4

102

204

816

2448

Total hits / sector

657

1314

5256

15768

Data rate / sector

3

0000

157680000

473040000

Data rate ( sector Mb/s)

157,68

473.1

Total

detector ( x30)

19710

4730.4

14191.2

Occupancy detector

0.14

Data rate to front end reading 3 samples

Use 4 Gigabit link = 512 MB/s not for safety

Workshop on future trends in Nuclear Physics Computing DAQ Trigger

15

3/17/2016

GEM event occupancy and sizeafter deconvolution

Sector

Rate

XY

Bytes

3

samples

( bytes)

0

23

46

184

552

1

12

24

96

288

2

10

20

80

240

3

9

18

72

216

4

9

18

72

216

Total hits / sector

63

126

504

1296

Data rate / sector

3

0000

1512000

77760000

Data rate ( sector Mb/s)

15.12

45.36

Total

detector (x30)

1620

453.6

1360.8

Occupancy detector

0.013

Rates with deconvolution 3 samples readout

Implementation in readout electronics

No issue for transfer up to 60 KHz ( 90 MB / s )

Data after processing going to tape

Workshop on future trends in Nuclear Physics Computing DAQ Trigger

16

3/17/2016

Future chip

VMM family

Zero suppression

ADC and TDC on chipLogic function with input from other chipsReduce data on the ASIC stage and then on front end stage using FPGA before sending to readout controllerWorkshop on future trends in Nuclear Physics Computing DAQ Trigger

17

3/17/2016

ROC

ROC

ROC

ROC

ROC

ROC

ROC

ROC

ROC

Front-End Crates

R

ead

O

ut

C

ontrollers

~60 crates

~50MB/s out per crate

EB1

Event Builder

stage 1

EB1

Event Builder

stage 1

EB1

Event Builder

stage 1

EB2

Event Builder

stage 2

EB2

Event Builder

stage 2

Staged Event Building

blocked event fragments

partially recombined event fragments

N x M array of nodes (exact number to be determined by available hardware at time of purchase)

Level-3 Trigger and monitoring

full events

L3 Farm

node

node

node

node

node

node

node

Raid Disk

ER

Event Recorder

Event Recording

3

00MB/s in

300MB/s out

All nodes connected with 1GB/s links

Switches connected with 10GB/s fiber optics

L3 Farm

Workshop on future trends in Nuclear Physics Computing DAQ Trigger

18

3/17/2016

PVDIS data rates

Simulation and trigger were checked and optimized

T

rigger rate estimated to be 27 KHzGEM data rate assuming 30 KHz after deconvolution around 46 MB/s FADC data 60 MB/s Total about 108 MB/s per sector to L3 x 30 for a total 3.24 GB/s Use L3 to reduce to 250 MB/s ( similar to Hall D )Workshop on future trends in Nuclear Physics Computing DAQ Trigger

19

3/17/2016

L3 data reduction

Can use slow detectors that cannot be used at L1

Pile

up detectionOnly record sample for event with pile up Calorimeter clusteringGEM readout and trackingTiming cutClusteringCrude trackingTracking

Improved timing to reduce accidentals

Workshop on future trends in Nuclear Physics Computing DAQ Trigger

20

3/17/2016

Tape size

Workshop on future trends in Nuclear Physics Computing DAQ Trigger

Days

Data rate

Seconds

Total data TB

Double

DLO5 in $

DLO6 in $

E12-11-108

Pol proton

120

250

10368000

2592

5184

259200

155520

E12-12-006

J/Psi

60

250

5184000

1296

2592

129600

77760

E12-10-006

Transv. Pol. 3He

90

250

7776000

1944

3888

194400

116640

E12-11-007

Long. Pol. 3 He

35

250

3024000

756

1512

75600

45360

E12-10-007

PVDIS

169

250

14601600

3650.4

7300.8

365040

219024

Total

474

40953600

10238.4

20476.8

1023840

614304

21

3/17/2016

Conclusion

SoLID requires high rates low dead time, flexible trigger capability

Use of JLab pipeline and CODA3 electronics allows almost deadtimeless electronics and makes this experiment possible

Serial readout and on board processing are additional available toolsL3 farm capability similar to HEP in smaller scale gives more flexibility on the triggerLimiting factor : readout speed of the data from the modules and tape priceWorkshop on future trends in Nuclear Physics Computing DAQ Trigger

22

3/17/2016