Sune Jakobsen on behalf of ATLASALFA Seminar at NBI 22062015 Trigger sources of latency in ALFA 2012 trigger system Total 2157 ns lt 2175 ns Time of Flight 800 ns Scintillator 3 ns ID: 815724
Download The PPT/PDF document "ALFA_CTPIN - a NBI hardware contrib..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
ALFA_CTPIN
- a NBI hardware contribution to ATLAS
Sune Jakobsen on behalf of ATLAS-ALFA
Seminar at NBI 22-06-2015
Slide2Trigger – sources of latency in ALFA
2012 trigger system
Total: 2157 ns
< 2175 ns
Time of Flight
800 ns
Scintillator
3 ns
Photomultiplier
6 ns
LEMO-cables (tunnel)
25 ns
MAROC2 (analog FE)
13 ns
Digital Front End
50 ns
LEMO-cables (USA15)
16 ns
MD/OD separation
60 ns
NIM_to_LVDS
5 ns
Air-core-cable
1030 ns
CTPIN logic
25 ns
CTP functionality
125 ns
Run2 without upgrade
Total: 2253 ns
> 2175 ns
Time of Flight
800 ns
Scintillator
3 ns
Photomultiplier
6 ns
LEMO-cables (tunnel)
25 ns
MAROC2 (analog FE)
13 ns
Digital Front End
50 ns
LEMO-cables (USA15)
16 ns
MD/OD separation
60 ns
NIM_to_LVDS
5 ns
CTP
PITbus
upgrade
Air-core-cable
1030 ns
CTPIN logic
25 ns
50 ns
CTP functionality
125 ns
New position of station
CTP OUT upgrade
25 ns
21 ns
Maximum latency for full ATLAS readout is:
87 BC = 2175 ns.
Upgrade needed for ALFA to trigger on level1! At least:
78 ns + margin
Latency sources
ALFA_CTPIN concept Early tests First module Revision 2 Timing Final latency First data Budget Thanks
237 m ATLAS IP
Cost in latency: 21 ns (
ToF
+ additional cable)
The B stations (furthers from ATLAS) have been move ~4 m away from the IP
Seminar at NBI 22-06-2015
ALFA_CTPIN - a
NBI hardware contribution to ATLAS
Sune Jakobsen
2
/26
Slide3Run2 with upgrades
Total: 2118 ns
< 2175 nsTime of Flight
800 ns
Scintillator
3 ns
Photomultiplier
6 ns
LEMO-cables (tunnel)
10 ns
MAROC2 (analog FE)
13 ns
Digital Front End
37.5 ns
LEMO-cables
(
USA15)
10 ns
ALFA_CTPIN
37.5 ns
Air-core-cable
1030 ns
CTP functionality
125 ns
New position of station
CTP OUT upgrade
25 ns
21 ns
Trigger – sources of latency in
ALFA with ALFA_CTPIN
2012 trigger system
Total: 2157 ns
< 2175 ns
Time of Flight
800 ns
Scintillator
3 ns
Photomultiplier
6 ns
LEMO-cables (tunnel)
25 ns
MAROC2 (analog FE)
13 ns
Digital Front End
50 ns
LEMO-cables (USA15)
16 ns
MD/OD separation
60 ns
NIM_to_LVDS
5 ns
Air-core-cable
1030 ns
CTPIN logic
25 ns
CTP functionality
125 ns
Run2 without upgrade
Total: 2253 ns
> 2175 ns
Time of Flight
800 ns
Scintillator
3 ns
Photomultiplier
6 ns
LEMO-cables (tunnel)
25 ns
MAROC2 (analog FE)
13 ns
Digital Front End
50 ns
LEMO-cables (USA15)
16 ns
MD/OD separation
60 ns
NIM_to_LVDS
5 ns
CTP
PITbus
upgrade
Air-core-cable
1030 ns
CTPIN logic
25 ns
50 ns
CTP functionality
125 ns
New position of station
CTP OUT upgrade
25 ns21 ns
Seminar at NBI 22-06-2015 ALFA_CTPIN - a NBI hardware contribution to ATLAS
Sune Jakobsen
3
/26
Latency sources
ALFA_CTPIN concept Early tests First module Revision 2 Timing Final latency First data Budget Thanks
Slide4ALFA_CTPIN board concept
Use electrical inputs on the new CTP_CORE+ (foreseen for topological triggers)
Intergrade the MD/OD separation
Integration near to the CTP
Add per bunch monitoring
Very low input/output time
Details in ATLAS note:
ATL-COM-LUM-2013-018
Add additional monitoring of rates
Latency sources
ALFA_CTPIN concept
Early tests First module Revision 2 Timing Final latency First data Budget Thanks
Seminar at NBI 22-06-2015 ALFA_CTPIN - a NBI hardware contribution to ATLAS
Sune Jakobsen
4
/26
Slide5ALFA_CTPIN board layout
Latency sources
ALFA_CTPIN concept
Early tests First module Revision 2 Timing Final latency First data Budget Thanks
Seminar at NBI 22-06-2015 ALFA_CTPIN - a NBI hardware contribution to ATLAS
Sune Jakobsen
5
/26
Slide6Seminar at NBI 22-06-2015 ALFA_CTPIN - a NBI hardware contribution to ATLAS
Sune Jakobsen
6
/26
2013: Forming of community for the ALFA_CTPIN
New manpower arriving at NBI (Jan) and good opportunity to immediately getting involved in a CERN project and establish connections to CERN.
Small, but highly important task with large visibility.
Agreement that NBI is responsible for the design, layout and production of the ALFA_CTPIN boards.
Niels Bohr Institute was already designing and producing the new ATLAS CTP_OUT modules (Henrik) and the design was used as a starting point for the ALFA_CTPIN board.
CERN is coordinating and testing.
A close collaboration is established with the ATLAS CTP group and in
particular the
engineer behind the CTP hardware.
Agreement made that ALFA_CTPIN is counted as an ATLAS upgrade and that the full cost (besides manpower which is counted as OTP class 3) count on the NBI contribution for phase 1 upgrade.
Shortly after the universities in Cracow committed to be responsible for the firmware, low level software and high level software.
Latency sources
ALFA_CTPIN concept
Early tests First module Revision 2 Timing Final latency First data Budget Thanks
Slide7ALFA_CTPIN board - approval
E-mail from David Francis 23-11-2013:
TDAQ welcomes the proposal to design and build an "ALFA_CTPIN interface board".
The implementation of such a board will not impact the Level-1 trigger performance.
A review of the board should be
organised
as soon as possible.
If you agree, TDAQ can take the lead in
organising
such a review.
Latency sources
ALFA_CTPIN concept
Early tests First module Revision 2 Timing Final latency First data Budget Thanks
Seminar at NBI 22-06-2015 ALFA_CTPIN - a NBI hardware contribution to ATLAS
Sune Jakobsen
7
/26
Slide8Seminar at NBI 22-06-2015 ALFA_CTPIN - a NBI hardware contribution to ATLAS
Sune Jakobsen
8
/26
2014: Early test
of small
parts of the design
at NBI - input
Test made on a LMH7322EVAL
board.
Measured delay 1.2
ns.
Datasheet delay: 0.700 ns.
Difference likely due to use of 200 MHz
oscilloscope.
Test of
LMH7322 to be used for converting the NIM signals to LVDS (FPGA in
).
Latency sources ALFA_CTPIN concept
Early tests
First module Revision 2 Timing Final latency First data Budget Thanks
Slide9Seminar at NBI 22-06-2015 ALFA_CTPIN - a NBI hardware contribution to ATLAS
Sune Jakobsen
9
/26
Test
made on a small custom build test
board.
Delay ~ 0.6
ns.
Rise and fall time ~1.5
ns.
Test of
SY100ELT24 to be used for converting the TTL (FPGA out) signals to
NIM.
The measurement's are likely biased by use of 200 MHz
oscilloscope.
2014: Early test
of small
parts of the design
at NBI - output
Latency sources ALFA_CTPIN concept
Early tests
First module Revision 2 Timing Final latency First data Budget Thanks
Slide10The conceptual design evolves
Latency sources ALFA_CTPIN concept
Early tests
First module Revision 2 Timing Final latency First data Budget Thanks
Seminar at NBI 22-06-2015 ALFA_CTPIN - a NBI hardware contribution to ATLAS
Sune Jakobsen
10
/26
Slide11Seminar at NBI 22-06-2015 ALFA_CTPIN - a NBI hardware contribution to ATLAS
Sune Jakobsen
11
/26
Design and layout
Due to time constrains it is decided to involve the CERN PCB workshop for the board layout.
A very close collaboration between NBI and the CERN PCB workshop
was
established.
From the conceptual design NBI makes the real board design.
To accommodate all the front panel connectors etc., a design with one main board and two daughter boards are made.
Latency sources ALFA_CTPIN concept
Early tests
First module Revision 2 Timing Final latency First data Budget Thanks
Slide12Seminar at NBI 22-06-2015 ALFA_CTPIN - a NBI hardware contribution to ATLAS
12/26
Production of the first ALFA_CTPIN module
Many parts
purchased though
CERN.
The production and mounting of the boards (fully organized by NBI) was
distributed to
3 companies:
Production a little delayed due to holiday period and problems at
HP
elektronik
montage
PCB3: PCB by
Printline
, mounting by NBI
PCB1 (main): C.B.
Svendsen
PCB2: PCB by
Printline
, mounting by
HP
elektronik
montage
Sune Jakobsen
2 modules produced.
Latency sources ALFA_CTPIN concept Early tests
First module
Revision 2 Timing Final latency First data Budget Thanks
Slide13Seminar at NBI 22-06-2015 ALFA_CTPIN - a NBI hardware contribution to ATLAS
13/26
Debugging for first module at
NBI
Several minor problems found and fixed in Copenhagen:
One major problem found: Not possible to program the FPGAs via the EPROM (used after power cuts).
Decision made to correct all known errors in the design and make new PCB layouts.
Termination of the compensator of the NIM inputs.
Termination of the NIM inputs.
Termination of the clock circuit.
Inversion of the clock outputs.
NBI committed to finance the new layout (at the CERN PCB workshop), the PCB production and mounting of components.
Important for commissioning: The first corrected module
could be
used for all tests.
Sune Jakobsen
Latency sources ALFA_CTPIN concept Early tests
First module
Revision 2 Timing Final latency First data Budget Thanks
Slide14Seminar at NBI 22-06-2015 ALFA_CTPIN - a NBI hardware contribution to ATLAS
14/26
First test at CERN in the ALFA laboratory
A dedicated test setup have been build in B251.
Numerous test of software, hardware and firmware made.
Software/Readout:
After the electrical test the modules were transported to CERN for further testing (October 2014).
5 front panels produced and first one mounted.
The phase of the input is shown in online histograms.
Values read out from the board with low level VME commands
.
A dedicated partition (like the ALFA standalone partition for the detector system) has been build.
The FPGAs where successfully programmed via J-TAG.
Sune Jakobsen
Latency sources ALFA_CTPIN concept Early tests
First module
Revision 2 Timing Final latency First data Budget Thanks
Slide15Seminar at NBI 22-06-2015 ALFA_CTPIN - a NBI hardware contribution to ATLAS
15/26
Early firmware
The work on the firmware started almost as the same time as the design.
Much work on optimization of internal timing.
Priority given to test of electrical parts of board (to check all before production of new boards).
MD/OD separation of signals.
Output of MD, OD and various internal signals for front
panel.
Measurement of the input phase using a 400 MHz clock (2.5 ns steep size).
Readout of values via VME bus.
The change of concepts is made to minimize input/output time and as to minimize the lines to the CTP (where before the number of LUTs in the CTP were minimized).
Input signal
FPGA interpret signal
MD output
OD output
Sune Jakobsen
Therefore the basic functionality was working only a few days after the first board arrived to CERN.
Latency sources ALFA_CTPIN concept Early tests
First module
Revision 2 Timing Final latency First data Budget Thanks
Slide16Seminar at NBI 22-06-2015 ALFA_CTPIN - a NBI hardware contribution to ATLAS
16/26
First test
w
ith the CTP_CORE+
In addition the stability of the data was checked (32 bits at 80
MHz) (not foreseen and ALFA_CTPIN board not optimized for timing yet)
The ALFA_CTPIN board was moved to the CTP laboratory and installed in there setup. Input signals generated by a
LTPi
.
The primary goal of the test was to test the electrical.
The electrical connection was verified (the CTPCORE+ could see the signals being input on the ALFA CTPIN board).
Some instabilities seen
=>
optimization of the timing in the ALFA_CTPIN board.
CTP_CORE+
ALFA_CTPIN
LTPi
Sune Jakobsen
Latency sources ALFA_CTPIN concept Early tests
First module
Revision 2 Timing Final latency First data Budget Thanks
Slide17Sune Jakobsen
Installation of revision 1 ALFA_CTPIN module in USA15
The connection to the CTP_CORE+ was established.
This
allowed for the next part of commissioning to start even without the final module.
While waiting for the revision 2 of the ALFA_CTPIN module to be produced, one of the revision 1 (prototype) was installed in USA15 and connected.
The old NIM crate was removed and the cables shorten and cables shorted and rerouted directly to the ALFA_CTPIN.
ALFA_CTPIN
CTP
First trigger signal passing though the ALFA_CTPIN from an ALFA detector to trigger ATLAS.
A first latency measurement without beam was made via the DAQ latency and
showed
~83 ± 1 BC, which was well below the hard cutoff of 87 BC.
Seminar at NBI 22-06-2015 ALFA_CTPIN - a NBI hardware contribution to ATLAS
17
/26
Latency sources ALFA_CTPIN concept Early tests
First module
Revision 2 Timing Final latency First data Budget Thanks
Slide18Sune Jakobsen
ALFA_CTPIN revision 2
Final boards produced by NBI (Copenhagen).
Boards basic functionality tested at NBI and was working.
Transport to CERN (
by
retired professor) unfortunately damaged two of the boards.
Spare capacitors send from NBI and the boards were repaired
at CERN.
All functionality checked and worked on module #2.
Module #1 and #3 does not load data from the
eprom
.
The problem was the order the components on the board was powered (the
voltage
generated on the board delayed compared to the voltages coming
directly).
Module #1 and #3 was fixed at NBI by NOT using the direct 3.3 V from the crate,
but instead
the 3.0 V generated from the board.
Module #2 installed in USA15.
Seminar at NBI 22-06-2015 ALFA_CTPIN - a NBI hardware contribution to ATLAS
18
/26
Latency sources ALFA_CTPIN concept Early tests First module
Revision 2
Timing Final latency First data Budget Thanks
Slide19Sune Jakobsen
Timing – Phase of each detector
Interval without jitter MUCH larger than in Run1 (due to new trigger PMFs).
Optimized for all detectors in first collisions at injection energy
.
The phase of each detector needs to be optimized to the clock
. (If not done correctly the
air-core trigger signal jitter). The scan is done via the TTC system.
Seminar at NBI 22-06-2015 ALFA_CTPIN - a NBI hardware contribution to ATLAS
19
/26
Delay of 0 ns
Delay of 2.5 ns
Delay of 5 ns
Delay of 7.5 ns
Delay of 10 ns
Delay of 12.5 ns
Delay of 15 ns
Delay of 17.5 ns
Delay of 20 ns
Delay of 22.5 ns
Delay of 25 ns
The check if phase is also optimal for MAPMTs pending (need much more beam).
The optimal phase is loaded by the DAQ system at “CONFIGURATION
”.
In Run1 the noise level for MDs was sharp 0 Hz.
Measurement attempted with circulating beam, but made impossible due to a <1 Hz random noise level on all detectors.
Radiation, light leaks or ?
Noise is not from
PMTs.
Latency sources ALFA_CTPIN concept Early tests First module Revision 2
Timing
Final latency First data Budget Thanks
Slide20Sune Jakobsen
Timing – Signal arriving time equalization
All signals
are
now arriving within
~1 ns.
The optimization was performed without beam, which was possible due to the synchronized LED system.
After the phase optimization the trigger the arrival time of the trigger signals to the ALFA_CTPIN module was equalized with cables delays.
Seminar at NBI 22-06-2015 ALFA_CTPIN - a NBI hardware contribution to ATLAS
20
/26
B7L1U
B7L1L
A
7L1U
A7L1L
A7R1U
A7R1L
B7R1U
B7R1L
10 ns
Latency sources ALFA_CTPIN concept Early tests First module Revision 2
Timing
Final latency First data Budget Thanks
Slide21Sune Jakobsen
Timing – Phase of ALFA_CTPIN
However optimizing the phase can gains up to ~25 ns in latency.
The excellent timing in the ALFA_CTPIN (after
lots
of hard work
on the firmware side)
makes it almost immune jitter.
ALFA trigger and timing ALFA general meeting in Copenhagen 11-05-2015
21
/26
The scan was performed by adding LEMO cables to the clock input.
Minimal latency with delay of 15.5 ns.
19 ns used as final value to account for differences of channels and contingency.
Delay of 2 ns
Delay of 4 ns
Delay of 6 ns
Delay of 8 ns
Delay of 10 ns
Delay of 12 ns
Delay of 14 ns
Delay of 15 ns
Delay of 15.5 ns
Delay of 16 ns
Delay of 18 ns
Delay of 20 ns
Delay of 22 ns
Delay of 24 ns
Air-core
LEMO-out ALFA_CTPIN
Latency sources ALFA_CTPIN concept Early tests First module Revision 2
Timing
Final latency First data Budget Thanks
Slide22Sune Jakobsen
Latency
The final latency was found with circulating beam in LHC: 84 BC (75 ns margin to the absolute limit).
The ALFA_CTPIN board was timed in with the CTP by sending a fixed pattern from memory on the ALFA_CTPIN board and see the pattern arrive correctly on the CTP_CORE+ (possible due to largely evolved firmware and software).
Shortest ATLAS latency without ALFA ~76
BC,
but L1topo
muon
are still not included.
From
A
TLAS weekly RC 28-04-2015 by Till
Eifert
:
Chocking first results from first complex
deadtime
settings: 5.5-11 % at 100
kHz, with the high number being with ALFA latency (plot missing as they are no yet pubic).
Likely ATLAS will run with the shorted possible latency to minimize dead time and only change to the ALFA latency for special runs.
ALFA trigger and timing ALFA general meeting in Copenhagen 11-05-2015
22
/26
Latency sources ALFA_CTPIN concept Early tests First module Revision 2 Timing
Final latency
First data Budget Thanks
Slide23Sune Jakobsen
First data taking and outlook
From a trigger point of view all worked.
ALFA was part of the second week of LHC Run2, which was special physics with
LHCf
.
The ALFA_CTPIN module insured stable level 1 triggering and thereby the possibility to take unique data combining ATLAS-ALFA-
LHCf
.
Unfortunately the ALFA interlock system had problems and ALFA was forced to not move the Roman Pots in, so only ~2hours
af
data was taking (bugs in the interlock system is now fixed).
Outlook:
The hardware part of the ALFA_CTPIN are in it final version and have successfully taking real data.
The firmware for the second FPGA, which is for bunch by bunch is not operational yet.
ALFA trigger and timing ALFA general meeting in Copenhagen 11-05-2015
23
/26
The software of both low and high level is still improving.
The board will be able to also monitor rates after logics, but software still in an early face.
Latency sources ALFA_CTPIN concept Early tests First module Revision 2 Timing Final latency
First data
Budget Thanks
Slide24Sune Jakobsen
Budget
Final total cost: 306160
kr
(exact value depending on the evolving exchange rate of the DKK/CHF)
Original estimate at NBI: 50
kkr
.
So this should be the lessen for next project: Better estimate of total cost.
Seminar at NBI 22-06-2015 ALFA_CTPIN - a NBI hardware contribution to ATLAS
24
/26
Latency sources ALFA_CTPIN concept Early tests First module Revision 2 Timing Final latency First data
Budget
Thanks
Slide25On behalf of ATLAS-ALFA I would like to say:
Thanks to NBI for this crucial hardware contribution that was finished in time for first data taking.
Seminar at NBI 22-06-2015 ALFA_CTPIN - a NBI hardware contribution to ATLAS
25
/26
Latency sources ALFA_CTPIN concept Early tests First module Revision 2 Timing Final latency First data Budget
Thanks
Sune Jakobsen
Slide2637/66
NIELS BOHR INSTITUTE
UNIVERSITY OF
COPENHAGEN
Commissioning of the Absolute Luminosity For ATLAS detector at the LHC
Sune
Jakobsen
Backup
Trigger
system upgrade winter 2011-2012
Luminosity ATLAS at LHC ALFA Installation Triggering 2011 Analysis
Trigger upgrade
2012 Impedance heating Conclusion/Outlook
Slide27Figure 22.2
Commissioning of the Absolute Luminosity For ATLAS detector at the LHC
NIELS BOHR INSTITUTE
UNIVERSITY OF COPENHAGEN
Situation in 2011
Sune Jakobsen
38/66
The OD signals arrive too late in USA15 to be used to trigger all of ATLAS => Can be used in ALFA standalone data taking only.
Fast coaxial cables were chosen over air-core cables due to price.
(
1 air-core cable is about 10000 CHF and 16 additional would have been needed).
Air-core cable signal
(Speed: 91 % of c)
Fast coaxial
cable
(
type
c-50-3-1. Speed: 83 % of c)
Fast coaxial
cable
(
type
c-50-3-1. Speed: 83 % of c)
Hit
Hit
Hit
Time
The rate of OD tracks in the 90 m run was much lower than expected. About 50 % of time used just for OD data taking.
Signal arriving in USA15
Luminosity ATLAS at LHC ALFA Installation Triggering 2011 Analysis
Trigger upgrade
2012 Impedance heating Conclusion/Outlook
Slide28Figure 22.4
Commissioning of the Absolute Luminosity For ATLAS detector at the LHC
NIELS BOHR INSTITUTE
UNIVERSITY OF COPENHAGEN
Upgrade to trigger combined on OD signal
Sune Jakobsen
39/66
Using only the already installed air-core cable also for all OD signals in combined running.
All signals arrive to USA15 in time and it is possible to distinguish OD and MD signals.
Air-core cable signal
(Speed: 91 % of c)
Hit
Hit
Hit
Time
Signal arriving in USA15
Only requires firmware upgrade of the Front
End electronics
(trigger mezzanine).
Different signal duration: 50 ns for MD signals, 25 ns for OD signals.
Luminosity ATLAS at LHC ALFA Installation Triggering 2011 Analysis
Trigger upgrade
2012 Impedance heating Conclusion/Outlook
Slide29Figure 22.6
Commissioning of the Absolute Luminosity For ATLAS detector at the LHC
NIELS BOHR INSTITUTE
UNIVERSITY OF COPENHAGEN
Distinguishing
MD and
OD signal
with standard
NIM electronics
Sune Jakobsen
40/66
Additional delay due to logic is 46 ns. From Front End firmware upgrade + CTP delay change: -50 ns.
Luminosity ATLAS at LHC ALFA Installation Triggering 2011 Analysis
Trigger upgrade
2012 Impedance heating Conclusion/Outlook
Slide30Figure 22.6
and 22.11
NIELS BOHR INSTITUTE
UNIVERSITY OF COPENHAGEN
NIM crate for MD/OD separation in ATLAS CTP rack
Sune Jakobsen
Commissioning of the Absolute Luminosity For ATLAS detector at the LHC
41/66
8 x
=
Luminosity ATLAS at LHC ALFA Installation Triggering 2011 Analysis
Trigger upgrade
2012 Impedance heating Conclusion/Outlook