A chieved results with the tests of ROD/BOC cards - PowerPoint Presentation

1. Achieved results with the tests of ROD/BOC cards1Marcello Bindi, on behalf of the Off-detector Working group

2. OutlineRead out system overviewCalibration and data taking requirementsSystem tests of the different labs Achieved results at CERN: TX Command lineRX Data communication and FormattingValidation of histogramming mechanism Histogrammer multiple chip read outOperation with TIMSummary and plan2

3. Why a new FE-I4 chip?The existing Pixel Detector readout architecture was designed for the nominal LHC peak luminosity of 1034 cm-2 s-1 and for an expected trigger rate of 100 kHz.Limitations from the old Pixel detector for larger occupancy and luminosities:  internal FE-I3 bandwidth (occupancy) RX link from the Module Controller Chip (MCC) to the off-detector electronics (occupancy and trigger rate)New FE-I4 designed: a smaller feature size, smaller pixel granularity, lower material budget leading to a simpler module design with MCC features integrated into FE-I4 added 8b/10b encoding before streaming out the data at 160 Mbit/s.3

4. Why a new Read-Out System?The old BOC cannot handle the data rate incoming.The hardware of the existing Pixel boards is designed to connect a maximum of 8x160 Mbit/s input links (from the modules) to 1 output S-Link (to the ROS), whilst IBL modularity foresees 32x160 Mbit/s input links to 4 S-Links have to be handled for each stave.Bandwidth limitation on the VME bus used in calibration runs. The obsolescence of the components. New design! The integration has increased by a factor 4.The calibration will be handled by FPGAs with embedded software + external PC farm connected via Gbit links instead of the DSPs4

5. Read Out System requirements2 FE-I4 chips form a detector module and are operated via a single clock (for synchronisation with the LHC machine) and a single input command line at 40MHz. There are 2 data links, one for each FE-I4.The Timing Trigger and Command (TTC) link, controlling the detector, runs at 40 Mbit/s. It uses Bi-Phase Mark (BPM) encoding in the optical path, same as the ATLAS Pixel Detector. It will be decoded by an opto-electrical converter (IBL Opto-Board).Data return per frontend runs with a 160 Mbit/s transmission, encoded using 8b/10b coding scheme.IBL will be placed within the same readout frame of Pixel (VME), using more powerful electronics. The IBL readout building block will deliver an increased bandwidth of 640 MB/s, serving a maximum of 16 modules with 32 frontend chips per building block.5

6. Read-out System OverviewNumber of IBL Staves /ROD-BOC pair14# DAQ Modules per ROD-BOC pair16# FE-I4s chip per ROD-BOC pair32Total # of FE-I4s in IBL448 (32*14)Number of Pixels per FE-I426880Total # of read-out channels~12 M6

7. Read-out System Overview/2DCS Section named: M[1:4] DAQ Modules Section named: [A/C][1:8] Chips attached to module names as [1:2] (from inside to outside) Chip naming example: M3_A6_2 or M1_C1_1. The TX Links have the module granularity : M4_A6 and M1_C1…The RX Links have the chips granularity : M4_C7_2 and M1_A1_27Stave + Optoboard Section: LI_S[01:15]_[A/C]

8. Calibration and data taking requirementsEach ROD/BOC pair has to deal with 32 FE-I4 chips, each sending 8b/10b encoded data at 160 Mb/s (128Mb/s of real data).Each pair is equipped with 4 Output S-Links, associated to 4 Histogrammer units, which individually share the serialized input from 8 chips 8*128Mb/s=1024 Gb/s.Since the FE-I4 uses 24 bit per dual-hit, we can produce a maximum of 1024/24*2 = 85 Mhits/s for each Histogrammer/S-Link. Digital, Threshold, Time over Threshold (ToT) and Analog scans have to be performed in a reasonable amount of time (<30 min), usually during the inter-fill periods.The calibration sequence foresees:Data Acquisition, typically histogramsAnalysis of the acquired data (fitting)Modification of the detector parameter (tuning) These operations have to be performed in parallel in the whole detector; each ROD will have 2 gigabit Ethernets (1 per Slave) for the calibration data transmission. No VME bottleneck of 7 MB/s is present anymore.8

9. ROD and BOC cards to be set up for system test and IBL assembly test.Full stave readout per card pair.Functionality of cards has to fulfil detector operation needs, both data taking and calibration.Multi card setup.Full readout chain tests with optical link.ROD and BOC card pairs should serve as the standard readout system in the CERN setups asap.9ROD/BOC cards goals

10. 10ROD (RevB) card1 Spartan6 (Xil.) Prog. Res. Man.1 Virtex5 (Xil.) Master: operation control (PowerPC)2 Spartan6 (Xil.) Slaves: data processing (MicroBlaze)Main Firmware components:3 Gbit Ethernet Connectors:1 receiving config. from PC2 sending histos to PC farm1 USB mezzanine1 TTCrq mezzanineControl, debug and interface with Calibration Farm/PC:Other components:4 SSRAM (2 per Sp6)1 Flash (Atmel) for the Vtx53 DDR2 (1 per each Sp6 + 1 Vtx5)P0/P2/P3 connectors 40-80MhzInterface with BOC (FE-I4&ROS):

11. 111 Quad-SFP link connector4 Single SFP link connectorsInterface with TDAQ (ROS)/FTK:1 ELMB connector1 Ethernet connector1 UART connectorControl, debug and monitoring:Interface with the FE-I4:Electrical: 2 FMC connectorsOptical:2 (1+1) SNAP12 TX Plugins4 (2+2) SNAP12 RX Plugins2 Old Fashion TX Plugins 1 FPGA (Xilinx) BOC control2 FPGAs (Xilinx) data encoding to/from RODFirmware components:BOC (RevB) card

12. Hardware availabilities for the system tests125 ROD revB (blue):1 in Mannheim1 in Wuppertal 1 in Genova1 at CERN ATLAS Pixel-lab1 in Bologna Card (revB) were distributed to the various labs in March 2012.5 BOC revB (black):1 in Mannheim1 in Wuppertal 1 in Bern1 at CERN ATLAS Pixel-lab1 in BolognaMODULES:Few FE-I4A Single Chip Cards (SCC) were available since the begin of the cards integration.By the end of 2012, FE-I4B SCC and also few Double Chip modules were distributed.Optical Components:nSQP (early) and IBL (later) Optoboards were available at CERN.Fibers and SNAP12 Plugins (TX and RX) were sent to CERN (from BERN) in late November, when an optical connection of the modules were installed.5 new ROD revC (red) available since January 2013:They were operated in Bologna for multiple card tests; recently 1 was sent to CERN (now in SR1) and 1 to Berkley.

13. IBL system tests in the different labsCERN (Pixel-Lab) : ROD + BOC + TIM-2C + SNAP12 TX/RX Plugins + Optical Fibers + IBL Optoboard + 2 FE-I4BBologna : ROD + BOC + TIM-2C + 1 FE-I4 (Elect. Conn.  BOC)Wuppertal : ROD + BOC + SNAP12 TX/RX Plugins + Optical Fibers + nSQP Optoboard + 1 FE-I4A Mannheim : ROD + BOC + FE-I4 (Elect. Conn. BOC) + ROBINGenova : ROD + FE-I4 (Elect. Conn.  ROD)Göttingen : ROD (revA)+ FE-I4 (Elect. Conn.  ROD)13Many functionalities have been tested in different system tests, depending on the local setups:

14. Chronological/Functional view of the testsROD and BOC Standalone (table top and on crate)ROD + BOC table topROD + BOC on crateROD + FE-I4BOC + FE-I4ROD + BOC + FE-I4 (electrical connection)ROD + BOC + FE-I4 (optical connection)14see previous talks from Alessandro and Andreas !!Look at this talk!!

15. IBL ROD/BOC initial tests with FE-I4PPC15The FE-I4A chip (mounted in a Single Chip Card) was electrically connected to the ROD in order to perform standalone development of the ROD firmware.The PowerPC (via Ethernet) was able to configure the FE-I4, send triggers and spy data output (accessing the Spartan-6 receiver with the ROD bus).Electrical Connection

16. The results obtained with the direct connection of the FE-I4A to the ROD were still valid after connecting the chip to the BOC.The direct connection to the ROD was kept in the Firmware; to be used for cross-checks or for those setups that didn’t have any BOC (Göttingen, Genova).We were able to configure the FE-I4 also using the Master-DSP via VME bus.16Decoded FE-I4 OUTTrigger to FE-I4Electrical connection to the BOCRx_rod_data = 3C IDLE from FE-I4 once it’s configuredRx_rod_addr = 0 ”Local” Chip Address/Link number, for each Gatherer (from 0 to 3)

17. Overview of the PixelLab test setup at CERN17 ROD revB BOC revBAgilent/Keithley Power Supplies: VpinVVDC, Viset 2 FE-I4BTXRXControl PC for sending front-end configurations, triggers and reading back of theHistogramsDDR2

18. Tx-ch 1 18Sketch of the system test at CERN (Pixel Lab)RxC Ch 0-7RxD Ch 8-FBMF NORTHBMF SOUTHRxA Ch 0-7RxB Ch 8-F TX1 Ch 0-7TX2 Ch 0-7 Eta CoordinateBOC(revB) ROD(revB) Rx-ch 0 Rx-ch 1SLAVE BSLAVE ANorth side used for LoopBack tests by BERN people!DAQ Module Tx-ch 0 Rx-ch 9Rx-ch 8 IBL OPTOBOARD PIN diode VCSEL#2 VCSEL#1DORIC VDC VDC

19. ROD/BOC Firmware and Software StatusThe actual Firmware version running on the ROD/BOC FPGAs are clearly a simplified version respect to the final one (especially for the ROD).Firmware workshop last May at CERN with round-table of the main experts many action items have been assigned.Full chain of digital/analog injection tests has been implemented; still work-in-progress the implementation of a complete scan of multiple chips.Software workshop at CERN (May 27-31)  many new tasks have been defined and assigned.19

20. rx decoderboc2rodgatherer Asimple EFB-0SLAVE BPPCserial portROD busV5RODxchisto012b @ 80 MHzfw: sp6fmtfw:rodMasterfw: RC1BMF_SOUTHBOCETHETHRAMBLOCKINMEM FIFOMBregister blockDDR2SSRAMBCF32gatherer Brx decoderboc2rod12b @ 80 MHzS-Link016b @ 80 MHz + 7 ctrlrx decoderboc2rodgatherer Csimple EFB-1histo112b @ 80 MHzgatherer Drx decoderboc2rod12b @ 80 MHzS-Link116b @ 80 MHz + 7 ctrlSSRAMDMAaxi histo dmaaxi epcaxi epcaxi epcaxi epcaxi epcaxi epcaxi epcregister block32321632323216Optical Connection:IBL OptoBoard + SNAP12 Plugin + Optical FibersBOC/ROD Firmware overview of the “South Hemisphere”

21. Gatherer ASLAVE BPPCROD busV5fw: sp6fmtfw:rodMasterETHETHRAMBLOCKINMEM FIFOMBSSRAMGatherer BS-Link0Gatherer CSimple EFB-1Gatherer DS-Link1SSRAMDMAaxi histo dmaaxi epcaxi epcaxi epcaxi epcaxi epcaxi epcregister blockHisto0axi epc2Multiple Output Debugging into the RODSimple EFB-021Histo1DDR2

22. RX data Path in the ROD22Multi-Sample trigger for HitEnable =1, that means good hit from the Front-End!Data from both chips are sent to the histogrammer block with right CHIP, ROW, COL and TOT values !Histo _0Histo_1Gatherer_A

23. Validation of the histogramming mechanismInjecting pseudo-Data into the Gatherer Input (Fac-simile of data coming from the BOC). The full data processing/formatting in the ROD is under test, including the Histogrammer. 2. Injecting pseudo-Data into the Histogrammer input (Fac-simile of data coming out of the Event Fragment Builder) Only the histogrammer block is under test.3. Injecting real-Data from the FE-I4 into the Gatherer Input The full data transmission/decoding/processing/formatting in the BOC and the ROD is under test, including the Histogrammer23Three main methods were used to test the histogrammer functionalities but also to test the correct behaviour of the main SLAVE registers:

24. Overview of the firmware for the Histogrammer testsGatherer Asimple EFB-0SLAVE 1PPCROD busV5 1 and 2 are ROD Standalone Tests no BOC, no FE-I4 are neededfw: sp6fmtfw:rodMasterETHETHRAMBLOCKINMEM FIFOMBDDR2SSRAMGatherer BS-Link0Gatherer Csimple EFB-1histo1Gatherer DS-Link1SSRAMDMAaxi histo dmaaxi epcaxi epcaxi epcaxi epcaxi epcaxi epcregister blockhisto0axi epc21The pseudo-data are correctly written and read back from the slaves.DMA transfer from BRAM to DDR2 was successfully tested.3

25. Digital Injection Scan ImplementationFlash the Firmware/Software binaries and start the slaves.2. Writing the Configuration of Histogrammer3. Start Histograming  Histogrammer in sampling mode4. Initialize the basic ROD/BOC Registers (V5Mask, Formatter Link Enable, TX/RX Control Registers) and FE-I4.25A simplified version of scan was implemented in order to test more complex operation, mostly FE-I4 related (mask shifting mechanism, multiple trigger or Double Column (DC) operation) but also to test the histogrammer configuration when operating a Multiple Chip setup:

26. 265. Setting of Scan Parameters:MASK_LOOP = par0CALPULSE/VCAL LOOP = par1DC_LOOP = par2TRIGGER_LOOP = par36. LOOPSMASK LOOP CALPULSE LOOPDC LOOPTRIGGER LOOPSend CalPulse and Trigger (8 clock fixed delay)Read Data in the Inmem Fifo and Fill the Plot every time7. Stop Histogram (start the DMA Transfer, entering the Read out mode and asserting ReadOut Completed (ROC) whenever the transfer is finished.Digital Injection Scan Implementation/2

27. Results of the mask shifting mechanism27To highlight the MASK Shifting, instead of LOOPING over the CALPULSE, a single CALPULSE was injected but function of the Mask stage : CalPulseLength= 4 + iMask  CalPulse= 4 BC for MaskStage=0  CAlPulse=5 BC for MaskStage =1  CalPulse= 6 BC for MaskStage =2 ……N = 8 in our caseCol 1 2 3 4 5 6 7 8

28. Histogrammer Output with Multiple ChipsCheck of the correct addressing of the memory depending on the Histogrammer Configuration.Check of the emptiness of the memory cell for CHIP and DC that were not operated.Check that the same memory location is overwrite for different Mask Stages of the same digital scan. It’s a software task to assign different row/col/chip number depending on the total # of Mask Steps chosen and witch mask stage of the scan is executed.Final check of the results comparing with the INMEM FIFO output. All the checks look OK !

29. CHIP #0 ¼ PixelMaskCal Pulse Length = 5 BCFirst Mask StageCHIP #1 ¼ PixelMaskCal Pulse Length = 6 BCFirst Mask StageFinal check with the Histogrammer memory

30. 30TIM – BOC - ROD communication seems ok! The TIM board, (TIM 2C) is the same as the one in Bologna, however with different firmware version. We were able to send trigger internally generated (100,150,250KHz) to the ROD.LVL1_ID are correctly incremented and received from the ROD. No error seen, no fluctuation of LVL1_ID has been detected so far.Changing the frequency of the trigger we see a coherent change in Chipscope signals.BC_ID reset to be checked.Operation with the TIM 100Khz250Khz

31. Summary and planROD(revC)/BOC tests performed in different labs (multi-chip configurations, digital/analog injections) show that the hardware specifics have been full-filled.We are able to configure and operate the 2 FE-I4B SCC, optically connected to the BOC via TX/RX Plugins + Fibers + IBL Optoboard.The PowerPC is able to configure in parallel the FE-I4B chips, send triggers and retrieve the data via RODBUS from the slave DDR2 memory.Firmware is behaving correctly; Data Formatter and Histogrammer basic functionalities have been successfully tested via multiple Digital Injections. However, it needs some extension for a more realistic version (BUSY/Timeout logic..) and further tests of the scan loop mechanism are necessary (memory transfer of histograms from external SSRAM to the Fit Server is under development).Multiple-card operations and interface with TIM have started.Full Scan Loop procedure with multiple and parallel chip operation is now our next (close) target !!31

32. Back up32

33. GathererHeader finderHeader finderHeader finderHeader finderdecode data FIFO1024x32 bitsFIFO readout controller323232323232323232valid_to_efbdata_to_efbIt expects to receive FeI4 24-bit Data Record M.S.Byte first.Column (7)Row (9)ToT (8)Data Record“E9X”LV1ID (4)BCID (8)Data Header1233

34. gatherer output data formatheader0010 | 00 | channel number (2) | 0 | L1ID (4) | BCID (8)hit1000 | 00 | channel number (2) | TOT (8) | col (7) | row (9)trailer0100 | 00 | channel number (2) | 0Example data slice:20000000808CAB5F4000000021000000818CAB5F41000000…all the data from module 0, then 1-2-3 in order

35. simple EFB it pops data from the 2 dual clock FIFOs, when the histo block is available (rfd = ‘1’) it removes header + trailer words it provides tot, row, col and chip#3232simple_EFBhitValidrowcolchiprfd973TotVal4

36. Histogrammer units36

37. What the INMEM FIFO says? At the same time, we observed the results from the INMEM FIFO output.This tell us that the SCAN mechanism works as expected and the data are sent back correctly from the FE-I4.Different Pixel Masks (or # Mask Steps) can be chosen; they would look the same for the histogrammer but in reality there are firing different pixels! 1/8 pixels 1/4 pixels 1/2 pixels Pixel Mask correspondingto the previous scan

38. Analog Injection tests38Vcal: 150  1000Vth-COARSE: 0x0  0x1Vth_FINE: 0x80  0xC0

39. 39RX and TX pluginSNAP 12 devices should provide communication between the Off detector readout and the on detector part, through optical links!RX from three different vendors have been tested. Avago, Tyco Electronics and Reflex Photonics.Reflex photonics shown problems coping with high light input power.Avago shows a drop in power reception with a power output reduce of ~3% after 1000 hours. Does not seem problematic.Turn on point better on Avago.Transition width basically equal.Avago and Tyco RX both qualify as good devices for signal reception on the BOC.Measurements show that they are capable of operating at the required frequency, and at the input power expected from the optoboard.

40. Optoboard View40