Y Schutz for L Betev August 26 2009 2 Outline Offline shifter basic responsibilities The shifter check list Systems and tools The dashboard see Costins talk The Shuttle see Chiaras talk ID: 559529
Download Presentation The PPT/PDF document "Offline shifter training tutorial" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Offline shifter training tutorial
Y. Schutz for L.
Betev
August 26, 2009Slide2
2
Outline
Offline shifter basic responsibilities
The shifter check list
Systems and tools
The dashboard (see Costin’s talk)
The Shuttle (see Chiara’s talk)
The reonctruction and visualization package (see Marco’s talk)Slide3
3
Basic responsibilities – RAW data
The RAW data path
DAQ online buffer @P2
Fast optical link to CERN CC
500MB/sec (p+p), 1.25GB/sec (Pb+Pb)
CASTOR2 disk buffer
CASTOR2 tape buffer
Step A
Step B
Reduced
100 MB/sec (
p+p
)Slide4
4
Step A – Online buffer -> CASTOR buffer
Automatic and well-exercised (it almost never goes wrong
)
At this step, the files are also registered in the
AliEn
catalogue
DAQ is nominally responsible for the transfers Offline provides the registration
gateway If not working, DAQ notifies the shifter and/or the alice-shift-alarms@cern.ch expert listSlide5
5
Step A – Shifter responsibilities
Monitors the fill of the CASTOR buffer (through the dashboard
)
Notify the run coordinator/shift leader if more than 80%
full
Clear disk space following instructions received from the SL (regular and selective basis
Follow the registration of RAW (through the dashboard
)
All files in PHYSICS partition typically go to CASTOR Follow the run screen and grow suspicious if
none of the runs are being registered Contact the DAQ shifter and ask what is going onSlide6
6
Step B – CASTOR buffer -> Tape storage
Selective copying
of runs to tape
1/5 of RAW data stream in
p+p
(100 MB/sec
)
Full data stream in Pb+Pb (1.25GB/sec)
The selection of runs to be copied is provided by the SL The Offline shifter will be responsible for the copy procedure (though dashboard tools) Also for the deletion of data from the CASTOR buffer
It will involve some automatic copying (calibration data for example)Slide7
7
Basic responsibilities – Shuttle
Covered in
Chiara’s
presentation
Here just to put it in the context of the basic responsibilitiesSlide8
8
Basic responsibilities – fast reco and event display
A quick method to check the reconstruction of data and display couple of events from recent
runs
NOT a tool to do analysis
Covered
in Marco’s
presentation
Here just to put it in the context of the basic responsibilitiesSlide9
9
Basic responsibilities – data replication
After RAW is recorded to tape in
CASTOR
A copy is made to a remote T1 centre
(out of 6 possible) for
custodial storage (and processing
)
The replication is an automatic process, triggered at EoR
Progress is displayed on the dashboard Beginning of data taking – automatic replication is
disabled In general – the Offline shifter should follow the replication and raise alarm in case of failuresSlide10
10
Basic responsibilities – prompt offline processing
After RAW is recorded to tape in
CASTOR
+ Shuttle is
done
Processing is
launched
The processing is an automatic process
Progress is displayed on the dashboard Beginning of data taking – automatic processing is disabled
Lists of runs to be processed is compiled by the run coordinator / shift leaderSlide11
11
Basic responsibilities – prompt offline processing (2)
The experiment logbook contains ‘hints’ - run quality flags
Per detector and global
The run quality flags are
filled manually by the SL based on the detector QA and the reconstruction/analysis QA collected by the QA shifter
Offline shifter responsibility is to follow for all PHYSICS runs the content of the quality flags and prompt the shift leader
to
fill
these.Slide12
12
Offline shifter check list
Registration of RAW (dashboard
)
Periodic check of status
Follow PHYSICS runs
Ask shift leader in case of doubt
Report registration errors to on-call expert
The run copy and removal procedure Shuttle (dashboard)
Follow on processing of all runs + global Shuttle messages In case of preprocessor failures, escalate to (concerned) detector shifters
In case of Shuttle failures first follow the restart/debug procedures, then report to on-call expertSlide13
13
Offline shifter check list (2)
Fast reconstruction and event display (processing scripts on shifter console)
Periodic check of PHYSICS runs (not the entire run!)
Run reconstruction and
analyse
the
AliRoot log files for errors/crashes
Note the above in the shifter report pages and send to alice-shift-alarms@cern.ch Visualize periodically events in PHYSICS runs
Note ‘strange’ event characteristics in the shifter report pages and send to alice-shift-alarms@cern.chSlide14
Run Coordination meeting
Day
shifter only
Attend
the
daily (2:00PM) Run coordination
meetingReport about issues related to the
OS tasks encountered since
the last meeting (see offline
log book)Send a mail to AIP
group summarizing offline issues
discussed during the meeting and important
notices CCRC F2F 10/01/2008
14Slide15
15
General shifter rules
Before pressing the
Read the procedures and rules, defined for each error type
Try out the remedies
If all fails, inform the on-call expert Slide16
16
Offline shifter check list (3)
Data replication (dashboard)
Periodic check of replication status
Note ‘stuck’ runs – not replicated 12 hours after registration – in the shifter report pages and sent list to
alice-shift-alarms@cern.ch
Prompt data processing (dashboard)
Periodic check of processing status Note ‘stuck’ runs – not processed 12 hours after registration – in the shifter report pages and sent list to
alice-shift-alarms@cern.ch Shift report (shifter system) At end of shift – summary of the operation and noteworthy eventsSlide17
17
Information sources for the shifter
The shifter manual – instructions
Shifter interface
(
http://
aloshi.cern.ch)
Monitoring – MonALISA (
http://alimonitor.cern.ch/)
Dashboard Shuttle
Processing and data management