Shifter Instructions - PowerPoint Presentation

Slide1

Shifter Instructions

Instructions for the SCT ACR Shifter

Version 0.4

– February

2010Slide2

Contents

Part I – Shifter Duties

Part II – ACR environment

Part III – Run Control & DAQ

Part IV – BeamPart V – DCSPart VI – Data Quality MonitoringPart VII – Problems

2

SCT Shifter

ManualSlide3

Shifter Duties

Part I - Instructions for the SCT ACR ShifterSlide4

Before your shift

Make

sure your training is up to date : courses

Level 1

,

Level 2

,

Level 3

and

Level 4A

and SCT Shifter QualificationYour access rights should be valid (ATL_CR) and SCT:shifter rights. Read documentation on Twiki page : http://atlasop.cern.ch/twiki/bin/view/Main/AtlasOperation and go to shifter manual.

4

SCT Shifter

ManualSlide5

Beginning of your shift

Come

to the

ACR at least

15 minutes

before your shift and consult with the previous shifter and check:

Current ATLAS conditions

Particular item under watch?

Where we stand respect to run plan?

LHC status and plan for the coming shift

Your access privileges will be granted automatically 2 hours before the beginning your shiftCheck the run planhttp://atlasop.cern.ch/atlas-point1/wmi/current/Run%20Status_wmi/AtlasRunCom.html Read previous shifter e-log and start your new entryLog in to the DCS, FSM Alarms and Run Com (Check-List Utility)5SCT Shifter ManualSlide6

During your shift

Do the checks requested in the Check-List

Monitor DCS alarms on the DCS Alarm Panel

Look for ERROR and FATAL messages at the DAQ coming from the SCT

Check Monitoring Histograms and DQM status flags

Check beam conditions are stable

Monitor Cooling status and alarms

Document in the

e

-log (SCT and

IDGen

) problems and happeningsIn case of problems contact relevant experts and/or Run Coordinator6SCT Shifter ManualSlide7

End of shift

Submit

e

-log entries SCT and

IDGen

(log out of the

e

-log)

Inform the next shifter

on the current situation of the running and about any special conditions or plans.

Log out of the Check-List, DCS and FSM Alarms

Do not leave if following shifter did not arrive (after 30 minutes call Run Coordinator if shifter has not arrived).7SCT Shifter ManualSlide8

ACR environment

Part II - Instructions for the SCT ACR ShifterSlide9

ACR Layout

SCT Shift Desk

ACR, CERN,

Meyrin

Shift Leader Desk

ACR, CERN,

Meyrin

9

SCT Shifter

ManualSlide10

ACR Desktop

TDAQ Screen

DQM Screen

DCS and Alarm Screens

Desks have 4 Screens to be used by the shifter

Get familiar with the environment

Some useful Utilities can be found in the

General

tab

Example of

ScreenShot

of the ACR Desktop10SCT Shifter ManualSlide11

SCT Shifter

Manual

RunCom

In the

General

tab find

RunCom

Look for the SCT tab and log yourself in

Log out of the Check-List and FSM

Read the checks and comments, some of the requirements have an information page

DQ checks will appear every hour

Relevant for ATLAS runs but NOT for SCT Standalone11Slide12

ATLAS e

-log

e

-log can be found under the tab

General

To be able to submit entries you have to login with your P1 login/password

Make entries if problems occur do not wait until the end of your shift

Do a

Shift Summary (

and select

ID-Gen and SCT)

for every shift Remember to select all options and attach plots (if any) before submitting12SCT Shifter ManualSlide13

ATLAS e

-log Sample

13

Short summary…..

*******************************************

15.00: start of shift

15.04: Recovered module Crate 22 Ch 4 which showed a trip error.

17.03: end of run 139303

17:05 Expert takes over DAQ to investigate XXX

18:30 DAQ tests concluded. The result is that blah blah blah...

a

nd more blah, blah, blah… Changes have been applied, blah, blah, blah… 18.40: SCT on stand-by19.17: Start of Physics run 13933419.30: SCT on during LB 6.20.00: Monitoring the number of errors, stable and around 20-30,see separate report from MON shifter.…21.12: Run 139336. Try a short cosmics run again(8 bins, 1000 triggers, timeout 60s)Result status -> no fail, but quite a few problematic modules?…SCT Shifter ManualSlide14

SCT Shifter

Manual

Atlas Operation

Information for shifters will be posted in the Operations page.

From outside P1 :

http:/

/

atlasop

.

cern.ch/twiki/bin/view/Main/AtlasOperation

SCT LinksGeneral Information14Slide15

SCT Shifter

Manual

SCT

WhiteBoard

Special announces, SCT DAQ Panel Configuration and SCT Phones

From outside P1 :

http:/

/

atlasop

.

cern.ch/twiki/bin/view/Main/SCTDetectorWhiteBoard

From P1 : ID Menu SCT WhiteBoard Important Announcements

Phones

15Slide16

SCT Shifter

Manual

SCT Shifter Manual

Information for your shift

From outside P1 :

http:/

/

atlasop

.

cern.ch/twiki/bin/view/Main/SctShifterManual

From P1 : Link from the Atlas Operation

Twiki 16Slide17

pplications

DAQ Panel

go to

TDAQ tab

FSM Alarms

and

DCS

go to

ID



SCT tabRunCom go to General tabe-log go to General tabkonqueror go to General tab ksnapshot go to General tab 17

SCT Shifter

ManualSlide18

Run Control & DAQ

Part III - Instructions for the SCT ACR ShifterSlide19

DAQ Panel

Log Window

Options

Applications

19

SCT Shifter

ManualSlide20

DAQ Panel

Select the setup script

(/

det/tdaq/scripts/setup_TDAQ.sh

will always point to the production release)

1

Select the partition db file

(

i.e.,

/atlas/oks/tdaq-02-00-03/combined/partitions/ATLAS.data.xml)2Get the partition(s) described in db(the Part Name list will be automatically filled)

3

Tell the

DAQPanel

to acquire the information

4

20

SCT Shifter

ManualSlide21

DAQ Panel : Main

Monitor Partition:

Use it to Spy on the ATLAS Partition status

Log Manager:

Browse the log messages produced by the applications

MRS:

It starts a viewer of the MRS (Message Reporting System) messages flowing through the system

Busy:

It starts the Busy Presenter. Use it to understand whether/why the system is in a busy state

21

SCT Shifter ManualSlide22

Monitor Partition GUI

22

RootController

Commands

Tab Panels

MRS Log Window

Run Information & Settings

SCT Shifter

ManualSlide23

MRS

23

Change subscription

Message format

Clone table

Start from

DAQPanel

– Main tab – MRS button

SCT Shifter

ManualSlide24

Busy Monitor

24

No data flowing through the system?

The system may be in a BUSY state

Start the Busy Presenter from the

DAQPanel

: Main tab, BUSY

RED: the corresponding subsystem is busy

PINK: the corresponding subsystem is busy but out of the PARTITION

GREY: the corresponding subsystem is not part of the run

If SCT is Busy and do not know what to do CALL DAQ EXPERT (

162034) for instructions SCT Shifter ManualSlide25

SCT Gui

25

DAQPanel

in the SCT Desk has an additional tab called

SCT

Use the button to launch the

SctGui

. Use the GUI to:

Run Mode Physics scans

Alternatively check DCS conditions

Probe and check modules : if OUT of ATLAS running and you have been instructed to do it

SCT Shifter ManualSlide26

SCT Gui

: Short Tour

26

Data

shows

different

options

depending on the kind of display selected (Module Configuration in this case)Views: Geometry and ROD cratesList of: Barrels/Wheels or Crates in the configurationDifferent module groupsEach of the squares is a module

SCT Shifter

ManualSlide27

SCT Shifter

Manual

SCT

Gui

: Short Tour

27

Example

: SCT

View

SIDE

Coordinates of the module under the mouse

0 – No Data1 – Bad Data2 – CLK/2a – Corrupt DataA – ABCD

Error code

C

–

Config

Readback

mode

E

–

Returning

events

J

–

Junk Data

O

–

OtherSlide28

SCT Gui

: Short Tour

28

You can access to the different menus using the toolbar or with a right click on the empty part of the screen

Module Configuration:

Change modules in the configuration

Probe Data:

Check for modules with errors and recover them. Using this display you can perform actions like Power Cycle, Reset, Configure…

Data Manager:

Allows you to access to results and plots from a test

Test Results:

Show test results for all the modules DCS Conditions: Several of the DCS parameters related with the modules are available through the SctGui (Temperature, LV, HV,...)SCT Shifter ManualSlide29

SCT Gui

: Short Tour

29

How to check the ROD which is BUSY?

Right Click

at the

TIM Module

and select

TTC Status

If

Right Click

on any of the Crates a menu will show three possibilities: Clear Latch Mask ROD Busy Raise ROD error (manual stop-less removal) : Use this one if a busy appears and is not picked up by the automatic removalOnly do this if DAQ EXPERT (162034) ask you to do itTTC StatusSlide30

Physics Mode Scan

30

In

SctGui

:

Tools Menu



Physics Mode Scan

Need to be in

Physics

mode (standalone) or in

ATLAS partition Uses triggers to acquire events, number of events and crates can be selected Results are displayed in the SctGui – Test ResultsPhysics ModeSlide31

Remember…

31

Important to be specially

vigilant

at the beginning of the run. Problem most likely will happen at the point. Look for:

FATAL / ERRORS from the SCT appearing in the MRS

Check BUSY status (and TTC Status). If no triggers are coming through one of the systems could be busy

Stop-less removal could happen at this stage. Ask to restart the run if:

3 Barrel

RODs

or 7

RODs (Barrel + EndCap) get disabledAlways write Information on SctGui Run comment window (pop-ups at the start and end of the run)Run a Physics Mode Scan and check for errorsDuring the run :Keep checking on disabled RODsCheck ROS Monitoring every hour. Look for errors and anomalous values in the resultsContinue to check for errors in MRS Document everythingSCT Shifter ManualSlide32

Beam

Part IV - Instructions for the SCT ACR ShifterSlide33

33

Also available on the WEB @

http:/

/

atlasop

.

cern.ch/atlas-

point1/operRef

.php?subs=

dcs/dcs/process.php?page

=ATL_LHC

Beam Intensity From LHC (P4) <#protons> / bunchBeam IntensityFrom Beam Pick Up <#protons> / bunchSCT Shifter ManualSlide34

ATLAS Stand-by Mode

34

ATLAS should ALWAYS be in stand-by mode except when:

STABLE BEAMS have been declared



Physics Mode

Beam is not possible due to LHC conditions



LHC NO BEAM Mode

In ALL OTHER situations ATLAS MUST be in stand-by mode"Stand-by Mode" is state of ATLAS hardwareit is decorrelated from the data aquisition state it is possible to take data (not physics quality) with ATLAS in stand-by SCT in Stand-by : LV ON and HV at 50 V SCT Shifter ManualSlide35

Fill Pattern

35Slide36

Beam Injection: I

36

ATLAS Shift Leader receive LHC INJECTION

WARNING

via the

LHC handshake panel

. Big Sound coming from the DCS Desk!

ATLAS shift leader replies ATLAS INJECTION = PREPARE, within few seconds, otherwise phone call from CCC

ATLAS shift leader asks the shifters to prepare for for INJECTION:

Make sure that SCT is in

Standby and Beam Protection is ACTIVE Check the INJECTION PERMIT mask. SCT should NOT be MASKEDSlide37

LHC Page

37

How to see if

SCT

is in STANDBY?

Masks

Link to the LHC Handshake pageSlide38

SCT Beam Page

38

SCT Status : Injection Permit and HV

Link to the LHC Handshake page

Beam Protection OFF / ONSlide39

SCT Beam Page

39

SCT Status : Injection Permit and HV

Beam Flag

Protection OFF / ON

LHC StatusSlide40

Beam Injection : II

40

ATLAS Shift Leader turns the Injection Key

Shift Leader uses hand-shake page and declares ATLAS INJECTION READY

LHC will inject Pilot bunch...Slide41

During Injection

41

Monitor

carefully

beam

losses

(

Backgrounds

) ATLAS BCM and BLM readingsLHC BLMsMonitor Beam parameters, position, ... BCM, LUCID, MBTS, L1Calo rates, Tile integrated currentsInform Shift Leader if some problem is found.Instructions and limits will be set soonSlide42

ATLAS in Beam Mode

42

Only with

STABLE BEAM

All systems should have Low Voltages (Front end powered, system can be read out)

All systems should have High Voltages / Bias Voltages, systems are providing signals from particles

SCT ON

only with

STABLE BEAM

and GOOD BEAM Conditions (TBD)Slide43

Stable Running

43

When STABLE BEAM Flag becomes TRUE. Shift Leader will ask systems to go ON

Check

STABLE BEAM

Flag

AND

Beam

BACKGROUNDS

:

Inform SCT Run Coordinator and ask for approval

HV can go ONInform Shift Leader that system is ONKeep Checking Beam Conditions at ALL TIMES during your shift.Slide44

Removal of Permit

44

If the Beam Mode= STABLE BEAMS, Injection Permit will be removed

AUTOMATICALLY

.

If a Beam Dump is requested anywhere around the LHC, our BCM receives a post mortem request which removes the ATLAS Injection Permit.

Without beam: If the

LHC Page 1

indicates that the beam will be absent from LHC for a period of a few hours “Quite Time”.

SCT will be informed that some “Quiet Time” is available. SCT Run Coordinator should be informed by shifter and/or ATLAS Run Coordinator.Slide45

Beam Dump

45

Planned Dump :

LHC send an ADJUST/ DUMP warning via the hand-shake

SCT will go to Standby

automatically

when BEAM DUMP state goes to INMINENT

Shift leader ask shifters PIX,

SCT

, MUO,

LArg

to put their systems in safe state. Send SCT to Standby if not alreadyNon planned Dump :Send system to Standby if not alreadyAsk Shift Leader for an explanation and post in the e-log as much information as you can gather. Do not wait up to the end of your shift to post it!Inform Run Coordinator and DAQ On-call expert (162034)If qualified perform the calibrations for a Non planned dump, if not wait for the On-call expertSlide46

Operation Flow-Chart

46

NO BEAM

Short Interruption

INJECTION PERMIT=

FALSE

PROTECTION ACTIVE

STABLE BEAM=False

NO BEAM

Long Interruption

INJECTION PERMIT=

FALSE

PROTECTION OFF

STABLE BEAM=False

INJECTION

INJECTION PERMIT=

True

PROTECTION ACTIVE

STABLE BEAM=False

STABLE BEAM

INJECTION PERMIT=

False

PROTECTION ACTIVE

STABLE BEAM=True

RAMP/SQUEEZE/..

INJECTION PERMIT=

True

PROTECTION ACTIVE

STABLE BEAM=False

Injection handshake

warning

Inject and dump

cycle

or lose the beam

(no dump)

LHC Page 1 announces NO BEAM for > 8hours

NO

YES

1) SL notifies ARC and shifters in the ACR

2) Shifters inform their SRC if instructed to

3) SRC requests to ARC to override STABLE BEAM flag

4) ARC checks LHC plan, gives permission for STABLE BEAM flag override

5) SRC can override STABLE BEAM flag and notifies SL

SL Removes

Injection Permit

Injection handshake

warning

1) SL notifies ARC

2) Remove STABLE BEAM override

LHC Page 1: Beam <2hours

1) SL notifies ARC

2) Remove STABLE BEAM override

3) SL goes through Injection handshake

Beam dump

ARC=

ATLAS Run Coordinator.

SRC=

Sub system Run Coordinator

SL =

Shift Leader

SL goes through Injection Handshake & brings ATLAS to STDBY, gives Injection Permit

Beam dump

AcronymsSlide47

ATLAS in Beam Mode

47

INJECTION PERMIT= is Injection into LHC Allowed?

If INJECTION PERMIT is FALSE:

Beam cannot be injected

Ongoing injection is STOPPED or PAUSED

But Beam is NOT dumped Slide48

Remember...

48

In NON SAFE situations SCT must be in Standby :

Single or Double beam operation during the commissioning phase

During beam injection and while STABLE BEAMS = False

During fills even if there is no beam in the machine

Standby :

LV ON and HV 20 V

Special Configuration applies for Beam Commissioning :

Splash Configuration

: LV ON,

EndCaps at 20 V and Barrel at 5 V (OUT of the data taking)This is the SAFE configuration up to further noticeOnly SCT Run Coordinator or someone designated by the Run Coordinator is allowed to give you permission to switch ON or change running conditions (Commissioning Configuration  Standby Configuration)ON Configuration : LV ON and HV 150 VSlide49

DCS

Part V - Instructions for the SCT ACR ShifterSlide50

FSM

50

State machine hierarchy

Detector hardware and hierarchical structure represented by FSM entities

State & command propagation

States upwards (using programmed logic with flexible rules)

Commands downwards

Status

Error handling upwards

Partitioning

Asynchronous operation of sub-trees possible

State/Status/Command propagation can be inhibitedAccess controlSlide51

Alarm Screen

51

Active and non-acknowledged alarms

Acknowledgement:

Only for some alarms

Operator needs to acknowledge

to remove it

For each alarm:

Get details for experts

Mask alarm on UI level until the alarm condition goes

Insert alarm to

Elog with commentDisplay trend plot of valueIn dev: alarm help on TWikiSummary alert:Hides several individual alerts in a single alarm entryAccess to individual alerts via “Details”Log in as yourself !Slide52

DCS Screen

52

Log in as yourself !

Beam Flags Summary

SCT Status

Beam Status

AlarmsSlide53

FSM Modes

53

Padlocks:

No control, Free

No control, Included

Exclusive, Owner

Shared, Owner

Exclusive, NOT Owner

Shared, NOT Owner

No control, Locked Out

Padlock background:

Tree completeExcluded children (partitionable trees)Disabled children(non-partitionable )Control:No propagation, NO commands

Propagation, NO commands

Propagation (if Included), Commands by you

Propagation (if Included), Commands by all

Propagation (if Included), NO commands

Propagation (if Included), Commands by all

No propagation, NO commands

Tree complete

Excluded children

(

partitionable

sub-trees)

Disabled children

(non-

partitionable

sub-trees

or

devices)Slide54

FSI Monitor in FSM

54

ON

/OFF flag shows when FSI laser is ON (light in SCT), in the Barrel and/or both End-caps

. Note

FSI laser is usually ON, even when FSI partition is not running.

Access under: SCT -> INFRASTRUCTURE -> FSI

History plots show peak and mean power in each SCT region:

fibres 0 & 1 are Barrel, fibres 2 & 3 are both End-caps.Slide55

FSI Monitor in FSM

55

Persistent HV trips on certain SCT end-cap modules.

Very rarely, overpower of the laser may cause persistent HV trips on certain end-cap modules. The watch list below shows which modules are expected to trip and in which order, if the laser power were to exceed upper safety levels.

4114, 5929, 7834, 7937,4231, 3743, 6731, 7926

If persistent HV trips are seen on the above modules, first check whether FSI is ON using the laser power monitor in the FSM (previous slide). If so call the FSI expert with a list of which modules have tripped and in which region (barrel or end-cap). The FSI expert will reduce the laser power to this region, so the modules can be recovered.

If the ID cooling is about to be stopped (e.g. imminent power intervention) or restarted, please notify the FSI expert who will ensure the alignment system is running to record any interesting thermal motions.Slide56

ROD Racks

56

Access to ROD Crates

PowerSlide57

Remember...

57

Use TOP Level of the Tree to send detector to ON (GOTO_READY) and Standby (GOTO_STANDBY)

Inform the Shift Leader when going to ON or STBY. On their side they should ask you to do the transition.

BE AWARE

of the beam STATUS.

Cooling loop 186 is OUT

. This loop has a leak and can not be repaired without access to the detector volume.

Do not try to take OUT modules of the configuration without expert advice.

Check that the FSM tree is complete. Nothing should be out of the Tree (grey). But be aware that a few modules will be out and one of the cooling loops.

Document problems in the

e-log Call the DCS expert (162035) if you get ALARMS. Investigate the WARNINGS.Slide58

Data Quality Monitoring

Part VI - Instructions for the SCT ACR ShifterSlide59

Check-List

59

DQ Check-List

Every hour the list will pop-up for you to fill

Read the comments and look to the documentation if have any question

Comment

Help

IF Problem found :

Check in the List of known problems and monitoring documentation

Call

Monitoring expert

(160037) if : plots stopped updating, monitoring processes crashed,... and you do not have instructions on how to actSlide60

Accessing Apps

60

Main Monitoring Application

s

.

From the

Main tab

in the DAQ Panel :

DQM Display

: this application displays a sub-set of the histograms with tests applied to them.

OHP : a larger set of histograms is available in it. Results of the check of the DQM are also shown there.OHP ConfigurationButtons to OHP and DQMMain tabSlide61

OHP

61

Online Histogram Presenter – OHP

If

NULL

monitoring could be not working or between updates

Colors show the result of the Checks from DQM

Links to different tabsSlide62

DQM Display

62

Data Quality Monitoring Display – DQMD

Colors :

GREEN

,

YELLOW

,

RED

show status of the checks

Grey

: Out of the Configuration or Masked (Link Masking + Stopless removal)SCT Tree SCT tabBad Module - REDSlide63

Histogram Display

63

Displays the content of the Information Server (IS)

Histograms will be found under

ATLAS

partition

Providers are called :

SCT-PT-X

while

SCT-PT

will be the gathered information from all of them

SCT-PT is the important one for the DQM. If this is not updated DQM will not update Contains all histograms created by Monitoring. Useful for further debuggingMon Advanced tabLast Update (60 s)Press to get last updateSlide64

MDA

64

Browse results for previous runs

Directory structure is the same as the one in the Monitoring files and in IS

For a given run files are saved every X (60) Luminosity Blocks (LB). An additional file is saved at the end of the run EOR

Useful to check if things are new or known

Mon Advanced tabSlide65

Summary Histogram

65

Displays a general view of the status of the system

Bins show an average over the current run

> 35 hits

> 70 % error rate

< 90 % efficient

> 0.001 NO

OUT + Masked Online

Large Number of Entries (>100) on ANY of the bins is

BAD!

Number of modules OUT of the configuration is around 30, rest is Masked which should be small.During WARM START, Noise and Inefficient will be high.Slide66

Remember…

66

Look at the Status of the DQM, if it turns red for Barrel or End-Caps, something really bad happened, if no clear reason –

call DAQ expert

(

162034

)

Check DQM if several modules (around 50) go

RED

. Investigate the problem –

call DAQ expert

(162034)Loop over the histograms in OHP especially:Summary histograms – Look at contents of each of the bins, check for increases. BAD! If an increase of more than 30Error summaries – Some modules could show some errors but a low rate, check the scale.BAD! If more than 30 modules with around 100% error rateNoise Occupancy – Check for a lot of modules with high occupancy (red) or white modules. Few white modules are worse than few red ones. BAD! White modules after enough statistics or lots of red ones.Document problems and read documentation during or previous to your shift. If found a problem call expert.Slide67

Problems

Part VII - Instructions for the SCT ACR ShifterSlide68

Problems – I

68

SCT sets a BUSY:

Stop-less removal will start and a window will appear in the Run Control Desk. If this does not happen you can use the TTC menu. Ask Shift Leader to restart a run if more than 3 Barrel

RODs

or 7

RODs

(Barrel +

EndCap

) get

disabled.

Frequency Veto: A bunch of messages like these or similar appear in MRS:Something is not working properly in the Trigger. Ask the RunControl shifter to check it. Good practice would be set triggers to HOLD while investigating.Monitoring stops updating:Easiest way is to restart the whole monitoring segment, but probably restarting the Gatherer would work.Slide69

Problems – II

69

Cooling goes OFF:

Call the Cooling expert (

162449

) and the SCT Run Coordinator. Document the alarms and errors from the FSM. Follow the error documentation procedure described in:

Shifter Manual page



Cooling



Infrastructure FailuresPower Cut:Same as above.….Slide70