ANT Team SKA Common Alarms Handling - PowerPoint Presentation

1. ANT TeamSKA Common Alarms HandlingAndrea DeMarco, Elisabetta Giani, Alessandro Marassi, Lize van den HeeverTANGO Harmonization Meeting (Edinburgh)TANGO Core AlarmsANT Team ObjectivesTANGO Alarm Extensions RationaleElettra Alarm SystemFormulaConfExperimental Work/Prototyping

2. ANT Team ObjectivesEvaluate the TANGO core alarm systemEvaluate the TANGO alarm extensions by Elettra (C++) and ALBA (Python)Discuss if it would be opportune to allow for LMC to use both these extensions, or only one of them (or none, meaning a custom SKA Alarm system should be implemented)Evaluate the LIG requirements for alarms and whether these can be satisfied to the full by the chosen systemLook into whether we can define a common interface (or function) within a future SKA Generic Device that allows for any element to declare/undeclare alarms in a uniform mannerDiscuss alarm hierarchies within element, and element to central coordination

3. TANGO Core Alarms: SetupThe standard mechanism to generate attribute alarms in the Tango core is the following:Alarm Configuration: define attributes and set their alarm configuration levelsAlarm Condition: When alarm/warning thresholds are crossed, the attribute quality factor is set to ATTR_ALARM/ATTR_WARNING and the device State attribute is switched to ALARM.Restore of Normal Conditions: When the attribute value returns to normal range, its quality factor is set to ATTR_VALID and the device State attribute set to ON.

4. TANGO Core Alarms: ClientsClients wanting to be notified that an attribute alarm condition occurred in a device server shall perform these actions:Subscribe to events (e.g. CHANGE_EVENT/PERIODIC_EVENT/...) defined for all server attributes for which an alarm configuration is set. When receiving attribute events, the attribute quality factor can be accessed and an alarm condition checked.Subscribe to CHANGE_EVENT for server State and Status attributes and parse them.

5. TANGO Core Alarms: DrawbacksClients need to subscribe and monitor all server attributes and parse the quality factorDetecting the State change alone would not provide sufficient information to know which of the attribute(s) was responsible for the alarmRequired: unique source of alarm informationRequired: difference classes of alarms e.g. Communication, Quality-Of-Service, Software Processing/Exception, Equipment, Environmental, UserRequired: different alarm priority levelsThe idea of a new device event, e.g. ALARM_EVENT, a standardized device ALARM_EVENT was therefore advanced for this purpose

6. TANGO Alarm Extensions RationaleSKA is not alone in requiring more robust alarm acknowledgement and more complex alarm definitionsElettra/Alba developed extensionsAlarms can still be setup in the TANGO coreExtensions have a concept of raising events, in the same way as requested by most elementsExtensions also execute specified callback functions when alarms go ON/OFFExtensions allow complex formulae, useful for logical/high-level alarms

7. ELETTRA:Alarm System

8. ELETTRA: Alarm FormulaeFormula can contain attribute names, numbers, Tango States, logical and mathematical operators. Expressions can be enclosed by round brackets.Attribute names have to be of four fields separated by “/”, ex: b/dev/test/currNumbers can be real numbers (ex: 35, -23.5, ...) or hexadecimal (ex: 0xaf, 0x1A)Tango States allowed are ON, OFF, CLOSE, OPEN, INSERT, EXTRACT, MOVING, STANDBY, FAULT, INIT, RUNNING, ALARM, DISABLE, UNKNOWNOperators allowed are the following in order of precedence: *, /, +, -, <<, >>, <=, >=, >, <, ==, !=, &, |, ^, &&, ||. Unary operators abs, ! are allowedAn example:(b/dev/test1/val * 2.5 >= abs(b/dev/test2/val) && (b/dev/test3/state == OFF) || b/dev/test4/stat & 0xA0)

9. ELETTRA: Callback Methods IThe Elettra formulae allow the definition of two callbacks for alarm configurations, one for alarm entry, another for alarm exit.These two commands can be specified as the last field of the alarm string given to the “Load” command and have to be separated by the “;” character. This field is optional.An example:b/sec/test/curr (b/sec/test/current > 100) 0 log gr_all “Alarm Curr! b/sec/dev1/action;b/sec/dev2/action”command b/sec/dev1/action is executed when changing state to ALARMcommand b/sec/dev2/action is executed when changing state to NORMAL

10. ELETTRA: Callback Methods IICallbacks either have no arguments (Tango::DEV_VOID), or take in a single string (Tango::DEV_STRING)E.g. For the alarm loaded with the following:b/sec/test/dev ((b/sec/test/current > 100) && (b/sec/test/voltage > 20)) 30 log gr_all “Alarm Device!” b/sec/dev/notif_alarm;b/sec/dev/notif_normalThe value passed to b/sec/dev/notif_alarm if it accepts a Tango::DEV_STRING as input would be:b/sec/test/dev;b/sec/test/current=101.75;b/sec/test/voltage=22.6

11. Experimental WorkStart polling on the attributes needed for a specific rule.Event firestarters need to be configured for the concerned attributes as well e.g. absolute/relative change criteria for change events to be fired by the TANGO kernel.An alarm rule is then configured in the Elettra Alarm device server - which automatically sets up subscriptions to the change events of the involved attributes etc.The TANGO core is only used as a source of events - the Elettra alarm server will itself make checks on specific values (and other logical conditions e.g. combined alarms) and decide for itself whether to raise an alarm event or not.The TANGO core alarm levels can still be used separately if needed (e.g. to change Quality factor, which in turn could be utilized by the alarm system), but can otherwise be bypassed.

12. FormulaConf IAnother TANGO device, works in tandem with Elettra Alarm SystemEssentially a parser generating dynamic attributes for each alarm formula. Types are (Bool, Long, Double, spectrum of the same types)Handles complex alarm formulaeProperties need to be set up:Attr_config_prop: an array of strings of the form:attr;type; dynamic attribute name; index; formula; description (optional)Attr_config_file: the path of the file with an array of stringsPolling_period: if = -1, disabled, else period (in sec.) of the internal polling threadRead_timeout: if internal polling is enabled, this is the time (sec) after which the attribute is read even if the corresponding event is not in error.

13. FormulaConf IISame rules of Elettra Alarm System apply – polling and eventingPerhaps a generic way of doing this can be formulated by the SKADevices have to be available prior to FormulaConf startup, otherwise a restart is neededE.g.admin_proxy = DeviceProxy(formula_conf_proxy.adm_name())admin_proxy.restartserver()

14. AAVS Alarm Stream Device PrototypeLFAA/AAVS has prototyped an Alarm Stream deviceAny number of stream devices can be deployedE.g. streams for different levelsCentralized/uniform alarm handlingMore on this in later sessions