Oslo Group Energy Statistics - PowerPoint Presentation

Oslo Group Energy Statistics8-11 May 2017ESS Data Validation and exchange (SDMX) Bart De NorreUnit E.5: EnergyEuropean Commission – DG Eurostat 1

OverviewContextESS shared validationSDMXDSD Energy Eurostat ESS Future2

ContextEuropean Energy Statistics: Energy Statistics RegulationsRegulation 223/2009 on European statisticsContent, Quality, Responsibilities (MS, Eurostat)International:IEA/EUROSTAT/UNECE data collectionsIRES / SIECCommon data needs / methodology (validation) / reportingEurostat and the ESS – the ESS Vision 2020ESS global policies / standards / projects (all domains)togetherEurostat, the ESS and world wide – SDMX3

ContextQuality:Criteria / Code of practice ( accuracy, coherence, timeliness, cost-effectiveness) Content: data and metadataProcessesActors / stakeholdersCommunication and exchangeExchanges at many levels: countries – international bodies, within country between different authorities; countries – countriesReporting countriesMost difficult part (subsidiarity / proportionality); many aspects in accuracy can only be done by collecting authorities; receiving aggregated data limits many accuracy checks to plausibility checks Increasing demands / stable or shrinking resources4

ContextQuality, the role of: Collaboration and mutual common understandingFormalising knowledge (data, metadata and validation logic)Who does what (formalising responsibilities)Information processing (IT)Facilitating exchange at collection, at dissemination (push/pull)Standardisation5

Context6 Standardisation: content (data, code lists, definitions and other domain methodology)methodology (validation typology and logic)business processes (GSBPM)generic outcomes (validation reports)information system design, IT "technology"within a domain / across domains (using/comparing data across several domains)understanding and applying similar "standardised" approaches is especially useful for NSI and Eurostat working in many domains ("industrial statistical processing"; metadata driven generic IT processes)Economics of scale

Context7 Eurostat / ESS global objectives for ESS shared validation and SDMX: increaseEffectiveness (result)Efficiency (process)Related projects to a higher quality at a lower cost:less errors, more certainty and more timely outcomesreduction in resources and elimination of redundant actions.A few examples of targeted changes:validation comprising rules which are formally adopted by all ESS stakeholders;sharing the same understanding via standards and more formalised knowledge;a more automatized validation process with clear responsibilities and avoiding multiple iterations (ping-pong)

ESS shared validationThe idea: collaborative approach of sharing validation between countries and Eurostatin fact, is this principle so new?Major objectives: increasingtransparency and clarity(shared and easily accessible documentation on validation procedures);effectiveness of the validation(the appropriate rules by the responsible authority close at the source, awareness of all possible errors and warnings, avoiding validation gaps, avoiding double work)efficiency(avoiding the number of iterations between sending – validating – requesting new sending)efficiency in the information processing and communication(The lack of common standards for validation solutions leads to a duplication of IT development and integration costs in the ESS).8

ESS shared validationESSC May 2016: "agreement and documentation at Working Group level of validation rules and responsibilities" Mandatory!"Use of shareable and reusable ESS services to validate data"What does this implyCommon validation policy / methodology / typologyvalidation levelsValidation responsibilitiesValidation rules and severityImproving inter-operability9

ESS shared validationValidation typology 10

ESS shared validationHow and whoBasic principles Design data structuresDesign validation rules (and severity level)A full description and agreement on the validation rules and their associated severity for each data collectionValidate data: 2 major steps/ actors (reporting country, Eurostat)An agreement on the responsibility of each national reporting organisation to perform the agreed validation rules beforeLong term process / in steps / Taskforce 11

12 ESS shared validation

13 ESS shared validationScenario: Autonomous / Interoperable validation services

14 ESS shared validationScenario: Replicated/Shared validation services

15 ESS shared validationScenario: shared validation process

16 ESS shared validation

SDMX Statistical data and metadata exchangeSDMX – adopted and used more and more in ESSOrganisational context and logistics Developments for energy statistics (Supply – transformation – consumption; Electricity and gas prices)17

SDMX Common open standards for data and metadataaccepted worldwide for exchanging and sharing statistical informationand as a general basis for statistical infrastructures. Started in 2001 by seven international organisations(BIS, ECB, Eurostat, IMF, OECD, UN and World Bank)ISO standard (ISO 17369:2013)SDMX Roadmap 2020 (March 2016):Strengthening implementation of SDMXMaking data usage easier via SDMXUsing SDMX to modernise statistical processesBetter communication and capacity-building (including interaction with ESS Vision 2020 and UNECE modernisation of official statistics)18

SDMX Promoted by the European Statistical System Policy and big enabler for the ESS VISION 2020reduce data errorsimprove timelinessimprove accessibilityimprove interpretabilityimprove coherencereduce the reporting burdenreduce IT development and maintenance costs (with open source approach, shared toolbox and improved interpretability).For future IT development: standards independent of domain specific structures"If each partner system were to use SDMX data structures and common IT building blocks, international information systems would be able to communicate ‘machine-to-machine’ as in industrial production processes."ESSC: SDMX one of the non-legislative normative documents to be recognised as ESS standards19

SDMXUse of SDMX in Eurostat – ESS statistical domains40% of European statistical production processes are now describing their data structures using the SDMX standards and concepts. ESS: 26 SDMX implementation projects, related to about ⅖ (38%) of all the data sets that Eurostat receives through EDAMIS Further increase: 2016: +10; 2017: +4GLOBAL DSD: 5https://webgate.ec.europa.eu/fpfis/mwikis/sdmx/index.php/SDMX_DSD_availability 20

SDMXstatistical content and the information processing (IT) perspectives Standards on logical and technical level:"Content""Container"Development of SDMX artefacts according SDMX methodology, guidelines and conceptsFor example information model, content oriented guidelines, cross-domain conceptsUse of SDMX objects by generic programs/services"Embed logic in data objects"21

SDMXOrganisation and logistics International agreement <= SDMX secretariatSupport and management by SDMX IT architecture and toolsfor example a central repository (Euro SDMX registry)Maintenance responsibility by a selected international agency22

SDMXFor Energy Statistics :Supply-consumption chain => "semi-global" DSD development with IEAFirstly collection/validation/production processAfterwards disseminationEurostat DSD on PricesFirstly DSD (data structure definition)Afterwards MSD (metadata structure definition)Still later: use of VTL (Validation and Transformation Language)23

DSD Energy statisticsSupply-Consumption chain: Joint annual data collections (JAQ) of IEA-Eurostat-UNECE; monthly data collections (joint, IEA, Eurostat)IRES and SIECInternational / Standards / Classification(s)SDMX guidelines:MODELLING A STATISTICAL DMAIN FOR DATA EXCHANGE IN SDMXhttps://sdmx.org/wp-content/uploads/Modelling-domain-SDMX-discussion-paper-v1-201503.pdfTHE DESIGN OF DATA STRUCTURE DEFINITIONShttps://sdmx.org/wp-content/uploads/SDMX_Guidelines_for_DSDs_1.0.pdfTHE CREATION AND MANAGEMENT OF SDMX CODE LISTShttp://sdmx.org/wp-content/uploads/SDMX_Guidelines_for_CDCL.doc24

DSD Energy statistics Joint development Eurostat - IEA IEA had done work in 2009-2012Eurostat proposed IEA in May 2015 to take up together DSD developmentSeries of video conferences / mail exchanges since October 2015Explanatory document + Inventory code listsTesting DSD mapping with some questionnaires on-goingGradual verification/update(revised monthly coal, new monthly electricity, revised JAQ 2017)25

DSD Energy statisticsConcepts in the energy domain Definition (concept code name and description)Role (dimension, primary measure, attribute)Level (attribute relevant at observation, series or dataset level)usage status (mandatory or conditional/optional attributes)code list / format26

DSD Energy statisticsDimensions / structural principles (SDMX guidelines) Parsimony: no redundant dimensions for identifyingSimplicity: keep identifiers short / keep number of dimensions lowPurity: dimensions relate to one pure concept, not to a combinationDensity and sparseness ("not available" values in the dimension combinations)Unambiguousness (avoid one observation to be expressed by multiple combinations of dimension values (keys))Exhaustiveness(includes every piece of information that is required to unambiguously represent a data point and to correctly interpret it outside its usual context)27

DSD Energy statisticsOrthogonality: independence of the meaning of a value of one dimension from the values of any other dimensionsUser friendliness (While a simple DSD consisting of a few dimensions only may be easier to understand by a human data consumer, a more complex, but purer DSD is typically more flexible in terms of further usage in automated processes.)Fitness for use throughout the entire statistical business processRe-use concepts / code lists(frequency, observation status)Extensible for potential future needs 28

DSD Energy statisticsFurther design principles:designed independently of the layout or technical features of existing EXCEL questionnaires and database structures in place at IEA and Eurostatone DSD which is based on a clear logical model and flexible enough to cover all data and metadata from all concerned questionnaires "Remarks" sheets (Attribute "COMMENT_OBS"; explanatory "free text" to future MSD) 29

DSD Energy statistics 14 DIMENSIONS (identifying concepts) QUEST_SOURCEREF_AREATIME_PERIODFREQENERGY_PRODUCTMAIN_FLOWFLOW_BREAKDOWNPLANT_TECHPLANT_TYPESTOCKSINFRASTRUCTURE_INDVIS_A_VIS_AREAMEASURE_VALUE_TYPEFACILITY_ID30

DSD Energy statistics REF_AREA SDMX promotes one code list across SDMX domainsBased on the one of National AccountsENERGY_PRODUCTall primary and secondary energy products or commodities and their aggregates as used in the energy questionnaires (and energy balances)Align to SIECSIEC doesn't contain all our productsAlign to SDMX code list guidelinesCodes based on SIEC hierarchical numbering 31

DSD Energy statistics MAIN_FLOW and FLOW_BREAKDOWN two-level hierarchical approach according IREScodes for MAIN_FLOWProductionNet production (of electricity or heat)Gross production (of electricity or heat)ImportsExportsInternational marine bunkersInternational aviation bunkersStocksTransfersSupplyAround 110 codes for FLOW_BREAKDOWN32 Statistical differences Demand Transformation Consumption Energy use Non-energy use Losses

DSD Energy statistics MAIN_FLOW and FLOW_BREAKDOWN Many main flows are split in more detailed flows There are multiple electricity and heat production flows because of all possible energy input, plant technologies and plant types.Stock changes apply to a broad diversity of types of stocksImport and export flows are detailed by the country/region from where is imported resp. the country/region to where is exportedFor some detailed flows additional dimensions are needed: PLANT_TECHPLANT_TYPESTOCKSVIS_A_VIS_AREA.33

DSD Energy statistics PLANT_TECH: technologies used in plants for production of electricity and / or heatThis code list is not a straight hierarchical classification: different perspectives are used in classifying/grouping power and heat plants (product based, single/multi-fired, and technical type of generation) COMBFUEL - Combustible Fuels; HYDRO - Hydro (all, unspecified)PLANT_TYPE:main classification of electricity and heat plantsMAINELEC - Main Activity Producer Electricity PlantsINFRASTRUCTURE_IND:A number of data in the questionnaires describe infrastructure characteristics GROSSCAP - Gross capacity (of electricity and/or heat);SOLARSUR - Solar collectors surface34

DSD Energy statistics MEASURE_VALUE_TYPE several measurement concepts used in reporting of energy data valuesENERGY Measure of Heat or Electricity; NCV Net Calorific Value VIS_A_VIS_AREA (COUNTERPART_AREA) FACILITY_IDan identifier key for the storage locations for gas and refineries (from the JAQ 2017 onwards)35

DSD Energy statistics OBS_VALUE 6 ATTRIBUTES:UNIT_MEASUREKT – Kilotonne; TJ_NCV - TeraJoule (NCV)OBS_STATUS (SDMX Standard)normal, missing, estimatedCONF_STATUS (SDMX Standard)SUBMISSION (date of the submission of the questionnaire)COMMENT_OBS (short free text related to one or more observations, dataset)FACILITY_TYPE (types of gas storage facilities and refineries)FACILITY_NAME36

SDMX SIEC based code list ENERGY_PRODUCT37

Example monthly coal38

Example monthly coal39

Example monthly electricity40

Example monthly electricity41

Example monthly electricity42

Example monthly electricity43

Eurostat ESS future44 A full global ESS Shared Validation / SDMX implementation is a long term issueAwareness and knowledge building is highly neededWell informed, active participation to be able to endorse together a realistic planningBefore real production some pilot projects with volunteering countries are neededImplementation shall take place in steps over yearsAvoid burden in the transition phases for reporting countriesAllow options for "fast implementers"

Eurostat ESS future45 "ESWG Task Force on the implementation of ESS shared validation and SDMX "the development and review of data structures definitions, code lists and other SDMX artefacts (including their validation and testing)development and assessment of the implementation (including timeline for the implementation)establishment of specific operational guidelines for the energy domain, including links to the ESS shared validationprepare and advise on recommended actions and implementations for the ESWG meetings

Bart.Denorre@ec.europa.eu 46