Quality of MIP indicators Assessment of data and metadata ESTP Course Luxembourg 911 December 2014 Ivana Jablonska amp Julien Bollati MIP TF Outline Quality framework of MIP Indicators Work ID: 765512
Download Presentation The PPT/PDF document "Quality of MIP indicators: Assessment of..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Quality of MIP indicators: Assessment of data and metadata ESTP Course Luxembourg 9-11 December 2014 Ivana Jablonska & Julien Bollati , MIP TF
OutlineQuality framework of MIP IndicatorsWork done by nowFuture steps / Discussion
Quality framework of MIP Indicators
(Draft) MIP RegulationCOM(2013) 342 FinalArticle 5The Commission (Eurostat) shall regularly assess the quality of the MIP relevant data (…). The quality assessments shall, as appropriate, make full use of the work carried out, and the results obtained, in the context of existing quality frameworks for MIP relevant data.
Quality principlesEurostat mission: to be the leading provider of high quality statistics on Europe European statistics Code of Practice ( 28th September 2011)
Quality principlesPublic commitment on European Statistics by the ESCB
What for?The big question:What about the quality of the MIP indicator?Our preliminary answer:In order to answer to this question we need to have a look to the inventories/quality report and run a risk assessment in the framework of a stocktaking exercise. Per country and per indicator
What for?The small question:How safe are the MIP indicator?Our preliminary answer:Let's run an expert opinion poll among our team in order to make a ranking. Per country and per indicator
Our safeness definitionsSafe: Information on sources and methods is clear in inventories/quality reportsSources cover the necessary basic informationThe compilation practices are in line with legal requirements and good/best practices
Our safeness definitionsRisk under control: Information on sources and methods is generally available and mostly clear in inventories/quality reportsSources cover most of the necessary basic information, estimation methods are only used to compensate for a small part of the basic informationCompilation practices are in line with legal requirements, but most other Member States use different practices
Our safeness definitionsPotential risk: Information on sources and methods is partially available in inventories/quality reports, or fully available but suggesting an incomplete implementation of the methodology The sources of basic information are incompleteCompilation practices are not adequate compared to other Member States or not in line with legal requirements
Our safeness definitionsNot known: Information on sources and methods is generally poorIt is not possible to quantify whether there is any risk of significant revisions to the data
Guiding principles Factual assessment of data and metadata (vs perception) Standardised approach towards different domains and countries
The outcomesBased on the reading of data, metadata and quality reportsFootnotes and proposed text for the Statistical Annex to the AMRCheck for quality improvements with respect to the previous yearClustering by MIP headlines according to risk profile
Work done by now
The questionnaireWe have developed a structured template with over 30 questions
Topics coveredInstitutional environmentAuthority responsibleLegal and institutional environmentSharing of responsibilitiesResourcesAdequacy of resourcesCost and burdenPrinciple 1Professional independencePrinciple 3Adequacy of resourcesPrinciple 2 Mandate for data collection Principle 6 Impartiality and objectivity Principle 5 Statistical confidentiality Principle 10 Cost effectiveness CoP / PC CoP / PC
Topics coveredQuality ManagementCompleteness and timeliness of information provided in the existing inventories and quality reportsQuality control proceduresClarityCommunication with EurostatPrinciple 4Commitment to qualityPrinciple 8Appropriate statistical proceduresCoP / PC
Topics coveredMethodological soundnessReliability of the methodologyImplementation of regulations/guidelines/recommendationsExpiry of derogationsUnexplained breaks in the seriesPrinciple 7Sound methodologyCoP / PC
Topics coveredRevisionsSize of routine revision policiesInformation on major revisionsData analysisCompletenessTimelinessPrinciple 12Accuracy and reliabilityPrinciple 13Timeliness and punctualityCoP / PCCoP / PC
Topics coveredInternal coherenceAggregation checks, outlier testsExternal coherenceConsistency with similar or related data setsOther risksOthersPrinciple 14Coherence and comparabilityCoP / PC
Sources usedBoP IndicatorsEurostat databaseBoP book 2007 (ECB)SDDS (IMF)Quality reports 2013 (Eurostat)Assessment of the QR 2012 (Eurostat)Quality Report on BoP of MS to European Parliament 2011 (Eurostat)Quality Report on Euroarea data 2013 (ECB)
Sources usedFinancial Sector IndicatorsEurostat databaseManual on sources and methods for the compilation of ESA95 financial accounts 2002 (Eurostat)Manual on sources and methods for the compilation of ESA95 financial accounts, 2nd edition – 2011 (Eurostat)Communication from the Commission to the European parliament, the Council and the Eurogroup (Eurostat)Other (websites, other documents)
Sources used General Government DebtEurostat databaseEDP Inventories on sources and methods (MS to Eurostat)EDP Mission reportsOther (websites, other available documents)
Sources usedShare of world exportEurostat databaseEuropean Union balance of payments/international investment position statistical methods (ECB)SDDS - Balance of Payments (IMF)BOP Quality Reports (Eurostat)
Sources used Nominal Unit Labour Cost IndexDatabases (Eurostat, OECD, National institutes)SDDS metadata (IMF)Joint OECD/Eurostat questionnaire on NA employment and hours workedTask Force Report on the Quality of the LFS 2009 (Eurostat)
Sources usedHouse Price IndexEurostat DatabaseHPI inventories mapped to ESMS (Eurostat)
Sources usedUnemployment rateEurostat databaseQuality reports (Eurostat / MS)
Sources usedReal Effective Exchange Rate"Self assessment" made by Eurostat
Fitness IndexOn the basis of the scores assigned to the different questions we have compiled a single Fitness for Purpose index.This was obtained aggregating the quantitative scores and the weights assigned to the questions. The final index is normalised and ranges between 0 (maximum risk) and 100 (totally safe).
BenchmarkingOur answers were checked by domain managers in Eurostat. More in detailed they were asked to:Challenge our scoresComplement with additional information (if any)Special case for General Government Debt
Assessment by CountryThe information is already partially available but it has not been fully exploited yet.Example
Evolution of the Safebook excercise20132014Only internal MIP projectEndorsed by Eurostat managementMIP team making assessmentBenchmarked by domain managersOnly IDR Countries (+ Croatia)All Member StatesImproved questionnaire
FutureSelf-assessment by domain (short term)Assessment by Member States (long term)Run the exercise for MIP auxiliary indicators
What is your experience?Brainstorming group workHow would you imagine doing a similar quality assessment exercise?Is a quality assessment done for your domain?If yes, how is it done?If not, would it be needed?