Eurostat Big Data Eurostat Effective Processing
MT
Published · 46 slides · 0 views
1 / 1
Description
Eurostat Big Data Eurostat Effective Processing and Analysis of Very Large and Unstructured data for Official Statistics. International activities on Big Data in Official Statistics Carlo Vaccari Istat (vaccariistat.it) THE CONTRACTOR IS
Related Topics
Share
Embed code
Download this presentation From Below
"Eurostat Big Data Eurostat Effective Processing" is the property of its rightful owner. Permission is granted to download and print the materials on this website for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Presentation Transcript
01
Eurostat Big Data Eurostat Effective Processing and Analysis of Very Large and Unstructured data
for Official Statistics. International activities on Big Data in
Official Statistics Carlo Vaccari
Istat (vaccari@istat.it) THE CONTRACTOR IS ACTING UNDER A FRAMEWORK CONTRACT CONCLUDED WITH THE COMMISSION<br>
for Official Statistics. International activities on Big Data in
Official Statistics Carlo Vaccari
Istat (vaccari@istat.it) THE CONTRACTOR IS ACTING UNDER A FRAMEWORK CONTRACT CONCLUDED WITH THE COMMISSION<br>
02
Eurostat International cooperation in Statistics Eurostat<br>
03
Eurostat High Level Group High Level Group for the Modernisation of Statistical Production and Services (HLG)
The members for 2017 are:
Pádraig Dalton (Ireland) – Chairman
Trevor Sutton (Australia)
Anil Aurora (Canada)
Giorgio Alleva (Italy)
Bert Kroese (Netherlands)
Liz MacPherson (New Zealand)
Yoo Gyeongjoon (Republic of Korea)
Genovefa Ružić (Slovenia)
Mariana Kotzeva(Eurostat)
Martine Durand (OECD)
Lidia Bratanova (UNECE)
Secretariat: UNECE Eurostat<br>
The members for 2017 are:
Pádraig Dalton (Ireland) – Chairman
Trevor Sutton (Australia)
Anil Aurora (Canada)
Giorgio Alleva (Italy)
Bert Kroese (Netherlands)
Liz MacPherson (New Zealand)
Yoo Gyeongjoon (Republic of Korea)
Genovefa Ružić (Slovenia)
Mariana Kotzeva(Eurostat)
Martine Durand (OECD)
Lidia Bratanova (UNECE)
Secretariat: UNECE Eurostat<br>
04
Eurostat HLG Vision Eurostat The product challenge (from the Document on Vision)
Changeover from a society with little or no data available to one that has an abundance of data.
In this light we have to rethink our traditional business values and the reasons of our existence.
Now other organisations developing statistics similar to ours but much, much quicker (e.g. Google) and on an almost global scale.
We also see other very interesting uses of statistics, prompted by the availability of so much data.<br>
Changeover from a society with little or no data available to one that has an abundance of data.
In this light we have to rethink our traditional business values and the reasons of our existence.
Now other organisations developing statistics similar to ours but much, much quicker (e.g. Google) and on an almost global scale.
We also see other very interesting uses of statistics, prompted by the availability of so much data.<br>
05
Eurostat HLG Vision Eurostat Vision on products:
Data are everywhere and are much cheaper than they used to be. Slowly people are beginning to understand the real value of this fact.
Social networking sites and search engines are now perceived as data collection devices. NSIs are in a unique position to connect to the data of the emerging information society and transform them into something useful.
As the global dimension of events gains importance we can no longer work on a national level only. We need to expand our work and deliver products that explain what is happening on a multinational level.<br>
Data are everywhere and are much cheaper than they used to be. Slowly people are beginning to understand the real value of this fact.
Social networking sites and search engines are now perceived as data collection devices. NSIs are in a unique position to connect to the data of the emerging information society and transform them into something useful.
As the global dimension of events gains importance we can no longer work on a national level only. We need to expand our work and deliver products that explain what is happening on a multinational level.<br>
06
Eurostat HLG Vision Eurostat Vision on products:
In the chapter dedicated to “Rationalising processes” there is a proposal
“(d) Develop new methodologies to reflect the changes in data acquisition and the dramatic increase of the volume of data available, for example, on topics such as noise and error reduction in large data sets, pattern recognition and other methodological tools appropriate for "Big Data"”.
in March 2013 the HLG published the paper “What does Big Data mean for Official Statistics”<br>
In the chapter dedicated to “Rationalising processes” there is a proposal
“(d) Develop new methodologies to reflect the changes in data acquisition and the dramatic increase of the volume of data available, for example, on topics such as noise and error reduction in large data sets, pattern recognition and other methodological tools appropriate for "Big Data"”.
in March 2013 the HLG published the paper “What does Big Data mean for Official Statistics”<br>
07
Eurostat MSIS 2013 Eurostat In Meeting on Statistical Information Systems (MSIS 2013) many points of discussion focused on Big Data. In the summary of the session on Collaboration, the Session Organizer asked “How can we collaborate on Big Data? Many initiatives and groups are involved in collaboration. How can the MSIS group contribute most effectively in this new structure?”
Also a special panel session was organized with the title: “Plug and play architecture and collaborative development will allow us to accelerate our Big Data programs by ...” The panel discussion followed a Pecha Kucha format: 20 slides for 20 seconds each.<br>
Also a special panel session was organized with the title: “Plug and play architecture and collaborative development will allow us to accelerate our Big Data programs by ...” The panel discussion followed a Pecha Kucha format: 20 slides for 20 seconds each.<br>
08
Eurostat Big Data Task Team Eurostat In May 2013 a temporary task team was set up, composed by members coming from thirteen organizations. The task team, working virtually through teleconferences and sharing documents on the wiki, started to define the key issues with using Big Data for official statistics, identify priority actions and formulate a project proposal.
Two preliminary activities:
a classification scheme for Big Data sources
the development of an inventory containing structured and searchable information about actual and planned use of Big Data in statistical organizations.<br>
Two preliminary activities:
a classification scheme for Big Data sources
the development of an inventory containing structured and searchable information about actual and planned use of Big Data in statistical organizations.<br>
09
Eurostat Big Data Project 2014 Project presented to HLG and CES
Task team composed by people from 13 organisations The project consists of four Work-packages:
WP1 - Issues and Methodology
WP2 - Shared computing environment ('sandbox') WP3 - Training and Dissemination
WP4 - Project Management Eurostat<br>
Task team composed by people from 13 organisations The project consists of four Work-packages:
WP1 - Issues and Methodology
WP2 - Shared computing environment ('sandbox') WP3 - Training and Dissemination
WP4 - Project Management Eurostat<br>
10
Eurostat Big Data Project 2014: the Sandbox Sandbox evaluates the feasibility of the following propositions:
'Big Data' sources can be obtained (in a stable and replicable way), and manipulated with relative ease and efficiency on the chosen platform, within realistic technological and financial constraints
The chosen sources can be processed to produce statistics which conform to the usual quality criteria used to assess official statistics
The resulting statistics correspond in a predictable way with existing mainstream products, such as price statistics, HBS indicators, etc.
Platforms, tools, methods and datasets can be used in similar ways to produce analogous statistics in different countries
The different participating countries can share tools, methods, datasets and results operating on the principles established in CSPA Eurostat<br>
'Big Data' sources can be obtained (in a stable and replicable way), and manipulated with relative ease and efficiency on the chosen platform, within realistic technological and financial constraints
The chosen sources can be processed to produce statistics which conform to the usual quality criteria used to assess official statistics
The resulting statistics correspond in a predictable way with existing mainstream products, such as price statistics, HBS indicators, etc.
Platforms, tools, methods and datasets can be used in similar ways to produce analogous statistics in different countries
The different participating countries can share tools, methods, datasets and results operating on the principles established in CSPA Eurostat<br>
11
Eurostat Big Data Project: 2014 Virtual Sprint (March 2014) → new document Workshop in Roma (April 2014)
Sandbox installation and verification
Testing scenarios for BD usage in Official Statistics:
use as auxiliary information to improve an existing survey replacing all or part of an existing survey with Big Data producing a predefined statistical output either with or without
supplementation of survey data
producing a statistical output guided by findings from the data Eurostat<br>
Sandbox installation and verification
Testing scenarios for BD usage in Official Statistics:
use as auxiliary information to improve an existing survey replacing all or part of an existing survey with Big Data producing a predefined statistical output either with or without
supplementation of survey data
producing a statistical output guided by findings from the data Eurostat<br>
12
Eurostat Shared computation environment for the storage and the analysis of large-scale datasets
Platform for collaboration across participating institutions ● ● Data-derived ● Explore tools and methods
Test feasibility of producing Big statistics
Replicate outputs across countries with support from the Eurostat Created Central Ireland Statistics office (CSO) of and the Irish Centre for High-End Computing (ICHEC) Cluster of 28 machines Accessible through web and SSH
Software: full Hadoop stack, visual analytics, R, RDBMS, NoSQL DB Objectives The Sandbox 2014<br>
Platform for collaboration across participating institutions ● ● Data-derived ● Explore tools and methods
Test feasibility of producing Big statistics
Replicate outputs across countries with support from the Eurostat Created Central Ireland Statistics office (CSO) of and the Irish Centre for High-End Computing (ICHEC) Cluster of 28 machines Accessible through web and SSH
Software: full Hadoop stack, visual analytics, R, RDBMS, NoSQL DB Objectives The Sandbox 2014<br>
13
Eurostat ICHEC Assisted the task team for the testing and evaluation of Hadoop work-flows and associated data analysis application software
Hortonworks provided free support on their open source Hadoop platform (Hortonworks Data Platform)
Italian distributor Bnova provided free license of the Platform product extension of trial Pentaho Enterprise for visual analytics Partnership Eurostat<br>
Hortonworks provided free support on their open source Hadoop platform (Hortonworks Data Platform)
Italian distributor Bnova provided free license of the Platform product extension of trial Pentaho Enterprise for visual analytics Partnership Eurostat<br>
14
Eurostat 7 Task Team for 2014 experiments Social Data
Mobile Phones Prices Smart Meters Job Vacancies Ads
Web Scraping Traffic Loops
Each experiment team produced a detailed report on its activity, available in the wiki Eurostat<br>
Mobile Phones Prices Smart Meters Job Vacancies Ads
Web Scraping Traffic Loops
Each experiment team produced a detailed report on its activity, available in the wiki Eurostat<br>
15
Eurostat Findings 2014 The results of the experiments were evaluated with respect to the initial objectives Eurostat SO1 Sources collection and manipulation
SO2 Production of quality statistics
SO3 Correspondence with existing products
SO4 Cross-country sharing
SO5 CSPA-based sharing
Details on the evaluation can be found in the Project Summary Report<br>
SO2 Production of quality statistics
SO3 Correspondence with existing products
SO4 Cross-country sharing
SO5 CSPA-based sharing
Details on the evaluation can be found in the Project Summary Report<br>
16
Eurostat Sandbox project in 2015 4 Data/Compute nodes
2 x 10 core Intel Xeon CPUs
128 GB RAM
4 x 4TB disks
56 Gbit InfiniBand network Installed Software
Hadoop (Hortonworks Data Platform)
R – Rstudio
RHadoop
Spark
ElasticSearch
and growing… 2 Service/login nodes
Similar hw as data nodes
10Gbit connection to Internet Eurostat<br>
2 x 10 core Intel Xeon CPUs
128 GB RAM
4 x 4TB disks
56 Gbit InfiniBand network Installed Software
Hadoop (Hortonworks Data Platform)
R – Rstudio
RHadoop
Spark
ElasticSearch
and growing… 2 Service/login nodes
Similar hw as data nodes
10Gbit connection to Internet Eurostat<br>
17
Eurostat Workgroups
Four main activities, for each task one multinational group Wikistats - Wikipedia hourly page views: use of an alternative data source
Twitter - Social media data: experiences comparison in tweets collection and analysis
Enterprise websites: the Web as data source - web scraping and business registers
Comtrade - UN global trade data: use of Big Data tools on a traditional data source Eurostat<br>
Four main activities, for each task one multinational group Wikistats - Wikipedia hourly page views: use of an alternative data source
Twitter - Social media data: experiences comparison in tweets collection and analysis
Enterprise websites: the Web as data source - web scraping and business registers
Comtrade - UN global trade data: use of Big Data tools on a traditional data source Eurostat<br>
18
Eurostat Wikistats Statistics on hourly page views of articles in the several language versions of Wikipedia Download and process to obtain manageable data
UNESCO heritage sites
Cities and Beaches
Popularity Touristic potential Eurostat Wikipedia page views is a potential source for many domains: tourism, culture, economic, …<br>
UNESCO heritage sites
Cities and Beaches
Popularity Touristic potential Eurostat Wikipedia page views is a potential source for many domains: tourism, culture, economic, …<br>
19
Eurostat Wikistats Widely used: 44% of EU 16-74
69% of EU 16-24 Public source (contents and metadata) Wikipedia seventh website ( Alexa)
Digital traces left by people in their activities Eurostat<br>
69% of EU 16-24 Public source (contents and metadata) Wikipedia seventh website ( Alexa)
Digital traces left by people in their activities Eurostat<br>
20
Eurostat Wikistats See peak for Vatican City in 2013 March for pope Francis election Eurostat<br>
21
Eurostat Twitter Source instability with technological changes
Comparison of Data Preprocessing
Data analysis: distinct targets, methods and tools Geolocated tweets 2015: two approaches: - - Public stream (Mexico , UK)
Buy data (UK) Eurostat<br>
Comparison of Data Preprocessing
Data analysis: distinct targets, methods and tools Geolocated tweets 2015: two approaches: - - Public stream (Mexico , UK)
Buy data (UK) Eurostat<br>
22
Eurostat Twitter Source instability with technological changes
Comparison of Data Preprocessing
Data analysis: distinct targets, methods and tools Geolocated tweets 2015: two approaches: - - Public stream (Mexico , UK)
Buy data (UK) Eurostat<br>
Comparison of Data Preprocessing
Data analysis: distinct targets, methods and tools Geolocated tweets 2015: two approaches: - - Public stream (Mexico , UK)
Buy data (UK) Eurostat<br>
23
Eurostat Twitter Eurostat<br>
24
Eurostat Twitter Eurostat<br>
25
Eurostat Twitter Filtered on tweets containing the words “Paris” or “Parigi” Eurostat Number of tweets generated in the Rome area (each bar = 12 hours)<br>
26
Eurostat Twitter Technology
Elastic (Elasticsearch) good tool to collect and index data
Kibana is a good tool to visualize data
Source
Continuity: technology outside NSOs control Continuity depends on even small changes
to technology
Rules for Data collection: fragile legal basis, better acquire data (cost UK
£25,000/year) Eurostat<br>
Elastic (Elasticsearch) good tool to collect and index data
Kibana is a good tool to visualize data
Source
Continuity: technology outside NSOs control Continuity depends on even small changes
to technology
Rules for Data collection: fragile legal basis, better acquire data (cost UK
£25,000/year) Eurostat<br>
27
Eurostat Enterprise Web sites Objective: using web sites of enterprises as a source to create statistics Issues Computing the statistics Implementing a method for scraping data from web sites Obtaining URLs of web sites Number of enterprises which advertise Job Vacancies
broken down by NACE activity and region Different approaches were experimented for both scraping and analysis phases Different approaches were experimented Obtain from registries/survey Obtain from search engines Eurostat<br>
broken down by NACE activity and region Different approaches were experimented for both scraping and analysis phases Different approaches were experimented Obtain from registries/survey Obtain from search engines Eurostat<br>
28
Eurostat Enterprise Web sites Development of an application for scraping the job vacancies data starting from URLs of enterprises Eurostat Not tied to language/method
Developed in Python
Available in the Sandbox Spider – Downloader
Start from a list of URLs, follow the links and download the content of the employment pages
Determinator – Classifier Implement a method for detecting and classifying job vacancies advertisements in the scrapped content<br>
Developed in Python
Available in the Sandbox Spider – Downloader
Start from a list of URLs, follow the links and download the content of the employment pages
Determinator – Classifier Implement a method for detecting and classifying job vacancies advertisements in the scrapped content<br>
29
Eurostat Enterprise Web sites Eurostat<br>
30
Eurostat Enterprise Web sites Methodology Comparison with distribution of job vacancies per NACE revealed coherence with survey results, indicating that the approach is solid Privacy
Retrieving and sharing URLs of web sites was not as easy as expected
Still not clear yet whether URLs collected as microdata from
Slovenian survey could be used in the Sandbox
Methodology
Promising results from the machine learning approach to identification Eurostat of job vacancies<br>
Retrieving and sharing URLs of web sites was not as easy as expected
Still not clear yet whether URLs collected as microdata from
Slovenian survey could be used in the Sandbox
Methodology
Promising results from the machine learning approach to identification Eurostat of job vacancies<br>
31
Eurostat ComTrade UN Comtrade compiles official trade statistics database since 1962 containing billions of records Due to interest in measuring economic globalization through trade, trade data has been used to analyse interlink between economies<br>
32
Eurostat ComTrade Trade of intermediate goods by geographical region Made with Gephi Eurostat Made with D3<br>
33
Eurostat ComTrade Network analysis Countries that were counted at least one time in the ten best ranks after application of the PageRank algorithm on each network for 2012 Eurostat<br>
34
Eurostat ComTrade Relevance
Comprehensive analysis of global value chain through trade networks in all economic sectors is crucial part to better understand international trade and new
approaches as those we experimented are needed
Technology
8 different tools/languages were used to work with the data
Starting from data in basic text format made easy to switch from one tool to another
Technology
Processing data with the Sandbox provided evident advantages in terms of processing time and manageable size wrt current tool used at UNSD (Relational DB) Eurostat<br>
Comprehensive analysis of global value chain through trade networks in all economic sectors is crucial part to better understand international trade and new
approaches as those we experimented are needed
Technology
8 different tools/languages were used to work with the data
Starting from data in basic text format made easy to switch from one tool to another
Technology
Processing data with the Sandbox provided evident advantages in terms of processing time and manageable size wrt current tool used at UNSD (Relational DB) Eurostat<br>
35
Eurostat The Sandbox today Eurostat Extended access to the Sandbox: still active
ICHEC goes on providing the Sandbox as a service to the international statistical community, on a non-profit basis
Users are required to pay an annual subscription to cover the costs of technical support, hardware upgrades and installation of software
Eurostat ESSnets are using the Sandbox for their experiments on Big Data<br>
ICHEC goes on providing the Sandbox as a service to the international statistical community, on a non-profit basis
Users are required to pay an annual subscription to cover the costs of technical support, hardware upgrades and installation of software
Eurostat ESSnets are using the Sandbox for their experiments on Big Data<br>
36
Eurostat Eurostat ESSnet on Big Data Eurostat Eurostat ESSnet inherits the experience of the UNECE Sandbox
Wiki available on Webgate site
Projects duration 2016 – 2018
22 partners: 20 NSIs and 2 statistical organisations, with CBS as coordinator
8 workpackages (WP) active, plus Coordination and Dissemination WPs<br>
Wiki available on Webgate site
Projects duration 2016 – 2018
22 partners: 20 NSIs and 2 statistical organisations, with CBS as coordinator
8 workpackages (WP) active, plus Coordination and Dissemination WPs<br>
37
Eurostat Eurostat ESSnet on Big Data Eurostat WP1 Webscraping Job Vacancies: techniques, and methodologies suitable to produce statistical estimates in the domain of job vacancies. Mix of sources including job portals, job adverts on enterprise websites, and job vacancy data from third party sources
WP2 Webscraping enterprise characteristics: webscraping, text mining and inference techniques used to collect, process and improve general information about enterprises. Unstructured data to improve business registers using webscraping techniques<br>
WP2 Webscraping enterprise characteristics: webscraping, text mining and inference techniques used to collect, process and improve general information about enterprises. Unstructured data to improve business registers using webscraping techniques<br>
38
Eurostat Eurostat ESSnet on Big Data Eurostat WP3 Smart meters: used to produce energy statistics but also census housing statistics, household costs, impact on environment, energy production. data is that these are currently available in a few countries only, but will be available in several countries before 2020.
WP4 AIS data: Automatic Identification System to improve ship traffic statistics. Data source generic world wide, contacts with other authorities<br>
WP4 AIS data: Automatic Identification System to improve ship traffic statistics. Data source generic world wide, contacts with other authorities<br>
39
Eurostat Eurostat ESSnet on Big Data Eurostat WP5 Mobile phone data: how to obtain stable access to this data source – today legal, privacy and contractual issues. Big potential of the source. Define concrete statistical outputs. Microdata or aggregated data from providers?
WP6 Early Estimates: combination of (early available) multiple Big Data sources and existing official statistical data used in order to create existing or new early estimates for statistics. Challenges are: representativity issues, linking to other datasets, metadata.<br>
WP6 Early Estimates: combination of (early available) multiple Big Data sources and existing official statistical data used in order to create existing or new early estimates for statistics. Challenges are: representativity issues, linking to other datasets, metadata.<br>
40
Eurostat Eurostat ESSnet on Big Data Eurostat WP7 Multiple domains: data used to improve current statistics and create new statistics in these statistical domains: Population, Tourism / border crossings, Agriculture.
WP8 Methodology: foundation in areas such as methodology, quality and IT infrastructure for future use of the selected big data sources. Literature overview of papers, presentations or webpages relevant to the application of Big Data for official statistics and link this literature overview with the findings of the pilots.<br>
WP8 Methodology: foundation in areas such as methodology, quality and IT infrastructure for future use of the selected big data sources. Literature overview of papers, presentations or webpages relevant to the application of Big Data for official statistics and link this literature overview with the findings of the pilots.<br>
41
Eurostat Future of the Sandbox
Running experiments and pilots The sandbox can be used for experiments involving creating and evaluating new software programmes, developing new methodologies and exploring the potential of new data sources
This use case extends the current role of the sandbox beyond Big Data, and encompasses all types of data sources Eurostat<br>
Running experiments and pilots The sandbox can be used for experiments involving creating and evaluating new software programmes, developing new methodologies and exploring the potential of new data sources
This use case extends the current role of the sandbox beyond Big Data, and encompasses all types of data sources Eurostat<br>
42
Eurostat Future of the Sandbox
Testing Eurostat Setting up and testing of statistical pre- production processes is also possible in the Sandbox, including simulating complete workflows and process interactions The environment could be used also for testing other kind of software, beyond Big Data tools<br>
Testing Eurostat Setting up and testing of statistical pre- production processes is also possible in the Sandbox, including simulating complete workflows and process interactions The environment could be used also for testing other kind of software, beyond Big Data tools<br>
43
Eurostat Future of the Sandbox Training The sandbox can be used as a platform for supporting training courses. It can run special software for high performance computing which cannot be installed or run on standard computers Non-confidential demonstration datasets can be uploaded and shared, facilitating shared training activities across organisations
The sandbox environment also allows statisticians opportunities for self-learning, e-learning and learning by doing Eurostat<br>
The sandbox environment also allows statisticians opportunities for self-learning, e-learning and learning by doing Eurostat<br>
44
Eurostat Future of the Sandbox Supporting the implementation of the Common Statistical Production Architecture (CSPA) Eurostat The sandbox can be used as a statistical laboratory where researchers can jointly develop and test new CSPA-compliant software<br>
45
Eurostat Future of the Sandbox
Data Hub Eurostat The sandbox also provides a shared data repository
(subject to confidentiality constraints) It can be used to share non-confidential data sets that cover multiple countries, as well as public- use micro-data sets<br>
Data Hub Eurostat The sandbox also provides a shared data repository
(subject to confidentiality constraints) It can be used to share non-confidential data sets that cover multiple countries, as well as public- use micro-data sets<br>
46
Eurostat Future of the Sandbox Eurostat Big Data technologies have proven to be usable for many of the elaborations standards of statistical data Parallel processing and NoSQL seem to be able to effectively backup the traditional software like RDBMS and old file systems Big Data Technologies for Statistics<br>