1 Big Data for Official Statistics* Herman Smith
Description: 1 Big Data for Official Statistics Herman Smith UNSD 10th Meeting of the Advisory Expert Group on National Accounts 13-15 April 2016, Paris Prepared by Ronald Jansen, UNSD Drivers Availability of automatically generated data in
Related Topics
Download Presentation
"1 Big Data for Official Statistics* Herman Smith" is the property of its rightful owner. Permission is granted to download and print the materials on this website for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Presentation Transcript
slide1. 1 Big Data for Official Statistics* Herman Smith
UNSD
10th Meeting of the Advisory Expert Group on National Accounts
13-15 April 2016, Paris * Prepared by Ronald Jansen, UNSD<br>
slide2. Drivers Availability of automatically generated data in electronic format, such as mobile phone, social media, electronic commercial transactions, sensor networks, smart meters, GPS tracking device, or satellite images
Higher frequency, more granularity, wider coverage, lower cost for data collection
Modernisation of statistical production and services 2<br>
slide3. Key messages Big Data for core national statistics – for integrated economic, social and environmental policies
Big Data for agile statistics – for emergency issues
Big Data to keep official statistics relevant – private sector moves fast
Big Data as part of modernization of statistical systems – new production processes and partnerships
Big Data to meet the data demand of the 2030 agenda – monitoring policies – “leave no one bend” 3<br>
slide4. Big Data for Official Statistics Benefits – Example of Social media data
Widespread use of social media, also in developing countries
Timely, high frequency and wide coverage
Great potential in tracking sentiments, such as consumer confidence
Potential use for tracking prices and outbreak of diseases, and useful in combination with other data, such as population census and geo-spatial data<br>
slide5. Examples of Big Data projects 5<br>
slide6. Examples 1: Telenor Big Data project on Poverty prediction (SDG 1) 6 Among the major mobile operators in the world Approaching 200 million mobile subscriptions
(e.g. in Bangladesh, India, Pakistan, Myanmar and Thailand)
33 000 employees Present in markets with 1.6 billion people A team of 9 Data scientists
Collaboration partners at leading academic research institutions
Bridge between academic research and all business units
Explore and develop new ways to utilize customer data across markets<br>
slide7. 7 IMSI: SIM card Type: Call, SMS, Data, etc Billions of data points collected each day Date & time B number – Receiving party Data volume<br>
slide8. 8 PREDICTION Survey data
Telco surveys
DHS
PPI # poor per km2
Prediction maps Satellite layers
Population
Aridity index
Evapotranspiration
Various animal densities
Night time lights
Elevation
Vegetation
Distance to roads/waterways
Urban/Rural
Land cover
Pregnancy data
Births
Ethnicity
Precipitation
Annual temperature
Global human settlement layer Mobile phone data
Basic phone usage
Advanced phone usage
Social Network
Mobility
Top-up
Revenue
Handset Introducing mobile phone data in Poverty prediction<br>
slide9. Introducing mobile phone data in Poverty prediction 9 Methods
Spatial prediction
Bayesian geostatistical modelling
Prediction maps
Individual classification using machine learning methods
RF
GBM
SVM
Deep learning Poverty Prediction map<br>
slide10. Example 2:
National Statistical Office of Tunisia
Big Data project
on Good Governance (SDG 16) 10<br>
slide11. October 2015 BIG DATA and Monitoring SDG 16 in Tunisia? SOCIAL MEDIA as a BIG data source Kamel ABDELLAOUI,
Direction de la diffusion , INS- Tunisie
Eduardo López-Mancisidor,
Programme des Nations Unies pour le développement - Tunisie<br>
slide12. Could social media data provide similar or new insights on public opinion to potentially complement or substitute household survey data? Free, public, easy access
No privacy issues
Express opinion Opinions in here Social media, WHY? Analyzing Social media for SDG 16: Why?<br>
slide13. Analyzing Social media for SDG 16: How?<br>
slide14. Sentiment Word Cloud Volume Data Sources Word Cluster Analyzing Social media for SDG 16: Outputs<br>
slide15. 15 Example 3:
Statistics Canada linking
Google Maps with the Statistical
Business Register (SDG 9)<br>
slide16. What can be gained from linking the SBR with Geo-spatial Information? Cross-sectional views of enterprise characteristics by (sub-national) regions:
Are there regional patterns of economic activity?
Are larger enterprises equally spread over the country?
Is FDI equally spread over the country? 16<br>
slide17. Statistics Canada– Geolocation of SBR data To study the potential of conducting economic analysis of small geographic areas by using Business Register (BR) microdata
Using BR data geocoded at the census subdivision (CSD) level, in combination with travel distance data generated from the Google Maps API
The identification of resource sectors is based on the aggregation of business data at the CSD level from the BR
A database was created, containing:
BR employment data, derived from payroll deduction files
BR revenue data, derived from the General Index of Financial Information, and
the six-digit North American Industrial Classification System (NAICS) code from the Business Register. 17<br>
slide18. Statistics Canada – employment by community & economic activity 08/04/2016 United Nations Statistics Division 18<br>
slide19. 19 GWG on Big Data for official statistics<br>
slide20. United Nations Global Working Group on Big Data for Official Statistics 20 Created in March 2014
32 Members –
22 Countries and 10 International Agencies<br>
slide21. Global survey on Big Data Projects 21<br>
slide22. Global survey on Big Data Projects 22<br>
slide23. 23 Thank you<br>
slide24. URLs to websites Telenor Research
http://www.telenor.com/media/press-releases/2015/telenor-research-deploys-big-data-against-dengue/
Mexico - Business Register on Google Earth
http://www3.inegi.org.mx/sistemas/mapa/denue/default.aspx
Geo-location of Business Register
https://www.unece.org/fileadmin/DAM/stats/documents/ece/ces/ge.42/2015/Session_III_Canada_-_Geolocation_of_BR_data__room_document_.pdf
Global Survey on Big Data
http://unstats.un.org/unsd/trade/events/2015/abudhabi/presentations/day1/04/UNSD%20-%20Global%20Survey%20on%20Big%20Data.pdf
Big Data Quality Framework
http://unstats.un.org/unsd/trade/events/2015/abudhabi/presentations/day3/01/3_Quality_Framework_Righiv3.pdf 24<br>
slide25. URLs to websites United Nations Statistics Division
http://unstats.un.org/unsd/
http://unstats.un.org/unsd/dnss/QualityNQAF/nqaf.aspx
United Nations Statistics Division/ Trade Statistics Branch
http://unstats.un.org/unsd/trade/default.asp
United Nations Statistical Commission
http://unstats.un.org/unsd/statcom/commission.htm
United Nations Global Working Group on Big Data for official statistics
http://unstats.un.org/unsd/bigdata/
http://unstats.un.org/unsd/trade/events/2014/Beijing/default.asp
http://unstats.un.org/unsd/trade/events/2015/abudhabi/default.asp
United Nations General Assembly Resolutions
http://www.un.org/en/ga/70/resolutions.shtml
United Nations History Publications
http://www.unhistory.org/publications/ 25<br>
slide26. URLs to websites United Nations Sustainable Development
https://sustainabledevelopment.un.org/
https://sustainabledevelopment.un.org/topics
United Nations Global Pulse
http://www.unglobalpulse.org/
Project 8
http://demandinstitute.org/projects/project-8/
United Nations Data Revolution
http://www.undatarevolution.org/
United Nations Statistics Division / SDG indicators
http://unstats.un.org/sdgs/
United Nations Statistics Division/ Modernization of Statistical Systems
http://unstats.un.org/unsd/nationalaccount/workshops/2015/NewYork/lod.asp 26<br>
slide27. URLs to websites United Nations Global Pulse
http://www.unglobalpulse.org/
World Pop
http://www.worldpop.org.uk/
Data Pop
http://datapopalliance.org/
Flowminder
http://www.flowminder.org/
UNU-EHS
http://ehs.unu.edu/
Future Earth
http://www.futureearth.org/
UProject
http://ureport.ug/ 27<br>
UNSD
10th Meeting of the Advisory Expert Group on National Accounts
13-15 April 2016, Paris * Prepared by Ronald Jansen, UNSD<br>
slide2. Drivers Availability of automatically generated data in electronic format, such as mobile phone, social media, electronic commercial transactions, sensor networks, smart meters, GPS tracking device, or satellite images
Higher frequency, more granularity, wider coverage, lower cost for data collection
Modernisation of statistical production and services 2<br>
slide3. Key messages Big Data for core national statistics – for integrated economic, social and environmental policies
Big Data for agile statistics – for emergency issues
Big Data to keep official statistics relevant – private sector moves fast
Big Data as part of modernization of statistical systems – new production processes and partnerships
Big Data to meet the data demand of the 2030 agenda – monitoring policies – “leave no one bend” 3<br>
slide4. Big Data for Official Statistics Benefits – Example of Social media data
Widespread use of social media, also in developing countries
Timely, high frequency and wide coverage
Great potential in tracking sentiments, such as consumer confidence
Potential use for tracking prices and outbreak of diseases, and useful in combination with other data, such as population census and geo-spatial data<br>
slide5. Examples of Big Data projects 5<br>
slide6. Examples 1: Telenor Big Data project on Poverty prediction (SDG 1) 6 Among the major mobile operators in the world Approaching 200 million mobile subscriptions
(e.g. in Bangladesh, India, Pakistan, Myanmar and Thailand)
33 000 employees Present in markets with 1.6 billion people A team of 9 Data scientists
Collaboration partners at leading academic research institutions
Bridge between academic research and all business units
Explore and develop new ways to utilize customer data across markets<br>
slide7. 7 IMSI: SIM card Type: Call, SMS, Data, etc Billions of data points collected each day Date & time B number – Receiving party Data volume<br>
slide8. 8 PREDICTION Survey data
Telco surveys
DHS
PPI # poor per km2
Prediction maps Satellite layers
Population
Aridity index
Evapotranspiration
Various animal densities
Night time lights
Elevation
Vegetation
Distance to roads/waterways
Urban/Rural
Land cover
Pregnancy data
Births
Ethnicity
Precipitation
Annual temperature
Global human settlement layer Mobile phone data
Basic phone usage
Advanced phone usage
Social Network
Mobility
Top-up
Revenue
Handset Introducing mobile phone data in Poverty prediction<br>
slide9. Introducing mobile phone data in Poverty prediction 9 Methods
Spatial prediction
Bayesian geostatistical modelling
Prediction maps
Individual classification using machine learning methods
RF
GBM
SVM
Deep learning Poverty Prediction map<br>
slide10. Example 2:
National Statistical Office of Tunisia
Big Data project
on Good Governance (SDG 16) 10<br>
slide11. October 2015 BIG DATA and Monitoring SDG 16 in Tunisia? SOCIAL MEDIA as a BIG data source Kamel ABDELLAOUI,
Direction de la diffusion , INS- Tunisie
Eduardo López-Mancisidor,
Programme des Nations Unies pour le développement - Tunisie<br>
slide12. Could social media data provide similar or new insights on public opinion to potentially complement or substitute household survey data? Free, public, easy access
No privacy issues
Express opinion Opinions in here Social media, WHY? Analyzing Social media for SDG 16: Why?<br>
slide13. Analyzing Social media for SDG 16: How?<br>
slide14. Sentiment Word Cloud Volume Data Sources Word Cluster Analyzing Social media for SDG 16: Outputs<br>
slide15. 15 Example 3:
Statistics Canada linking
Google Maps with the Statistical
Business Register (SDG 9)<br>
slide16. What can be gained from linking the SBR with Geo-spatial Information? Cross-sectional views of enterprise characteristics by (sub-national) regions:
Are there regional patterns of economic activity?
Are larger enterprises equally spread over the country?
Is FDI equally spread over the country? 16<br>
slide17. Statistics Canada– Geolocation of SBR data To study the potential of conducting economic analysis of small geographic areas by using Business Register (BR) microdata
Using BR data geocoded at the census subdivision (CSD) level, in combination with travel distance data generated from the Google Maps API
The identification of resource sectors is based on the aggregation of business data at the CSD level from the BR
A database was created, containing:
BR employment data, derived from payroll deduction files
BR revenue data, derived from the General Index of Financial Information, and
the six-digit North American Industrial Classification System (NAICS) code from the Business Register. 17<br>
slide18. Statistics Canada – employment by community & economic activity 08/04/2016 United Nations Statistics Division 18<br>
slide19. 19 GWG on Big Data for official statistics<br>
slide20. United Nations Global Working Group on Big Data for Official Statistics 20 Created in March 2014
32 Members –
22 Countries and 10 International Agencies<br>
slide21. Global survey on Big Data Projects 21<br>
slide22. Global survey on Big Data Projects 22<br>
slide23. 23 Thank you<br>
slide24. URLs to websites Telenor Research
http://www.telenor.com/media/press-releases/2015/telenor-research-deploys-big-data-against-dengue/
Mexico - Business Register on Google Earth
http://www3.inegi.org.mx/sistemas/mapa/denue/default.aspx
Geo-location of Business Register
https://www.unece.org/fileadmin/DAM/stats/documents/ece/ces/ge.42/2015/Session_III_Canada_-_Geolocation_of_BR_data__room_document_.pdf
Global Survey on Big Data
http://unstats.un.org/unsd/trade/events/2015/abudhabi/presentations/day1/04/UNSD%20-%20Global%20Survey%20on%20Big%20Data.pdf
Big Data Quality Framework
http://unstats.un.org/unsd/trade/events/2015/abudhabi/presentations/day3/01/3_Quality_Framework_Righiv3.pdf 24<br>
slide25. URLs to websites United Nations Statistics Division
http://unstats.un.org/unsd/
http://unstats.un.org/unsd/dnss/QualityNQAF/nqaf.aspx
United Nations Statistics Division/ Trade Statistics Branch
http://unstats.un.org/unsd/trade/default.asp
United Nations Statistical Commission
http://unstats.un.org/unsd/statcom/commission.htm
United Nations Global Working Group on Big Data for official statistics
http://unstats.un.org/unsd/bigdata/
http://unstats.un.org/unsd/trade/events/2014/Beijing/default.asp
http://unstats.un.org/unsd/trade/events/2015/abudhabi/default.asp
United Nations General Assembly Resolutions
http://www.un.org/en/ga/70/resolutions.shtml
United Nations History Publications
http://www.unhistory.org/publications/ 25<br>
slide26. URLs to websites United Nations Sustainable Development
https://sustainabledevelopment.un.org/
https://sustainabledevelopment.un.org/topics
United Nations Global Pulse
http://www.unglobalpulse.org/
Project 8
http://demandinstitute.org/projects/project-8/
United Nations Data Revolution
http://www.undatarevolution.org/
United Nations Statistics Division / SDG indicators
http://unstats.un.org/sdgs/
United Nations Statistics Division/ Modernization of Statistical Systems
http://unstats.un.org/unsd/nationalaccount/workshops/2015/NewYork/lod.asp 26<br>
slide27. URLs to websites United Nations Global Pulse
http://www.unglobalpulse.org/
World Pop
http://www.worldpop.org.uk/
Data Pop
http://datapopalliance.org/
Flowminder
http://www.flowminder.org/
UNU-EHS
http://ehs.unu.edu/
Future Earth
http://www.futureearth.org/
UProject
http://ureport.ug/ 27<br>