1 Big Data for Official Statistics* Herman Smith

Transcript:1 Big Data for Official Statistics* Herman Smith:
1 Big Data for Official Statistics* Herman Smith UNSD 10th Meeting of the Advisory Expert Group on National Accounts 13-15 April 2016, Paris * Prepared by Ronald Jansen, UNSD Drivers Availability of automatically generated data in electronic format, such as mobile phone, social media, electronic commercial transactions, sensor networks, smart meters, GPS tracking device, or satellite images Higher frequency, more granularity, wider coverage, lower cost for data collection Modernisation of statistical production and services 2 Key messages Big Data for core national statistics – for integrated economic, social and environmental policies Big Data for agile statistics – for emergency issues Big Data to keep official statistics relevant – private sector moves fast Big Data as part of modernization of statistical systems – new production processes and partnerships Big Data to meet the data demand of the 2030 agenda – monitoring policies – “leave no one bend” 3 Big Data for Official Statistics Benefits – Example of Social media data Widespread use of social media, also in developing countries Timely, high frequency and wide coverage Great potential in tracking sentiments, such as consumer confidence Potential use for tracking prices and outbreak of diseases, and useful in combination with other data, such as population census and geo-spatial data Examples of Big Data projects 5 Examples 1: Telenor Big Data project on Poverty prediction (SDG 1) 6 Among the major mobile operators in the world Approaching 200 million mobile subscriptions (e.g. in Bangladesh, India, Pakistan, Myanmar and Thailand) 33 000 employees Present in markets with 1.6 billion people A team of 9 Data scientists Collaboration partners at leading academic research institutions Bridge between academic research and all business units Explore and develop new ways to utilize customer data across markets 7 IMSI: SIM card Type: Call, SMS, Data, etc Billions of data points collected each day Date & time B number – Receiving party Data volume 8 PREDICTION Survey data Telco surveys DHS PPI # poor per km2 Prediction maps Satellite layers Population Aridity index Evapotranspiration Various animal densities Night time lights Elevation Vegetation Distance to roads/waterways Urban/Rural Land cover Pregnancy data Births Ethnicity Precipitation Annual temperature Global human settlement layer Mobile phone data Basic phone usage Advanced phone usage Social Network Mobility Top-up Revenue Handset Introducing mobile phone data in Poverty prediction Introducing mobile phone data in Poverty prediction 9 Methods Spatial prediction Bayesian geostatistical modelling Prediction maps Individual classification using machine