Data Management - PowerPoint Presentation

Slide1

Data Management forGeoinformaticsA short course on good data management for taught postgraduate students in geoinformatics and related data sciences.

John Murtagh, UELSlide2

Data CollectionSlide3

Data sources3Slide4

1. Finding data – this involves searching and finding data that has already been released2. Getting hold of more data

 – asking for ‘new’ data from official sources e.g. through Freedom of Information requests. 3. Collecting data yourself – This means gathering data and entering it into a database or a spreadsheet – whether you work alone or collaboratively

Sometimes data is public on a website but there is not a download link to get hold of it in bulk – but don’t give up! This data can be liberated with what

datawranglers

call 

scraping

. More later…Slide5

Finding already released data or…. open dataSlide6

Open Data – a definition“A piece of data is open if anyone is free to use, reuse, and redistribute it — subject only, at most, to the requirement to attribute and/or share-alike.”Slide7

Where can I find open data?Slide8

Lots of places!Slide9
Slide10
Slide11
Slide12

ScraperWiki…is an online tool to make the process of extracting "useful bits of data easier so they can be reused in other apps, or rummaged through by journalists and researchers.“

Most of the scrapers and their databases are public and can be re-used.Slide13

Geospatial data…A group for open geospatial data with an emphasis on use in teaching and research.Slide14
Slide15

Data.ac.uk“a landmark site for academia providing a single point of contact for linked open data development.”

It not only provides access to the know-how and tools to discuss and create linked data and data aggregation sites, but also enables access to, and the creation of, large aggregated data sets providing powerful and flexible collections of information.Slide16
Slide17

Other developmentsSlide18

A number of startups

are emerging, that aim to build communities around data sharing and re-sale. This includes Buzzdata and Figshare — a place to share and collaborate on private and public datasets — and data shops such as 

Infochimps and 

DataMarket

.

DataCouch

 — A place to upload, refine, share & visualize your data

.

The 

World Bank

 and 

United Nations

 data portals provide high-level indicators for all countries, often for many years in the past

.Slide19

An interesting Google subsidiary, Freebase, provides "an entity graph of people, places and things, built by a community that loves open data.“

Research data. There are numerous national and disciplinary aggregators of research data, such as the UK Data Archive

. While there will be lots of data that is free at the point of access, there will also be much data that requires a subscription, or which cannot be reused or redistributed without asking permission first.Slide20

While they may not always be easy to find, many databases on the web are indexed by search engines, whether the publisher intended this or not. Here are a few tips:Slide21

Tips for searching for data (from the Data Journalism Handbook)When searching for data, make sure that you include both search terms relating to the content of the data you’re trying to find as well as some

information on the format or source that you would expect it to be in.

Google and other search engines allow you to search by file type.

http://datajournalismhandbook.org/1.0/en/getting_data_0.htmlSlide22

For example, you can look only for…Spreadsheets (by appending your search with ‘filetype:XLS filetype:CSV’)

Geodata (‘filetype:shp’)

Database extracts (‘filetype:MDB

,

filetype:SQL

,

filetype:DB

’).

PDFs (‘

filetype:pdf

’).Slide23

You can also search by part of a URL. Googling for ‘inurl:downloads filetype:xls

’ will try to find all Excel files that have “downloads” in their web addressAnother popular trick is not to search for content directly, but for places where bulk data may be available.

(if you find a single download, it’s often worth just checking what other results exist for the same folder on the web server). You can also limit your search to only those results on a single domain name, by searching for, e.g. ‘

site:agency.gov

’.

For example, ‘

site:agency.gov

Directory Listing’ may give you some listings generated by the web server with easy access to raw files, while ‘

site:agency.gov

Database Download’ will look for intentionally created listings.Slide24

Getting hold of new dataSlide25

The information requested must be provided unless an exemption or exception allows the institution not to disclose it.The request could be addressed to anyone in the University organisation, & there are only 20 working days to respond.

Freedom of Information (FoI) & Environmental Information (EIR) legislation provides the public with a right to access information (also research data) held by a UK public authority, which includes most

universities, colleges, or publicly-funded research institutions. Slide26

You can make an FOI request using a website whatdotheyknow.comSlide27
Slide28

Statistics and Registration Services Act 2007The Act is mainly concerned with the UK Statistics Authority and applies only to data designated as Official Statistics. It defines how 'personal information' can be disclosed to an 'Approved Researcher' i.e. an individual to whom the Statistics Authority has granted access, for the purposes of statistical research, to personal information held by it.

Although the Act does not apply to individual researchers managing confidential research data not designated as Official Statistics, such researchers might wish to adapt the Approved Researcher model for access to confidential data.Slide29

Environmental Information Regulations 2004This Act gives the public access rights to environmental information held by a public authority (including universities) in response to requests (similar to the Freedom of Information Act). Freedom of access does not imply free access. There are circumstances under which requests may or must be refused, for example if the data contain personal information.Slide30

Collecting data yourself – This means gathering data and entering it into a database or a spreadsheet – whether you work alone or collaboratively.Slide31

Getting data in the format you need it

Finding more Data using GoogleYou can search for CSV files on Google by typing +filetype:csv in the search bar. Searching for "South Africa +filetype:csv" will result in CSV files mentioning South Africa. You can try different other

filetypes as well (such as: "

xls

" for excel spreadsheets or "

pdf

“)Slide32

Permissions and Licensing dataSlide33

Licensing of Open Data - reusePublic domain dedications, which also serve as maximally permissive licenses; there are no conditions put upon using the work;Permissive or attribution-only licenses; giving credit is the only substantial condition;

Copyleft, reciprocal, or share-alike licenses; these also require that modified works, if published, be shared under the same license.

As defined by Open Knowledge FoundationSlide34

Using dataSlide35

Ethics of carrying out research with dataUniversity's Research Ethics Committee (UREC) has specific responsibility for institutional oversight of matters relating to ethics and governance in research undertaken by both staff and postgraduate research students that involves 

human participation personal sensitive data or human material.

Further information from the Quality Assurance and Enhancement officer

researchethics@uel.ac.uk

http://www.uel.ac.uk/qa/research/Slide36

Collecting personal or sensitive dataThe Data Protection Act (1998) covers personal or sensitive personal data, but not

to all research data in general, nor to anonymised data or if the participants are no longer livingSlide37

Questions to ask yourself in your data creation.

Is personal data needed? Names and addresses, for example? Store this data, if required,

separately

.

Inform your participants about use of personal data.

Not all research data obtained from participants constitute personal data. If data are

anonymised

!Slide38

Lastly: To gain access to police data or records you may be subject to a Disclosure or a DBS check (previously 

CRB check) which provides details of any criminal record data held on you.

https://www.gov.uk/government/organisations/disclosure-and-barring-serviceSlide39

Other sessions as part of Data Management in Geoinformatics:Data IntegrationData ManagementData Sharing

Data Management for Geoinformatics by John Murtagh as part of the

Jisc funded project TraD (University of East London

is

licensed under a Creative Commons Attribution

Share Alike Licence