Data reproducibility and good research practices - PowerPoint Presentation

Slide1

Data reproducibility and good research practices

Navigating the University: Research and Resources Management RetreatAugust 19, 2016Michael Jamieson DRScAssociate Director, International Center for Regulatory SciencesAssistant Professor, Clinical PharmacyUSC School of PharmacySlide2

2

PloS Medicine 2005doi:10.1371/journal.pmed.0020124

Nature 2012 doi:10.1038/483531a

October 2013

11% ReproducibilitySlide3

In the News!!!!If a Job is Worth Going, it is Worth Doing Twice

Researchers and funding agencies need to put a premium on enduring that results are reproducible.NIH Plans to Enhance ReproducibilityFrances S. Collins and Lawrence A. Tabak discuss initiatives that the US National Institutes of Health is exploring to restore the self-correcting nature of preclinical research.Believe it or Not: How Much Can We Rely on Published Data on Potential Drug Targets Slide4

In the News (contn’d)

Unreliable Research: Trouble at the LabScientists Like to Think of Science as Self-Correcting. To an alarming degree, it is not.Hedging Against Academic RiskInvestors in early stage companies must often take a leap of faith that the academic research behind a company’s strategy is well validated and can be reproduced by independent hands.Raise Standards for Preclinical Cancer Research.C. Glenn Begley and Lee M. Ellis propose how methods, publications and incentives must change if patients are to benefit.Slide5

Amgen researchers tried to confirm the findings of published academic research, and could only do so in 6 of the 53 papers

they considered: a result the commentary authors call "shocking." The authors don't claim fraud. Instead they blame increasingly lax standards in results reporting.Nature; (483) 531-533, 2012 Slide6

Robust research: Institutions must do their part for reproducibility

C. Glenn Begley, Alastair M. Buchan & Ulrich Dirnagl

Nature |

Comment 01 Sep 2015

GIPSlide7

NIH

Plans to Enhance ReproducibilityA growing chorus of concern, from scientists and laypeople, contends that the complex system for ensuring the reproducibility of biomedical research is failing and is in need of restructuring. Nature; Francis S. Collins & Lawrence A.Tabak

27 January 2014Slide8

NIH

Plans to Enhance Reproducibility (contn’d)Preclinical research, especially work that uses animal models, seems to be the area that is currently most susceptible to reproducibility issues. Many of these failures have simple and practical explanations: different animal strains, different lab environments or subtle changes in protocol.Nature; Francis S. Collins & Lawrence A. Tabak

27 January 2014Slide9

NIH

Plans to Enhance Reproducibility (contn’d)As a funding agency, the NIH is deeply concerned about this problem. Because poor training is probably responsible for at least some of the challenges, the NIH is developing a training module on enhancing reproducibility and transparency of research findings, with an emphasis on good experimental design. Nature; Francis S. Collins & Lawrence A. Tabak 27 January 2014Slide10

The “unspoken rule” among early stage VCs is that

at least 50% of published studies, even those in top-tier academic journals, can’t be repeated with the same conclusions by an industrial lab.As a result, many VCs now insist on external validation studies of a new company’s basic science as a precondition to further investment.Osherovich, L. (2011) SciBX 4(15) Slide11

The Fundamental Issue

There is no well established quality standard for the conduct of non regulated biomedical research.Slide12

So where is a good place to start? Slide13

Good Laboratory Practices (GLP) and Good Research Practices (GRP) are both centered around the principles of good documentation

so they are a very good place to start.GLP embodies a set of principles that provides a framework within which laboratory studies are planned, performed, monitored, recorded, reported and archived.Good Laboratory Practices (GLP) and/or Good Research Practices (GRP)Slide14

GLP and GRP (contn’d)

GRP highlights non-regulatory practices that can be easily institutionalised with very little extra expense.Incorporating GLP and/or GRP helps assure regulatory authorities and potential industry partners that the data generated is a true reflection of the results obtained during the study and that the methods used in the study can be reproduced.Slide15

Handbook on Quality Practices in Basic Biomedical Research (QPBR).Slide16

“The handbook is designed to address the very important topic of quality in basic biomedical research. The quality practices outlined in this handbook provide the basis for

a non-regulatory quality management system, which if applied properly, will enable institutions and individuals to produce credible research data. The quality management system is designed as an aid to research institutions and individual researchers wishing to improve the quality of their research data.”WHO: Handbook on Quality Practices in Basic Biomedical Research (QPBR)Slide17

Some Simple First Steps

Insist on good laboratory notebook practices in your lab.Develop a standardized format for all of your S.O.P’s.Validate analytical methods where possible.Ensure traceability for all batches/runs of your drug/device.“GLP like” documentation practices for animal work.Maintenance/calibration books for your equipment.Slide18

Nature and Nature Research Journals introduce editorial measures to address the problem of irreproducibility.

April 25, 2013Slide19

Nature’s Initiative To ease the interpretation and

improve the reliability of published results we will more systematically ensure that key methodological details are reported, and we will give more space to methods sections. We will examine statistics more closely and encourage authors to be transparent, for example by including their raw data.Slide20

Nature’s Initiative Checklist

Central to this initiative is a checklist intended to prompt authors to disclose technical and statistical information in their submissions, and to encourage referees to consider aspects important for research reproducibility.The checklist focuses on a few experimental and analytical design elements that are crucial for the interpretation of research results but are often reported incompletely. For example, authors will need to describe methodological parameters that can introduce bias or influence robustness, and provide precise characterization of key reagents that may be subject to biological variability, such as cell lines and antibodies.Slide21

Checklist (contn’d)

The journals will demand more precise descriptions of statistics, and we will commission statisticians as consultants on certain papers, at the editor’s discretion and at the referees’ suggestion.To allow authors to describe their experimental design and methods in as much detail as necessary, the participating journals, including Nature, will abolish space restrictions on the methods section.The journal will encourage authors to provide tables of the data behind graphs and figures. This builds on their established data-deposition policy for specific experiments and large data sets. The source data will be made available directly from the figure legend, for easy access. Slide22

Contact InfoMichael W Jamieson

DRScAssociate Director, International Center for Regulatory SciencesAssistant Professor, Clinical PharmacyUniversity of Southern California, School of Pharmacy1540 Alcazar Street, CHP 140Los Angeles, CA 90089-9014Tel: 323-442-1276Fax: 323-442-2333E-mail: mjamieso@usc.edu