Questionable Research Practices Part 4: HARKing 1
Description: Questionable Research Practices Part 4: HARKing 1 HARKing: Hypothesizing After the Results are Known 2 AKA: Forking Paths Researchers Degrees of Freedom HARKing: Hypothesizing After the Results are Known 3 HARKing includes i presenting
Related Topics
Download Presentation
"Questionable Research Practices Part 4: HARKing 1" is the property of its rightful owner. Permission is granted to download and print the materials on this website for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Presentation Transcript
slide1. Questionable Research Practices Part 4: HARKing 1<br>
slide2. HARKing: Hypothesizing After the Results are Known 2 AKA:
Forking Paths
Researchers Degrees of Freedom<br>
slide3. HARKing: Hypothesizing After the Results are Known 3 “HARKing includes [i] presenting ad hoc and/or unexpected findings as though they had been predicted all along (Kerr 1998); and [ii] presenting exploratory work as though it was confirmatory hypothesis testing (Wagenmakers et al. 2012).”<br>
slide4. HARKing: Hypothesizing After the Results are Known 4<br>
slide5. Exploratory vs. Confirmatory Research 5 Exploratory research:
Analysis steps not specified in advance.
Confirmatory research:
Analysis steps have been specified in advance of data collection.<br>
slide6. Exploratory vs. Confirmatory Research. Both are essential. 6 Exploratory research:
Explore (‘get to know’) your biological system:
Correlations?
Which dependent variables seem to respond to independent variables?
Anything unusual or surprising?
…What appears to be happening?
Generate new hypotheses
Confirmatory research:
A focused test of that hypothesis.
We obtain a dataset in order to test a focal hypothesis in a pre-determined way.<br>
slide7. Exploratory vs. Confirmatory Research. Both are essential. 7 Exploratory research:
Explore (‘get to know’) your biological system:
Correlations?
Which dependent variables seem to respond to independent variables?
Anything unusual or surprising?
…What appears to be happening?
Generate new hypotheses
Confirmatory research:
A focused test of that hypothesis.
We obtain a dataset in order to test a focal hypothesis in a pre-determined way. Problems arise when an exploratory analysis is confused for, or explained as, a confirmatory hypothesis.<br>
slide8. Exploratory Confirmatory 8<br>
slide9. HARKing: Why a problem? 9 (At least) Two, non-mutually exclusive problems:
Problem 1:
In an exploratory analysis, our interaction with the data will generate hypotheses.
Generally, generating hypothesis is useful. But we may generate and test many more hypotheses than we would have if we had a single hypothesis in mind before looking at the data.
Testing multiple hypotheses increases opportunities for False Positives (Type 1 error).
If we present an exploratory result as confirmatory, we hide the fact that multiple hypotheses were investigated (and associated risks).<br>
slide10. HARKing: Why a problem? 10 (At least) Two, non-mutually exclusive problems:
Problem 2:
As we look at the data to find interesting patterns we use the data to generate hypotheses.
At this point, the data are not longer appropriate to test the hypothesis that was generated by these data. Using the data to test a hypothesis generated by the data involves circular reasoning.
We need a fresh dataset to test the hypothesis that was generated by exploring the data.
If we partake in this circular reasoning (and do not obtain a fresh dataset), this process inflates type 1 errors and p-values cannot be trusted.<br>
slide11. HARKing: Why a problem? 11<br>
slide12. How common is HARKing? 12<br>
slide13. How common is HARKing? 13 n = 573 Ecologists, 299 Evolutionary biologists
Q3: “Reporting an unexpected finding or a result from exploratory analysis as
having been predicted from the start.”<br>
slide14. HARKing 14 Reported having done at least once<br>
slide15. HARKing 15<br>
slide16. Why is HARKing relatively common? 16 Training may not equip to avoid HARKing.
Training might encourage HARKing.<br>
slide17. How should we do science? 17 Only confirmatory? This video might give that impression, but no.
Confirmatory analyses are essential, but we need exploratory analyses, too.
It is vital to distinguish between the two.
Declare which analyses were planned a priori in a paper
Pre-registration provides evidence to support such claims.<br>
slide2. HARKing: Hypothesizing After the Results are Known 2 AKA:
Forking Paths
Researchers Degrees of Freedom<br>
slide3. HARKing: Hypothesizing After the Results are Known 3 “HARKing includes [i] presenting ad hoc and/or unexpected findings as though they had been predicted all along (Kerr 1998); and [ii] presenting exploratory work as though it was confirmatory hypothesis testing (Wagenmakers et al. 2012).”<br>
slide4. HARKing: Hypothesizing After the Results are Known 4<br>
slide5. Exploratory vs. Confirmatory Research 5 Exploratory research:
Analysis steps not specified in advance.
Confirmatory research:
Analysis steps have been specified in advance of data collection.<br>
slide6. Exploratory vs. Confirmatory Research. Both are essential. 6 Exploratory research:
Explore (‘get to know’) your biological system:
Correlations?
Which dependent variables seem to respond to independent variables?
Anything unusual or surprising?
…What appears to be happening?
Generate new hypotheses
Confirmatory research:
A focused test of that hypothesis.
We obtain a dataset in order to test a focal hypothesis in a pre-determined way.<br>
slide7. Exploratory vs. Confirmatory Research. Both are essential. 7 Exploratory research:
Explore (‘get to know’) your biological system:
Correlations?
Which dependent variables seem to respond to independent variables?
Anything unusual or surprising?
…What appears to be happening?
Generate new hypotheses
Confirmatory research:
A focused test of that hypothesis.
We obtain a dataset in order to test a focal hypothesis in a pre-determined way. Problems arise when an exploratory analysis is confused for, or explained as, a confirmatory hypothesis.<br>
slide8. Exploratory Confirmatory 8<br>
slide9. HARKing: Why a problem? 9 (At least) Two, non-mutually exclusive problems:
Problem 1:
In an exploratory analysis, our interaction with the data will generate hypotheses.
Generally, generating hypothesis is useful. But we may generate and test many more hypotheses than we would have if we had a single hypothesis in mind before looking at the data.
Testing multiple hypotheses increases opportunities for False Positives (Type 1 error).
If we present an exploratory result as confirmatory, we hide the fact that multiple hypotheses were investigated (and associated risks).<br>
slide10. HARKing: Why a problem? 10 (At least) Two, non-mutually exclusive problems:
Problem 2:
As we look at the data to find interesting patterns we use the data to generate hypotheses.
At this point, the data are not longer appropriate to test the hypothesis that was generated by these data. Using the data to test a hypothesis generated by the data involves circular reasoning.
We need a fresh dataset to test the hypothesis that was generated by exploring the data.
If we partake in this circular reasoning (and do not obtain a fresh dataset), this process inflates type 1 errors and p-values cannot be trusted.<br>
slide11. HARKing: Why a problem? 11<br>
slide12. How common is HARKing? 12<br>
slide13. How common is HARKing? 13 n = 573 Ecologists, 299 Evolutionary biologists
Q3: “Reporting an unexpected finding or a result from exploratory analysis as
having been predicted from the start.”<br>
slide14. HARKing 14 Reported having done at least once<br>
slide15. HARKing 15<br>
slide16. Why is HARKing relatively common? 16 Training may not equip to avoid HARKing.
Training might encourage HARKing.<br>
slide17. How should we do science? 17 Only confirmatory? This video might give that impression, but no.
Confirmatory analyses are essential, but we need exploratory analyses, too.
It is vital to distinguish between the two.
Declare which analyses were planned a priori in a paper
Pre-registration provides evidence to support such claims.<br>