Dismissive Reviews, Citation Cartels, and the
Description: Dismissive Reviews, Citation Cartels, and the Replication Crisis Richard P. Phelps The Replication Crisis smaller the sample size. smaller the effect. greater the number of tested relationships. greater the flexibility in design,
Related Topics
Download Presentation
"Dismissive Reviews, Citation Cartels, and the" is the property of its rightful owner. Permission is granted to download and print the materials on this website for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Presentation Transcript
slide1. Dismissive Reviews, Citation Cartels, and the Replication Crisis Richard P. Phelps<br>
slide2. The Replication Crisis<br>
slide3. … smaller the sample size.
… smaller the effect.
… greater the number of tested relationships.
… greater the flexibility in design, definitions, outcomes, and analytical modes.
… greater the financial and other interests and prejudices.
… hotter the scientific field.
… greater the publicity. According to John Ioannidis … … research findings are less likely to be true, the …<br>
slide4. Other researchers found “gaming the system” to be widespread
“P-hacking”
Only requiring P < .05
Non-replicable findings
Many scholars refusing to share data sets
I add 2 more: Dismissive Reviews, Citation Cartels Ioannidis exaggerated,
but much of what he wrote was true<br>
slide5. Proportion of studies replicated at p < .05,
in three fields of study, 2015 to 2018<br>
slide7. Number of citations to replicated and not replicated studies, as of 2021<br>
slide8. The growth of “Metascience” as a field of study Research Policy (journal), 1971
Scientometrics (journal), 1978
Cochrane Library, 1993
Metascience (journal), 1996
Campbell Collaboration, 2000
Research Ethics (journal), 2005
Centre for Journalology, 2005
Journal of Informetrics, 2007
Retraction Watch, 2010
PubPeer, 2012
Science Advances (journal), 2015
Research Waste/EQUATOR, 2015
Research Integrity and Peer Review (journal), 2016<br>
slide9. Information proliferation Pro-Quest UMI dissertation publishing:
3 + 5 million dissertations and theses
250,000 new works each year
STM reports:
34,000 journals
+4%/year
10,000 publishers<br>
slide10. Still more proliferation As of 2021…
…3 million articles are published in peer-reviewed journals every year.
More than 5 new articles per minute.<br>
slide11. The proliferation of researchers Residing in the United States alone (2008):
2.5 million with doctoral degrees
5.5 million with professional degrees
14.9 million with masters’ degrees<br>
slide12. Knowing ALL the research literature on a topic There is so much, is anyone qualified to speak for all of it?
It is genuinely difficult to do something new and unique<br>
slide13. Knowledge is Unlimited? It may be, but there are limits to the amount that we can use.
So, we filter it. Two ways to filter:
Summarize all of it
Accept only a certain amount, a certain type, …or only from certain people<br>
slide14. The World Wide Web is an information filter 20th Century:
Reference librarians provides wide range of information
21st Century:
Search Engine Optimization (SEO) narrows range<br>
slide15. Definitions: Firstness Claims & Dismissive Reviews With a firstness claim, a researcher insists that s/he is the first to study a topic.
With a dismissive literature review, a researcher assures the reader that no one else has conducted a study on a topic.<br>
slide16. The Effect of Firstness Claims and Dismissive Reviews The public is told that no other research exists on a topic, ergo, there is no reason to look for it.<br>
slide17. How difficult is a literature review? Not analytically difficult
But a genuinely thorough review requires a substantial amount of time, and some money
Generally, neither search nor ethics are part of US professors’ training.<br>
slide18. Professional incentives to do a thorough literature review THERE ARE NONE?
Scholars get little credit for a thorough literature review, much more for “original work”
In “publish or perish” environments, lit reviews are impediments to progress<br>
slide19. Why do a thorough lit review? huge burden in time and distraction
little to no benefit professionally
no punishment for not doing it<br>
slide20. Dismissive Reviews and Citation Cartels in Education Policy Research – A List https://nonpartisaneducation.org/Review/Resources/DismissiveList.htm A database of > 1,000 dismissive reviews from top US scholars in education policy 3 types:
Firstness Claims
Dismissals
Denigrations<br>
slide21. Examples of Firstness Claims “This report is unique.”
“This is the only study to …”
“We provide the first evidence …”
“We construct the first nationwide measures of …”
“Our study is the first to precisely estimate …”
“We are the first to isolate the impact of …”
“Ours is the first comprehensive analysis of …”
“To our knowledge, this has not been studied before …”<br>
slide22. Examples of Dismissals “There is very little research …”
“Surprisingly little is known about …”
“The debate is mostly supported by anecdotes.”
“Scant empirical evidence has been provided.”
“Studies of … are only relatively recent.”
“Knowledge of … has remained nonexistent.”
“There has been little systematic effort to study …”<br>
slide23. Examples of Denigrations “Previous work is very limited …”
“To provide more rigorous evidence, we …”
“Existing studies focus on a very small number of ….”
“[we] found the bulk of studies to be flawed …”
“… significant measurement problems in previous studies.”
“[earlier] studies were unable to control for ....”
“[earlier] studies had major deficiencies …”
“There are statistical shortcomings in … the studies.”<br>
slide24. “absence of evidence is not evidence of absence” Eyewitness fallacy – what you see represents all
Composition fallacy – the whole must have a property because its parts have the property.
Amazing Familiarity – the speaker seems to have information that there is no possible way for him to get
Argumentum ad ignorantiam – a proposition is true if it has not been proven false, or false if not proven true.
Wishful Thinking – things are (or will be) some way because that is how we wish them to be.<br>
slide25. Raw Declaration
No evidence of a literature search provided Dismissed Work Not Identified
Providing no help in finding it
Discouraging search for it
Avoiding debate Key Characteristics of Dismissive Reviews<br>
slide26. “absence of research” ~33,000
“absence of studies” ~26,000
“this is the first study” ~1,610,000
“little research” ~931,000
“paucity of research” ~121,000 Search engine counts (Google Scholar), 9.5.2023<br>
slide27. Dismissive reviews worse than plagiarism Misrepresent the work of one person
(by plagiarizing)
reward is small
(saves some work & time)
risk is large
(could ruin one’s reputation and career) Misrepresent the work of hundreds (in dismissive reviews)
reward is large
(for being first & unopposed)
risk is nil<br>
slide28. Ethics of dismissive reviews “Whatever you allow, you encourage.”
Michael Josephson<br>
slide29. Definition: Citation Cartels In a Citation Cartel, scholars…
… cite and reference each other and
… ignore, dismiss, or denigrate other research.<br>
slide30. Definition: “Sincere” Scholarship Sincere scholars behave ethically:
Conduct forthright and thorough literature reviews,
Cite all other, relevant research regardless of:
their own personal opinions of the individual scholars their own personal preferences for their research results.
They provide full and accurate citations of all that previous work,
so that readers will have no trouble finding it.
Focus is on accuracy and adding knowledge<br>
slide31. Definition: “Strategic” Scholarship Strategic scholars behave strategically:
Avoid literature searches and reviews
Either:
Declare ”firstness” – no previous research
Reference only work from within their cartel
Do not pass up opportunities to promote their work…
…even if they lack expertise on a topic, they will say something
Focus is on advancing their careers<br>
slide32. Comparing Strategic and Sincere Scholarship:
Citation accumulation over thirty years<br>
slide33. How can they do this? No legal restrictions on education research quality –
no Hippocratic Oath, no license or certification
Rewards still lie within one’s primary field –
economics for economists, psychology for psychologists, etc.
Scholars are rewarded for what they add; but they are not punished for what they subtract
Many who could “blow the whistle” are afraid, or they might dislike the research literature and like dismissing it<br>
slide34. Journalists help to suppress information When they print one researcher’s firstness claim or dismissive review,
they help to suppress others’ work and competing evidence<br>
slide35. * saves time, avoids tedium of reading the research literature
* adds to own citation totals, or those of one’s citation cartel,
… while not adding to rivals’ citation totals
* gives readers no help in finding rival evidence (by not even citing it)
* establishes (false) bona fides as an “expert” on the topic
* attention by allegedly being "first," "original," "a pioneer."
* increases the likelihood of press coverage for the same reason.
* increases prospects for grant funding to "fill knowledge gaps." Benefits accrue to…
…individual scholars
and small groups:<br>
slide36. Costs accrue to… society Society loses information; remaining information is skewed in favor of the powerful
Policy decisions are based on information that is limited and skewed
Government and foundations may pay again for research that has already been done<br>
slide37. Research most vulnerable to dismissal That done by those below the “celebrity threshold”*
Studies by civil servants (government agencies do not promote or defend their work)
That done by the deceased
All become:
“Zombie Researchers” * Researchers below the celebrity threshold lack the resources and media access to successfully counter dismissals of their work – they can easily be ignored.<br>
slide38. Paradox of research proliferation As the amount of research grows…
…so does the amount declared nonexistent
…so does the incentive to dismiss it
…so does the opportunity to dismiss it<br>
slide39. What Can be Done?<br>
slide40. Transparency Too easy to publish dismissive reviews:
Find 1 editor and 1-2 reviewers among the hundreds of thousands of journals available
If first journal one tries objects, there are many others.
Just submit same paperwork to another journal. A genuine lit review takes weeks or months.
Preprints and PubPeer are the only current, effective deterrents<br>
slide41. Ban firstness claims and dismissive reviews Add ban to the ethics codes of…
…journalists
…foundation research funders
…government research funders In most cases, editors, reviewers, & journalists have neither the time nor the resources to verify<br>
slide42. Real punishment for false firstness claims and dismissive reviews Make literature reviews optional for getting funding, but…
…make their accuracy mandatory,
…suspend violators from any
further funding<br>
slide43. Remove any literature review obligation from research articles Removes temptation
Most do more harm than good anyway because they are partial and selective<br>
slide44. Is meta-analysis the solution? Let meta-analysts do all literature reviews
Meta-analysis review model:
Identify where you have looked before making summary claims<br>
slide45. Merci !<br>
slide2. The Replication Crisis<br>
slide3. … smaller the sample size.
… smaller the effect.
… greater the number of tested relationships.
… greater the flexibility in design, definitions, outcomes, and analytical modes.
… greater the financial and other interests and prejudices.
… hotter the scientific field.
… greater the publicity. According to John Ioannidis … … research findings are less likely to be true, the …<br>
slide4. Other researchers found “gaming the system” to be widespread
“P-hacking”
Only requiring P < .05
Non-replicable findings
Many scholars refusing to share data sets
I add 2 more: Dismissive Reviews, Citation Cartels Ioannidis exaggerated,
but much of what he wrote was true<br>
slide5. Proportion of studies replicated at p < .05,
in three fields of study, 2015 to 2018<br>
slide7. Number of citations to replicated and not replicated studies, as of 2021<br>
slide8. The growth of “Metascience” as a field of study Research Policy (journal), 1971
Scientometrics (journal), 1978
Cochrane Library, 1993
Metascience (journal), 1996
Campbell Collaboration, 2000
Research Ethics (journal), 2005
Centre for Journalology, 2005
Journal of Informetrics, 2007
Retraction Watch, 2010
PubPeer, 2012
Science Advances (journal), 2015
Research Waste/EQUATOR, 2015
Research Integrity and Peer Review (journal), 2016<br>
slide9. Information proliferation Pro-Quest UMI dissertation publishing:
3 + 5 million dissertations and theses
250,000 new works each year
STM reports:
34,000 journals
+4%/year
10,000 publishers<br>
slide10. Still more proliferation As of 2021…
…3 million articles are published in peer-reviewed journals every year.
More than 5 new articles per minute.<br>
slide11. The proliferation of researchers Residing in the United States alone (2008):
2.5 million with doctoral degrees
5.5 million with professional degrees
14.9 million with masters’ degrees<br>
slide12. Knowing ALL the research literature on a topic There is so much, is anyone qualified to speak for all of it?
It is genuinely difficult to do something new and unique<br>
slide13. Knowledge is Unlimited? It may be, but there are limits to the amount that we can use.
So, we filter it. Two ways to filter:
Summarize all of it
Accept only a certain amount, a certain type, …or only from certain people<br>
slide14. The World Wide Web is an information filter 20th Century:
Reference librarians provides wide range of information
21st Century:
Search Engine Optimization (SEO) narrows range<br>
slide15. Definitions: Firstness Claims & Dismissive Reviews With a firstness claim, a researcher insists that s/he is the first to study a topic.
With a dismissive literature review, a researcher assures the reader that no one else has conducted a study on a topic.<br>
slide16. The Effect of Firstness Claims and Dismissive Reviews The public is told that no other research exists on a topic, ergo, there is no reason to look for it.<br>
slide17. How difficult is a literature review? Not analytically difficult
But a genuinely thorough review requires a substantial amount of time, and some money
Generally, neither search nor ethics are part of US professors’ training.<br>
slide18. Professional incentives to do a thorough literature review THERE ARE NONE?
Scholars get little credit for a thorough literature review, much more for “original work”
In “publish or perish” environments, lit reviews are impediments to progress<br>
slide19. Why do a thorough lit review? huge burden in time and distraction
little to no benefit professionally
no punishment for not doing it<br>
slide20. Dismissive Reviews and Citation Cartels in Education Policy Research – A List https://nonpartisaneducation.org/Review/Resources/DismissiveList.htm A database of > 1,000 dismissive reviews from top US scholars in education policy 3 types:
Firstness Claims
Dismissals
Denigrations<br>
slide21. Examples of Firstness Claims “This report is unique.”
“This is the only study to …”
“We provide the first evidence …”
“We construct the first nationwide measures of …”
“Our study is the first to precisely estimate …”
“We are the first to isolate the impact of …”
“Ours is the first comprehensive analysis of …”
“To our knowledge, this has not been studied before …”<br>
slide22. Examples of Dismissals “There is very little research …”
“Surprisingly little is known about …”
“The debate is mostly supported by anecdotes.”
“Scant empirical evidence has been provided.”
“Studies of … are only relatively recent.”
“Knowledge of … has remained nonexistent.”
“There has been little systematic effort to study …”<br>
slide23. Examples of Denigrations “Previous work is very limited …”
“To provide more rigorous evidence, we …”
“Existing studies focus on a very small number of ….”
“[we] found the bulk of studies to be flawed …”
“… significant measurement problems in previous studies.”
“[earlier] studies were unable to control for ....”
“[earlier] studies had major deficiencies …”
“There are statistical shortcomings in … the studies.”<br>
slide24. “absence of evidence is not evidence of absence” Eyewitness fallacy – what you see represents all
Composition fallacy – the whole must have a property because its parts have the property.
Amazing Familiarity – the speaker seems to have information that there is no possible way for him to get
Argumentum ad ignorantiam – a proposition is true if it has not been proven false, or false if not proven true.
Wishful Thinking – things are (or will be) some way because that is how we wish them to be.<br>
slide25. Raw Declaration
No evidence of a literature search provided Dismissed Work Not Identified
Providing no help in finding it
Discouraging search for it
Avoiding debate Key Characteristics of Dismissive Reviews<br>
slide26. “absence of research” ~33,000
“absence of studies” ~26,000
“this is the first study” ~1,610,000
“little research” ~931,000
“paucity of research” ~121,000 Search engine counts (Google Scholar), 9.5.2023<br>
slide27. Dismissive reviews worse than plagiarism Misrepresent the work of one person
(by plagiarizing)
reward is small
(saves some work & time)
risk is large
(could ruin one’s reputation and career) Misrepresent the work of hundreds (in dismissive reviews)
reward is large
(for being first & unopposed)
risk is nil<br>
slide28. Ethics of dismissive reviews “Whatever you allow, you encourage.”
Michael Josephson<br>
slide29. Definition: Citation Cartels In a Citation Cartel, scholars…
… cite and reference each other and
… ignore, dismiss, or denigrate other research.<br>
slide30. Definition: “Sincere” Scholarship Sincere scholars behave ethically:
Conduct forthright and thorough literature reviews,
Cite all other, relevant research regardless of:
their own personal opinions of the individual scholars their own personal preferences for their research results.
They provide full and accurate citations of all that previous work,
so that readers will have no trouble finding it.
Focus is on accuracy and adding knowledge<br>
slide31. Definition: “Strategic” Scholarship Strategic scholars behave strategically:
Avoid literature searches and reviews
Either:
Declare ”firstness” – no previous research
Reference only work from within their cartel
Do not pass up opportunities to promote their work…
…even if they lack expertise on a topic, they will say something
Focus is on advancing their careers<br>
slide32. Comparing Strategic and Sincere Scholarship:
Citation accumulation over thirty years<br>
slide33. How can they do this? No legal restrictions on education research quality –
no Hippocratic Oath, no license or certification
Rewards still lie within one’s primary field –
economics for economists, psychology for psychologists, etc.
Scholars are rewarded for what they add; but they are not punished for what they subtract
Many who could “blow the whistle” are afraid, or they might dislike the research literature and like dismissing it<br>
slide34. Journalists help to suppress information When they print one researcher’s firstness claim or dismissive review,
they help to suppress others’ work and competing evidence<br>
slide35. * saves time, avoids tedium of reading the research literature
* adds to own citation totals, or those of one’s citation cartel,
… while not adding to rivals’ citation totals
* gives readers no help in finding rival evidence (by not even citing it)
* establishes (false) bona fides as an “expert” on the topic
* attention by allegedly being "first," "original," "a pioneer."
* increases the likelihood of press coverage for the same reason.
* increases prospects for grant funding to "fill knowledge gaps." Benefits accrue to…
…individual scholars
and small groups:<br>
slide36. Costs accrue to… society Society loses information; remaining information is skewed in favor of the powerful
Policy decisions are based on information that is limited and skewed
Government and foundations may pay again for research that has already been done<br>
slide37. Research most vulnerable to dismissal That done by those below the “celebrity threshold”*
Studies by civil servants (government agencies do not promote or defend their work)
That done by the deceased
All become:
“Zombie Researchers” * Researchers below the celebrity threshold lack the resources and media access to successfully counter dismissals of their work – they can easily be ignored.<br>
slide38. Paradox of research proliferation As the amount of research grows…
…so does the amount declared nonexistent
…so does the incentive to dismiss it
…so does the opportunity to dismiss it<br>
slide39. What Can be Done?<br>
slide40. Transparency Too easy to publish dismissive reviews:
Find 1 editor and 1-2 reviewers among the hundreds of thousands of journals available
If first journal one tries objects, there are many others.
Just submit same paperwork to another journal. A genuine lit review takes weeks or months.
Preprints and PubPeer are the only current, effective deterrents<br>
slide41. Ban firstness claims and dismissive reviews Add ban to the ethics codes of…
…journalists
…foundation research funders
…government research funders In most cases, editors, reviewers, & journalists have neither the time nor the resources to verify<br>
slide42. Real punishment for false firstness claims and dismissive reviews Make literature reviews optional for getting funding, but…
…make their accuracy mandatory,
…suspend violators from any
further funding<br>
slide43. Remove any literature review obligation from research articles Removes temptation
Most do more harm than good anyway because they are partial and selective<br>
slide44. Is meta-analysis the solution? Let meta-analysts do all literature reviews
Meta-analysis review model:
Identify where you have looked before making summary claims<br>
slide45. Merci !<br>