How risk averse is our - PowerPoint Presentation

1. How risk averse is our research funding? Reinhilde VeugelersProf@KULeuven-MSI; ERC Scientific Council Member

2. Low growth & austerity leading to shrinking public (research) budgets in many countries, exceptions:China’s expanding budgetFlanders FWO budget EU FP/Horizon budgetMonitoring, evaluation of public research budgetsMore emphasis on (measuring) impact of (public) research (funding) on societyMore emphasis on contribution of public research to (local) economic & societal development Public funders more short-term impact oriented, Public funders more risk averse…Context of public funding for research

3. Risk aversion in fundingSelection procedures increasingly accused of favoring “safe” projects which exploit existing knowledge at expense of novel projects that explore untested watersYet:Fundamental reason for government support of public research is to promote risk taking (Arrow 1962)

4. James Rothman, 2013 Nobel Laureate in Physiology/Medicine, Comments on Risk Taking Rothman told interviewer that “he was grateful started work in the early 1970s when the federal government was willing to take much bigger risks in handing out funding to young scientists”“I had five years of failure, really, before I had the first initial sign of success. And I’d like to think that that kind of support existed today, but I think there’s less of it. And it’s actually becoming a pressing national issue, if not an international issue.” Nobel Laureate, Physiology or Medicine, 2013Interview on NPR

5. Roger Kornberg on Risk TakingTo quote Nobel laureate Roger Kornberg, “If the work that you propose to do isn’t virtually certain of success, then it won’t be funded.” Paula Stephan, Georgia State University, pstephan@gsu.edu

6. Greg PetskoToo ambitious—suggest he go to Portugal, instead. Lack of preliminary dataFailure would be disastrous for funder-- “think of how it would look if we funded something that didn’t pan out.”Poor fit for reviewers: Experts (da Gamma and Magellan) too busy to review proposalLimited fundsFunds are used for data collection (“Grape Vine Sequencing”) rather than hypothesis testing—data collection projects are “guaranteed to work”Why Columbus’s proposal “Finding a New Route to the Indies by Sailing West” is (hypothetically) rejectedGenome Biology 2012 13:155

7. Variety of ReasonsFocus on funding “successful” projects-Obsession with “preliminary” findings— “no crystal no grant” Higher success rate for recurrent, established grant holdersIncreased reliance on bibliometric measures—particularly short-term bibliometric measures (Leiden Manifesto)“Instant bibliometrics for reviewers” JIF, Three-year citation window

8. Goes Beyond Funding DecisionsHiring and promotion decisions based in part on short-term bibliometric measuresAllocation of research funds to universities and departments based on such measuresNetherlands, UK, Australia, Finland, Italy, NorwayBOF Key in Flanders

9. The Case of Novel ScienceScientific breakthroughs advance the knowledge frontier and contribute disproportionally to economic growth.Research that underpins scientific breakthroughs often requires novel approaches.Novel research has high risk and is often controversial.“High risk/high gain” novel research  public support.Bias against novel research in funding?Bias against novel research in bibliometric indicators?

10. Wang, J, Veugelers, R., Stephan, P. 2017, Bias against novelty in science: a cautionary tale for users of bibliometric indicators, Research Policy, 46, 1416-1436. http://voxeu.org/article/bias-against-novelty-science; http://www.nber.org/digest/jun16/w22180.htmlDevelop a bibliometric measure of novelty: papers making new combinations of journal references, taking into account the difficulty of making such new combinations through the distance between the journalsStudy relationship between novelty and citations, using 2001 WoS journal articles.Stephan, Veugelers, Wang, 2017, Evaluators blinkered by bibliometrics, Nature, 544, 411-412.

11. Find a ‘high risk/high gain” profile of novel researchMore average citations but also higher variance in citations; More likely to become top cited (top 1%)but only when using a long enough time window (at least 4 years); More likely to stimulate follow-on breakthroughs;Appreciation of novel research comes from outside its own field; not within its field.Characteristics one expects if novelty is correlated with breakthrough researchAlso find bias against novelty in standard bibliometric indicatorsLess likely to be highly cited in typically short-term citation windowMore likely to be published in journals with lower Journal Impact FactorFindings in a Nutshell

12. Findings in a Nutshell

13. Novelty and Journal Impact FactorNovel papers are less likely to be published in high JIF journalsAnd even if they get into high JIF, they face a delayed recognitionFindings in a Nutshell

14. ImplicationsScience policy based on short term bibliometric indicators and journal Impact Factor has a bias against noveltyOver-reliance on such measuresDirectly discourages novel research that might be of great value.Indirectly misses follow-on breakthroughs built on novel research.Findings may help explain why funding agencies who are increasing relying on bibliometric indicators are at the same time perceived as being increasingly risk averseResults also point to importance of having interdisciplinary panels evaluate research

15. Funders should not provide (or ask to provide) short-term bibliometric measures and prevent them from being used as decisive in reviews of grant proposals. They should insist on multiple ways to assess applicants’ and institutions’ publicationsThey should resist evaluating the success based on short-term citation counts and journal-impact factors. They should also include experts with outside field expertise.Panel members should resist seeking out and relying too much on metrics, especially when calculated over less than a three-year windowImplications

16. Funding frontier research:some evidence from the European Research Council (ERC)

17. Excellence as the only criterionSupport for the individual scientist – no networks!Global peer-reviewNo predetermined subjects (bottom-up)Support of frontier research in all fields of science and humanities│ 2What is ERC? °2007LegislationStrategyScientific governance: independent Scientific Council with 22 members; full authority over funding strategyBudget: € 13 billion (2014-2020) - 1.9 billion €/year € 7.5billion (2007-2013) - 1.1 billion €/yearERC represents 17% of total EC-H2020 budget

18. ERC’s ambitions “the ERC aims at reinforcing excellence, dynamism and creativity in European research by funding investigator-driven projects of the highest quality at the frontiers of knowledge”.“its grants will help to bring about new and unpredictable scientific and technological discoveries - the kind that can form the basis of new industries, markets, and broader social innovations of the future”. “Scientific excellence is the sole selection criterion. In particular, high risk/high gain pioneering proposals which go beyond the state of the art, address new and emerging fields of research, introduce unconventional, innovative approaches are encouraged”.

19. ERC’s modus operandusThe evaluation of ERC grant applications is conducted by peer review panels composed of scholars selected by the ERC Scientific Council from all over the world; They are assisted by remote referees.Reviewers are asked to evaluate the proposals on their ground breaking nature, their level of ambition to go beyond the state of the art and push the frontier. Panels decide on the ranking/who-gets-fundedERC does not provide or asks for bibliometric indicators (JIF, Citations, ERC instructs its panel to only consider submitted material (ie not look up/use other information...Nevertheless, PIs often self-report in their applications (often advised by their host institutions/peers)Panel members are often found to self-search for bibliometric indicators│ 19

20. Evaluating ERC’s ambition of supporting frontier research? Insights from quantitative analysisEx Ante: What does ERC select? Check big impact, novelty, interdisciplinarity of grantees on pre grant publicationsComparing granted vs rejected ERC applicantsComparing marginally accepted vs marginally rejected ERC applicantsEx Post: What is the impact of ERC funding? Check big impact, novelty, interdisciplinarity of grantees on post-grant publications Compared to counterfactual: similar grantees without ERC fundingDifferenceinDifference, Note: ERC only just starting to have finished grants (2007, 2008, 2009)

21. Pre-SelectionPost SelectionAll StGControlGranteesDiff (T-C)ControlGranteesDiff (T-C)Diff-in-DiffHigh Gain: Highest JIF pubJIF_MAX6.6911.234.54***7.5111.544.03***-0.51High Gain: Share of pubs in Top1%RATIO_C3TOP010.030.060.03***0.020.040.02***-0.01High Risk: NoveltyDUM_HIGH_NOV0.270.19-0.08***0.220.20-0.020.06**High Risk: Interdisciplinarityrao _MAX0.300.29-0.010.300.30-0.010.00All GrantsHigh Gain: Highest JIF pubJIF_MAX7.8912.054.17***8.8513.214.37***0.20High Gain: Share of pubs in Top1%RATIO_C3TOP010.030.050.02***0.030.040.02***0.00High Risk: NoveltyDUM_HIGH_NOV0.270.24-0.03**0.220.240.010.05**High Risk: Interdisciplinarityrao stirling_MAX0.300.300.000.310.310.000.00Source: Own calculations on basis of ERCEA; Unit of analysis: project. Measures based on 3 years publication profile before/after call year (excl.).Preliminary results (not to be quoted); Control=Rejected applicantsDiD analysis of Publication profile of All (StG)Selection on High GainTreatment on High Risk (less risk aversion)

22. Caveats/ConcernsNovelty only one measure of frontier research; others neededNot all frontier research is “novel”Important for public agencies to have a portfolio that includes risk; not all research funded should be risky. Real role for “ditch diggers”

23. Thank you for you attentionComments very welcome !!Reinhilde.Veugelers@kuleuven.be