Improving Undergraduate STEM Education Program

slide1. Improving Undergraduate STEM Education Program (IUSE: EHR) Program Solicitation: NSF 14-588 IUSE Reviewer Orientation<br>
slide2. Outline IUSE: EHR Overview

Panel Review Process Expectations

Ratings & Panel Summary Criteria

Panel Review Process

COI/Confidentiality & Implicit Bias<br>
slide3. IUSE: EHR Overview<br>
slide4. IUSE: EHR supports 2 NSF strategic goals: 1. Transform the Frontiers of Science and Engineering:

Supporting fundamental, high-risk and potentially transformative research in STEM
Education of the next generation of the STEM workforce to continue this transformation<br>
slide5. IUSE: EHR supports 2 NSF strategic goals: 2. Stimulate Innovation and Address Societal Needs through Research and Education:

supports research and development on STEM education

prepare a diverse, globally competent STEM workforce and a STEM-literate citizenry<br>
slide6. IUSE: EHR Program Goals Improve STEM Learning & Learning Environments – Improve the knowledge base for defining, identifying, and innovating effective undergraduate STEM education teaching and learning
Broaden Participation & Institutional Capacity for STEM Learning – Increase the number and diversity of undergraduate students
Build the Professional STEM Workforce for Tomorrow – Improve the preparation of undergraduate students<br>
slide7. Educational Practice and Research Cycle Which help improve... That results in... Which lead to... Identifies and motivates... STEM Learning Environment<br>
slide8. IUSE: EHR 2 tracks Engaged Student Learning Institutional and Community Transformation Research, Design, and Development studies that involve creation, exploration, and implementation of tools, resources, or models Projects that use innovative approaches to substantially increase the propagation of highly effective methods of STEM teaching and learning in institutions of higher education<br>
slide9. Panel Review Process Expectations<br>
slide10. Who reviews - What and How? In our panels we include reviewers that are
Experts in fields of STEM education, STEM content, methodology, cognitive science, and administration
Panelists read 10 – 12 proposals, but panels may include more than 12 proposals and thus you will not read every proposal assigned to a panel
IUSE is using a mix of virtual and face to face panels. Often the more complex Exploration proposals are being discussed face to face in panels here in the DC area.<br>
slide11. The Proposer Receives:<br>
slide12. NSF Merit Review CriteriaGuiding Principles All NSF projects should be of the highest quality and have the potential to advance, if not transform, the frontiers of knowledge.
Meaningful assessment and evaluation of NSF-funded projects should be based on appropriate metrics, keeping in mind the likely correlation between the effect of broader impacts and the resources provided to implement projects.<br>
slide13. NSF Merit Review CriteriaRequired Both criteria are to be given full consideration during the review and decision-making processes; each criterion is necessary but neither, by itself, is sufficient.
Intellectual Merit: The Intellectual Merit criterion encompasses the potential to advance knowledge

Broader Impacts: The Broader Impacts criterion encompasses the potential to benefit society and contribute to the achievement of specific, desired societal outcomes
http://www.nsf.gov/pubs/policydocs/pappguide/nsf14001/gpg_3.jsp#IIIA<br>
slide14. Merit Review Considerations The following elements should be considered in the review for both Intellectual Merit & Broader Impacts:
What is the potential for the proposed activity to:

Advance knowledge and understanding within its own field or across different fields (Intellectual Merit)?
and
Benefit society or advance desired societal outcomes (Broader Impacts)?

To what extent does the proposed activity suggest and explore creative, original or potentially transformative concepts?<br>
slide15. (Merit Review Considerations cont.) The following elements should be considered in the review for both Intellectual Merit & Broader Impacts:

Is the plan for carrying out the proposed activities well-reasoned, well-organized, and based on a sound rationale?

Does the plan incorporate mechanisms to assess success?

How well qualified is the individual, team, or institution to conduct the proposed activities?

Are there adequate resources available to the PI (either at the home institution or through collaborations) to carry out the proposed activities?<br>
slide16. Common Guidelines for EducationResearch & Development: NSF13-126 Offer guidance on building the evidence base in STEM learning

Are not a “requirement” for IUSE: EHR proposals, but may be helpful to proposers and reviewers

We encourage you to be familiar with them<br>
slide17. Should be both “knowledge -using” and “knowledge-producing.”
Using: Is the project building on prior work?
Producing: What will “we” learn from the planned conduct of this project?

*Perspective: It is OK for IUSE: EHR projects to serve the PI’s institution, but they should also serve all of us by providing useful knowledge for broad constituencies of educators Basic perspectives from the Common Guidelines for IUSE: EHR Projects<br>
slide18. Applied work (e.g. development) is fine as long as it adheres to the principles of the preceding slide. [NSF budgets are not large enough to support simple local improvement projects.]

While we call for innovation, replication is also acceptable and an important part of science. [How do we know that something is effective if has been tested at only one place?] (Basic Perspectives cont.)<br>
slide19. IUSE: EHR accepts projects of different size. They should be judged relative to their funds request. 
Institutional transformation projects by their very nature seek major local impacts. These also need to provide insights about how to accomplish the transformation, and thus the proposal should deal explicitly with the theory of change that underlies its promise, and provide an applied research design. (Basic Perspectives cont.)<br>
slide20. As you explore the project budget request and compare this with the scope of the project, please note that:
Budgets that seem out of line with the proposed project can be negotiated by the program officer if the project itself seems promising.
Hence, we request that you don’t assign a “killer rating” due to budget alone. [A review might provide a rating conditional on budget reforms.] The Project Budget & Summary Rating<br>
slide21. Reviewer RatingsandPanel SummaryCriteria<br>
slide22. Use the Entire Proposal to Inform Your Review: Project Summary
Project Description
Biographical Sketches
Budget
Letters of Commitment
Project Data Form
Current & Pending Support<br>
slide23. Individual Review Process: Rating the Proposal Assign only ONE rating: E, V, G, F, P
Do NOT assign split ratings, such as E/V or V/G
Your rating MUST be consistent with your
written review<br>
slide24. Within FastLane:
In your own words, write aspects you find to be compelling and those that you believe will benefit from improvement; Do NOT cut and paste text from the proposal into your review
Be sure that your written review is consistent with your rating
It is OK to modify reviews during the panel meetings, including a change of rating
Be sure any modifications to reviews are recorded in FastLane! - MUST be done BEFORE the panel is closed.
Note that many MS Word symbols do not translate in FastLane - check your work after submitting Individual Review Process: Writing Your Review<br>
slide25. Characteristics of Informative Reviews Include a section for IM, BI and summary
Use appropriate style (positive tone)
Contain adequate details
Write understandable, specific, and complete statements
Relate strengths and weaknesses, supported by detailed explanation, associated with review criteria
If an item would have helped to strengthen the proposal, a clear statement should be included as to why that would be the case.<br>
slide26. Panel Review Process<br>
slide27. During the panel: Each proposal will typically have four reviews
Each proposal will have a reviewer assigned as the Scribe/Lead Discussant (called the primary panelist)
The Scribe/Lead Discussant will write the panel summary as well as write an individual review
Panel members assigned to review the proposal are called secondary panelists.
Panel members not assigned to review the proposal are simply identified as “panelists.”<br>
slide28. Proposal Discussion The Scribe/ primary panelist is the lead discussant who presents an overview of the project being proposed.
The Scribe then provides the “highlights” of her/his own review
Each assigned “secondary” panelist will provide his/her individual “highlights” while the scribe takes notes
Additional discussion will include all panelists providing the scribe with sufficient information to be able to compose the Panel Summary<br>
slide29. Panel Summaries For each proposal:
Initially framed by primary panelist who serves as scribe
Should reflect the panel discussion (not just restate individual reviews)
Include short, clear comments to help PIs improve their projects
Should be written in 3rd-person, as the panel, and will be read and agreed to by all panelists<br>
slide30. Characteristics of Informative Panel Summaries Includes a section for IM, BI and summary
Uses appropriate style (positive tone)
Contains adequate details
Contains understandable, specific, and complete statements
Relates strengths and weaknesses, supported by detailed explanation, associated with review criteria
If an item would have helped to strengthen the proposal, a clear statement should be included as to why that would be the case<br>
slide31. COI/Confidentiality&Implicit Bias<br>
slide32. Primary purpose is to remove or limit the influence (or appearance of influence) of ties to an applicant institution or investigator that could affect reviewer advice
Second purpose is to preserve the trust of the scientific community, Congress, and the general public in the integrity, effectiveness, and evenhandedness of NSF’s peer review process Conflict of Interest (COI)<br>
slide33. Statement on Confidentiality NSF receives proposals in confidence and protects the confidentiality of their contents

Proposals should not be discussed before the review process or after the review process with anyone outside of the panel

You must not copy, quote from, or otherwise use or disclose (to anyone) material from any proposal you are asked to review

The fact that you are reviewing IUSE proposals now should also be kept confidential<br>
slide34. Implicit Bias Implicit bias toward a group (“schemas”)
Non-conscious hypotheses/stereotypes, often about competence
Implicit institutional or geographic bias
Lack of critical mass leads to greater reliance on schemas
Fewer women and minorities in sciences
Accumulation of disadvantage
Accumulation of small biases in the same direction has large effects over time
Very small differences in treatment can have major consequences in salary, promotion, and prestige (Valian 1998)

https://implicit.harvard.edu/implicit/<br>
slide35. Implicit Bias Schemas are: Widely shared culturally:
All people, including members of underrepresented groups, hold schemas about these groups
People are often not aware of them
Applied more under circumstances of:
Lack of information
Stress from completing tasks
Time pressure
Lack of critical mass
Fiske (2002). Current Directions in Psychological Science, 11, 123-128.<br>
slide36. Division of Undergraduate EducationImproving Undergraduate STEM Education Program (IUSE: EHR) Questions?

Comments?<br>
slide37. Division of Undergraduate EducationImproving Undergraduate STEM Education Program (IUSE: EHR) Thank You!<br>