/
Inside graduate admissions: Inside graduate admissions:

Inside graduate admissions: - PowerPoint Presentation

alexa-scheidler
alexa-scheidler . @alexa-scheidler
Follow
343 views
Uploaded On 2019-12-15

Inside graduate admissions: - PPT Presentation

Inside graduate admissions Merit diversity and FACULTY GATEKEEPING Julie Posselt PhD University of Southern California posseltuscedu JuliePosselt Doctoral Degree Attainment ID: 770441

diversity amp faculty gre amp diversity gre faculty race ets admissions 2006 test program scores research students gender performance

Share:

Link:

Embed:

Download Presentation from below link

Download Presentation The PPT/PDF document "Inside graduate admissions:" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.


Presentation Transcript

Inside graduate admissions:Merit, diversity, and FACULTY GATEKEEPING Julie Posselt, Ph.D.University of Southern California posselt@usc.edu @JuliePosselt

Doctoral Degree Attainment BY GENDER, SELECTED FIELDS Field % Female All fields 46%Biological sciences52%Physics18%Psychology71%Economics34%Sociology60%Philosophy29%Linguistics60%Engineering21% BY RACE, ACROSS FIELDS Racial/ethnic groupU.S. Doctorate recipientsBlack12%7%Latino14%6%Native American1%<1%Asian American4%9%White67%78% Source: National Science Foundation, 2013, Doctorate Recipients from US Universities.

Institutions profess the importance of racial & gender diversity in... Undergraduate admissions & other types of selection 1 Mission statements & websites 2 There is strong evidence...1 Grodsky, 2007; Lamont, 2009; Karabel, 2005; Stevens, 20082 Morphew & Hartley, 2006; Osei-Kofi, et al., 2013Empirical research suggested basic tensions.

Institutions profess the importance of racial & gender diversity in... Undergraduate admissions & other types of selection 1 Mission statements & websites 2 Work of graduate diversity officers is often stymied.32 of the 3 best predictors of admission are high GRE scores and selective college attendance.4Unconscious bias in responses to emails from prospective students.5 There is strong evidence... At the same time....1 Grodsky, 2007; Lamont, 2009; Karabel, 2005; Stevens, 20082 Morphew & Hartley, 2006; Osei-Kofi, et al., 20133 Griffin & Muñiz, 20114 Garces, 2012; Attiyeh & Attiyeh, 1997 5 Milkman, et al., 2014Empirical research suggested basic tensions.

Why?

Research Questions : -How do faculty individually judge & collectively select applicants to highly ranked Ph.D. programs? -What assumptions about merit guide faculty judgment?-How do disciplinary norms shape faculty judgment? Comparative ethnographic case study 10 programs in 3 public & private universities 85 interviews with professors & a few graduate students22 hours of admissions meeting observations in six of the programsHarvard University Press, 2016

Humanities Social Sciences Natural Sciences High Consensus Philosophy (2 programs)EconomicsPhysicsModerate ConsensusClassics SociologyAstrophysicsLow ConsensusLinguisticsPolitical ScienceBiology

Field N Female International Scholars of ColorDomestic Scholars of ColorHumanities2433%26%7%4%Social Sciences2415%35% 16%4%Natural Sciences196%46%21%0%Total6718%36%15%3%

Preference for specific criteria was rooted in beliefs about what they signal. Those beliefs were tightly linked to their own identities as scholars in highly ranked programs. Preference for a process that is efficient and collegial. They wanted to quantify quality & minimize conflict. In high-consensus fields, defended decisions to themselves and one another using disciplinary logics (def: shared disciplinary norms about epistemology/methods, theories & metaphors, and practical priorities)In low-consensus fields, the strong role of individual preferences included four clear patterns of homophily (pedigree, cool, social mobility, international).Ambivalence about organizational change, especially reforms related to diversity and equity. Faculty mindsets toward diversity, equity, and inclusion must be addressed.Evaluative cultures explain apparent tensions between (definitions of) merit & (valuing) diversity.

Initial screening Conceptualizing merit Conventional achievers with low perceived risk of attrition Important criteria “Numbers” in context of undergraduate prestige & curriculum rigorRelationship of merit & diversityStandard of merit may be in tension with racial/gender diversity aims.Two-tiered review is used in most places.

Initial screening Later rounds of review Conceptualizing merit Conventional achievers with low perceived risk of attrition Future of the disciplineImportant criteria“Numbers” in context of undergraduate prestige & curriculum rigorExperience with and dispositions for research; Unique perspective; research interests alignRelationship of merit & diversityStandard of merit may be in tension with racial/gender diversity aims.Diversity is a component of merit. Two-tiered review is used in most places.

PROBLEMS WITH THE FIRST CUT

Theory of cultural & evaluative scripts ( Goffman, 1959; Lamont, 2009)GRE scores and Grades | Institutional prestige  Intelligence  Belonging in an elite intellectual communityRisk profile Scripts about GRE SCORES

GRE SCORES & INTELLIGENCE In interviews, 50% of the sample volunteered some idea about intelligence when asked what GRE scores signal (e.g., “sheer intellectual horsepower”, “native intelligence”)In meetings, >50% of GRE mentions were what I classified as smart talk.

Risk aversion Risk aversion as obligation & luxuryExamples of challenging risk aversion Prof. Bob: “Her GREs [of 690, 740, & 4.5] present a risk for her not succeeding” particularly because she “didn’t attend a top-rated university.” Prof. Lynn: “She may have undershot…This is an area that can be gendered…We have to be very careful here.” Prof. Bob: “All in all, it gives me doubt.” [Student ultimately waitlisted]PHILOSOPHY

Prof. Denise: “She might be a bet, but it could be a good bet… If we are going to increase diversity, these are the students we need to take seriously.” Prof. Jack: (Tentatively) “What’s the diversity?” Dept. Chair Nancy: “Family financial hardship.”[Committee agrees to move her forward, but discussion continues.]Dept. Chair Nancy: “It will be good for the whole faculty to take a look at her file. It seems pretty clear that she’s a risk, but if we’re going to increase diversity, we have to take risks.” Prof. Denise: “And she seems like a good bet.”[Student ultimately rejected after being waitlisted and attending recruitment weekend]LINGUISTICS

BLINDSPOTS ABOUT RISK Informal assumptions about risk might not be entirely accurate. Untested Difficult to reliably predict PhD completion for populations who rarely enroll (i.e., problem of small n’s) Validity of graduate entrance exams varies by test and graduate school outcome, but is consistently strongest for first year grad school GPA. 1 Student outcomes result from what they bring to the table AND from the educational experience we provide. Women who did not complete the PhD had higher mean GPA than men who didn’t complete, but left programs in higher numbers.21 Kuncel & Hezlett, 20072 Lovitts & Nelson, 2000

GRE Quantitative Score (2006-2007) SOURCE: ETS, "Factors that can influence performance on the GRE general test 2006-2007”

GRE Quantitative Score (2006-2007) SOURCE: ETS, "Factors that can influence performance on the GRE general test 2006-2007”

GRE Quantitative Score (2006-2007) SOURCE: ETS, "Factors that can influence performance on the GRE general test 2006-2007”

GRE Quantitative Score (2006-2007) SOURCE: ETS, "Factors that can influence performance on the GRE general test 2006-2007”

GRE Quantitative Score (2006-2007) SOURCE: ETS, "Factors that can influence performance on the GRE general test 2006-2007”

GRE Quantitative Score (2006-2007) SOURCE: ETS, "Factors that can influence performance on the GRE general test 2006-2007”

Technically not “bias” Nearly independent of intended graduate major Qualitatively unchanged when controlling for undergraduate GPA Qualitatively the same for GRE Subject test SAT Math8th grade math achievement tests4 th grade math achievement tests A feature of standardized testingGRE Test Disparities Are…Miller, C., & Stassun, K. (2014). A test that fails. Nature, 510 (7504), 303-304.

Pop Quiz Folder AGRE-Q: 740 (80%) Folder B GRE-Q: 800 (perfect) With all else equal, which folder do you admit?

“It is an inexact measure; only score differences that exceed the standard error of measurement of a given score can serve as a reliable indication of real differences in applicants' academic knowledge and developed abilities.” S.E.M. ~60 points (on old GRE scale, 200-800) 740 = 800 From ETS Guide to Use of Scores: Source: http:// www.ets.org/gre/institutions/scores/guidelines/

What does the literature say about THE GRE & predicting student success? Meta-analyses come to differing conclusions. Morrison & Morrison, 1995; Kuncel , et al., 2001; Kuncel & Hezlett, 2010Orlando, 2005Why? Studies draw upon different methods, different disciplinary and institutional contexts, and different populations.Only a few correct for attenuation bias; ETS continues to revise the test.What do we know?There has never been a true validity study: Denied students aren’t studied.Correlations vary by exam and by graduate school outcome (Kuncel & Hezlett, 2007). The longer the time between the test and the outcome, the weaker the validity.A flurry of discipline-specific studies: some find relationships with first year graduate school GPA, none with later outcomes, race or gender (despite score gaps)

r = 0.33; N = 1743 r = 0.02 [ETS 0.04]; N = 2133 r = 0.15; N = 2133 r = 0.33; N = 1743 r = 0.24 [ETS 0.27]; N = 1686 r = 0.15 [ETS 0.18]; N = 2133 Miller et al., submitted

r = 0.33; N = 1743 r = 0.02 [ETS 0.04]; N = 2133 r = 0.15; N = 2133 r = 0.33; N = 1743 r = 0.24 [ETS 0.27]; N = 1686 r = 0.15 [ETS 0.18]; N = 2133 Miller et al., submitted

Practical Significance?

UNDISCIPLINED ”HOLISTIC” REVIEW

Astrophysics committee:BLINDSPOTS in Holistic review

Juan: Is it enough to be a woman in science? (Discussion of how different perspectives might affect the community.)Prabhat: Lisa said she wants to be a role model because she never received explicit encouragement until recently. She wrote about the importance of providing active support, not just the absence of discrimination.Wayne: Shawna says she needs to develop self-confidence and overcome self-doubt.Juan: And then there’s Amy, who claimed to experience teasing and bigotry from her peers and a high school science teacher. She went to an all-women’s college so she could still study science. Chris: I’m less persuaded by that story. Maybe the teacher was young and inexperienced in handling high school boys. She might come to the program with an axe to grind. Juan: Either way, now she’s taking action, organizing a lecture series on women in science... We need to read between the lines on these things. (Person eventually nominated had started an astronomy outreach program and had letters of recommendation from familiar names.)

Prabhat: He grew up in a yurt in the Himalayas, was raised by his mom and grandma after his father died at an early age, and the next neighbors were two mountains over. He then found his way to a major U.S. public research university and has since started the only organization for the discipline in the Himalayan region. Jeff: But do we think he can succeed? [long pause] Prabhat: He’s the most amazing case we’ve ever seen. George: He would bring some personality to the department. I commit to look after him and fund him through the prelims…. He presents himself as quite intelligent. Chris: Excellent idea to give him a chance. [Student ultimately admitted and enrolled.]

IMPROVING HOLISTIC REVIEW

”Beware the tyranny of best practices.”-Paul Courant

What is holistic review? “…the consideration of a broad range of candidate qualities including ’noncognitive’ or personal attributes” (CGS, 2016, p. iii)Three types:Whole file: Considers all parts of the application. Whole person: Considers many facets of the applicant.Whole context: Considers the context in which the applicant achieved what they did. ( Bastedo , Bowman, & Glasener, 2018)http://cgsnet.org/ckfinder/userfiles/files/CGS_HolisticReview_final_web.pdf

Comprehensive Systematic Contextualized

Comprehensive Numerous and diverse criteria Consider the whole person and the sum of their potential Note that diverse perspectives improve scholarly work Consider that socio-emotional skills are necessary for outstanding professional performance

Non-Cognitive Competencies Social and emotional skills that we use to navigate life. Measurable! Results from decades of psychology research (developmental, social, and industrial-organizational) Predict academic/job performance Little, if any, group differences by gender and race Orthogonal to cognitive measures (e.g., GPA, SAT/GRE)Domain specific. Some will be specific to academia, grad school, and/or fields of study.

Self-Management competencies correlate with clinical grade. “Cognitive ability and knowledge are threshold aspects of professional work, necessary but not sufficient for outstanding professional performance.”Victoroff and Boyatzis , J. Dent. Ed 77, 416 (2013)Achievement Orientation AdaptabilityInitiativeEmotional Self-ControlTrustworthinessConscientiousnessOptimismDidacticClinical CognitiveYesNoNon-CognitiveMaybe YesCorrelating professional performance with admissions criteria and non-cognitive competencies

Self-Confidence : A strong sense of one’s self-worth and capabilities.Accurate Self-Assessment: Knowing one’s strengths and limits.Emotional Awareness: Recognizing one’s emotions and their effects. Optimism : Persistence in pursuing goals despite obstacles and setbacks. Trustworthiness : Maintaining integrity.Achievement Orientation: Striving to improve or meeting a standard of excellence.Conscientiousness: Taking responsibility for personal performance.Adaptability: Flexibility in handling change.Emotional Self-Control: Keeping disruptive emotions/impulses in check.Initiative: Readiness to act on opportunities. Cultural Awareness: Respecting and relating well to people from varied backgrounds.Organizational Awareness: Reading a group’s emotional currents and power relationships.Empathy: Sensing others’ feelings and perspectives, and taking an active interest in their concerns.Service Orientation: Anticipating, recognizing, and meeting customers’ needs.Teamwork and Collaboration: Working with others toward shared goals and creating group synergy in pursuing collective goals.Communication: Listening openly and sending convincing messages.Building Bonds: Nurturing instrumental relationships.Conflict Management: Negotiating and resolving disagreements.Influence: Wielding effective tactics for persuasion. Change Catalyst: Initiating or managing change.Inspirational Leadership: Inspiring and guiding individuals and groups.Developing Others: Sensing others’ development needs, bolstering their abilities.Self ManagementSelf Awareness Relationship ManagementSocial Awareness

Metrics in context Note intrinsic errorNote societal patterns Achievements in context Distributions of opportunities relative to societal patterns Achievements don’t always signal aptitude or ability Admissions in contextHow students aid your program’s identity/mission and broader goalsBase review on shared, predefined criteria with structured protocols, for efficiency & consistency.Build in safeguards & checks to promote equity and limit biases.Carefully select & train gatekeepers Coordinate evaluation with recruitment and yield effortsContextualizedSYSTEMATIC

Rubrics: Comprehensive, Contextualized, & Systematic

Rubrics offer benefits that redress common drawbacks in many programs’ process. EFFICIENCY is enhanced by expediting review, reducing faculty load. STRUCTURE for a process in which many applicants are compared on multiple dimensions.SPECIFICITY about what reviewers should be looking for may reduce implicit bias and prevent unseemly considerations from creeping in. TRANSPARENCY about evaluation criteria is good for decision makers, their colleagues, and applicants themselves.RELIABILITY across raters can be assessed.ACCOUNTABILITY heads off charges that the process is unfair.

Developing a rubric:Identify qualities on which everyone should be evaluated. Here, knowing your program mission can be very helpful.Qualities can be broad if you want to leave room for individual interpretation & multiple ways for people to fulfill themOr, qualities, can be narrowly defined if you want a highly structured process. Examples: Research experience, Academic preparation, Clearly defined goals align with program expertiseRecommended: If you choose to require GRE scores, fold GRE scores and grades into a single judgment of academic preparation, to prevent anchoring bias and/or attributing small differences in scores/grades into large differences in overall quality.

Developing a rubric:Define how you will measure/ operationalize the qualities named above. What does it means for an applicant to be outstanding, strong, acceptable, or weak on each of these?The more concrete your definitions, the more consistent you can expect your judgments to be.Recommended: Create space for comments to justify assessments; Leave open the possibility of naming unique strengths that merit special consideration. Optional: Weight some qualities more than others.

Using the rubric A rubric is only as beneficial as users’ fidelity to it.Calibrate and increase inter-rater reliability by having all members independently rate two applications, then meet to discuss how they came to their scores.Ensure each application is reviewed by 2+ people. If there is significant divergence in the ratings, bring in a third reader.Prepare in advance a plan to subject very unique cases to a different sort of evaluation.

Consider your committee Composition: Larger committees with a variety of voices require less effort per person & reduce risks of groupthink and homophily . Would your committee consider including students, staff, or alumni in decision making?Discussion norms Discuss how race and gender may be relevant: Easier if faculty talk candidly elsewhere in department life about race and gender dynamics. Disagree with each other: Where trust is higher, friendly debate & challenging assumptions may be less threatening.Provide reasons for decisions: Central to deliberative democracy; legitimizes the outcome.Get comfortable with discomfort: Develop capacity & collective will to talk about how race, gender, and other power dynamics.

Faculty & administrative Coordination Obtaining institutional data, disaggregated by race and gender, to test evidence for common assumptions about risk.Facilitating opportunities for learning & create accountability structures Considering with you the biases primed or mitigated through the application’s design, content & layout.Incentivizing admissions & recruitment for diversity through fellowships

Work together to strengthen recruitment: build the pool & close the deal Coordinate admissions, recruitment, and outreach.1Assess cues about the value of diversity sent by your online presence.2Remember that prestige & financial aid are not enough to attract students of color.3 Faculty composition may send cues about the qualities necessary for success.4Faculty responsiveness & early contact.3Student ambassadors can play an important role. 5 Positive climate as a competitive advantage. 51 Griffin & Muñiz, 20092 Slay et al., forthcoming3 Bersola et al, 20144 Correll, 20015 Posselt et al., in press

conclusion

discussion Julie Posseltposselt@usc.edu

LETTER GRADE CRITERIA A Shows an ability to handle ancient material in an intelligent, precise and informed manner; use of relevant reading, which has been absorbed and analyzed critically; shows an ability to argue a case in a well-organized manner; the very best work will also show some original thought well expressed. There may be shortcomings in any one of these criteria, but the first class work must clearly and significantly outweigh any such flaws. No spelling errors. No or almost no typographical errors.BAnswers all questions citing adequate sources and arguments and opinions have logical justifications; has analyzed sources critically in most cases. Very few typographical errors. CWhile the assignment is complete, it is poorly supported by source citation. There is little or no sophistication of argument, or critical acumen. The overall strengths of the piece clearly do not outweigh its flaws; Typographical errors.DΑ clear and significant failure to achieve an adequate level in the criteria. Ancient sources/material are handled in an incorrect and ill-informed manner; assignment is grossly incomplete; full of errors.Grading Rubric for Written AssignmentsCLCIV 372 Fall 2015

DIMENSION OF FIT HIGH MEDIUM LOW research research interests align with multiple faculty in multiple subfieldsresearch interests align with multiple faculty in one subfieldlimited alignment between student interests and faculty expertise facultysomeone wants to hire as RA now and/or there is a clear fit with current faculty expertisesomeone could supervise, but interests do not directly support a faculty member's work faculty aligned with applicant's interests are not seeking studentscommunityhas clearly contributed positively to prior department/school culture, and would do the same for our programsome evidence of participating in service activitiesapplicant only discusses him/herself; no evidence of engagement in department or university activitiesdiversity applicant has been an active advocate for diversity in physics belongs to an underrepresented identity group; first generation in college or low SES; and/or contributes to another type of diversity the department seeks contributions to diversity are unclear from the application program mission fits a unique program goal, e.g., wants to be a community college teacher, entreprenuer, etc. Background and interests generally a good fit for the program's focus Student interests and experience do not seem well-suited for this particular program's focus geography student has expressed a significant reason for wanting to be at/near our location some desire for our location no mention of geography potential exemplar consistent excellence in and beyond coursework and research strong and well rounded in and out of school limited evidence of potential for exemplary scholarship and/or program participation

Legal landscape

Bakke Fisher Gratz & Grutter 1978 2013, 2016 2003University of CA-DavisMedical School University of Michigan undergraduate education & law schoolUniversity of Texas undergraduate education

Bakke Fisher Gratz & Grutter 1978 2013, 2016 2003Racial quotas are unconstitutional.Race is a permissible “plus factor,” BUT policies must be “narrowly tailored” to achieve diversity,which is the only “compelling state interest” for aff action.

Bakke Fisher Gratz & Grutter 1978 2013, 2016 2003Racial quotas are unconstitutional.Race is a permissible “plus factor,” BUT policies must be “narrowly tailored” to achieve diversity,which is the only “compelling state interest” for aff action. Redressing the “present effects of past injustice” was ruled to be an unconstitutional basis for affirmative action.

Bakke Fisher Gratz & Grutter 1978 2013, 2016 2003Predetermined points for race/ethnicity unconstitutional (Gratz), BUTrace can be considered as one of many factors (Grutter) in a holistic way.

Bakke Fisher Gratz & Grutter 1978 2013, 2016 2003 Colleges must offer a “reasoned, principled explanation” for diversity.Race-conscious admissions must …be narrowly tailored to achieve diversity goals.…withstand strict scrutiny (i.e., demonstrate that diversity can’t be achieved through means that don’t require the consideration of race).

Meanwhile, 8 states have banned affirmative action. BALLOT INITIATIVESArizonaCaliforniaMichiganNebraskaOklahoma Washington LEGISLATURE / GOVERNOR New Hampshire Florida Institution-SPECIFIC University of Georgia

In other states, key PRINCIPLES for practice FROM BAKKE STILL STAND. Reserving seats or shares of seats for underrepresented students is not permissible.Reviewers should use a common evaluation process for all applicants.Race should be just one of several individual characteristics assessed as a plus factor.Every applicant should be evaluated as an individual, not assumed to represent a broader identity category. Programs should not single out specific racial/ethnic groups, but consider contributions that all groups make to diversity. Source : UCLA Civil Rights Project, 2002

Legal Landscape in sum Under specific conditions, race-conscious admissions policy is constitutional outside the states mentioned, as part of how SCOTUS conceives of academic freedom. However, the parameters are arguably tightening. Universities & graduate programs MUST seek to build diversity in multiple ways, and they should have a “reasoned, principled explanation” for why diversity has value in their context.Weighing race as a formal admissions consideration is different than accounting for how dynamics of race in America may shape……t he distributions of grades, test scores, and institutional affiliations our applicants hold.…the kind of opportunities and viewpoints our students contribute.Our admissions committees need not be colormute , & they will be best protected legally if admissions policy is defined. Ad hoc policy is hard to defend.

RECRUITMENT

Institutional actions Student actions

Importance placed on various institutional characteristics by two hypothetical students.Bersola et al. (2014).

Importance placed on various institutional characteristics by two hypothetical students.Consider: Which one will be easiest to attract, if the students were considering your program? Importance placed on various studentcharacteristics by two hypothetical professors.Consider: Which one would be most admissible to your program? Bersola et al. (2014).

What faculty thoughtFinancial aid is paramount What non-matriculants said77% of non-matriculants said they would have still enrolled at their current institution if Western University had matched their current institution’s package. Faculty may misjudge what is important to students’ matriculation decisions. Bersola et al. (2014).

What faculty thought85% rated their yield activities as “strong” or “above average.” What non-matriculants saidWhen asked ”which institution gave a more favorable impression,” 60% named their current institution, 27% rated them the same, and only 13% rated Western University higher. Bersola et al. (2014). Faculty may misjudge what is important to students’ matriculation decisions.

Recruitment strategies used by high-diversity STEM programs in research universities PsychologyWebsite revampCreation of a diversity-focused curriculum trackCoffee hour during campus visit weekend for “straight talk” about diversity in the department.Beware the risk of bait & switch Slay, Posselt, & Reyes (2017) Applied physics Individualized curriculum Prominent role of administrative staff in all facets of program life.“Eyes & ears of the department”Family-like roles with prospective & current studentsCultural translators to aid faculty in serving students across race & genderClimate as a ”competitive advantage” in the admissions process.Posselt, Reyes, Slay, Kamimura, & Porter (2017)

Domains of recruitment work Online messagingProgramming & points of connection for studentsFinancial aidFaculty compositionFaculty responsiveness & one-on-one contact Student ambassadorsClimate DISCUSS: Which of these are strengths & weaknesses in your department? How could you shore up weaknesses?