The Role of Unmet Financial Need in Retention - PowerPoint Presentation

Slide1

The Role of Unmet Financial Need in Retention

Reuben

Ternes

Oakland University

MI/AIR Nov. 2017Slide2

Overview

Primary Question:

If we met the financial needs of all of our students, what would their retention rates be?

Or, alternately, what is the role of unmet financial need in (first year) retention rates? Slide3

Definitions & Demographics

We are mostly concerned with Full-Time, First Time

cohorted

students.

The same group that is reported for typical IPEDS first year retention rates.

Oakland University is a large Midwestern doctoral research university

19,333 students (fall 2017)

Heavy undergraduate population

A large working-class student body

Many of whom work….A lotSlide4

Working HoursSlide5

What is unmet financial need?

We define unmet need in the following way:

Cost of Attendance – non-loan aid – EFC

Or, more specifically:

Unmet Need = (Tuition + Housing Cost + Miscellaneous Costs) – (Scholarships + Need

Aid +

Expected Family Contribution

)

Scholarships would include all institutional scholarships and outside scholarships

Need aid includes both federal (Pell, Work Study) and institutional need aid

Tuition should be exact (not estimated).

Housing should be exact (if on campus) or estimated (if off campus)

Misc. costs often depend on housing status (on/off), but is generally estimated by the financial aid office. Slide6

A Brief Tangent About Measurement IssuesSlide7

Are Cost of Attendance Estimates Accurate?

Measurement is an often overlooked but critically important piece of all statistical analyses.

It’s worth examining if the cost of attendance

(COA)

estimates at your institution are an accurate depiction of real costs.

But, how can we tell if the estimate are accurate? Triangulate!

Find an independent estimate to compare with.

One possible source was recently featured on the Chronicle:

http://

www.chronicle.com/interactives/cost-of-living

Created by the Wisconsin HOPE Lab.

OU’s COA was within $7 of what HOPE Lab found.

That made me feel pretty good using our COA estimates as is. Slide8

COA Differences MI Public Universities

Name

Institution Estimate

HOPE Estimate (County Data)

Difference

OU

$

11,292

$11,300

$7

UM-D

$

11,286

$11,111

$176

MSU

$

10,742

$10,544

$199

GVSU

$

10,716

$10,202

$515

LSSU

$9,143

$9,734

$590

WMU

$

10,872

$10,197

$675

NMU

$10,436

$9,743

$694

UM-AA

$12,050

$11,201

$850

FSU

$10,526

$9,473

$1,054

EMU

$9,898

$11,201

$1,302

CMU

$8,240

$9,743

$1,502

SVSU

$11,612

$9,842

$1,771

MTU

$11,447

$9,491

$1,957

UM-F

$12,670

$9,878

$2,793

WSU

$

14,170

$11,111

$3,060Slide9

More Measurement Issues!

For many SISs, COA is a variable that can be attached to an individual student’s record.

But we couldn’t use that variable.

Why?

Because, tuition, a huge component of COA, was estimated.

A student taking 18 credits was getting the same tuition estimate as a student with 14 credits. But our tuition is based entirely on credits!

In addition, COA estimates sometime change if the student indicates they are attending only one semester.

So the data needed further cleaning to make sure that COA estimates were comparable for students that attended for one semester

vs. those that attended for the whole year.

All COAs were recalculated for students based on actual credits.

Students that attended only for the fall semester had their COA’s doubled (as well as their disbursed aid) to keep them in relative sync with students that attended for both fall and winter. Slide10

Measurement issues abound!

Expected family contribution is an important part of the formula used here.

What about students that did not complete the FAFSA?

It was then assumed that these students had no unmet need.

Is that a true statement? Probably not.

But it’s hard to assume otherwise. Slide11

Now back to our regularly scheduled program…Slide12

The Sample

Examined 2015 FTIAC class

Full time only.

70 students were excluded because they started in fall 2015 with a part time status.

Also excluded international students and out of state students. Their COA’s are way different, and probably deserve special analysis.

Plus, we don’t have many of them. 56 total students excluded here.

Also excluded students that started full time, but then dropped to part time in winter.

This decision could be criticized. I’m still not sure if it was the right decision or not. But I found it difficult to create an appropriate COA for these students.

63 students met this exclusion criteria.

Final Sample was 2,084

Total exclusions represented only 7% of the total 2015 class. Slide13

Ok. Some Demographics

Table

1.

2015

Full-Time FTIAC Outcomes by Financial Need Category

Unmet

Financial Need

N

% of Total

Retention Rate

1st Semester Probation Rate

Pell Rate

% URM

Avg. HS GPA

None

1033

40%

79%

8%

2%

9%

3.48

$1-$3k

222

9%

86%

8%

23%

13%

3.70

$3k-$6k

308

12%

84%

5%

45%

12%

3.56

$6k-$9k

355

14%

79%

9%

60%

19%

3.39

$9k-$12k

259

10%

74%

16%

65%

22%

3.23

$12k+

411

16%

50%

30%

72%

43%

3.06Slide14

Table 1 Highlights (A)

That’s a lot of info in a little table. Let’s break it down.

Retention rates are (mostly) stable until the highest unmet need category

This suggests that moderate unmet financial need may not be an insurmountable retention hurdle. Extreme need is clearly disruptive. Slide15

Table 1 (A) – Higher unmet need = lower retention rates

Table

1.

2015

Full-Time FTIAC Outcomes by Financial Need Category

Unmet

Financial Need

N

% of Total

Retention Rate

1st Semester Probation Rate

Pell Rate

% URM

Avg. HS GPA

None

1033

40%

79%

8%

2%

9%

3.48

$1-$3k

222

9%

86%

8%

23%

13%

3.70$3k-$6k30812%84%5%45%12%3.56$6k-$9k35514%79%9%60%19%3.39$9k-$12k25910%74%16%65%22%3.23$12k+41116%50%30%72%43%3.06Slide16

Table 1 Highlights (B)

Higher unmet need is correlated with a host of other risk characteristics:

Higher Pell Rates

Lower HS GPA

Higher first semester probation rates

Minority statusSlide17

Table 1 (B) - Higher unmet need is correlated with a host of other risk characteristics

:

Table

1.

2015

Full-Time FTIAC Outcomes by Financial Need Category

Unmet

Financial Need

N

% of Total

Retention Rate

1st Semester Probation Rate

Pell Rate

% URM

Avg. HS GPA

None

1033

40%

79%

8%

2%

9%

3.48

$1-$3k

222

9%

86%

8%

23%

13%3.70$3k-$6k30812%84%5%45%12%3.56$6k-$9k35514%79%9%60%19%3.39$9k-$12k25910%74%16%65%22%3.23$12k+41116%50%30%72%43%3.06Slide18

Table 1 (C) – The highest category of financial need is also one of the largest.

Table

1.

2015

Full-Time FTIAC Outcomes by Financial Need Category

Unmet

Financial Need

N

% of Total

Retention Rate

1st Semester Probation Rate

Pell Rate

% URM

Avg. HS GPA

None

1033

40%

79%

8%

2%

9%

3.48

$1-$3k

222

9%

86%

8%

23%

13%

3.70$3k-$6k30812%84%5%45%12%3.56$6k-$9k35514%79%9%60%19%3.39$9k-$12k25910%74%16%65%22%3.23$12k+41116%50%30%72%43%3.06Slide19

What does Table 1 really tell us?

So, what do these things all mean?

Basically, Table 1 illustrates the complicated relationship between unmet need, student characteristics, and student outcomes.

It’s not going to be as simple as ‘if we give these students more money, our retention problems will go away’.

However, the question remains: if we did address their financial struggles, how much improvement would we expect to see in first year retention rates? Slide20

Regression to the Rescue!

With some appropriate control variables, we can start to address this question using a basic logistic regression.

Though we may not be able to answer it with complete confidence.

For the most part, regression is still a correlational tool. And correlation does not imply causation. Slide21

Sample (yet again)

2015 FTIACs

(First Time in Any College)

N = 2,084

Excludes

Part-time students

Non-resident/out of state students

Includes

Students that did not fill out the FAFSA

EFC set to zero in these cases

This is a large group of students! Slide22

Variables

Outcome of Interest

First Year Retention (Binary)

Control Variables

Unmet Need (in thousands of USD)

Expected Family Contribution (in thousands of USD)

ACT Composite Scores

High School GPAs

First Term Credits

Housing Indicator (binary yes/no)

First Generation Status (binary yes/no)

Underrepresented Minority Stats (binary yes/no)Slide23

Why Not Add More Control Variables?

I could have added more control variables

For example, SES estimates or interaction terms

I chose not to

I have two main concerns with adding a large number of additional control variables

Interpretability

Overfitting

My goal is not to necessarily develop a comprehensive explanatory model, but rather, to estimate a generalized impact of changing financial aid dollarsSlide24

Results

Variable

B

S.E.

Wald

Sig.

Exp

(B)

Unmet Need

-0.091

0.010

78.652

0.000

0.913

EFC

-0.007

0.003

6.090

0.014

0.993

ACT

-0.021

0.018

1.343

0.247

0.980

HS GPA

1.044

0.147

50.7600.0002.841First Term Credits0.1240.03412.9720.0001.132Housing Indicator-0.1160.1160.9980.3180.890First Generation Indicator-0.0010.1540.0000.9940.999URM Status0.2580.1413.3400.0681.295Constant-3.2090.67722.4830.0000.040Nagelkerke R2 = 0.192Slide25

How is this even remotely useful?

Normally in a regression study, one would want to examine the significance level of some variable of interest and then write a really long paper about it.

The only variable I am interested in is unmet financial need.

And I don’t need to do a regression analysis to understand that this is an important variable. I think everyone already knows that.

Instead, I want to probe what happens to the outcome variable (retention rates) when (unmet need) is systematically decreased.

Q: How to do this?

A: MathSlide26

Logistic Regression Math

Retention Rate =

Those beta weights correspond to the B column in the results slide. All you have to do is plug them in and go.

In order to get our final answer though, we have to run some estimates on what the average might be for the other variables.

Some variables, like first generation, have basically no impact on the outcome at all.

But you can run a number of different scenarios to probe how the retention rate changes for various student demographics.

 Slide27

Results

Variable

B

S.E.

Wald

Sig.

Exp

(B)

Unmet Need

-0.091

0.010

78.652

0.000

0.913

EFC

-0.007

0.003

6.090

0.014

0.993

ACT

-0.021

0.018

1.343

0.247

0.980

HS GPA

1.044

0.147

50.7600.0002.841First Term Credits0.1240.03412.9720.0001.132Housing Indicator-0.1160.1160.9980.3180.890First Generation Indicator-0.0010.1540.0000.9940.999URM Status0.2580.1413.3400.0681.295Constant-3.2090.67722.4830.0000.040Nagelkerke R2 = 0.192Slide28

Bottom Line

Each $1000 that we decrease Unmet Need by, retention rates increase by about 1.6%.

This finding is very consistent with previous research that suggests that every $1000 equates to somewhere between 1% and 2% change in retention rates.

Includes other regression studies

Includes Regression Discontinuity studies

Includes Interrupted Time Series studies

Even if we

add $5000

of additional aid to the most needy group, we would expect their retention rates to improve from 50% to less than 60%.

That is a lot smaller of an impact than I suspect many other administrators would expect. Slide29

Policy

Results like these can really impact policy.

At $5k per student, can we do better than a 10% improvement in retention rates with other services?

A money only policy to student retention is unlikely to have the impact that many administrators might intuitively think.

To address retention, especially for those with high unmet need, a more comprehensive approach will be required. Slide30

Confidence (Error Estimates)

Q: As this is only a correlational design, how confident am I in my own conclusions?

Can’t use p-values to answer this question

A: Somewhat

Pros

Several control variables were added to reduce confounding factors

Results are consistent with previous research

I’m not testing multiple hypotheses (or any really)

Strong theoretical foundation (we know money matters)

Cons

Lots of data decisions to make. Unlikely that all of them were ideal.

A high amount of ‘Researcher Degrees of Freedom’ existed due to all of the data decisions.

Overall effect size is modest to begin with. Changing the Unmet Need by a single unit of standard error results (approx.) in a 0.7% change in retention rates. So two units of standard error would be almost the same as a $1000 change (1.6%). Slide31

Limitations/Criticisms

Doubling the COA (and aid) for students that attended only fall.

I think that one could criticize this decision. Alternatively, you could analyze only fall data (COA and aid).

Doing so is awkward, but it might be more ‘accurate’

I excluded part time students.

I also excluded non-resident students and out of state students.

I did not include any additional SES estimates – which I usually do.

This means the ACT/HS GPA data is also conflated with general SES. So, the concept of ‘academic preparedness’ is conceptually conflated with general SES.

I also included all students – setting those students without FASFA information to an EFC of zero.

I’m unsure if this is the right thing to do, or the wrong thing to do. Slide32

The End

Questions?