1 Delaware Cost Study Progress Report: - PowerPoint Presentation

1. 1Delaware Cost Study Progress Report: Beyond Average BenchmarkingUse of the Delaware Cost Study data with Data Envelopment Analysis (DEA) to Effect Productivity Improvement and Excellence in Resource UtilizationTom Eleuterio, M.S.Manager, Higher Education Consortiatommyu@udel.edu Ti Yan, PHD.Research Analystyant@udel.edu

2. What is the Delaware Cost Study?The National Study of Instructional Costs and Productivity (Delaware Cost Study) is a benchmarking project and data sharing consortia among four-year colleges and universities with over 200 institutions participating annually. Since 1996, over 700 institutions have participated, and over the past two decades, the Delaware Cost Study has become the “tool of choice” for comparative analysis of faculty teaching loads, direct instructional costs, and separately budgeted scholarly activity, within academic disciplines. The Cost Study has been a major data source for state agencies including:Association of American Universities Data Exchange (AAUDE)Southern Universities Group (SUG)University of North Carolina (UNC) SystemPennsylvania State System of Higher Education (PaSSHE)University of Missouri SystemUniversity of Nebraska SystemConnecticut State University System (CSUS)City University of New York (CUNY) System

3. Big PictureLeadership at institutional and departmental levels need to evaluate the instructional costs and teaching loads and to make decisions on allocation of faculty and other resources.Benchmarking is supposed to provide straightforward information for program evaluation and strategic planning.Compared to whom and on what metrics.What is the best practice target allowed by the current resources?

4. Benchmarking on WhatDelaware Cost Study surveys four-year institutions on annual instructional activities and costs and provides benchmarking results at the discipline (CIP) level.Who teaches what to whom at what cost?Refined Means of Cost $ per Student Credit Hour by CIPRefined Means of Cost $ per FTE student by CIP

5. Benchmarking with WhomOur participants traditionally decide their peers for benchmarking purposes based on Carnegie Classifications and regional factors.

6. Accumulating EvidenceInstructional cost of an academic program can be affected byThe discipline: 70%-80% variation explained ( Middaugh, M. etc, 2003)The highest degree offeredThe percentage of bachelor degrees offered per total degreesThe percentage of tenured/tenure-track faculty members per total faculty……

7. Problem Space 1: Benchmarking with WhomIndividual Academic programs might not be aligned with their institutional classification. A program that only offers bachelor degrees is affiliated with a research high university.Evaluating departmental/program-level cost of instruction requires comparison across program peers.Programs peers are not equivalent to institutional peers.

8. Problem Space 2: Benchmarking on What The averaged Cost per SCH or per Student does not reflect your exact position as compared to your peers.Benchmarking results should inform decision makers of specific, quantitative and measurable goals for continuing improvement.

9. Research Questions1. At a discipline level, which institutions offer programs that are the most comparable in terms of instructional productivity?Solution: Data-driven Peer Selection by discipline. 2. Among those program peers, who produces the optimal combination of instructional outputs within their given inputs? Who can do it better and how?Solution: Identifying the best-performing units among peers and the gaps between them and every one else in a selected group of metrics.

10. Research Question 1At a discipline level, which institutions offer programs that are highly comparable on instructional productivity?Solution: Data-driven Peer Selection by discipline. Assumptions:“Psychology might select Psychology peers and Biology might select a different set of Biology peers..”(Chatman, 2017)Program peers have similar characteristics of instructional productivity, highest degree offered, % of bachelors per total degrees…

11. Research Question 2Among those program peers, who produced the optimal combination of instructional outputs at a given input? Who can do it better and how?Solution: Identifying the best-performing units among peers and the gaps between them and every one else in a selected group of metrics.

12. California Cost Study Participants 1996-20179697989901234567891011121314151617Institution                 XXXXXAzusa Pacific University                  X   Biola University                    X California Institute of the Arts                 X    California Lutheran University                   XX California State University - Dominquez Hills                     XCalifornia State University - East Bay  X X                 California State University - Fresno                     XCalifornia State University - Long Beach            X         California State University - Los AngelesXX           X        California State University - Monterey Bay          XXXXXXXXXX  California State University - San Marcos            X X     X California State University - StanislausXX                  X Concordia University - IrvineX        XX X XXX     Dominican University of California              X X  XXXLa Sierra University X                    Loma Linda University X                    Loyola Marymount UniversityXX      X             Naval Postgraduate SchoolXX             XXXX  XPoint Loma Nazarene University                 X    Simpson University        X  X          Sonoma State University     X           XXXXXUniversity of California - Irvine                 XX XXUniversity of California - Merced X     XXXXX          University of San Francisco               X X   XUniversity of the Pacific

13. Data & Measurement: Data

14. Data & Measurement: Data

15. Major Findings from the 2003 NCES StudyAcross almost all disciplines, the volume of teaching activity, measured in student credit hours taught, is always associated with direct instructional expense. Cost decreases as the volume of teaching increases.Department size, measured in terms of total number of faculty, and total number of tenured and tenure track faculty, is consistently associated with cost across the disciplines. The larger the faculty size, the higher the cost.

16. Major Findings from the 2003 NCES StudyThe proportion of total faculty who are tenured or who are on tenure track is associated with cost. The higher that proportion, the higher the cost.Among variables that measure faculty teaching workload, the mean number of student credit hours taught per FTE faculty is the most common cost factor across the disciplines. The larger the average number of student credit hours taught, the lower the cost.

17. Findings from the 2014 Delaware Cost StudyAccording to the National Center for Education Statistics, 76 – 82 percent of the variation in cost is located at the academic disciplinary level.

18. OutlineData Envelopment Analysis (DEA) – Basics and ApplicationLatent Class Analysis (LCA), a subset of Structural Equation Modeling (SEM), was used to identify “refined” groups for a longitudinal dataset obtained from our study.Naïve Bayesian method was used to classify one-year participants to the settled groups by LCA.DEA results and discussion: case studyDiscussion of next steps possible for use by colleges or departments

19. Research Question 2Among those program peers, who produced the optimal combination of instructional outputs at a given input? Who can do it better and how?Solution: Identifying the best-performing units among peers and the gaps between them and every one else in a selected group of metrics.Approach: Data Envelopment Analysis (DEA)An efficiency number is determined by both input (cost of instruction) and output (teaching productivity).There could be more than one efficient unit in a group of peers.Efficient units produce the best outputs with the given input.The other units will have a best-practical output with the given input.

20. DEA (Data Envelopment Analysis)Efficient FrontierEfficiencies: Efficiency Number

21.

22. Tenure-tracked FTE are quite different from other types of FTE.No difference between SCHs taught by different types of faculty members.SCHs taught at different student levels are not replaceable. The ability to gain separate budget indicates the research and service ability.Total SCHs taught at different levels - Lower/Upper Division, Graduate, Individual InstructionsInput variables Number of FTE Instructional Faculty -Tenured/Tenure-eligible and All othersTotal direct expenditures for instructionOutput variablesApplying DEA to Delaware Cost Study Results: Assumptions & MetricsAssumptionsMetrics

23. DEA: An example DMUInputOutput1Output2A100400B100205C1001020E (29.2,7.3)A,C: Efficient; B: InefficientE: “Best” Virtual DMU of B Decision Making Units (DMU) represent a group of organizational units that are compared in the process of DEA based on their measurements on a certain set of inputs and outputs. Two weeks ( 100 hours) labor as constant input, output 1 is undergraduate SCH, output 2 is grad SCH

24. Fig 1 shows a set of DMUs with each consuming the same amount of a single resource (e.g. a fixed number of faculty members) and producing different amounts of two outputs: y1 (e.g. Lower Division Student Credit Hours) and y2 (e.g. Upper Division Student Credit Hours).Solid lines construct the efficiency frontier on which all DMUs are efficient ( efficiency # =1).DEA: IllustrationTwo possible best-practice virtual producers for P5 are labeled as P’5 and P’’5.P’5 can be achieved if y1 and y2 could be increased proportionally as depicted in last slide.If y2 could not be increased for P5, the alternative is to increase y1 solely to reach the efficiency frontier, shown as P’’5.In the case of P6, only one possible best virtual producer is labeled as P’6.Figure retrieved 5/25/2017 from http://deazone.com/en/resources/tutorial/graphical-representationOutput 2Output 1

25. Discipline Group - 4 digit CIPRange of Efficency numberTotal in Group1.0 0.99 to 0.800.79 to 0.600.59 to 0.40Humanities51 4Social Sciences62121Arts321Health Sciences44Engineering7214Natural Sciences514 Education & Human Development211Business & Economics51211Agriculture and Natural Resources4211Earth Ocean Environment3111Total44166148Percent100%36%14%32%18%Results: University of Delaware

26. Pre-requisites for using DEA: Quality data and representative peer groups DEA depends upon using reliable data ( data cleaning) and comparing similar decision making units (DMUs) ( cluster selection from population).Carnegie classifications are done at the institutional level not the academic discipline level; some departments are poorly aligned with a Carnegie class category.Using longitudinal data over two or more years of the Delaware Cost Study provides for data smoothing that reduces ‘one-time’ data anomalies or distortions that may occurSelecting data-informed peer groups for use in DEA improves the reliability of the efficiency measures and provides focused targets to improve teaching activity.

27.

28.

29. 1. Highest degree granted ( Bachelors, Masters, Doctorate or Professional Degree for each study year 2012 through 2015) 2. The proportion of bachelors degrees of the total degrees awarded (2012-15)3. The proportion of undergraduate class sections compared to the total class sections (2012-15)4. The proportion of class sections taught by tenured-tenure track faculty of the total class sections taught (2012-15)5. Research & Public Service : Expenditures in Externally Sponsored grant activity Normalized (0,1) across the four years from 2012 through 20156. The total number of all degrees awarded by the departmentVariables used in the latent class analysis model (LCA)

30. How many clusters are optimal?Information Criterial The Akaike Information Criterion (AIC) is a measure of thegoodness of fit of a model that considers the number ofmodel parameters (q).AIC = 2q − 2 log Ll Schwarz’s Information Criterion (also called the BayesianInformation Criterion or the Schwarz-Bayesian InformationCriterion) is a measure of the goodness of fit of a model thatconsiders the number of parameters (q) and the number ofobservations (N):BIC = q log(N) − 2 log Ll Other information criteria exist - most are based on theconcept of entropy.l For another example, look up Shannon Entropy.

31.

32.

33. Model produced average total degrees(degtot) and average bachelors degrees(nb) for each year

34. OutlineData Envelopment Analysis (DEA) – Basics and ApplicationLatent Class Analysis (LCA), a subset of Structural Equation Modeling (SEM), was used to identify “refined” groups for a longitudinal dataset obtained from our study.Naïve Bayesian method was used to classify one-year participants to the settled groups by LCA.DEA results and discussion: case studyDiscussion of next steps possible for use by colleges or departments

35. Naïve Bayesian Clusters – Peers in one yearGoal: to classify all participants on a CIP reported in 2015 into the four latent classes generated by LCA. Data Feature: one-year participants are made of two subsets. Subset 1 are those who participated across all four years and were involved in the LCA procedure. The remaining are Subset 2 who were excluded from LCA due to missing data in previous years. Action: treat Subset 1 in which cases were classified into four groups by LCA as the training set, and then use the naïve Bayesian method to predict the probabilities of belonging to a certain group for Subset 2. Outcomes: 174 cases were classified into 4 groups. UD is in Cluster #1 with 30 peers which are highly comparable in terms of the selected metrics.

36. Naïve Bayesian Clusters to incorporate all 2015 Delaware Cost Study Participants

37. Latent Class Analysis CIP-specificFour-year participants identified by FICENo missing data6 metricsHighest degree (categorical)# of all degrees% of bachelors in all degrees% of UG OCS in total OCS% of TT OCS in total OCSStandardized research and public service $.71 cases, 4 groups.Data Envelopment AnalysisNaïve Bayes ClustersCIP-specific2015 participants identified by FICEMissing data allowed in earlier years.Identifying refined groups of peers in one-year participation.Group peers are comparable in terms of the 7 metrics and ready for efficiency assessment in the next step.CIP-specificEach individual group peers from 2015 participants. Inputs: faculty number and rank, direct instructional $.Outputs: SCHs at UG and GR levels.Efficiency numbers and Best-practice virtual DMU generated.

38. A Map of 174 participants in a specific CIP for The Year of 2015 Delaware Cost Study, classified by four groups.

39. 31 Cluster #1 Institutions from the 2015 Delaware Cost Study with 174 total participants in this specific CIPunivnameclasshidegalldegstateunivnameclasshidegalldegstateBoise State UniversityR32BMIDUniversity of DelawareR11BMDDEDePaul UniversityR32BMILUniversity of KansasR11BMDKSFlorida International UniversityR12BMFLUniversity of Massachusetts AmherstR11BMDMAGeorgia State UniversityR11BMDGAUniversity of Missouri - ColumbiaR11BMDMOGrand Valley State UniversityM12BMMIUniversity of Missouri - St. LouisR22BMMOKansas State UniversityR12BMKSUniversity of New Hampshire Main CampusR21BMDNHMiami University - OxfordR21BMDOHUniversity of North Carolina at Chapel HillR11BMDNCNorth Carolina State University at RaleighR12BMNCUniversity of Tennessee - KnoxvilleR11BMDTNNorthern Arizona UniversityR21BMDAZUniversity of UtahR11BMDUTNorthern Illinois UniversityR21BMDILUniversity of VermontR22BMVTOhio State University - Main CampusR11BMDOHUniversity of Virginia - CharlottesvilleR11BMDVASimon Fraser UniversityR21BMDBCVirginia Polytechnic Institute & State UniversityR11BMDVASUNY at AlbanyR11BMDNYWilfrid Laurier UniversityM11BMDONTUniversity of California - IrvineR11BMDCAWright State University - Main CampusR32BMOHUniversity of Central FloridaR11BMDFLYoungstown State UniversityM12BMOHUniversity of ConnecticutR11BMDCT

40. Descriptive Summary of Clusters 1 & 4Of the twelve AAU participants in the 2015 Delaware Cost Study, three did not provide a complete dataset so they were not included in the DEA. Six are included in cluster 1 and are highlighted in yellow in the summary table and 1 was the lone member of cluster 4. This one department was literally in a class by itself due to large expenditures on research and public service relative to the other 173 data submissions for this discipline.80.6% of the programs included in cluster 1 are classified as either R1 or R2. The other 19.4% are either R3 or M1 institutions.

41. OutlineData Envelopment Analysis (DEA) – Basics and ApplicationLatent Class Analysis (LCA), a subset of Structural Equation Modeling (SEM), was used to identify “refined” groups for a longitudinal dataset obtained from our study.Naïve Bayesian method was used to classify one-year participants to the settled groups by LCA.DEA results and discussion: case studyDiscussion of next steps possible for use by colleges or departments

42. Comparison of similar DMUs in close proximity to each other and the total Cluster # 1 averages (Grp Avg) Efficiency NumberCarnegie ClassUG DegreesTotal DegreesTT FacultyTotal FacultyUG SCHGR SCHCost per SCHFTE StudentsRes & Ps per TT FacDMU10.837R1161.7175.130.6368.6528481477$285847.2$5803DMU20.836R1193.0210.336.60108.85358291024$2571238.7$6278DMU31R1190.020627.8186.50343651063$1051310.6$32387Grp Avg  142.6177.033.883.4273361632$251955.0$8033Applying DEA to Cluster 1:16 out of 31 or 51.6% of the programs in cluster 1 had efficiency numbers of 1.00. These 16 form the data envelope.

43. DMU 1 with efficiency # = 0.837 would have reached 1.00 by teaching 110 additional graduate student credit hours in the Fall 2015 semester EFF#REF_FICE_ID WeightTT FacultyAll_Other FacultyDirect CostsLD SCHUD SCHUG IND SCHGR SCHGR IND SCH 19600150.58089727.845$3,710,499 12,9535,406408342193 15500150.1735265257.7$4,743,934 5,1901,9927526251 15510150.332293730.56.9$5,233,877 5,4842,53823268101FOC_ID Best-practice Virtual DMU30.629.8$4,717,801 10,2474,329258333155140150.837Focal DMU30.629.8$8,259,454 8,5783,6248022333  Best-practice Virtual - Focal00($3,541,653)1,669705178110122Target Increased Number of Grad Students = 12

44. Graduate SCH teaching in lecture sections & as individual instruction by Focal DMU #1 is well below 3 efficient neighbors and best-practice DMU

45. Web portal view of workload & cost benchmarks for DMU #1 showing potential for growth in Graduate SCH versus Carnegie 3-year and average cost/SCH 2012-2014 5.4 times 46 regular faculty FTE yields 249 Graduate SCH Target goal: 12 of 31tenured faculty members take on one new grad student

46.

47.

48. DMU 2 with efficiency # = 0.836 would have reached 1.00 by teaching 56 additional graduate student credit hours in the fall 2015 semester EFF#REF_FICE_ID WeightTT FacultyAll_Other FacultyDirect CostsLD SCHUD SCHUG IND SCHGR SCHGR IND SCH 11620150.001692755$8,073,872 13,5482,04428117636 13200150.9857634.769$8,769,570 17,6513,54991312283 19600150.0850327.845$3,710,499 12,9535,406408342193FOC_ID Best-practice Virtual DMU36.671.9$8,973,798 18,5243,962124339295720150.837Focal DMU36.672.3$9,485,001 15,4812,49910428315  Best-practice Virtual - Focal0(0.4)($511,203)3,0431,4632056280Target Increased Number of Grad Students = 6

49. Existing departments Operating on the Efficiency Frontier in Cluster 1Programs with Instructional Expenses from $ 8 to 9 Million in blue

50. ReferencesChatman, S. (2017). Constructing a Peer Institution: A New Peer Methodology. AIR Professional File, Summer 2017(55).Middaugh, M.F., Graham, R., & Shahid, A. (2003). A Study of Higher Education Instructional Expenditures: The Delaware Study of Instructional Costs and Productivity (NCES 2003-161). Washington, DC: U.S. Department of Education.