Results of some statistical analysis Normalization procedure 1 Get percentile P of a student on the basis of 5subject aggregate scores in board Determine the JEEMain aggregate marks corresponding to percentile P at the AllIndia level Regard this as the normalized board score of the st ID: 135414
Download Presentation The PPT/PDF document "Comparison of two Normalization Procedur..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Comparison of two Normalization Procedures
Results of some statistical analysisSlide2
Normalization procedure 1
Get percentile (
P
) of a student on the basis of 5-subject aggregate scores in board.
Determine the JEE-Main aggregate marks corresponding to percentile P at the All-India level. Regard this as the normalized board score of the student (
B
1
).
Get JEE-Main aggregate marks (
A
0
) of the student.
The composite marks used for drawing merit list is
A
1
= 0.6
A
0
+ 0.4
B
1
.Slide3
Normalization procedure 2
Get percentile (
P
) of a student on the basis of 5-subject aggregate scores in board.
Determine the JEE-Main aggregate marks corresponding to percentile
P
from the
set of aggregate marks obtained by different students of that board in JEE-Main
. Regard this as the normalized board score of the student (
B
2
).
Get JEE-Main aggregate marks (
A
0
) of the student.
The composite marks used for drawing merit list is
A
2
= 0.6
A
0
+ 0.4
B
2
.Slide4
Common assumptions
Common assumptions
aggregate scores of students in a board examination are in the order of their general merit within that board (scholastic achievement indicates ability).
aggregate scores of the students in JEE-Main are in the order of their merit in respect of engineering admissions (achievement indicates aptitude).
Assumptions may be violated if
a student has a poor exam,
A student has a special advantage.
If these effects are confounded with merit, violations become difficult to rectify.
No fairer assumption is possible.Slide5
Exclusive assumptions
Procedure 1
General merit distribution does not vary from one board to another.
Procedure 2
Merit distribution varies from board to board; this difference is captured by
differences in performance patterns of students of various boards in JEE-Main
.Slide6
Data for analysis
One data common data set:
Aggregate marks in AIEEE 2012 and the corresponding percentiles computed from the set of all students taking the examination.
Three sets of data in respect of six boards:
Aggregate marks in board examinations 2012 (expressed as percentage) and the corresponding percentiles for all students taking the examination.
Aggregate marks in AIEEE 2012 and the corresponding percentiles computed from the set of students from the chosen board.
Matched pair of AIEEE aggregate marks and board aggregate percentage.Slide7
Issue 1: Extent of ‘compensation’ for board percentile ‘disparity’Slide8
‘Compensation’ for ‘disparity’: some specific examples
Transformed score of a student at the 50th percentile of CBSE is more than that of
a student at the 80th percentile of Maharashtra Board,
a student at the 85th percentile of Jharkhand Board.
Transformed board scores of top five students of three boards:
CBSE: 346,
Maharashtra board : 331,
Jharkhand board : 274.
Amount of compensation (by Procedure 2) is LARGE!Slide9
What is being ‘compensated’?
Poor performance of students of some boards in AIEEE may be due to
Poor merit/ability (not a tenable explanation for large boards),
Non-alignment of board examination pattern with AIEEE (rank correlations indicate this),
Lack of instruction in English and Hindi (only available languages for AIEEE/JEE-Main),
Less access to coaching,
Load of an extra subject in board (for some boards).
All these effects are confounded.
If confounding factors are ignored, solution may be worse than the problem.Slide10
Issue 2: Performance pattern of different boards (distribution of students in different percentile ranges)
Current
procedure
Procedure 1
Procedure 2
Disparity among the distributions
is the least for Procedure 1.Slide11
Issue 3: The question of fairness
In 2012, five students (4 from CBSE, 1 from Maharashtra board) were tied at
AIEEE aggregate marks of 130 ,
board percentile of 93.1.
According to Procedure 1, they shared rank 20,352.
According to Procedure 2
CBSE students had shared rank 18,077,
Maharashtra board students had rank 34,175.
The differential treatment is entirely due to performance of other students of the two boards in AIEEE.
This goes against the principle of fairness.Slide12
Issue 4: Sampling bias
Procedure 2 requires the assumption
Board to board difference in merit distribution is captured by
differences in performance patterns of students of various boards in JEE-Main
.
Proportion of AIEEE candidates in percentile range (board aggregate)
Board
0-20
20-50
50-80
Above 80
Total
Assam
0%
0%
2%
98%
100%
CBSE
29%
15%
19%
37%
100%
Jharkhand
16%
5%
5%
74%
100%
Maharashtra11%8%11%69%100%Mizoram1%1%2%96%100%Uttarakhand14%3%4%79%100%
For this assumption to hold, sampling of AIEEE candidates from the boards should follow the same pattern.
However, 2012 data shows that AIEEE attracts different profiles of students from different boards.Slide13
In summary
Students of
different
boards
face different types/levels of difficulty
in AIEEE.
The presence of these factors confound any presumed board-to-board difference in merit distributions.
Board-to-board difference in performance patterns in AIEEE is vast.
Use of AIEEE marks for ‘tracking’ would produce drastic amount of compensation.
Such levels of unequal treatment of percentiles of different boards, based on presumptions, is unfair.
Procedure 2 also penalizes a student for poor performance of peers from same board.
For large boards, there is a strong reason to presume a common merit distribution.
Procedure 1 produces a more equitable pattern of performance of students of different boards.Slide14
Criteria that are NOT appropriate for choosing between procedures
Correlation of new and old ranks.
How much correlation is ideal?
The more the merrier?
Movement of students from one merit list to another.
How much movement is ideal?
The less the merrier?Slide15
A benchmark for rank correlations
Rank correlation between board and AIEEE marks
Board
Spearman
rank
correlation with AIEEE aggregate marks
Assam
0.4124
CBSE
0.6891
Jharkhand
0.1926
Maharashtra
0.4322
Mizoram
0.2919
Uttarakhand
0.3235
All matched samples
0.4841
Observations
No rank correlation is more than 0.7.
Board examinations are different from AIEEE (measure something else).
CBSE marks have highest rank correlation with AIEEE:
Similarity of examination pattern,
Same agency administers CBSE exams and AIEEE.Slide16
Correlation of new and old ranks
Board
Rank correlation between
A
0
and A
1
Rank correlation between
A
0
and A
2
Assam
0.7283
0.8347
CBSE
0.9464
0.9302
Jharkhand
0.5782
0.7163
Maharashtra
0.7691
0.8337
Mizoram
0.5936
0.7025
Uttarakhand
0.5996
0.7435All matched samples0.85590.9106Not much difference overall, and for CBSE.For some boards, Procedure 1 produces lower correlation .This is in line with observed low correlation between board and AIEEE marks.Why should large correlation be a good thing?Slide17
Movement from old to new rank list
Procedure 1 produces greater movement.
This is in line with observed disparity between board and AIEEE marks.Slide18
Some questions of interest to CBSE
What do the correlation patterns and rank movements indicate about the fate of CBSE students?
Will the CBSE students be at a disadvantage under Procedure 1?
Will they get a better deal under Procedure 2?
Will smaller boards gain unfairly from Procedure 1?
Is there a possibility of ‘board hopping’ if Procedure 1 is used?Slide19
Q1: Correlation patterns and rank movements
Correlation and movement patterns for the two procedures are SIMILAR.
Correlations indicate degree of association/alignment between new and old ranks.
Correlations do not reveal ANYTHING about the relative advantage or disadvantage to CBSE students.
If all CBSE students are pushed down 10,000 places in AIEEE merit list, their rank correlation with board scores would stay the same.
Scatter-plots of rank movement show that students exchange fate with one another . – no clue about advantage/disadvantage to CBSE students.Slide20
New simulations to answer Q2-Q5
Need simulated data for ALL the boards
Simulation conditions
10% Random samples chosen from AIEEE 2012 scores data.
Presumed correlation between AIEEE and board scores:
Observed correlations for AS, CBSE, JK, MR, MZ, UK
0.4 for other boards.
100 separate simulation runs.Slide21
Actual performance of CBSE students in AIEEE 2012
30.1% of all AIEEE candidates are CBSE students.
However, this groups constitutes
66 of the top 100 AIEEE rand holders,
608 of the top 1,000 AIEEE rand holders,
5,977 of the top 10,000 AIEEE rand holders,
54,430 of the top 100,000 AIEEE rand holders.Slide22
Share of boards in different percentile ranges of AIEEE 2012Slide23
How Procedure 1 or 2 would change the shares
Procedure 1 reduces marginally the dominance of CBSE students in merit list ,
Procedure 2 increases drastically that dominance.Slide24
…and the numbers behind the charts
Procedure 1: Share of boards in different percentile
ranges of AIEEE 2012
Board
0-10
10-20
20-30
30-40
40-50
50-60
60-70
70-80
80-90
90-100
CBSE
15%
18%
21%
24%
26%
29%
33%
38%
44%
53%
MR
17%
16%
14%
13%
12%11%
10%
8%
7%
5%
AP
7%
8%
8%
8%
8%
8%
9%
9%
10%
13%
BR
10%
9%
9%
8%
8%
7%
6%
6%
5%
3%
All other
51%
49%
48%
47%
46%
44%
42%
39%
35%
26%
Procedure 2: Share of boards in different percentile ranges of AIEEE 2012Board0-1010-2020-3030-4040-5050-6060-7070-8080-9090-100CBSE11%14%17%21%25%29%35%42%48%59%MR 20%18%16%14%12%11%9%7%5%4%AP 6%6%7%8%8%9%9%10%11%13%BR 11%10%10%9%8%7%6%5%4%2%All other53%51%50%49%47%44%41%36%31%22%
Current scenario: share of boards in different percentile
ranges
of AIEEE 2012
Board
0-10
10-20
20-30
30-40
40-50
50-60
60-70
70-80
80-90
90-100
Overall
CBSE
14%
17%
20%
22%
26%
28%
33%
39%
46%
54%
30%
MR
17%
17%
15%
14%
12%
11%
9%
8%
6%
5%
11%
AP
7%
8%
8%
8%
8%
8%
9%
9%
10%
13%
9%
BR
10%
9%
9%
8%
8%
7%
6%
5%
4%
3%
7%
All other
51%
49%
48%
48%
46%
45%
42%
38%
33%
25%
43%Slide25
Findings from simulation
Present scenario
Procedure 1
Procedure 2
Top 0.01%
65.1%
61.9%
82.7%
Top 0.1%
60.5%
60.4%
80.1%
Top 1%
60.0%
60.0%
69.6%
Top 10%
54.4%
53.4%
58.8%
Procedure 1 produces a marginal reduction in CBSE’s share at the top; 2% reduction at the top 0.01%.
Procedure 2 brings a great windfall to CBSE students.
CBSE students gain more and more from Procedure 2 as one goes up the merit list (share at top 1% and top 0.1% increases by 10% and 20%, respectively ).
If interest of top CBSE students had been the criterion, Procedure 2 would be a clear winner!Slide26
How does Procedure 2 dramatically increase
CBSE’s share of top-rankers?
It does so by practically turning the board score into another AIEEE score.
The composite score
A
2
for each board has same mean as
A
0
, but has much smaller spread.
Each set of board scores get more concentrated around their respective means.
In effect, every students gets a second chance to perform at the same mean level.
As a result, CBSE students gain substantially.Slide27
Why does Procedure 1 have only a mild effect on CBSE’s share of top-rankers?
For CBSE, the students who do well in AIEEE are generally those who do well in board exam.
They mostly hold on to their position in merit list.
For boards with less rank correlation with AIEEE, this is not the case.
Many AIEEE top-rankers from other boards move towards middle in terms of composite score.
If their board exams had been better aligned with AIEEE, they would have gained more from Procedure 1.Slide28
Answers to Q2 and Q3
Q1: Will the CBSE students be at a disadvantage under Procedure 1?
Yes, but marginally so.
Q2: Will they get a better deal under Procedure 2?
Yes, the top third of the merit list gets PACKED with CBSE students.
It is not just a better deal, it is a
great deal
.
It would not be fair to other boards!Slide29
Q4: Will smaller boards gain unfairly from Procedure 1?
A small board may not have same merit distribution as others. It may have
Stronger students, or
Weaker students.
To say that they gain more (from Procedure 1) than larger boards is to PRESUME that they have weaker students.
Percentiles generally correct for board size.
Here, percentile of a student is defined as percentage of candidates
strictly below
that student.
Toppers of a smaller board will always have a smaller percentile than the toppers of a larger board.
Students of smaller boards are also more vulnerable to variations of all kinds.Slide30
Q5: Will Procedure 1 encourage board hopping?
Yes, but TOWARDS CBSE (continuation of current trend)
CBSE’s loss and other boards’ gain is marginal.
This loss is only in relation to the
present scenario
; 2013 students cannot hop into 2012 system.
Even after this minor loss, CBSE students will have continued advantage over students of most other boards in terms of
alignment of examination pattern
alignment of language
access to coaching
a well administered examination system (less uncertainty)
If a CBSE student hops to a smaller board, good marks are not guaranteed (exam is likely to be of a different type)
Confounding advantages in AIEEE should not give them false confidence.Slide31
Composition of different percentile groups in 2012 by Procedure 1