Community for Advancing Discovery Research in
Description: Community for Advancing Discovery Research in Education (CADRE) A Review of Research Instruments for STEM Teacher Practices, PCK, and Content Knowledge Presented by Daphne Minner Alina Martinez Abt Associates June 15, 2012 DR-K12 Principal
Related Topics
Download Presentation
"Community for Advancing Discovery Research in" is the property of its rightful owner. Permission is granted to download and print the materials on this website for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Presentation Transcript
slide1. Community for Advancing Discovery Research in Education (CADRE) A Review of Research Instruments for STEM Teacher Practices, PCK, and Content Knowledge Presented by
Daphne Minner
Alina Martinez
Abt Associates
June 15, 2012
DR-K12 Principal Investigator Annual Meeting
Arlington, VA<br>
slide2. DR K-12 Portfolio Overview June 15, 2012<br>
slide3. Data from 248 Projects Includes projects that submitted at least a proposal
Sources:
Provided by PIs voluntarily in response to requests
Found in the public domain June 15, 2012<br>
slide4. Project Materials were Reviewed and Coded June 15, 2012<br>
slide5. RFP Guides Innovation in STEM Education June 15, 2012<br>
slide6. Disciplines Addressed in Projects June 15, 2012<br>
slide7. Target Segment of the Education Population June 15, 2012<br>
slide8. Outcomes Studied June 15, 2012<br>
slide9. Types of Student and Teacher Outcomes Studied Students Teachers June 15, 2012 Achievement<br>
slide10. Teacher Instrument Study Overview Rationale
Methods June 15, 2012<br>
slide11. 75 projects assessed teacher outcomes with existing instruments June 15, 2012 Table A Table B<br>
slide12. 82 unique instruments identified 42 used to assess Practices
Reformed Teaching Observation Protocol (modified)—15 studies
Inside the Classroom Observation and Analytic Protocol—8
Surveys of Enacted Curriculum (modified)—5
24 used to assess Content
Views of Nature of Science (Form C)—3 studies
National Assessment of Educational Progress (modified)—3
Praxis content tests/ Earth & physical science (modified)—2
TIMSS content tests (modified)—2 June 15, 2012<br>
slide13. 82 instruments identified 22 used to assess Pedagogical Content Knowledge
Mathematical Knowledge for Teaching—14 studies
Knowledge of Algebra for Teaching (modified)—2
Science Teaching Efficacy Belief Instrument—2
Reformed Teaching Observation Protocol (modified)—2 June 15, 2012<br>
slide14. More Refined Look at Constructs Instruction practices
Content knowledge
Instructional beliefs; teaching self-efficacy
Class management
Assessment practices/ skills
Classroom social context
Classroom physical context
Administrative or policy context
Student / teacher demographics
Instructional planning June 15, 2012<br>
slide15. Instructional Practices June 15, 2012<br>
slide16. Instructional Practices plus June 15, 2012<br>
slide17. Teachers’ Content Knowledge June 15, 2012<br>
slide18. Teachers’ Content Knowledge June 15, 2012<br>
slide19. Instructional Beliefs June 15, 2012<br>
slide20. Multidimensional June 15, 2012<br>
slide21. System-wide Reform Focused June 15, 2012<br>
slide22. June 15, 2012<br>
slide23. Dialogue Focused June 15, 2012<br>
slide24. Reliability Information June 15, 2012 Reliability type
missing information (19, 38%)
internal stability alpha (11, 22%) (6 acceptable, 5 good)
interrater, % agreement (7, 14%) (2 low, 2 accept., 3 good)
interrater, Kappa (2, 4%) (acceptable)
Interrater, Spearman rank-order correlation (1, good)
Other (2, good and N/A)
More than one type (8, 16%) (all acceptable to good)
Level
missing information
<.59 or equivalent: low
.6-.79 or equivalent: acceptable
> .8 or equivalent: good<br>
slide25. Validity Evidence June 15, 2012 Validity Type
Missing information (28, 54%)
Content/ face (11, 21%)
Criterion/ predictive (9, 17%)
Construct (8, 15%)
Concurrent/ convergent (5, 10%)
Discriminant /divergent (2, 4%)<br>
slide26. Trends Instructional practices (n=22)
11 instruments predominantly observation
11 instruments with 1-2 other constructs, observation supplemented with survey methods
Teachers’ content knowledge (n=25)
primarily assessed domain-specific conceptual or procedural knowledge by multiple choice test
majority of these instruments being used for teachers were originally designed for students
8 science, 3 math, 1 technology-specific tests June 15, 2012<br>
slide27. Trends cont. Instructional beliefs (n=9)
Predominated by survey methods
6/9 were science specific with only 2 for math
Multidimensional (n=23)
System-wide reform (predominantly surveys) (10)
Dialogue focused observation instruments to capture classroom climate and social aspects (13)
Making reliability and validity evidence more readily available with information about instruments is essential June 15, 2012<br>
slide28. Discussion Surprises?
Gaps?
Strategies?
Resources? June 15, 2012<br>
slide29. Contact Information Abt Associates June 15, 2012<br>
Daphne Minner
Alina Martinez
Abt Associates
June 15, 2012
DR-K12 Principal Investigator Annual Meeting
Arlington, VA<br>
slide2. DR K-12 Portfolio Overview June 15, 2012<br>
slide3. Data from 248 Projects Includes projects that submitted at least a proposal
Sources:
Provided by PIs voluntarily in response to requests
Found in the public domain June 15, 2012<br>
slide4. Project Materials were Reviewed and Coded June 15, 2012<br>
slide5. RFP Guides Innovation in STEM Education June 15, 2012<br>
slide6. Disciplines Addressed in Projects June 15, 2012<br>
slide7. Target Segment of the Education Population June 15, 2012<br>
slide8. Outcomes Studied June 15, 2012<br>
slide9. Types of Student and Teacher Outcomes Studied Students Teachers June 15, 2012 Achievement<br>
slide10. Teacher Instrument Study Overview Rationale
Methods June 15, 2012<br>
slide11. 75 projects assessed teacher outcomes with existing instruments June 15, 2012 Table A Table B<br>
slide12. 82 unique instruments identified 42 used to assess Practices
Reformed Teaching Observation Protocol (modified)—15 studies
Inside the Classroom Observation and Analytic Protocol—8
Surveys of Enacted Curriculum (modified)—5
24 used to assess Content
Views of Nature of Science (Form C)—3 studies
National Assessment of Educational Progress (modified)—3
Praxis content tests/ Earth & physical science (modified)—2
TIMSS content tests (modified)—2 June 15, 2012<br>
slide13. 82 instruments identified 22 used to assess Pedagogical Content Knowledge
Mathematical Knowledge for Teaching—14 studies
Knowledge of Algebra for Teaching (modified)—2
Science Teaching Efficacy Belief Instrument—2
Reformed Teaching Observation Protocol (modified)—2 June 15, 2012<br>
slide14. More Refined Look at Constructs Instruction practices
Content knowledge
Instructional beliefs; teaching self-efficacy
Class management
Assessment practices/ skills
Classroom social context
Classroom physical context
Administrative or policy context
Student / teacher demographics
Instructional planning June 15, 2012<br>
slide15. Instructional Practices June 15, 2012<br>
slide16. Instructional Practices plus June 15, 2012<br>
slide17. Teachers’ Content Knowledge June 15, 2012<br>
slide18. Teachers’ Content Knowledge June 15, 2012<br>
slide19. Instructional Beliefs June 15, 2012<br>
slide20. Multidimensional June 15, 2012<br>
slide21. System-wide Reform Focused June 15, 2012<br>
slide22. June 15, 2012<br>
slide23. Dialogue Focused June 15, 2012<br>
slide24. Reliability Information June 15, 2012 Reliability type
missing information (19, 38%)
internal stability alpha (11, 22%) (6 acceptable, 5 good)
interrater, % agreement (7, 14%) (2 low, 2 accept., 3 good)
interrater, Kappa (2, 4%) (acceptable)
Interrater, Spearman rank-order correlation (1, good)
Other (2, good and N/A)
More than one type (8, 16%) (all acceptable to good)
Level
missing information
<.59 or equivalent: low
.6-.79 or equivalent: acceptable
> .8 or equivalent: good<br>
slide25. Validity Evidence June 15, 2012 Validity Type
Missing information (28, 54%)
Content/ face (11, 21%)
Criterion/ predictive (9, 17%)
Construct (8, 15%)
Concurrent/ convergent (5, 10%)
Discriminant /divergent (2, 4%)<br>
slide26. Trends Instructional practices (n=22)
11 instruments predominantly observation
11 instruments with 1-2 other constructs, observation supplemented with survey methods
Teachers’ content knowledge (n=25)
primarily assessed domain-specific conceptual or procedural knowledge by multiple choice test
majority of these instruments being used for teachers were originally designed for students
8 science, 3 math, 1 technology-specific tests June 15, 2012<br>
slide27. Trends cont. Instructional beliefs (n=9)
Predominated by survey methods
6/9 were science specific with only 2 for math
Multidimensional (n=23)
System-wide reform (predominantly surveys) (10)
Dialogue focused observation instruments to capture classroom climate and social aspects (13)
Making reliability and validity evidence more readily available with information about instruments is essential June 15, 2012<br>
slide28. Discussion Surprises?
Gaps?
Strategies?
Resources? June 15, 2012<br>
slide29. Contact Information Abt Associates June 15, 2012<br>