Research Analysis of Faculty Peer Review Comments

slide1. Research Analysis of Faculty Peer Review CommentsQM Conference 2014 Dr. Andria Schwegler
Dr. Barb Altman
Texas A&M University – Central Texas<br>
slide2. Presentation Agenda Research Overview and Background
TAMUCT Internal Peer Review Process
Research Design
Coding Scheme and Demonstration
Summary Data of Our Reviewer Comments
Thematic Analysis of Our Reviewer Comments
Potential Solutions to Issues Found – External Reviews?
Potential Solutions to Issues Found – What does Theory Tell Us?
Implications – Revisions to Peer Review Process and Training
Future Research Schwegler & Altman 2014
QM Conference Presentation 2<br>
slide3. 1. 1. Research Overview & Background Goal: To empirically examine the content of the peer review comments elicited in our first round of internal Quality Matters peer reviews at Texas A&M University Central Texas.
Purpose: To better understand what it was about the peer review comments that led to the most effective improvements in reviewed courses.
Comments were evaluated with three frames in mind:
QM standards of peer review comments that are constructive, specific,
measurable, and balanced.
2. Qualitative thematic analysis of overall comments.
3. Theoretical framework of literature on effective feedback for learning
from the cognitive psychology literature. Schwegler & Altman 2014
QM Conference Presentation 3<br>
slide4. 2. 2. TAMUCT Peer Review Process Details on our process and a related research project on our faculty perceptions regarding participating in peer review can be found here:
Schwegler, A. F., Altman, B. W., & Bunkowski, L. M. (2013). Beliefs regarding faculty participation in peer reviews of online courses. Internet Learning, 3, 95-116.
http://www.ipsonet.org/images/Westphalia_Press/Internet_Learning_Journal_2-2/3-1/8.%20Schwegler%20ILJ%203-1.pdf Schwegler & Altman 2014
QM Conference Presentation 4<br>
slide5. 2. 2. TAMUCT Peer Review Process An internal peer review process based on the Quality Matters (QM) framework was initiated over a 1.5 year period ending in Fall 2013.
We had to incentivize process to overcome faculty resistance.
We had to keep all reviewers internal to Colleges.
Though faculty were trained on QM, we had to reinforce differences between design and delivery.
Faculty completed a self-review prior to full review commencing; College Online Coordinator provided input that was voluntary to correct. Schwegler & Altman 2014
QM Conference Presentation 5<br>
slide6. 2. 2. TAMUCT Peer Review Process Across 4 semesters:
37 courses were peer reviewed
19 different faculty course developers
26 additional faculty members did not offer a course for peer review, but did volunteer to serve as peer reviewers Schwegler & Altman 2014
QM Conference Presentation 6<br>
slide7. 2. 2. TAMUCT Peer Review Process Impetus for research:
Distance Learning Advisory Committee meeting/debrief raised anecdotal concerns about:
Quality of comments and associated course improvements.
Lack of learning objectives.
Hostile exchanges between faculty related to peer reviews and faculty roles.
Collusion between faculty members who were “buddies” to go “easy” on each other.
Subcommittee was formed to evaluate process and whether improvements justified costs associated with program. Schwegler & Altman 2014
QM Conference Presentation 7<br>
slide8. 3. Research Design Study followed protocols for “exempt” archival research.
QM recommendations for writing effective comments was used as primary analysis tool.
Associated research questions:
Do the peer reviewers’ recommendations meet QM standards of being constructive, specific, measurable, sensitive and balanced?
Do the recommendations cite content or information from the course reviewed?
Is specific information from the QM standards quoted or referenced?
Are there identifiable trends in comments (i.e., strengths or weaknesses) that might inform future trainings? Schwegler & Altman 2014
QM Conference Presentation 8<br>
slide9. 3. Research Design Data set from Quality Matters of all peer review comments was de-identified
Coding scheme devised using QM guidelines for effective Peer Review comments:
Content from the QM Standards or Annotations
Content from the course under review
Recommendation that is constructive
Recommendation that is specific
Recommendation that is measurable
Recommendation that is sensitive
Recommendation that is balanced Schwegler & Altman 2014
QM Conference Presentation 9<br>
slide10. 3. Research Design – Coding Scheme Schwegler & Altman 2014
QM Conference Presentation 10 Coding Rules for Evaluating Comments
Standard/ Annotation Evidence
. = missing data, no comment is provided
0 = comment with no mention of Standard/Annotation
1 = comment with example, phrasing or reference to Standard/Annotation
Course Evidence
. = missing data, no comment is provided
0 = comment with no mention of course content
1 = comment with at least one element from course
Constructive Recommendation
. = missing data, no comment is provided
0 = comment provides no recommendation
1 = comment provides at least one potential recommendation/suggestion or
includes a thought provoking question<br>
slide11. 3. Research Design – Coding Scheme Schwegler & Altman 2014
QM Conference Presentation 11 Coding Rules for Evaluating Comments (cont.)
Specific Recommendation
. = missing data, no comment is provided
0 = comment provides only vague or general statements
1 = comment provides at least one example of the recommendation
Measurable Recommendation
. = missing data, no comment is provided
0 = comment with no recommendation
1 = comment with recommendation for an observable suggestion to implement
Sensitive Comment***
. = missing data, no comment is provided
1 = comment includes more negatively-valenced than positively-valenced words
2 = comment includes only neutral words, no emotion provoking words
3 = comment includes more positively-valenced than negatively-valenced words<br>
slide12. 3. Research Design – Coding Scheme Schwegler & Altman 2014
QM Conference Presentation 12 Coding Rules for Evaluating Comments (cont.)
Balanced Comment***
. = missing data, no comment is provided
0 = comment points out only weakness/revision
1 = comment points out only strengths/assets
2 = comment points out at least one strength/asset and weakness/revision
Grammatically Correct Sentence
. = missing data, no comment is provided
0 = comment is not a complete sentence
1 = comment is grammatically correct (includes you understood; ignore punctuation)<br>
slide13. 4. Coding Scheme and Demonstration Review Comment Coding Rules Handout
Connection for posting your Coding decision Schwegler & Altman 2014
QM Conference Presentation 13<br>
slide14. 4. Coding Demonstration – Sample 1 Schwegler & Altman 2014
QM Conference Presentation 14 Course 4, Standard 4.5 “Very few instructor explanations or secondary readings.” (Not met)
2011- 2013 Standard 4.5: The instructional materials present a variety of perspectives on the course content. (1 point)

Audience Results<br>
slide15. 4. Coding Demonstration – Sample 2 Schwegler & Altman 2014
QM Conference Presentation 15 Course7, Standard 1.3 “A line about email should be included. There is a nice statement about writing for an academic audience, but it doesn’t apply to the online discussions or email.” (Not met)
2011 – 2013 Standard 1.3: Etiquette expectations (sometimes called “netiquette”) for online discussions, email, and other forms of communication are stated clearly. (2 points)

Audience Results<br>
slide16. 4. Coding Demonstration – Sample 3 Schwegler & Altman 2014
QM Conference Presentation 16 Course 11, Standard 3.5 “Might want to consider doing some review quizzes.” (Met)
2011-2013 Standard 3.5 Students have multiple opportunities to measure their own learning progress.

Audience Results<br>
slide17. 4. Coding Demonstration – Sample 4 Schwegler & Altman 2014
QM Conference Presentation 17 Course 23, Standard 2.1 “Suggest the knowledge objective (improve) be revised and made more measureable.” (Met)
2011- 2013 Standard 2.1 The course learning objectives describe outcomes that are measurable. (3 points)

Audience Results<br>
slide18. 4. Coding Demonstration – Sample 5 Schwegler & Altman 2014
QM Conference Presentation 18 Course 29, Standard 8.2 “Not much auditory.” (Not met)
2011- 2013 Standard 8.2 The course contains equivalent alternatives to auditory and visual content. (2 points)

Audience Results<br>
slide19. 4. Coding Demonstration – Wrap-Up Schwegler & Altman 2014
QM Conference Presentation 19 And you think those were challenging, try this one!!
Course 26, Standard 3.4 “I’m not sure how the sequencing work (sic) in this course. But I’m not sure it is needed either.” (Not met)
2011-2013 Standard 3.4 The assessment instruments selected are sequenced, varied, and appropriate to the student work being assessed.<br>
slide20. 5. Summary Data 3034 – Total Number of Reviewer Comments Possible
925 – Peer Review Comments Provided (30.49%)
2109 – Missed Opportunities to Comment (69.51%) Schwegler & Altman 2014
QM Conference Presentation 20<br>
slide21. Missing data was distributed throughout courses, except
Course 15 – no comments at all
Course 35 – one comment total
Course 11 and Course 30 – two comments total
Course 34 – three comments total
Course 24 – five comments total
Course 14 – six comments total Schwegler & Altman
2014 QM Conference Presentation 21 5. Summary Data<br>
slide22. Schwegler & Altman
2014 QM Conference Presentation 22 5. Summary Data Missing data was distributed throughout reviewers, except
Reviewers who provided no comments: 7 courses
Reviewers who provided one comment: 4 courses
Reviewers who provided two comments: 6 courses
Reviewers who provided three comments: 1 course
Some reviewers provided extensive comments
Reviewer provided 41/41 comments for Course 20
Reviewer provided 40/41 comments for Courses 2, 8, 31<br>
slide23. Schwegler & Altman
2014 QM Conference Presentation 23 5. Summary Data by Code<br>
slide24. 5. Data-Based Answers to Research Questions Schwegler & Altman
2014 QM Conference Presentation 24 Do the peer reviewers’ recommendations meet QM standards of being constructive, specific, measurable, sensitive and balanced?
Generally, no.
Do the recommendations cite content or information from the course reviewed?
Generally, yes.
Is specific information from the QM standards quoted or referenced?
Generally, yes.
Are there identifiable trends in comments (i.e., strengths or weaknesses) that might inform future trainings?
OH YES!<br>
slide25. 6. Thematic Analysis of Comments: Strengths and Weaknesses Faculty members willing to give each other “benefit of the doubt” even though does not follow process
Situations where evidence was not found yet course determined to meet standard
Course 28, Standard 1.5, Reviewer 2 “I could not find a place where prerequisites are stated on the syllabus or in Blackboard. Dr. X does address a number of kinds of required competencies, so I have no problem saying this standard is met. It might be good to add course prerequisites, though.” (Assigned “Met”)
Course 9, Standard 1.3, Reviewer 3 “I think when looking at the course, it just depends on the perspective of the reviewer. Dr. X doesn’t from what I can see state “etiquette” rules but within his instructions, he tells the student what the expects.” (Assigned “Met”) Schwegler & Altman 2014
QM Conference Presentation 25<br>
slide26. 6. Thematic Analysis of Comments:  Strengths and Weaknesses Lack of module level objectives
Either not found or professor deliberately chooses not to include them
Course 6, Standard 2.2, Reviewer 1 “Dr. X has made a good argument for the lack of module learning objectives being explicitly stated.” (Assigned “Met”) AND Reviewer 3 “I mark this met even though technically it’s not accurate. I accept and believe in Dr. X’s assertions about modular objectives in this level of course. They can inhibit the imagining of creative alternatives.” (Assigned “Met”)
Course 27, Standard 2.2, Reviewer 1 “ There are no module/unit learning objectives. All assignments are geared toward accomplishment of the overall course objectives. One could argue that there are 38 units (Dr. X calls these “blocks” in the running notes section), and that perhaps each unit should have specific learning objectives attached. Some of these blocks do ask the students to do things (e.g., reflect, describe, practice), so these could be considered learning objectives that are not clearly stated as such. Perhaps providing a simple learning objectives statement for each block would solve this problem? “(Assigned “Met”) Schwegler & Altman 2014
QM Conference Presentation 26<br>
slide27. Confusion in Distinguishing between Design and Delivery
Reviewers frame comments related to the specific professor as opposed to the course
Course 6, Standard 4.5, Reviewer 3 “One of Dr. X’s strengths as a teacher is that he values quality of argument over rigid agreement with his own point of view – something I consider essential in academia.”
Course 20, Standard 1.1, Reviewer 1 “I am impressed with the clarity of what Dr. X does to meet this standard.” Standard 2.2 “I am impressed with the detail Dr. X uses for his discussion items and for his writing assignments.” Standard 3.1 “I am impressed with Dr. X’s use of multiple choice quizzes as supplements to his heavy writing requirement.” Standard 4.2 “I am particularly impressed with Dr. X’s use of oral reinforcement of written materials.” Standard 5.2 “I particularly like Dr. X’s requirement that students respond to each other’s discussion posts.” Schwegler & Altman 2014
QM Conference Presentation 27 6. Thematic Analysis of Comments:  Strengths and Weaknesses<br>
slide28. Not Taking Student’s Perspective
Comments related to media access demonstrate where reviewer “took the blame” rather than advocating for students; final comment does have a recommendation
Course 19, Standard 6.4, Reviewer 2 “The link to the films would not work for me when I tried but there may be a problem on my end!” (Assigned “Met”)
Course 27, Standard 1.1 Reviewer 1 “I like Dr. X’s introduction and course overview – these help set the tone for the class. Both audio and readable formats are provided – this is good. I could not pull up the .mp3 file, but I think my computer is just being slow.” (Assigned “Met”)
Course 36, Standard 6.4, Reviewer 3 “The mp3 files would not open on my Mac. It could be that the files are large, and I did not wait long enough. If that is the case (i.e., large files create longer downloads), then stating as much might reduce student frustration.” (Assigned “Met”) Schwegler & Altman 2014
QM Conference Presentation 28 6. Thematic Analysis of Comments:  Strengths and Weaknesses<br>
slide29. 6. Thematic Analysis of Comments:  Strengths and Weaknesses - Hints Issues/concerns that arose but not to level of full theme
Confusion on copyright and fair use
Universal design standards (unclear how to evaluate for Standard 8)
Institutional policies on standards for course design (particularly as relates to the appropriate level objectives and assignments in a undergraduate vs. graduate classes) Schwegler & Altman 2014
QM Conference Presentation 29<br>
slide30. 6. Thematic Analysis of Comments: Strengths and Weaknesses Strengths – some faculty took the process very seriously and wrote very effective comments!
Course 2, Standard 2.2, Reviewer 3 “As stated in the annotations for module/unit learning objectives, ‘…these objectives are prominently stated in the corresponding module or unit so that they are accessible to the student from within the online classroom.’ The module level objectives are stated on the Course Information page, on the syllabus, and on the learning module tab, but they are not nested within each learning module to provide students with focused directions as they work on specific content. For example, I see for Module 1 that I am to “conduct and apply the statistical foundations…” but I’m not sure exactly what I should be getting out of the Lecture 5 link specifically. The addition of objectives for each lesson in the Learning Module would make the targeted objectives clear for each set of content. I noticed that for objectives 2.151 and on (on the Course Information page; the order is different on the syllabus) you used action verbs to describe what students would do with the content (e.g., calculate confidence intervals…, conduct hypothesis testing…). This level of specificity is helpful. Add such specificity to all objectives. For example, when listing “qualitative” and “measures of central tendency,” it is not clear what students need to do with the information (perhaps “define and recognize” and “calculate,” respectively?).” Schwegler & Altman 2014
QM Conference Presentation 30<br>
slide31. 6. Thematic Analysis of Comments: Strengths and Weaknesses Another example of an exemplar comment:
Course 31, Standard 2.2, Reviewer 2 “Stating the module level objectives on the weekly modules is helpful for students. On the surface, the module level objectives are well-stated. They describe specific outcomes regarding content and lead with behavioral verbs. However, the module-level objectives need revision regarding alignment as explained in my comments for Standard 3.1. (Given the discrepancy of wording for module level objectives between the Course Content links and the syllabus, have the objectives stated on the syllabus already been revised as I recommend below?) In addition, being naive to course content, it is not clear how each set of module level objectives maps onto the broader course level objective. Numbering the course level objectives and adding the relevant number to the end of each module level objective would help students understand the connection between the module level and course level objectives.” Schwegler & Altman 2014
QM Conference Presentation 31<br>
slide32. 7. Potential Solutions – External Reviews? Internal vs. External Peer Reviews: Would “Official” QM Reviews Address Found Issues?
In an official review, the Master Reviewer as Chair would provide more direction and peer coaching for newer peer reviewers not skilled in writing comments or that do not fully understand design/delivery distinction.
In an official review, the review would not proceed without module level objectives.
In an official review, other reviewers would not know and have a relationship with course developer and perhaps better maintain objectivity (e.g. no granting "benefit of the doubt," decisions would strictly be made on course content). Schwegler & Altman 2014
QM Conference Presentation 32<br>
slide33. 7. Potential Solutions – External Reviews? But
Official reviews would not provide a feedback loop to connect us with the needs of our faculty.
Internal reviews help us identify the training needs of our faculty.
Internal reviews help our faculty practice distinguishing between design and delivery.
Internal reviews provide feedback on the peer review process that comes from performing the task itself instead of coming from other people (i.e., task-generated feedback). Feedback implicit in the task is an important source of feedback that has implications for learning – in addition to feedback provided by others. Schwegler & Altman 2014
QM Conference Presentation 33<br>
slide34. 8. Potential Solutions: What does Theory  Tell Us? Acknowledging Our Goals for Peer Review
Improvement of online course offerings
Faculty buy-in of process
Distinguishing Our Goals during the Review Process
Attempting to achieve both goals may have hampered our efforts to obtain them
Uncoupling our goals may help us better achieve them in the future
Integrating Lessons from Feedback Intervention Theory Schwegler & Altman 2014
QM Conference Presentation 34<br>
slide35. Kluger &DeNisi (1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119(2), 254-284.
“We argue that a considerable body of evidence suggesting that feedback intervention (FI) effects on performance are quite variable has been historically disregarded by most FI researchers. This disregard has led to a widely shared assumption that FIs consistently improve performance” (p. 254). Schwegler & Altman
2014 QM Conference Presentation 35 8. Potential Solutions: Feedback Intervention Theory<br>
slide36. Feedback Interventions (FI) are “actions taken by external agents to provide information regarding some aspect of one’s task performance” (Kluger & DeNisi, 1996, p. 255).
Though the theory is most readily applied to teachers providing feedback to students on coursework, the definition is consistent with QM peer reviewer comments to course developers. Schwegler & Altman
2014 QM Conference Presentation 36 8. Potential Solutions: Feedback Intervention Theory<br>
slide37. Schwegler & Altman
2014 QM Conference Presentation 37 8. Potential Solutions: Feedback Intervention Theory FI Cues Self Focal Task Affect Performance Non-Focal Tasks Learning Motivation Cognitive Resources<br>
slide38. Schwegler & Altman
2014 QM Conference Presentation 38 Self Self Goals
Self-esteem,
Control,
Impression management Is focal task important for self goals? Reduce Performance Performance
Interference Performance
Enhancement Is task easy? YES NO 8. Integrating Theory with Data<br>
slide39. Comments targeting self goals that may have hampered performance:
Course 5, Standard 2.4, Reviewer 1 “You do a terrific job of connecting specific assignments to particular learning objectives. Good work!”
Course 29, Standard 6.3, Reviewer 3 “One of the better courses I have seen!”
Course 28, Standard 2.5, Reviewer 3 “Dr. X does a terrific job keeping the objectives in front of her students on every level of the course.” Schwegler & Altman
2014 QM Conference Presentation 39 8. Integrating Theory with Data<br>
slide40. Schwegler & Altman
2014 QM Conference Presentation 40 Focal
Task Feedback-Standard Discrepancy Attain other self goals? Effort reduce discrepancy? Positive Negative Reduce Effort Increase Effort YES NO None Maintain Effort Maintain or Increase Effort YES NO Learning or
Self 8. Integrating Theory with Data<br>
slide41. Schwegler & Altman
2014 QM Conference Presentation 41 8. Integrating Theory with Data<br>
slide42. Schwegler & Altman
2014 QM Conference Presentation 42 8. Integrating Theory with Data Task
Details Is the task well known? Temporary
Performance
Interference YES Interruption NO Generate
and test hypotheses Hypotheses match reality? Positive
Learning No or Negative
Learning Quit Hypothesescorrect? YES NO<br>
slide43. Schwegler & Altman
2014 QM Conference Presentation 43 8. Integrating Theory with Data Feedback that is constructive, specific, and measurable should direct/maintain attention at task detail level.
Comments that help course developers test hypotheses:
Course 14, Standard 1.8, Reviewer 1 “I do see that Dr. X has students introduce themselves through course content and I can accept this as a legitimate introduction... I share Dr. X’s worry about the redundancy of student introductions in my classes. I reluctantly incorporated introductions in my classes two or three semesters ago, but did so as a first attempt for students to think through the over-arching questions of the class. I was pleased with how this worked. Students began selling the online experience and the benefit of taking “my” classes... There might be a way to let students “introduce” themselves to others and you… This is what I have tried to do in a sly way.” (Assigned “Not Met”<br>
slide44. Schwegler & Altman
2014 QM Conference Presentation 44 Comments that refute hypotheses for the course developer are likely to be ineffective.
Comments that do NOT help course developers test hypotheses:
Course 5, Standard 8.2, Reviewer 3 “This is something that, as a university, we need to address…it is also something that faculty should not be required to be responsible for. That being said, I did not see any mention of alternative formats for content being available.” (Assigned “Not Met”)
Course 9, Standard 1.8, Reviewer 1 “I do not see a place where students are asked to introduce themselves. I suspect they ‘introduce’ themselves, though, in the ways that are most meaningful (through discussion). I can’t remember ever introducing myself in a graduate class.” (Not met
Course 24, Standard 1.8, Reviewer 1 “Dr. X, I know that we have talked about this already but you may want to include something to satisfy this requirement for the sake of QM but this is up to you of course.” (Assigned “Not Met”) 8. Integrating Theory with Data<br>
slide45. Comments focused on task details that help course developer test hypotheses are most likely to promote positive learning.
Empirical question: Are these types of comments equally effective for positive and negative feedback-standard discrepancies?
QM is moving toward recommending (and requiring) comments for all standards regardless of “Met” or “Not Met” verdict. How helpful is this?
Positive feedback to course developer for participation should be separated from constructive comments directed to course improvement.
Empirical question: Are the sensitive and balanced recommendations for writing effective comments distracting? Do these comments need to be shifted from asynchronous to synchronous contact only? Schwegler & Altman 2014
QM Conference Presentation 45 8. Theoretical Implications<br>
slide46. 9. Practical Implications - Revisions to Peer Review Process So, choice between internal vs. external reviews going forward is not an easy one
Internal reviews are still more palatable to the majority of faculty, although this differs by College
One College will be “experimenting” with external reviews this coming year
Primary discussion becomes how to proceed with a revised internal review process that would have better safeguards for improved comments and, therefore, course improvements Schwegler & Altman 2014
QM Conference Presentation 46<br>
slide47. 8. Practical Implications – Revisions to Faculty Training Training Going Forward
The APPQMR these faculty took was prior to changes that “beefed up” sections on writing effective peer review comments.
Provide a refresher training with writing peer review comments emphasized with lots of examples
Training needs to provide a much better distinction between design and delivery.
New Course Development Program where faculty work more directly with an Instructional Designer has been adopted and clearly shows distinction
Training needs to emphasize importance of module level objectives and taking the student perspective.
New Course Development Program requires module level objectives and includes questions regarding student perspective Schwegler & Altman 2014
QM Conference Presentation 47<br>
slide48. 8. Practical Implications – Training Training going forward (cont.)
Training needs to be offered on
Universal design
Copyright and fair use Schwegler & Altman 2014
QM Conference Presentation 48<br>
slide49. 9. Future Research Peer Review Coding
Comparative study of other institutions doing Internal Reviews
Would highlight whether issues we found are unique to TAMUCT or more widespread
Study of QM Official Reviews
Would highlight whether these same issues or different ones exist with official reviews
Would help QM further refine process and training
Comparison of training programs to determine which training ameliorates the issues identified
Tests of Feedback Intervention Theory in Peer Review Context Schwegler & Altman 2014
QM Conference Presentation 49<br>
slide50. Research Analysis of Faculty Peer Review CommentsQM Conference 2014 Dr. Andria Schwegler
Dr. Barb Altman
Texas A&M University – Central Texas

Questions?

Contact information:
schwegler@tamuct.edu

Thank you for attending!<br>