Internet coaches for problem-solving in introductory - PowerPoint Presentation

Internet coaches for problem-solving in introductory physics: Usability, Usefulness and Design Qing (Xu) Ryan1, Ken Heller1, Leon Hsu1, Jia-Ling Lin1, Bijaya Aryal21.University of Minnesota–Twin Cities2.University of Minnesota–Rochester Supported by NSF DUE #0715615 and DUE-1226197. and by the University of Minnesota Presented by: Evan Frodermann1 WAPT Fall 2013 Meeting Eau Claire, Wisconsin

Shameless PlugNational AAPT 2014 Summer MeetingJuly 26-30, 2014University of Minnesota – Twin CitiesMinneapolis, MN More coaches (and possible workshop)

What are the computer coaches?Version 1 – Online Computer software which coach students in expert-like behavior in physics problem solving.3 different types Type 1: Computer coaches student for full problem.Type 2: Student coaches computer, computer gives feedbackType 3: The computer only coaches where students request coaching.Version 2 – In-development. This study focuses on version 1 data.

Design componentsSoftware Framework: “Intelligent” TutorsPedagogy Framework: Cognitive Apprenticeship Modeling CoachingFadingScaffoldingResearch Basis: Expert & Novices differences in problem solvingProblem solving framework:Competent Problem Solver – K & P Heller; University of Minnesota

Design Process CyclePrototypes AssessmentUsage & Usability Large-scale Implementation

Usage, Usefulness and Design StudiesWho will naturally use such coaches? subpopulation characteristicsHow do students use them? usage characteristicsDo the students perceive them as useful? Do the coaches improve student problem solving?How to use this information to guide future development of the next version?

Implementation tests35 coached problems 2 sections of calculus-based introductory mechanics course Spring 2013 (148 students/103 students).Homework with WebAssign.netVarious problem had available computer coaches Data collected includes:Pre/post-test scores (FCI/Math/CLASS)13 written problem solutions from 4 midterms (2 problems each) and a final exam (5 problems)Two surveys of student opinions regarding the coaches (mid & end semester)Keystrokes data monitoring students’ useNo specific credit for using the coaches

User Characteristics3 User populations (m:male; f:female) H group (heavy user): 80-100% ( nH=49 (20% of N), 65%m,35%f) M group (medium user) : 40-60% (nM=38 (15% of N), 55%m,45%f) L group (Light/Non user) : 0-20% (nL=72 (29% of N), 85%m,15%f) Entire class (N=251, 70%m,30%f)

Expectations of the class L group: Expects A grade, study lessM group: Expects A grade, study moreH group: Expects B grade, study moreTypical average grade in the course is B/B-

Pre-class FCI & Gender Total PreFCI: L: 58%3% M:49%4% H:41%3%The gender, pre class FCI scores, expected study time and expected grade will serve as the primary identifiers of these usage patterns.TestL (N=48)M (N=27) H (N=35)MaleFemaleMaleFemaleMaleFemale85%15% 67%33%66%34%FCI58%3% 59%11% 53%5%42%7%46%3%31%3%MATH 57%3%66%8% 53% 4% 61% 7% 54% 4% 45% 4% CLASS 62% 3% 55% 6% 66% 3% 66% 5% 65% 3% 56% 4%

Did the students improve?Final exam scoresHeavy users  Light users in final examHowever, the confidence and FCI score of the students started lower. Need to examine this closer.Final ExamMFTotalH (N=49)72.8  2.067.5  4.1 71.0  1.9M (N=38)64.8  3.568.3  4.466.4  2.7 L (N=72)69.2  2.770.9  6.2 69.5  2.5

Matching Historical DataMatched similar students to the Spring 2013 coach user class.Match on pre-class FCI, gender, expected grade, and expected study time.~85% student perfect match. 3145 students from Spring 2009 to Fall 2011. Allowed FCI difference of 2 for the remaining baseline.Normalize the exam scores and vary the selection process.Reduce unintended selection bias or specific class bias into the results.Retained the exam distributions.

Matched Final Exam ScoresFall  SpringSpring  FallSpring 2013Matched L(N=48)71.9%  1.4%71.6%  1.5%70.3%  3.0%Matched M(N=27)68.2%  1.9%69.3%  2.0%66.1%  3.1%Matched H(N=35)61.4%  1.6%62.3%  1.6% 69.9%  2.6%Only students with FCI scores were considered.Lower overall statistics in Spring 2013Increased baseline to compensate by matching 4 students to Spring 2013. Still a 85% “perfect match” with 4 to 1 match.

Motivation for future coach designSelected data from survey, End-term Spring 2013The computer coaches did not help improve my problem solving in this class. (135 responses)Selected data from survey, Midterm Spring 2013Free response: What do you like least about the computer coaches ?“Too repetitive” or “too long”: 49% of the 183 responses30 minutesFree response: What do you like most about the computer coaches?“Step by step” or “Guide beginning to end”: 23% of the responses.Both free response categories were the most frequent.Not ImproveNeitherImproveH (65) 21%12%67%M (27)11%15%74%L (43) 34%24%41% H: Heavy User (80-100%)M: Medium User (40-60%) L: Light user (0-20%)

Student FlexibilityTransparent student dictated solution paths.Students are free to finish any part of the “unlocked” coach at any point.More “human-like” coaching; This is accomplished due to a shift towards object-oriented programmingStudent adjusted grain sizePrevious version either forced students into completing steps they may not need or did not give full detailed guide even if they desired. Areas of the coaches are streamlined to reduce repetitionMore detail can be accessed if desired to retain the “step-by-step” nature of the coaches.

Instructor FlexibilityThe individual elements of the coach are easier to edit. No flash programming needed. Everything is edited in a graphical interface.Instructors can choose to add or remove forces, add objects, change colors, shapes, etc. to their diagrams based on their own structure. (ie, choose own pedagogy) Instructors edit pictures and diagrams within the GUI. Diagrams are constructed by dragging elements from the picture into the diagram box. Picture elements are added to the database through the interface menus.

SummaryA significant number of students use the coaches.Female, less-prepared students, and students with less-confidence tend to use the coaches.Different usage patterns.These identified usage patterns have indications that coaches may improve problem solving.Need more coached student data. FCI gains for both groups. (FCI gain for Spring 2013 done)Correlation between TA grades and Expert rubric assessment.Compare baseline with two different coach implementation.Version 2 is being developed which improves flexibility for instructor and student.

Backup SlidesVersion 1 data

FCI post and gain Heavy users absolute gain is higher.We expect to see specific gains in problem solving aspects which isn’t addressed with this diagnostic tool.,  

User approach

Forced rankingRank the components of the physics class in order from the most useful (10) to least useful (1) to your learning. Do not use any ties. Top 3: Lectures, homework, Computer coaches (High & Medium Users) ahead of textbook, labs, and problem-solving discussion sectionsLectures, Discussion, Labs, Computer coaches, Text book, Tutor room, Doing the homework, Clicker questions, The Competent Problem solver, Feedback from WebAssign (in the order that was given on the survey)

End-semester surveyA: Strongly agree B: Agree C: Neither D: Disagree E: Strongly disagreeA or BNeitherD or EH21%12% 67%M11%15%74%L34%24% 41%

End-semester surveyA: Strongly agree B: Agree C: Neither D: Disagree E: Strongly disagreeA or BNeitherD or EH70%12% 19%M63%33%4%L45%28% 28%

End-semester surveyA: Strongly agree B: Agree C: Neither D: Disagree E: Strongly disagreeA or BNeitherD or EH70%14% 16%M70%26%4%L47%28% 26%

Developing version 2Better address the needs of the H and M user population H_ encourage bypassing detailed coaching M_ adjust decision grain sizes

Backup Slides Version 2 talk from AAPT

Developing a Framework for Problem Solving Computer CoachesPresenter: Evan Frodermann University of Minnesota – Twin CitiesCollaborators: Qing (Xu) Ryan, Kristin Crouse, Ken Heller, Leon Hsu, Jia -ling Lin, Bijaya AryalAAPT Summer 2013 Meeting Portland, Oregon

From Version 1 (v1) to Version 2 (v2)What do we want to retain?Cognitive apprenticeship frameworkModeling, Coaching, Fading, ScaffoldingWhat do we want to improve?A more flexible framework in which to build coaches.Remove some of the technological limitations of the previous version. We used instructor and student feedback to improve the design and structure of the coaches.

Motivation from Student ResponsesSelected data from survey, End-term Spring 2013The computer coaches did not help improve my problem solving in this class. (135 responses)Selected data from survey, Midterm Spring 2013Free response: What do you like least about the computer coaches ?“Too repetitive” or “too long”: 49% of the 183 responses30 minutesFree response: What do you like most about the computer coaches?“Step by step” or “Guide beginning to end”: 23% of the responses.Both free response categories were the most frequent.Not ImproveNeitherImproveH (65) 21%12%67%M (27)11%15%74%L (43) 34%24%41% H: Heavy User (80-100%)M: Medium User (40-60%) L: Light user (0-20%)

Developing version 2Better address the needs of the Heavy and Medium user population (Qing’s Talk) H: encourage bypassing detailed coachingM: adjust decision grain sizes

Student FlexibilityTransparent student dictated solution paths.Students are free to finish any part of the “unlocked” coach at any point.More “human-like” coaching; This is accomplished due to a shift towards object-oriented programmingStudent adjusted grain sizePrevious version either forced students into completing steps they may not need or did not give full detailed guide even if they desired. Areas of the coaches are streamlined to reduce repetitionMore detail can be accessed if desired to retain the “step-by-step” nature of the coaches.

Grain-size with Quantity ModuleThe quantity module is a small example of this “grain size”.Students choose which ones to define. Only minimal set required.More quantities (specifically those associated with other solution paths) can be defined depending on what students want to define. Quantity categories are defined by instructor which include a list of required responses. Students choose which quantities to define with these minimal responses. Quantities required to solve the problem must be defined. Other quantities can also be defined.

Instructor FlexibilityThe individual elements of the coach are easier to edit. No flash programming needed. Everything is edited in a graphical interface.Instructors can choose to add or remove forces, add objects, change colors, shapes, etc. to their diagrams based on their own structure. Instructors edit pictures and diagrams within the GUI. Diagrams are constructed by dragging elements from the picture into the diagram box. Picture elements are added to the database through the interface menus.

Problem Solving PhilosophyPreviously, pedagogical changes/rewrites amounts to completely rewriting the software.The instructors can adapt the coach solving structure using the graphical interface to fit the needs of their own course. “Children” elements are only unlocked for the student with correct responses to questions. Users navigate to different “parent” elements at any time through this menu. “Primitive” elements are unlocked for the student after correct responses.

ConclusionsVersion 1 coaches had some limitations.Version 2 is far more versatile.Flexible for students with adjusting grain sizes.Transparent freedom of choice.More human-like coachingAdaptable to an instructor’s individual courses.Future work.Develop coaches for calculus based electromagnetism course. Explore overall usefulness for the identified populations,More in Qing’s talk about the ID’ed population.A new interface not using Flash (version 3 and beyond) for more general use on tablet computers.

ThanksSupported in part by National Science Foundation, DUE-1226197 and the University of MN For a demonstration of portions of the new version of the online coaches, visit the Wednesday poster session, poster number PST2C10.Qing’s poster on usage data is PST2C14

General Coach DesignOur coaches follow the Minnesota model for problem solving.Pictured is just the “focus” part of the process.Other instructors are not locked to this model and may adjust coaches how they see fit.

Program DesignVersion 1 required programming knowledge.Difficult editing or changing of existing coaches.New graphical user interface (GUI) designed to construct coaches.Backend data stored in SQL database and accessed through Java web server. (Free software)Clients access the java client through Adobe Flash. (Free software) Java Web server Apache TomcatMySQL database Server HostClients Instructor Adobe Flash Player and internet connection Students

Program Structure LayoutGeneral terms.“Parent” and “Children” elementsLogic for solving the problems“Primitive” elementsAnything you would naturally write for a paper solution of a physics problem.Coordinate system, approach, picture, diagrams, etc.Students unlock elements by answering questions in the problem solving process.