Qatar National Research FundQNRF Undergraduate Research Experience Program Eighth Cycle Session Agenda 2 Collaborative Research Model The Study Findings Test Reviews The Researchers ID: 1033153
Download Presentation The PPT/PDF document "A Review of Placement Tests for EFL EAP ..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
1. A Review of Placement Tests for EFL EAP ContextsQatar National Research Fund(QNRF) Undergraduate Research Experience Program - Eighth Cycle.
2. Session Agenda2Collaborative Research Model The StudyFindingsTest Reviews
3. The ResearchersStudents (University of Calgary-Qatar)Muna AdenNoof Al KuwariMihad IbrahimOmaima SouayahFaculty (University of Calgary-Qatar )Karen BrookeVirginia ChristopherDr. Brad JohnsonExternal AdvisorsDr. Dudley Reynolds (Carnegie Mellon University-Qatar)Dr. Anne Nebel (Georgetown University-Qatar)Dr. Peter Brown (College of the North Atlantic-Qatar)3
4. Context4University of Calgary, Qatar (B.A. Nursing)English the medium of all instructionMost students do not have English as a first languageMost students require additional English language classes before starting regular studies
5. The CR Model in UseUndergrad Research Experience Project (UREP)Sponsored by the Qatar National Research Fund (QNRF)MissionUndergraduate research Learning by doingSkills Development Problem-solvingCommunicationWorking independentlyUnderstanding research methodsEthics & rules of conduct5
6. Collaborative Research Model6Key Characteristics Students as CO-Researchers Collaborative Deliberation Group Reflection Multiple PerspectivesCore Elements Guided Inquiry Collaborative Safe, supported, mentoredMain Benefits Actively Engages Students and Faculty Promotes Effective Group Processes Provides a Process for Conducting All Phases of Research
7. The StudyAssessing English Language Placement Tests for Use in Qatari Post-Secondary InstitutionsThe ProblemWestern cultural perspectiveChoosing best test for Gulf The Goal Assessment rubricThe Research Team4 Undergrads3 faculty members3 External advisors7
8. The Study: Stages All stages implement Collaborative Research methodology… 1. Getting started2. Literature review 3. Development of assessment rubric/ Seeking advisor’s input4. Applying the rubric 5. Recruiting subjects & piloting tests 6. Compare Test Scores to Expert Raters7. Reflecting & assessing/ Writing reports & articles / Presenting8
9. Stage 2: Literature Review9
10. Stage 3: Development of Assessment Rubric10 Brainstorm rubric criteria Categorize Develop rubric sections
11. Applying the Rubric: Scores11Test NameTest ModeRubric Score /100Holistic Score /10AccuplacerComputer898CompassComputer859University of Michigan English Placement TestPaper718PasswordComputer667OPT Online TestComputer515OPT Placement TestPaper465
12. Recruiting Subjects and Piloting Tests 12Oxford Placement Test: 23 participants Oxford Online Test: 26 participantsAccuplacer: 16 participants
13. Evaluating the Rubric: Expert Raters13AccuplacerOPT OnlineOPT PaperCohen’s Kappa.3766.2632.2477Weighted Kappa (linear).4839.4257.2562
14. Evaluating the Rubric: Correlations14Oxford Online TestListeningUse of EnglishTotalAccuplacerLanguage Use0.453699030.575370370.54591973Listening0.627457380.544467150.62689216Reading Skills 0.70796080.574750010.69046848Sentence Meaning0.496544480.685785130.62420157Writeplacer0.441798120.59391448 0.5437714Total Excluding Writeplacer0.636014770.675775070.69895996
15. Evaluating the Rubric: Correlations15OPT PaperListeningGrammarTotalAccuplacerLanguage Use0.079920.690980.62999298Listening-0.1540.741580.54316763Reading Skills-0.1090.66164 0.5003664 Sentence Meaning0.182870.734040.72358122Writeplacer0.104280.651950.61041479Total Excluding Writeplacer0.015290.790290.67832386
16. Evaluating the Rubric: Correlations16Oxford Online TestListeningUse of EnglishTotalOPT PaperListening-0.0272-0.028-0.02Grammar0.768370.702840.78485Total0.632830.577450.64989
17. Findings17OPT Paper Based Listening not functioning well, as we had suspected
18. Findings18Culturally inappropriate material not common, rarely caused incorrect responsesEvery test mentioned wineOne question required students to interpret the relationship between 2 room-mates – our students could notOne question relied on the knowledge that dogs’ fur gets thicker in winter – our students did not know this
19. Findings19Test Conditions Computerised delivery not a problem for most test sectionsComputerised writing tests may be seriously compromised by slow typingTyping test showed average typing speed of only 11 wpm for one group of incoming studentsLack of familiarity with English keyboard
20. Findings20Possible mismatch between test content and skills needed for success in non-native classesTests have heavy emphasis on grammar. Teachers in this context may be more tolerant. Some tests include knowledge of idioms. Teachers in this context learn to avoid idioms. Our students knew few common idioms.One test had a section on collocations. Our students did very poorly on this section, despite advanced level of English.
21. Test Reviews21Accuplacer ESL:American/ College BoardInternet BasedComputer Adaptive
22. Test Reviews22Accuplacer ESL:Reading SkillsSentence Meaning (Vocabulary in Context)Language UseListeningWritePlacer ESL
23. Test Reviews23Accuplacer ESLProsGood range of academic contentFamiliar multiple choice formatInternet basedReading tested a variety of skills (paraphrasing/ inferences/ fact-opinion)Realistic listening conversations with pictures for contextWriting included short reading to give contextDescription of what elements of writing will be gradedVery flexible delivery by institution
24. Test Reviews24Accuplacer ESLConsWriting Prompt difficultAmericanisms (Miranda rights, American city names, Labor Day)Difficult instructions Very unhelpful tutorials
25. Test Reviews25Compass: American/ ACTSecure Browser on InternetComputer Adaptive
26. Test Reviews26Compass: ListeningReading Grammar/ UsageEssay
27. Test Reviews27CompassProsGood range of very realistic academic content from range of subjectsListening items moved from short conversations in familiar context to long abstract lectures and dialoguesReading items moved from short texts to long academic texts Reading items tested a variety of skills (main ideas/ details, inferences, conclusions)Grammar included not only sentence level grammar but relationships between paragraphs
28. Test Reviews28CompassConsTest software must be downloaded to each computerScreen resolution must be reconfigured before test is run
29. Test Reviews29Password: British/ English Language Testing, Ltd.Computerised and Unique, but not Computer AdaptiveSecure Browser
30. Test Reviews30Password: 5 sections of grammar and vocabulary, optional writing and reading sections2 sections of multiple choice grammar Vocabulary – choose best synonymCollocationsIdentify grammatically correct sentences
31. Test Reviews31PasswordProsVocabulary from Academic Word ListConsIntended student profile did not match oursPoor performance by our students on collocationsLast section confusing and tiring
32. Test Reviews32Oxford Online Placement Test: Computer AdaptiveInternet BasedUse of English (grammar, vocabulary, reading)Listening
33. Test Reviews33Oxford Online Placement TestProsVery easy to conductConsNot delivered on a secure browserTest content less suitable for our students – range of settlement and business situations Emphasis on idiomsBritish language and accents
34. Test Reviews34University of Michigan English Placement TestListening, reading, grammar, vocabularyProsVery easy to conduct (paper based)Vocabulary from Academic Word ListConsStudents found listening confusing (choose appropriate response or best paraphrase)Reading passages a few sentences at most
35. Test Reviews35Oxford Placement Test (paper)Listening, grammarProsVery easy to conduct (paper based)ConsListening based on distinguishing similar sounding wordsVery low frequency words in listening section
36. Value of Using CR36For Students Actively Engaged in Research Process Learn Effective Group Processes Learn a Process for Conducting All Phases of ResearchFor Faculty Emphasizes Mentorship Role Makes Research Process Explicit Provides a Process for Conducting All Phases of ResearchFor Institutions Provides a Best-Practices Model Promotes Faculty-Student Working Relationships Prepares Students for Graduate Work
37. ReferencesBrooke, K., Aden, M., Al-Kuwari, N., Christopher, V., Ibrahim, M., Johnson, B., & Souyah, O. (2012). Placement Testing in an EFL Context. TESOL Arabia, 19 (2), 13-20.37
38. Questions and DiscussionContact kbrooke@vcc.caThank You for Your Attention38