Chapter 5 Human Error? No, Bad Design - PowerPoint Presentation

1. Chapter 5Human Error? No, Bad Design

2. Today’s TopicsHumans, systems, and errorMore on data gatheringDesigning for communication and coordination

3. Part 1: Humans, Systems and ErrorRoot cause analysis for errorWhy did Air Force stop at human error?Do not view the system being designed as one combining human and computer/machine actors (e.g. sociotechnical or cyber-human system)The five whys methodHumans are good at creative and adaptive behavior. The more mechanistic the activity is required to be, …Deliberate ViolationsOccur in all types of contexts (e.g. driving, officework, security, …)Sometimes because official procedures are developed for legal or accountability reasonsIn many situations, work cannot get done without exceptions

4. Errors: Slips and MistakesSlips: the goal is correct but the actions not done correctlyAction based – inaccurate performance of planned actionMemory lapse – forgetting to take a planned actionMistakes: the goal or plan is wrongRule based – wrong plan due to poor planning (“wrong rule”)Knowledge based – wrong plan due to inaccurate/incomplete knowledge Memory lapse – forgetting at stage of goals, plans, or evaluationExamples from elevatorPressing button below correct one along with correct onePressing the 1st floor button when going to the lobby

5. SlipsSlips are more common for experts than novicesResult of more aspects of a task being handled unconsciouslyCapture slipsWhen similar action sequences get confusedGenerally a more familiar or recent action sequence replaces anotherExample might be finishing a password with more common suffixMore common for expertsDescription-similarity slipsPerforming an action with a similar item to the one intendedDesigners should reduce similar items and reduce number to choose fromMode errorsOccurs when design has modes where the same user action results in different system actionsDesigners should try to avoid modes and make the mode visible when they are necessaryMemory lapse slipsForgetting where one is in procedure (repeating or not performing steps)Often caused by interruptionsDesigners can show steps and indicate where one is or can create forcing functions enforcing prerequeisits

6. MistakesRule-based mistakesSituation is misinterpreted so wrong rule is selectedCorrect rule is invoked, but it is faulty or did not account for current situationCorrect rule is invoked, but the outcome is incorrectly evaluatedKnowledge-based mistakesOccurs in situations requiring more general problem solvingCan be due to a mistaken mental modelDesigners need to provide clear conceptual models Systems need to support users when they encounter previously unknown situationsCooperative problem solving systemMemory-lapse mistakesOccur when user forgets previously gathered information about situation, or forgets the plan altogether

7. Social Issues and ErrorSocial, time, and economic pressure lead to bad decisionsSocial pressure also can lead to errors not being reportedAdding more responsible people can cause less attention to tasksEach may assume others are being more responsibleNeed to encourage reporting of errors to fix their causes Jidoka and Poka-YokeNASA aviation safety reporting system

8. Detecting Errors is Non-TrivialErrors are generally preceded by a number of uncommon events that each by itself is not strongly indicative of errorFalse alarmResponse to correct action interpreted as failureMay occur When there are conflicting indicatorsDue to different interpretations of plan/taskGarden PathIncorrect action not discovered until laterLosing context of original problemMachine can interpret action as correct for some alternative pathUsers can assume they know the process without machineTrivial breaches of understanding can become “fatal”

9. Designing for ErrorWarningsNeed to be clear from one another and coordinated with one anotherNeed to gain needed attention without being obnoxious/disabilitatingComputer systems can often addDifferent visual cues and locations for different actions/featuresSensibility checks on user actionsUndo and multi-stage undoChecklistsCommon in aviation but not other industriesDesign of checklists is still a difficult taskMore generally, processes can focus on creating resilienceThe Swiss cheese model to minimize likelihoodNotifications when features are lining up for error

10. Part 2: More on Data GatheringInterviewsQuestionnairesObservationChoosing and combining techniques

11. Four key issuesSetting goalsDecide how to analyze data once collectedRelationship with participantsClear and professionalInformed consent when appropriateTriangulationUse more than one approachPilot studiesSmall trial of main study

12. Data recordingNotes, audio, video, photographsNotes plus photographsAudio plus photographsVideo

13. InterviewsUnstructured - are not directed by a script. Rich but not replicable. Structured - are tightly scripted, often like a questionnaire. Replicable but may lack richness.Semi-structured - guided by a script but interesting issues can be explored in more depth. Can provide a good balance between richness and replicability.

14. Interview questionsTwo types:‘closed questions’ have a predetermined answer format, e.g., ‘yes’ or ‘no’‘open questions’ do not have a predetermined formatClosed questions are easier to analyzeAvoid:Long questionsCompound sentences - split them into twoJargon and language that the interviewee may not understand Leading questions that make assumptions e.g., why do you like …?Unconscious biases e.g., gender stereotypes

15. Running the interviewIntroduction – introduce yourself, explain the goals of the interview, reassure about the ethical issues, ask to record, present any informed consent form.Warm-up – make first questions easy and non-threatening. Main body – present questions in a logical orderA cool-off period – include a few easy questions to defuse tension at the endClosure – thank interviewee, signal the end, e.g, switch recorder off.

16. Enriching the interview processProps - devices for prompting interviewee, e.g., a prototype, scenario

17. QuestionnairesQuestions can be closed or openClosed questions are easier to analyze, and may be done by computerCan be administered to large populationsPaper, email and the web used for disseminationSampling can be a problem when the size of a population is unknown as is common online

18. Questionnaire designThe impact of a question can be influenced by question order.Do you need different versions of the questionnaire for different populations?Provide clear instructions on how to complete the questionnaire. Strike a balance between using white space and keeping the questionnaire compact.Decide on whether phrases will all be positive, all negative or mixed.

19. Question and response format‘Yes’ and ‘No’ checkboxesCheckboxes that offer many optionsRating scalesLikert scalessemantic scales3, 5, 7 or more points?Open-ended responses

20. Encouraging a good responseMake sure purpose of study is clearPromise anonymityEnsure questionnaire is well designedOffer a short version for those who do not have time to complete a long questionnaireIf mailed, include a stamped addressed envelopeFollow-up with emails, phone calls, lettersProvide an incentive40% response rate is high, 20% is often acceptable

21. Advantages of online questionnairesResponses are usually received quicklyNo copying and postage costsData can be collected in database for analysisTime required for data analysis is reducedErrors can be corrected easily

22. Problems with online questionnairesSampling is problematic if population size is unknownPreventing individuals from responding more than onceIndividuals have also been known to change questions in email questionnaires

23. ObservationDirect observation in the fieldStructuring frameworksDegree of participation (insider or outsider)EthnographyDirect observation in controlled environmentsIndirect observation: tracking users’ activitiesDiariesInteraction logging

24. Structuring frameworks to guide observation- The person. Who? - The place. Where?- The thing. What? The Goetz and LeCompte (1984) framework:- Who is present? - What is their role? - What is happening? - When does the activity occur?- Where is it happening? - Why is it happening? - How is the activity organized?

25. Ethnography (1)Ethnography is a philosophy with a set of techniques that include participant observation and interviewsDebate about differences between participant observation and ethnographyEthnographers immerse themselves in the culture that they studyA researcher’s degree of participation can vary along a scale from ‘outside’ to ‘inside’Analyzing video and data logs can be time-consumingCollections of comments, incidents, and artifacts are made

26. Ethnography (2)Co-operation of people being observed is requiredInformants are usefulData analysis is continuousInterpretivist techniqueQuestions get refined as understanding growsReports usually contain examples

27. Other Observational MethodsDirect observation in a controlled settingThink-aloud techniqueIndirect observationDiariesInteraction logs

28. Choosing and combining techniquesDepends onThe focus of the studyThe participants involvedThe nature of the techniqueThe resources available

29. Data Gathering SummaryThree main data gathering methods: interviews, questionnaires, observationFour key issues of data gathering: goals, triangulation, participant relationship, pilotInterviews may be structured, semi-structured or unstructuredQuestionnaires may be on paper, online or telephoneObservation may be direct or indirect, in the field or in controlled settingTechniques can be combined depending on study focus, participants, nature of technique and available resources

30. Part 3: Designing for Conversation and CoordinationConversational mechanismsCoordination mechanismsAwareness mechanismsExamples of technologies designed to extend how peopletalk and socialisework togetherplay and learn together

31. Conversational mechanismsVarious mechanisms and ‘rules’ are followed when holding a conversation, e.g. mutual greetingsA: Hi thereB: Hi!C: HiA: All right?C: Good, how’s it going?A: Fine, how are you?C: OKB: So-so. How’s life treating you?

32. Conversational rulesSacks et al. (1978) work on conversation analysis describe three basic rules:Rule 1: the current speaker chooses the next speaker by asking an opinion, question, or requestRule 2: another person decides to start speakingRule 3: the current speaker continues talking

33. Conversational rulesTurn-taking used to coordinate conversationA: Shall we meet at 8?B: Um, can we meet a bit later?A: Shall we meet at 8?B: Wow, look at him?A: Yes what a funny hairdo!B: Um, can we meet a bit later?Back channeling to signal to continue and followingUh-uh, umm, ahh

34. More conversational rulesFarewell ritualsBye then, see you, yer bye, see you later….Implicit and explicit cuese.g., looking at watch, fidgeting with coat and bags explicitly saying “Oh dear, must go, look at the time, I’m late…”

35. Breakdowns in conversationWhen someone says something that is misunderstood:Speaker will repeat with emphasis:A: “this one?”B: “no, I meant that one!”Also use tokens:Eh? Quoi? Huh? What?

36. What happens in technology-mediated conversations?Do same conversational rules apply?Are there more breakdowns?How do people repair them for:Phone?Email?IM?Texting?

37. Synchronous computer-mediated communicationConversations are supported in real-time through voice and/or typingExamples include video conferencing, VOIP, MUDs and chatBenefits include:Not having to physically face people may increase shy people’s confidenceAllows people to keep abreast of the goings-on in an organization without having to move from their officeProblems:Difficult to establish eye contact with images of othersPeople can behave badly when behind the mask of an avatar

38. Asynchronous computer-mediated communicationCommunication takes place remotely at different timesEmail, newsgroups, textingBenefits include:Read any place any timeFlexible as to how to deal with itCan make saying things easierProblems include:FLAMING!!!Message overloadFalse expectations as to when people will reply

39. Coordination mechanismsWhen a group of people act or interact together they need to coordinate themselvese.g., playing football, navigating a shipThey use:verbal and non-verbal communicationschedules, rules, and conventionsshared external representations

40. Verbal and non-verbal communicationTalk is centralNon-verbal also used to emphasize and as substitutee.g., nods, shakes, winks, glances, gestures and hand-raisingFormal meetingsexplicit structures such as agendas, memos, and minutes are employed to coordinate the activity

41. Schedules, rules and conventionsSchedules used to organize regular activities in large organizationsFormal rules, like the writing of monthly reports enable organizations to maintain order and keep trackConventions, like keeping quiet in a library, are a form of courtesy to others

42. Shared external representationsCommon method used to coordinate collaborative activities, e.g., checklists, tables, to-do listsThey can provide external information on:who is working on what when it is being worked onwhere it is being worked onwhen a piece of work is supposed to be finished whom it goes to next

43. Collaborative technologies to support coordinationThere are a variety of software tools designed to support scheduling, planning and coordinating e.g., group calendars, electronic schedulers, project management tools, and workflow tools Need to get balance between human and system controltoo much system control and the users will rebeltoo little control and the system breaks down

44. Awareness mechanismsInvolves knowing who is around, what is happening, and who is talking with whomPeripheral awarenessKeeping an eye on things happening in the periphery of visionOverhearing and overseeing - allows tracking of what others are doing without explicit cues

45. Lo tech awareness mechanism

46. Designing technologies to support awarenessProvide awareness of others who are in different locationsEarly example was media spaces “extend the world of desks, chairs, walls and ceilings” (Harrison et al, 1997) Examples: Clearboard and Portholes

47. Portholes (Xerox PARC)Regularly updated digitized images of people in their offices appeared on everyone’s desktop machines throughout day and night

48. Notification systemsUsers notify others as opposed to being constantly monitoredProvide information about shared objects and progress of collaborative tasksexamples: Tickertape, Babble

49. Elvin Elvin is a distributed awareness system that provides a range of client services (Segall and Arnold, 1997)It includes Tickertape, one of the first lightweight messaging systems

50. Conversation and Coordination Key PointsSocial mechanisms, like turn-taking, conventions, etc., enable us to collaborate and coordinate our activitiesKeeping aware of what others are doing and letting others know what you are doing are important aspects of collaborative working and socialisingMany collaborative technologies systems have been built to support collaboration