Usability Test Findings and Recommendations DRAFT 1 Project Overview 2 Amplifire usability test objectives Validate usability of the overall design approach and interactions Gather feedback on specific design innovations including the progress bar transition between questions and explana ID: 515156
Download Presentation The PPT/PDF document "Amplifire" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Amplifire
Usability Test Findings and RecommendationsDRAFT
1Slide2
Project Overview
2Slide3
Amplifire usability test objectives
Validate usability of the overall design approach and interactions
Gather feedback on specific design innovations, including the progress bar, transition between questions and explanations, and confidence indicators
Identify opportunities to improve ease of use:
aspects of the design that hinder task completion, cause confusion, or negatively impact user
experience
3Slide4
Project structure
Acclaro and Knowledge Factor recruited 6 participants who matched the amplifire target demographic.2
college students with some experience using online study
tools and
flashcard mobile
apps
2 recent college graduates, with work experience and an interest in graduate-level
programs
1 college
instructor
1 hospital nurse, familiar with online training coursesEach participant used amplifire during a one-on-one session with the test moderators:Molly moderatedJames took notes and analyzed dataParticipants completed a module unassisted and were interviewed about their experiences.Sessions were recorded.
4Slide5
Themes
1:
Usability findings and recommendations align
with eight broad categories
5
Dashboards are simple and easy
Welcome video and module introduction pages provide inadequate orientation
Feelings of being tested discourage honesty
Answering questions is difficult to learn
Streamlined question sets and explanation pages were liked
Progress indicators are too subtle
Review, refresh, and flashcards are on track
Going back
introduces confusion
2:
3:
4:
5
:
6:
7
:
8:Slide6
1:
Positive feedback:General
impressions were positive – simple and clean
Launching modules and changing courses
were
easy
The
new dashboards were useful
and easy to
interpret
The term “module” is intuitiveOpportunities:The Learn link is hidden when a course contains chapters The Change Book control was not noticed, so assignments were missedDashboards are simple and easy6Slide7
Positive feedback
Participants described amplifire as “clean” and “easy to follow”.
Participants were able to understand the purpose of amplifire by looking at the dashboard.
Most participants had no trouble locating the Learn link to begin modules.
All participants found and successfully used the My Courses drop down in the header.
All
participants were comfortable with the term “Module”.
1: Dashboards
7Slide8
New dashboard mockups well received
1: Dashboards
8Slide9
New dashboard mockups well received
The new dashboards were perceived as useful and easy to interpret.
Participants
liked that estimated time remaining was
shown, and commented
that they would start a module only if they had enough time to finish it in one study session.
Participants thought that showing progress through each book (rather than for the entire course) was appropriate.
Participants understood and liked the “Knew It” / “Learned It” chart. They said it would guide them in determining what
to
study.
One participant was an instructor, so we showed him the instructor dashboard mockup. He thought the information presented would help him decide which topics he needs to address in class. He would like it to be broken down by class section.1: Dashboards
9Slide10
Opportunities
After switching courses, participants hesitated because the Learn link is hidden.Only one participant noticed the Change Book link without prompting. To ensure that assignments are not missed, the control needs to be improved.
10Slide11
Recommendations: Dashboards
If a course contains multiple books, make the list of books and method for changing books more obvious.
On
page load, open the first (or most recently-used) chapter to show modules
.
11
1: DashboardsSlide12
2:
Positive feedback:Participants thought the introduction page was informative
Participants liked the video
Opportunities:
The introduction page is too long its purpose is unclear
The
video is
shown at the wrong time and can’t be accessed again
later
Welcome video and module introduction pages provide inadequate orientation
12Slide13
Positive feedback
Participants liked the style and tone of the welcome video.Participants who read the introduction page found the information about how amplifire works to be informative.
2. Video and introduction
13Slide14
Opportunities: Introduction page
Participants said they would not read the introduction page because it is too long.
The purpose of the
page
was unclear:
Is this page about
the course content or about
amplifire
?
Will I be tested on this information?
(While answering questions) Can I go back to review the Introduction page? 2. Video and introduction14Slide15
Opportunities: Welcome video
“This is not a test” is a catchy introduction, but is presented out of context.
Few
retained the information that
the video presented, and there is no way to review the video.
One
thought the video was a commercial, and tried to close
it immediately
. She then wished she could view it again.
2. Video and introduction
15
Highlights: Participants’ reactions to the
introduction and welcome videoSlide16
Recommendations
Introduction:Improve labeling on the introduction.
Break the page into two clear
sections (or pages?)
– about this module and how online learning
works
Or, make the introduction only about the module content, and pull how online learning works into a separate opt-in tutorial.
Video:
Wait until the user has answered a few questions to show a prompt for the video. (Put the video into context and allow the user to opt in.)
Add a Help option in the global navigation to allow users to access the video and informational overlays later.
Include the video in the dashboard as a widget to access later.16
2. Video and introductionSlide17
Prompt to watch video
17Slide18
Suggested revisions to video script
Revise the video script to clarify why amplifire uses a test-like format:
Answering
questions makes your brain learn faster (line graph spiking ever upward)
And shows where you need to spend time (question mark over head, furrowed eyebrows)
And what you can skip (check mark, “got it!”, lightbulb)
So you learn
better…faster
18
Thanks, Matt and Charles!Slide19
Video widget in dashboard
19Slide20
3:
Opportunities:Participants felt like modules were tests
Participants were more inclined to guess than choose “I Don’t Know”
Feelings of being tested discourage honesty
20Slide21
Students bring a strong mental
model that any system asking questions is testing them.
Aspects
that created a test-like feeling:
The correct answer is
withheld.
Learners can’t go back to
view a previous question.
Learners can’t change their responses after learning that they missed a question
Participants wondered
how the modules would “be graded” and whether instructors would be able to see students’ grades.3. Feelings of being tested21
Opportunities: Feelings of being testedSlide22
Participants didn’t use “I Don’t Know”
Participants were more inclined to guess than choose “I Don’t Know
”.
Participants
’ past experience with test taking tells them that it’s better to guess and possibly get the answer correct than to not select an answer.
3. Feelings of being tested
22
Highlights: Participants’ feelings of being tested
“I’m not going to risk getting it wrong
”.
- Recent college graduateSlide23
Partially sure
encouraged dishonesty
Learners relied on
partially sure to make a random guess that covers 2/3 of the answers, even without any knowledge on the topic.
23
Highlights: GuessingSlide24
Recommendations
Show the prompt for the video later in the Q&A flow.Detect patterns in learners’ answer selection behavior and display a prompt to remind them that this is not a test.
If learners repeatedly answer questions by selecting two responses, encourage them to choose I Don’t Know Yet if they truly don’t know.
If a learner repeatedly selects just one answer and is wrong, ask them if they are sure before they click Submit.
How important is this problem? Does a little test anxiety improve learning?
24
3. Feelings of being testedSlide25
4:
Positive feedback:Once participants learned how to select answers and indicate certainty, they moved quickly through questions
Opportunities:
Answer selection
was difficult to learn
When
unsure,
participants wanted to select only one answer
C
onfidence
dots to the left of the Submit button seemed unnecessaryAnswering questions is difficult to learn25Slide26
Positive feedback
Once participants figured out how to answer questions, they were able to move through questions efficiently.
4. Answering questions
26Slide27
Opportunities: Answer selection difficult to learn
Most participants dismissed the answer selection instruction overlay without fully reading it. One
later wished
she could re-open it to review the
instructions.
4. Answering questions
27Slide28
Opportunities: Answer selection difficult to learn
Participants didn’t initially realize how to activate the Submit
button.
Three
participants clicked one answer and then sat, wondering why the Submit button was
disabled.
They hovered the mouse over the submit button, and nothing happened.
4. Answering questions
28Slide29
Answer selection difficult
to learnTwo accidentally discovered that clicking two responses would activate the Submit button. Those participants answered the next few questions by clicking two answers, unclear why the system “required” this.
One
participant clicked an answer twice
because she thought that was required for activating the Submit button, even
though she wasn’t sure of her
response.
4. Answering questions
29
Highlights: A participant struggling with answer selectionSlide30
Participants wanted to select one answer and “I am unsure”
Selecting two answers didn’t match the way participants think about uncertainty.
Most participants attempted to
guess
by clicking submit after selecting only one partially sure answer. The system doesn’t allow this, which forces them to choose a second
guess.
Two participants told us they wanted to
be able to select just one answer, and indicate that they weren’t sure
.
4. Answering questions
30Slide31
Confidence dots are unnecessary
None of the participants noticed the confidence
dots to the left of the Submit button.
When
asked,
all
but one said they seemed redundant with the click twice/click two answers
convention.
4. Answering questions
31Slide32
Recommendations: Answering questions
Minimum:
Add
tooltip over the select
button
Add
a
Help or Tips option in the global navigation to allow users to access the video and informational overlays later
.
Remove the confidence dots
324. Answering questionsSlide33
Recommendations: Answering questions
Bigger Picture:
Consider changing
answer selection to allow single click and
submit. Include a tip explaining that clicking an answer once means you’re unsure, and twice means you’re sure.
Detect
patterns in learners’ answer selection
behavior. If
the user is confident and incorrect more than 2 or 3 times, display a prompt – if you’re not sure, try selecting
your answer once or selecting two
answers.334. Answering questionsSlide34
Recommendations (continued)
Bigger Picture:
Consider separating confidence from answer selection.
Brainstorm additional solutions…
34
4. Answering questionsSlide35
5:
Positive feedback:Participants liked the small question sets and alternating between question and explanation modes
Opportunities:
Participants
wondered about question order and spacing
The combined answer/explanation page was understood, but page layout caused repeated scrolling
Streamlined question sets and explanation pages were liked
35Slide36
Positive feedback: Small question sets
Participants liked the small question sets and alternating between question and explanation
modes
.
The learning
experience
felt manageable
, and not overwhelming. Participants appreciated the format of “small chunks”
of learning
.
Participants appreciated that questions they struggled with were repeated frequently, while at the same time, new questions were introduced.The transition between learning and review/explanations caught some participants by surprise. But, they were not annoyed by it. One said he thought he had “finished the test”, but then realized that there were more questions when amplifire automatically moved into the next question set. This is a success in terms of not letting students stop after the first round
.
5. Question sets and explanations
36Slide37
Opportunity: Question order and spacing
Participants were aware that there was something special about question order and spacing.
Most participants expressed
interest in understanding the underlying factors behind the determined question order
.
Participants were unclear
on
how the answer choices they made affected their progress.
5. Question sets and
explanations
37
Highlights: Comments on question order and spacingSlide38
Positive feedback: Combined answer/explanation page
Participants appreciated the availability of additional learning (but didn’t use it).
Placement of the Review Correct Answers link at the end of the review cycle seemed appropriate to participants, and a few said they might use it.
Participants repeatedly scrolled up and down
between the answer choices and the explanation.
When asked
why,
one participant confirmed that it was so that he could see the correct
answer in
context and contrast it with the other answer choices
.5. Question sets and explanations
38
Highlights: Scrolling on explanation pagesSlide39
Image use in explanation was effective
Participants appreciated the use of images in the explanation, and wanted to zoom or enlarge the image (on both the question and explanation pages
).
39
Highlights: Scrolling on explanation pages
“The image in the explanation was much more helpful than the text itself”.
- Recent college graduateSlide40
Recommendations: Question sets and explanations
In the Help / Tips section of the app, provide a link to an overlay or interactive tutorial describing:
Indicating certainty
Benefits of being honest
What
determines question order
.
Add a section to the dashboard with links to topics about the science behind the learning.
40
5. Question sets and explanationsSlide41
Recommendations (continued)
Consider developing a Screen Tips mode that allows the user to click items on any screen to see an explanation.
41
5. Question sets and explanations
Thanks, Nick!Slide42
Recommendations (continued)
Add zoom functionality to images (in question and explanation views).
42
Zooming to see fabric texture on Athleta.comSlide43
6:
Positive feedback:Once participants understood the progress bar, they liked the affirmation that they were progressing toward completing the module
Opportunities:
The
number of questions and color coding in the progress bar was not understood
The overlay after completing the first question was not noticed or was
misleading
Progress indicators are too subtle
43Slide44
Positive feedback
Once participants understood the progress bar, they liked the feedback as they completed questions progressed through the module.
44Slide45
Opportunity: Number of questions and color coding was not understood
The number of questions and color coding in the progress bar was not understood
Initially,
participants were
unable to identify the meaning for the two shades of green.
One tried
to use
the progress bar to
jump back to a previous question.
Because they didn’t know how many questions were in the module, participants didn’t know what the number of tick marks in the progress bar corresponded to
.6. Progress indicators45Slide46
Opportunity: The Your Progress button wasn’t used
Only one participant found the
“
My Progress” overlay without prompting. When viewing it, they found it helpful, and wanted estimated time remaining to be added.
Most participants seemed not to recognize
that the button was clickable.
One participant was confused because he had answered one question correctly but the Completed number in the overlay was zero.
6. Progress indicators
46
Highlights: Reactions to the progress barSlide47
Overlay after completing the first question was confusing
Two
participants
didn’t
notice the overlay when
they
completed their first question.
(bug / cache issue?)
The language in the overlay confused participants:
One thought it meant she had completed the module. The word “completed” seemed suggest that she was finished.Another was confused because she had answered several questions correctly, so didn’t know why amplifire considered only one question “completed”.6. Progress indicators
47
Highlights: Dismissing and misunderstanding overlaysSlide48
Recommendations: Progress indicators and overlay
Improve click affordance on the Your Progress button.Include a “What does “Completed mean”
link or info icon in the Your Progress overlay.
Change the wording on the first question completed overlay to
“You answered that question correctly twice, so won’t see it again
.”)
Add estimated time remaining to the Your Progress overlay.
48
6. Progress indicatorsSlide49
7:
Positive feedback:Refresh, review, and flashcards were well
received
Opportunities:
Upcoming changes to review should be re-evaluated
Flashcards should be developed
Review, refresh, and flashcards are on track
49Slide50
Positive feedback: Refresh, review, and flashcards are useful concepts
All participants correctly guessed what refresh
and review links on the dashboard would display.
All participants thought that refresh and review would be useful.
Opportunity: Participants weren’t sure whether they would need to answer questions correctly once or twice in Refresh.
All participants except one said that digital flashcards would be useful.
7. Review, refresh, and flashcards
50Slide51
Upcoming changes to Review should be re-considered
All participants preferred the current presentation of the review
screen:
7. Review, refresh, and flashcards
51
Over the future design:Slide52
Recommendations
Review:
Keep the current design
Refresh:
Clarify for users how many times a question must be answered correctly in Refresh mode.
52
7. Review, refresh, and flashcardsSlide53
Recommendations
Flashcards:
Begin competitive analysis of flashcard apps.
Add flashcards to the upcoming feature list
.
53
7. Review, refresh, and flashcardsSlide54
8:
Positive feedback:Participants were largely successful in navigating the appOpportunities:
The
browser Back button behaved
unexpectedly
Save and Return introduces unnecessary doubt
Going back introduces confusion
54Slide55
Positive feedback
Overall, participants navigated successfully among and within modules.
55Slide56
Opportunity: Browser back button behaves unexpectedly
Participants wanted to page back through questions and explanations, and were naturally drawn to use the browser back button.
In most cases the back button crashes amplifire.
8. General navigation
56Slide57
Opportunity: Save and return introduces unnecessary doubt
It was unclear to participants whether using the Save and Return feature was the only way to save their progress. The presence of the Save and Return link seems to plant uncertainty about whether their progress is automatically saved
.
8. General navigation
57Slide58
Recommendations: General navigation
Change Save and Return to a Dashboard link.Fix the browser Back button.
Question – what would happen when a learner clicks Back when answering questions?
58
8: General navigationSlide59
Priorities
This report provided many recommendations and illustrations. The highest priorities are:
List will go here.
59