Bryce Boe 20130307 University of California Santa Barbara Bryce Boe Charlotte Hill Michelle Len Greg Dreschler Phillip Conrad Diana Franklin Motivation Scratch project assessment is tedious and error prone ID: 332535
Download Presentation The PPT/PDF document "Hairball: Lint-inspired Static Analysis ..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Hairball: Lint-inspired Static Analysis of Scratch Projects
Bryce Boe2013/03/07University of California Santa Barbara
Bryce Boe, Charlotte Hill, Michelle Len, Greg Dreschler, Phillip Conrad, Diana FranklinSlide2
Motivation
Scratch project assessmentis tedious and error pronetakes away from student interaction timeScratch programming
becomes relatively more difficult to manage as the project size grows
has nearly no tools to check for
correctnessSlide3
Related Work
J. C. Adams and A. R. Webster. What do students learn about programming from game, music video and storytelling projects? SIGCSE 2012.Q. Burke and Y. B.
Kafai
. The writers’ workshop for youth programmers: digital storytelling with scratch in middle school classrooms. SIGCSE 2012.Slide4
Background
Assessed four Scratch concepts from a two week summer camp58 projects across 5 assignmentsSee tomorrow’s talk:
Assessment of Computer Science Learning in a Scratch-Based Outreach Program
11:30 in Governors 16Slide5
Hairball
A Scratch program static analysis toolFlag items that are potentially
incorrect
can be extended through Python plugins
Goals
Provide automated
assistance
for manual analysis
Warn students about
potential
mistakesSlide6
Methodology
Manual Analysis (intended ground truth)For each concept, 3 staff members each manually counted and classified instances
of the CS concept
Reconciled any discrepancies
Hairball Analysis
Programmed hairball plugins to attempt detect and classify the same
instances
Actual Ground Truth
Set of similarly classified instances between manual and hairball, plus the result of a second manual analysis for any discrepanciesSlide7
Instance Classification
Correct
Properly demonstrates the Scratch concept
Semantically incorrect
May
appear
to work correctly upon execution, but implemented in a non-robust way
Incorrect
Implemented in way that doesn’t work
Incomplete
Missing necessary componentsSlide8
Terminology
False negatives
Instances that are
not
labeled
correct
when they in fact are
False Positives
Instances that are labeled
correct
that are
not
actually correctSlide9
Hairball PluginsSlide10
Initialization
Checks that the project initializes attributes that are modified
INCORRECT
CORRECT
Initialization ZoneSlide11
Initialization Evaluation
32 false positives
33 false negativesSlide12
Say and Sound Synchronization
Checks that say bubbles are synchronized with sound files
S. INCORRECT
CORRECTSlide13
Say and Sound Synchronization Evaluation
4 false positives
4 missing instances
2 missing instancesSlide14
Broadcast and Receive
Checks that each event has matching broadcast and receive blocks and only one broadcast through any one path of a scriptSlide15
Broadcast and Receive Evaluation
3 false positives
79 false positives
100% detection
12 missing instancesSlide16
Complex Animation
Checks that a sequence of position and/or orientation changes occur along with costume changes and a delaySlide17
Complex Animation
3 missing instances
2 false negatives
11 extra instancesSlide18
Hairball SummarySlide19
Hairball SummarySlide20
Hairball SummarySlide21
Live Demo
http://hairball.herokuapp.com/Slide22
Conclusions
Manual assessment is both time-consuming and quite error-proneHairball is useful to augment manual analysis (finds things that humans miss)Hairball is incredibly accurate at detecting
correct
itemsSlide23
Future Work
Add additional plugins for other sorts of analysisTest Hairball on a larger set of assignments(Anyone have Scratch projects they need assessed?)Measure effectiveness of Hairball as a lint toolSlide24
Questions
Contact Informationbboe@cs.ucsb.eduhttps://twitter.com/bboe
Links
http://hairball.herokuapp.com/
https://github.com/ucsb-cs-education/hairball
Tomorrow’s talk (11:30 in Governors 16)
“Assessment
of Computer Science Learning in a Scratch-Based Outreach
Program”Slide25
Bonus SlidesSlide26
Initialization Check Weakness
Visibility initialization properly detectedPosition and orientation
initialization does not occur in the
initialization zoneSlide27
Say Sound Sync Weakness
Blocks between say and sound block
Resulting code
may
still produce desired effect