Marco Canini KAUST and Jon Crowcroft University of Cambridge What it takes to do reproducible research It is actually hard work for both authors and reproducers Produce solid artifacts Create package share reproducible experiments ID: 634625
Download Presentation The PPT/PDF document "Learning Reproducibility with a Yearly N..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Learning Reproducibility with a Yearly Networking Contest
Marco
Canini (KAUST) and Jon Crowcroft (
University of Cambridge)Slide2
What it takes to do reproducible research?
It is actually hard work for both authors and reproducers
Produce solid artifacts
Create, package, share reproducible experiments
It’s about the ecosystem
Mindset
–
motivations, incentives
, expectations (“golden standards
”)
Skillset
–
the art of reproducible research: skills, methods, technologiesSlide3
Position
Goal:
increase
pervasiveness of reproducible networking research
Proposal
: yearly international contest with SIGCOMM branding
Targeted at plurality of students
, especially
at
early stage
career
Enable
students to learn how to perform reproducible networking research
Desired
outcome
is two-fold:
Instill
in students an
appreciation
for reproducible
research
Provide a
learning experience of what reproducibility
entailsSlide4
Non Goals
Substitute or conflict with other processes
like reproducibility evaluation committees, special-issue journals, etc.
Establish a “best of reproductions” competition
Add
yet another
checkbox for reproducible research
Focus on advancing the state of the art on a big problem
i.e., it’s not a DARPA Grand Challenge for networking
We view our proposal as complementary
be a cog in the wheel for making research reproducibility more
pervasiveSlide5
What you’ll find in the paper
A short review of existing contests in networking and why they do not apply to our
proposal
Various contests are industry focused or have incompatible goals
We connect how
a contest
would aid
with
reproducibility
Supporting examples from Stanford, MIT and
UCLouvain
Describe
a possible format of the
contest
Some other implications (longer term)Slide6
What you’ll NOT find in the paper
It is not a complete blue print or spec to follow
We have not trialed it
yetSlide7
What you’ll NOT find in the paper
It is not a complete blue print or spec to follow
We have not trialed it
yet
Your feedback!Slide8
So, why a contest?
Learn something of the skills, methods and technologies
Participants
: gain understanding of what’s required to reproduce an experiment and make it reproducible
Work with a sim/emulator, set up an environment, do traffic generation, etc.
Package and share solution with organizersSlide9
So, why a contest?
Learn something of the skills, methods and technologies
Participants
: gain understanding of what’s required to reproduce an experiment and make it reproducible
Work with a sim/emulator, set up an environment, do traffic generation, etc.
Package and share solution with organizers
Contest brings excitement and challenges to stimulate students
Makes learning reproducibility a by-product of participatingSlide10
So, why a contest?
Learn something of the skills, methods and technologies
Participants
: gain understanding of what’s required to reproduce an experiment and make it reproducible
Work with a sim/emulator, set up an environment, do traffic generation, etc.
Package and share solution with organizers
Enrich testbeds, test harnesses, datasets, and platforms
Organizers
: gain experiences with evaluation of submissions and share these experiences with the community (reviewing committees)
Advance our “best practices” for research reproducibility
S
upport the sociotechnical system, including archives, testbeds, budget, and staffSlide11
On the contest format
Reproduce experiments from published papers
+ independent verification of results
+ gain understanding of difficulties and recommendations of what to improve
- unclear how to rank participants (success might be high)
Well-defined problem with quantitative goal
+ score submissions objectively
- reproducing research is not an integral partSlide12
Our proposal
Objective is not to reproduce research per se
but a first phase creates a baseline by building upon an existing approach and reproducing a certain experiment from a paper
in a second phase focus is to find a better solution
Contest design has to promote clean and well-structured solutions, packaged and shared with organizers
Disincentive “hack it together” mentality
Make reproduction integral part but take it a step furtherSlide13
Some details
Students compete in team
Not a hackathon; should last a few weeks
Possible run down for research group R during year Y
Acquainted with skeleton code and small exercises
Reproduce an experiment and validate on supplied workload
Start to improve upon baseline solution; submit and watch the scoreboard
Fine tune solutions; submit and get ranked on a final (secret) workload
Final prize and award at SIGCOMM
Give more visibility to reproducibility in CVs
Y+=1; R = select()Slide14
Summary and open questions
Make the case that a contest promotes better research reproducibility ecosystem
Learn the skillset and mindset
Aid the reproduction evaluation committees with tooling and training
Success: an increase of reproducible research by contest participants
How does all this sound?
How to define tractable and interesting problems for the contest?
Any other ideas and contributors?
Would you encourage your students to participate?