A Randomized Scheduler with Probabilistic Guarantees - PowerPoint Presentation

1. A Randomized Scheduler with Probabilistic Guarantees of Finding BugsSebastian BurckhardtMicrosoft ResearchPravesh KothariIndian Institute of Technology, KanpurSantosh NagarakatteUniversity of PennsylvaniaMadanlal MusuvathiMicrosoft Research

2. What is Concurrency Testing?Whether a test finds a bug depends onthe configurationthe inputsthe scheduleConcurrency bugs are bugs that surface only for some schedulesThe Concurrency Testing ProblemHow to cover buggy schedules as best we can?Testing all schedules is infeasible!

3. Idea: Randomize the Schedulevoid* p = 0; CreateThd(child);p = malloc(…);Init();DoMoreWork();p->f ++;ParentChildInstrument code with calls to insert random delaysIf we are lucky, delay exposes bugsBut: how long to delay? where not to delay?void* p = 0;RandDelay();CreateThd(child);RandDelay();p = malloc(…);Init();RandDelay();DoMoreWork();RandDelay();p->f ++;void* p = 0;RandDelay(); CreateThd(child);RandDelay();p = malloc(…);Init();RandDelay();DoMoreWork();RandDelay();p->f ++;void* p = 0;RandDelay(); Start(child);RandDelay();p = malloc(…);Init();RandDelay();DoMoreWork();RandDelay();p->f ++;

4. What is a Randomized Algorithm?A randomized algorithm:“An algorithm that makes nondeterministic choices”An algorithm using a random source with a precisely defined distributionA probabilistic guarantee:“A guarantee that doesn’t always hold”A lower bound on the probability of success

5. What we did / Talk OutlineDefine bug depth in such a way that common bugs have low depthDevelop PCT algorithm (probabilistic concurrency testing), a randomized scheduling algorithm with a good probabilistic guarantee to find bugs of low depthBuild it into Cuzz, a concurrency fuzzing tool that improves the efficiency of stress testing

6. Bug depthPart I

7. Bug DepthBug Depth = the number of ordering constraints a schedule has to satisfy to find the bug.More constraints means more things have to go “just right” to find the bug.Conjecture: many typical bugs have low depth.Let’s look at 3 examples.

8. Ordering Violation Example: A Bug of Depth 1Bug depth = the number of ordering constraints sufficient to find the bug.All schedules that satisfy the “” find the bug.…start(child);p = malloc();…Parent Thread…do_init();p->f ++;…Child Thread

9. Atomicity Violation Example: A Bug of Depth 2Bug depth = the number of ordering constraints sufficient to find the bug.All schedules that satisfy both “” find the bug.p = malloc();start(child);…If (p != null) p->f++…Parent Thread…p = null;…Child Thread

10. Deadlock Example: A Bug of Depth 2Bug depth = the number of ordering constraints sufficient to find the bug.All schedules that satisfy both “” find the bug.…Lock(A);…Lock(B);…Parent Thread…Lock(B);…Lock(A);…Child Thread

11. the PCT ALGORITHMPart II

12. PCT Algorithm: Randomly Assign & Change Thread PrioritiesInput: int k; // no. of steps - guessed from previous runs int d; // target bug depth - randomly chosenState: int pri[]; // thread priorities int change[]; // when to change priorities int stepCnt; // current step countPCT::Init() { stepCnt = 0; foreach tid pri[tid] = rand() + d; for( i=0; i<d-1; i++ ) change[i] = rand() % k;} PCT::RandDelay( tid ) { stepCnt ++; if stepCnt == change[i] for some i pri[tid] = i; if (tid is not highest pri enabled thread) spin;}

13. The PCT GuaranteeGiven a program withn threads (~tens)k steps (~millions)a bug of depth d (1,2)Each run PCT finds the bug with a probability of at least (this is a worst-case guarantee)

14. the cuzz Tool& ResultsPart III

15. How it WorksIntercept at synchronization pointsDetour win32 synchronization callsOptionally instrument data accessesNo manual instrumentation requiredProgramKernel SchedulerWin32 APICuzzRandomizedAlgorithmbinary instrumentationfor data accesses(optional)

16. Some Results

17. Practice Beats Worst-CaseMeasured Probability often significantly better than worst-case guaranteed probability

18. Why Does Practice Beat Worst-Case?Worst-case guarantee applies to hardest-to-find bug of given depthIf bugs can be found in multiple ways, probabilities add up!Example: Increasing the number of threads helps:

19. Internal Tool StatusThe Cuzz tool is available internally at MicrosoftWe are working with several product groups that actively use Cuzz to improve their stress testing

20. DEmo

21. Demo ConclusionMeasure probabilities on clusterWithout Cuzz: 1 Fail in 238’820 runs ratio = 0.000004817With Cuzz: 12 Fails in 320 runs ratio = 0.0375Resource Savings: factor 7,8001 day of stress testing = 11 seconds of Cuzz testing

22. ConclusionsBug depth is a useful metric to focus testing effortsSystematic randomization improves concurrency testingNo reason not to use CuzzThank You For Your Attention.

23.