Detecting Assumptions on Deterministic Implementations of N - PowerPoint Presentation

Slide1

Detecting Assumptions on Deterministic Implementations of Non-deterministic Specifications

August Shi, Alex Gyori, Owolabi Legunsen, Darko Marinov4/12/2016ICST 2016Chicago, Illinois

CCF-1012759

, CCF-1409423,

CCF-1421503, CCF-1439957Slide2

Example Code and Test

2public

class Book {

String

author; String title; public Book(String author, String title) {

this

.author

= author;

this.title = title; } public String getStringRep() { JSONObject j = new JSONObject(); // JSONObject extends java.util.HashMap j.put("author", this.author); j.put("title", this.title); return j.toString(); // toString() iterates through entries }}

public class BookTest { @Test public void testGetStringRep() { Book b = new Book("A", "T"); assertEquals("{\"author\":\"A\",\"title\":\"T\"}", b.getStringRep()); }}

/** … This class makes no guarantees as to the order of the map … */

p

ublic class

java.util.HashMap

…Slide3

Non-deterministic Specifications

A specification (spec) that allows multiple implementations with different outputs for a given inputGood: Allow freedom of implementationAlthough specs are non-deterministic, underlying implementations are often deterministicBad: Code that Assumes a

Deterministic Implementation of a Non-deterministic Specification (

ADINS)Such code can behave unexpectedly when run using a different underlying implementation that still satisfies the specSuch code can be a cause of

flaky tests3Slide4

Library

NonDex (Non-Deterministic Explorer)

A simple technique for detecting ADINS code4

T

estMethod with Deterministic Speccallsoutputs

Method with Non-deterministic Spec

calls

outputs

…Slide5

Library

NonDex (Non-Deterministic Explorer)

A simple technique for detecting ADINS code5

NonDex

Modelcalls…Method with Deterministic Spec

calls

Implementation 1

Implementation 2

Implementation n…outputsNonDexoutputsTestSlide6

Library

NonDex (Non-Deterministic Explorer)

A simple technique for detecting ADINS code6

…

Method with Deterministic SpecExplore: Implementation 1 … Implementation n

T

est

NonDex

ModelImplementation 1Implementation 2Implementation n…NonDexoutputscallscallsoutputsSlide7

Finding Non-deterministic Specs

Searched Java Standard Library for method specs that are non-deterministicSearched JavaDocs for these keywords: “order”, “deterministic

”, “not specified”Searched for methods that return an arrayExamined JavaDocs for all found methodsIdentified

31 method specs that are non-deterministic

7Slide8

Some Non-deterministic Specs

java.util.Set.iterator():“Returns an iterator over the elements in this set. The elements are returned in no particular order…”

java.util.HashMap.entrySet():“Returns a Set view of the mappings contained in this map…”“This class makes no guarantees as to the order of the map

; in particular, it does not guarantee that the order will remain constant over time.”java.lang.Class.getDeclaredMethods

():“Returns an array containing Method objects reflecting all the declared methods of the class … The elements in the returned array are not sorted and are not in any particular order.”8Slide9

9

Class

MethodsCategory

java.lang.ObjecthashCode

Randomjava.util.HashMap*keySet, values, entrySetPermutejava.util.ConcurrentHashMap*keySet, values, entrySet, keys, elementsPermute

java.io.File

list,

listFiles, listRootsPermutejava.lang.ClassgetClasses, getDeclaredMethods, …Permutejava.lang.reflect.MethodgetParameterAnnotationsPermutejava.lang.reflect.FieldgetDeclaredAnnotationsPermutejava.text.DateFormatSymbolsgetAvailableLocales,getZoneStringsPermute, Extendjava.text.BreakIteratorgetAvailableLocalesPermutejava.text.DecimalFormatSymbolsgetAvailableLocalesPermutejava.text.NumberFormatgetAvailableLocalesPermutejava.text.DateFormatgetAvailableLocalesPermute* Non-determinism is actually in internal iterator, exposed to outside by these methodsNonDex Models for SpecsPermute: method returns array/collection whose order is not specifiedNonDex model explores different permutations of elementsSlide10

Different Levels of Non-determinism

Some assumptions of determinism may be “acceptable”For example, if two calls of iterator() are made on the same, unmodified Set, should the iteration orders be the same?We introduce four “levels” for NonDex to “Permute”

ONE: assumes deterministic implementation, but shuffles once, potentially different than underlying implementationEQ: shuffles differently only when objects are not “equal

”ID: shuffles differently only when object address is different, or object has been modified

FULL: shuffles differently with every call10Slide11

01.

Set<Integer> s =

new HashSet

<Integer

>();02. s.add(1); s.add(2);03. Integer[] a = s.toArray();

04.

//

assertArrayEquals

(a, new Integer[]{1, 2}); // FULL, ID, EQ, ONE can fail05.06. // assertArrayEquals(a, s.toArray()); // FULL can fail07.08. s.contains(1);09. // assertArrayEquals(a, s.toArray()); // FULL can fail10.11. s.add(3); s.remove(3);12. // assertArrayEquals(a, s.toArray());

// FULL, ID can fail13.14. Set<Integer> t = new HashSet<Integer>();15. t.add(1); t.add(2);16. // assertArrayEquals(a, t.toArray()); // FULL, ID can fail17.18. Set<Integer> u = new HashSet

<Integer>();

19.

u.add

(3);

u.add

(4);

20.

Integer

[] b =

u.toArray

();

21.

//

assertEquals

(a[0] < a[1], b[0] < b[1]);

//

FULL, ID, EQ can fail

11Slide12

NonDex Model Exploration

In our prototype, NonDex models use seeds to control which implementations to useModels use seeds for different java.util.Random calls that represent the different implementationsWe execute code with NonDex

multiple times, each using different seeds, to explore different behaviors12Slide13

Research Questions

How many flaky tests are due to ADINS code?In open-source projects?In student submissions?How many seeds should be used to likely find all flaky tests due to ADINS code?How often are developers making “acceptable” assumptions about non-determinism?

13Slide14

Experimental Setup

Tool (also called NonDex) is implemented for OpenJDK 8Models implemented by modifying code in OpenJDK 8Use modified

NonDex JVM in place of the OpenJDK 8 JVMEvaluation projects:195 open-source GitHub projects that build with MavenFirst run each project’s tests with 10 different seeds

If a test with different behavior is found, run project’s tests with 100 different seeds

72 student submissions from Software Engineering I courseRun each student submission’s tests with 100 different seeds14Slide15

Open-source Project Results

15

FULL

ID

EQONEFlaky Tests Detected60545454Total Seeds Detecting4362/60003744/60003643/6000

3590/6000

Min Seeds

Detecting

8000Max Seeds Detecting100100100100Slide16

16

Project

TestClass#TestName

FULL

IDEQONEreflectasmFieldAccessTest#testIndexSetAndGet48000joda-timeTestDateTimeZone#testGetShortName

35

53

53

53oryxTextUtilsTest#testJSONMap51526053visualeeJPAExaminerTest#testFindAndSetAttributes…8556commons-cliOptionGroupTest#testToString42000commons-langMultilineRecursiveToStringStyleTest#boolArray100100100100easy-batchPrinterTest#testPrettyPrinting69735453scribe-javaMapUtilsTest#shouldPrettyPrintMap97949797geoserver-managerGSLLayerEncoder21Test#testMetadata848171100handlebars.javaTagTypeTest#collectSectionAndVars100100100100jscep

DefaultCertStoreInspectorTest#example92945953junitMethodSorterTest#testJvmMethodSorter100000org-jsonTestSuite#testJSONStringerObject79778384slf4jEventLoggerTest#testEventLogger100000wsdocCollectionTests#testBasicSets100969184

Probability of

not

detecting in

10 runs is

(

1 −

)

10

= 0.434

Greater than 0.5 chance of

detecting

Subset of Flaky Tests DetectedSlide17

Example Flaky Test (from commons-cli)

17public

class OptionGroupTest

{

public void testToString() { OptionGroup g1 = new OptionGroup

();

g1.addOption(

new

Option(null, "foo", false, "Foo")); g1.addOption(new Option(null, "bar", false, "Bar")); if (!"[--bar Bar, --foo Foo]".equals(g1.toString())) { assertEquals("[--foo Foo, --bar Bar]", g1.toString()); } ...}}public class OptionGroup ... { Map om = new HashMap(); public OptionGroup

addOption(Option option) { om.put(option.getKey(), option); return this;} public String toString() { StringBuilder buff = new StringBuilder(); Iterator iter = getOptions().iterator(); buff.append("["); while (iter.hasNext

()) { /* ... populate buff with the values in

iter ... */

}

return

buff.toString

();}} Slide18

Student Submission Results

Flaky tests mostly due to asserting result of method call is exactly equal to some String (similar beginning example)We also used Java PathFinder (JPF) to do systematic explorationReplace java.util.Random calls with JPF choice pointsUsing JPF choice points did not detect any more flaky tests than using

java.util.Random (with 100 seeds)18

FULL

IDEQONEFlaky Tests Found110883434Total Seeds Detected

8159/11000

6785/11000

2031/11000

1827/11000Min Seeds Detected37000Max Seeds Detected1001008178Slide19

Conclusion

Non-deterministic specs give freedom to implementers, but code that assumes deterministic implementations (ADINS) leads to undesirable behaviorWe develop a technique NonDex to detect ADINS codeWe detected 60 flaky tests in open-source projects, and 110 flaky tests in student submissionsWe plan to further research how to better detect, debug, and repair ADINS codeWe plan on releasing

NonDex, but are currently having issues with OpenJDK/Oracle licensing19

August Shi: awshi2@illinois.eduSlide20

BACKUP

20Slide21

21

class HashMap

{ ...

class

HashIterator { ... /* Rename hasNext() -> original_hasNext() */ /* Rename nextNode

() ->

original_nextNode

() */

Iterator<Node> NonDex_iter; HashIterator() { ... List<Node> original = new ArrayList<>(); while (original_hasNext()) original.add(original_nextNode()); NonDex.shuffle(original, (NonDex.level == ID) ? System.identityHashCode(HashMap.this) + modCount : (NonDex.level == EQ) ? HashMap.this.hashCode() : 0); NonDex_iter = original.iterator(); } public

final boolean hasNext() { return NonDex_iter.hasNext(); } final Node nextNode() { if (modCount != expectedModCount) throw new ConcurrentModificationException(); current = NonDex_iter.next(); return current; } }

}Slide22

22

class NonDex

{ static

int

level; // FULL, ID, EQ, or ONE int seed = ...; static Random

full

=

new

Random(seed); public static List shuffle(List l, int v) { int size = l.size(); Random rand = (level == FULL) ? full : // Full (level == ID) ? new Random(seed + v) : // Same object (level == EQ) ? new Random(seed + v) : // Equal object (level == ONE) ? new Random(seed); // Once for (int i = 0; i

< size - 1; i++) { int s = rand.getNext(i, size); if (s == i) continue; T obj = l.get(i); l.set(i, l.get(s)); l.set(s, obj); } return l; }} Slide23

Example Flaky Test (from scribe-java)

23public

class MapUtilsTest

{

@Test public void shouldPrettyPrintMap() { Map map =

new

HashMap

<>(); map.put(1, "one"); map.put(2, "two"); map.put(3, "three"); map.put(4, "four"); assertEquals( "{ 1 -> one , 2 -> two , 3 -> three , 4 -> four }", MapUtils.toString(map)); }}public class MapUtils { public static String toString(Map map) { ... StringBuilder result = new

StringBuilder(); for (Map.Entry entry : map.entrySet()) { result.append(String.format(", %s -> %s ", entry.getKey().toString(), entry.getValue().toString())); } return "{" + result.substring(1) + "}"; }}