Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix flaky test RefactorTest #47

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

jcoben
Copy link

@jcoben jcoben commented Oct 19, 2023

What is the purpose of this PR

  • This PR fixes the error resulting from the flaky test: com.insightfullogic.java8.examples.chapter3.RefactorTest#allStringJoins
  • The mentioned test may fail or pass without changes made to the source code when it is run in different JVMs due to HashSet's non-deterministic iteration order.
  • The NonDex tool detects possible issues with "under-determined Java APIs." This means that this problem may come up if in the future someone tries to run the code in a different environment (JVM version, operating system, etc.). Fixing the test now will prevent developers from spending time on this in the future.

Why the tests fail

  • This test fails because the test requests a set of strings returned from code in the Refactor class. This code returns a HashSet containing the string data. The test compares this result to a hard-coded string value. The problem is that the order of iteration is not guaranteed for HashSets. So, the test may fail as the given order can be different from the expected order.

Reproduce the test failure

  • Run the tests with NonDex maven plugin. The commands to recreate the flaky test failures are:
    mvn edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=com.insightfullogic.java8.examples.chapter3.RefactorTest#allStringJoins

  • To run the test without NonDex:
    mvn test -Dtest=com.insightfullogic.java8.examples.chapter3.RefactorTest#allStringJoins

Expected results

  • This test should run successfully when run with NonDex.

Actual Result

  • We get the following failure:
-------------------------------------------------------------------------------
Test set: com.insightfullogic.java8.examples.chapter3.RefactorTest
-------------------------------------------------------------------------------
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.029 sec <<< FAILURE! - in com.insightfullogic.java8.examples.chapter3.RefactorTest
allStringJoins(com.insightfullogic.java8.examples.chapter3.RefactorTest)  Time elapsed: 0.028 sec  <<< FAILURE!
org.junit.ComparisonFailure: expected:<[[Acknowledgement, Resolution]]> but was:<[[Resolution, Acknowledgement]]>
	at org.junit.Assert.assertEquals(Assert.java:115)
	at org.junit.Assert.assertEquals(Assert.java:144)
	at com.insightfullogic.java8.examples.chapter3.RefactorTest.lambda$allStringJoins$0(RefactorTest.java:38)

Description of Fix

  • I added some code to convert the HashSet into a list and then used Collections.sort() to sort it before comparing to the hard-coded string value.

This test was passing when run on its own but flaky when run with other
tests.

Discovered with the nondex tool: https://github.com/TestingResearchIllinois/NonDex
Ran test on its own with command: mvn test -Dtest=com.insightfullogic.java8.examples.chapter3.RefactorTest#allStringJoins
Ran tests in groups with command: mvn edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=com.insightfullogic.java8.examples.chapter3.RefactorTest#allStringJoins

Fixed test by adding some code to sort the HashSet before comparing it
to a hard-coded string.
@jcoben jcoben changed the title [SWE637 FA23] Fix flaky test RefactorTest Fix flaky test RefactorTest Oct 19, 2023
@aswin-devil
Copy link

feab683

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants