Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Fix flaky test in LogUtilTest.groovy #92

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

KiruthikaJanakiraman
Copy link

I confirm that this contribution is made under the terms of the license found in the root directory of this repository's source tree and that I have the authority necessary to make this contribution on behalf of its copyright owner.

Problem

This PR aims to fix the flaky test: com.yahoo.parsec.logging.LogUtilTest."test generateLog with custom data object should return info contain custom json data"

def "test generateLog with custom data object should return info contain custom json data"() {
when:
def result = LogUtil.generateLog(null, null, null, new SimpleDataObject("key1", "value1"))
then:
result.matches(/.* timestamp=".+", log_data=\{"key":"key1","value":"value1"\}/)
}

The above check fails as the order of the items in log_data varies upon non-determintic shuffling.

I found and confirmed the flaky behaviour of the test using an open-source research tool NonDex, which shuffles implementations of nondeterminism operations.

Solution

This PR fixes the test by checking for both possibilities of log_data.

result.matches(/.* timestamp=".+", log_data=\{"key":"key1","value":"value1"\}/) || result.matches(/.* timestamp=".+", log_data=\{"value":"value1","key":"key1"\}/)

Steps to reproduce

The following command can be used to reproduce the assertion errors and verify the fix.

Integrate NonDex:

Add the following snippet to the top of the build.gradle in $PROJ_DIR:

plugins {
    id 'edu.illinois.nondex' version '2.1.1-1'
}

Append to build.gradle in $PROJ_DIR:

apply plugin: 'edu.illinois.nondex'

Execute Test with Gradle:

./gradlew test --info parsec-logging:test --tests=com.yahoo.parsec.logging.LogUtilTest."test generateLog with custom data object should return info contain custom json data"

Run NonDex:

./gradlew test --info parsec-logging:nondexTest --tests=com.yahoo.parsec.logging.LogUtilTest."test generateLog with custom data object should return info contain custom json data" --nondexRuns=50

Test Environment:

Java version "1.8.0_381"
macOS Venture Version 13.4.1 (22F82)

Please let me know if you have any concerns or questions.

@KiruthikaJanakiraman KiruthikaJanakiraman changed the title fix flaky test in LogUtilTest.groovy Fix flaky test in LogUtilTest.groovy Nov 9, 2023
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant