feat: Add logic to verify xml results #1362

adamfilipow92 · 2020-11-27T17:44:34Z

Fixes #1315

Test Plan

How do we know the code works?

run ./gradlew integrationTests
integration tests should pass

Description

Flank should also verify XML results for android and ios integration tests.

This ticket should be merged after #1316 when we update all tests.

filter all tests - ios is disabled and should be updated in #1388

Checklist

Add verification for basic android test
Add verification for tests from test: [IT] Add test cases #1316

… android test

codecov-io · 2020-11-27T17:49:58Z

Codecov Report

Merging #1362 (1c6c083) into master (9d1ba8d) will increase coverage by 0.02%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #1362      +/-   ##
============================================
+ Coverage     77.56%   77.58%   +0.02%     
  Complexity      717      717              
============================================
  Files           248      248              
  Lines          4788     4788              
  Branches        915      915              
============================================
+ Hits           3714     3715       +1     
  Misses          572      572              
+ Partials        502      501       -1

github-actions · 2020-11-27T17:51:48Z

Timestamp: 2020-12-11 12:58:58
Buildscan url for ubuntu-workflow run 415427673
https://gradle.com/s/vf5wovfx6vsbg

piotradamczyk5 · 2020-12-10T12:39:05Z

integration_tests/src/test/kotlin/integration/AllTestFilteredIT.kt

@@ -29,6 +30,7 @@ class AllTestFilteredIT {
    }

    @Test
+    @Ignore("Should be fixed in https://github.com/Flank/flank/issues/1388")


should we ignore this test for all OS ?

only for ios, android works but output could change after #1388

I wonder if this will be picked up and removed in that ticket. Can we add a // TODO with a ticket number

piotradamczyk5 · 2020-12-10T12:42:14Z

integration_tests/src/test/resources/flank_ios.yml

@@ -1,3 +1,6 @@
 gcloud:
  test: ../test_runner/src/test/kotlin/ftl/fixtures/tmp/earlgrey_example.zip
  xctestrun-file: ../test_runner/src/test/kotlin/ftl/fixtures/tmp/EarlGreyExampleSwiftTests_iphoneos13.4-arm64e.xctestrun
+  device:
+    - model: iphone8
+      version: 13.6


should we force to use this device?

I think we should use the default device. Thanks! Changed

jan-goral

The code looks ok, but IT tests are failing on my local run. I guess it's related to my local configuration rather than the branch.

https://gist.github.com/jan-gogo/5b52b7a2c6fc0d085175bfa0b2b749c7

pawelpasterz · 2020-12-11T11:00:37Z

integration_tests/src/test/resources/compare/RunTimeoutIT-compare

@@ -65,16 +65,5 @@ RunTests
 Saved 1 shards to android_shards.json
  Uploading app-debug.apk \.*
  Uploading app-single-success-debug-androidTest.apk \.*
-  1 test / 1 shard


Why those lines are removed?
I launched it locally and the timeout case failed because of it 🤔

Locally I get the output without removed lines.

I fixed this test by using test and apk from gcs. Using local files can produce different outputs related to file uploading speed.

Sloox · 2020-12-11T12:17:14Z

integration_tests/src/test/kotlin/utils/OutputHelper.kt

+
+fun TestSuites.assertCountOfSkippedTests(expectedCount: Int) = assertEquals(expectedCount, testSuites.sumBy { it.skipped })
+
+fun TestSuites.assertCountOfFailedTests(expectedCount: Int) = assertEquals(expectedCount, testSuites.count { it.failures > 0 })


does detekt not pick this up as too long? can you run it?

Sloox · 2020-12-11T12:18:47Z

integration_tests/src/test/kotlin/integration/AllTestFilteredIT.kt

@@ -29,6 +30,7 @@ class AllTestFilteredIT {
    }

    @Test
+    @Ignore("Should be fixed in https://github.com/Flank/flank/issues/1388")


I wonder if this will be picked up and removed in that ticket. Can we add a // TODO with a ticket number

Sloox · 2020-12-11T12:19:23Z

integration_tests/src/test/kotlin/utils/LoadTestSuites.kt

+import utils.testResults.TestSuites
+import java.io.File
+
+fun File.loadAsTestSuite(): TestSuites = XmlMapper().registerModule(KotlinModule()).readValue(this, TestSuites::class.java)


Again with long lines and detekt failing. Do you have githooks installed?

Yes, I have installed it.
Strange. To be sure I run detekt and check and both pass.

Sloox · 2020-12-11T12:19:35Z

integration_tests/src/test/kotlin/utils/OutputHelper.kt

+import java.io.File
+import java.nio.file.Paths
+
+fun String.findTestDirectoryFromOutput() = "results-dir:\\s.*\\s".toRegex().find(this)?.value.orEmpty().trim().replace("results-dir: ", "")


jan-goral

changed to request changes

jan-goral

One test was failed on my local run

https://gist.github.com/jan-gogo/9e2f4d96f05f1efc3a1831bd1709d95e

[EDIT]
...due to Insufficient test quota

adamfilipow92 added 2 commits November 26, 2020 19:40

Deserialize testsuite

60c5777

Added asserts for test results, fixed ios regex, add asserts to basic…

3700c05

… android test

adamfilipow92 self-assigned this Nov 27, 2020

github-actions bot assigned piotradamczyk5 Nov 27, 2020

github-actions bot added Dragon 🐲 Feature labels Nov 27, 2020

Merge branch 'master' into 1315-it-verify-xml-results

09f9eda

bootstraponline changed the title ~~feat: Add logic to veryfi xml results~~ feat: Add logic to verify xml results Nov 30, 2020

adamfilipow92 added Tiger 🐯 and removed Dragon 🐲 labels Nov 30, 2020

adamfilipow92 and others added 10 commits November 30, 2020 12:10

Merge branch 'master' into 1315-it-verify-xml-results

f5f3725

Merge branch 'master' into 1315-it-verify-xml-results

d6c0853

assert weblinks from xml result

c620396

Merge branch 'master' into 1315-it-verify-xml-results

a2e96bf

Add veryfication of xml results for Sanity, TestFIltering, RunTImeOut

9be7437

Validate passed tests

1556b5b

Update SanityRoboIT.kt

070d80c

Update AllTestFilteredIT.kt

a4f3584

Merge branch 'master' into 1315-it-verify-xml-results

1f1bdf0

Update RunTimeoutIT-compare

ee57289

adamfilipow92 marked this pull request as ready for review December 10, 2020 11:36

adamfilipow92 requested review from pawelpasterz, Sloox, jan-goral, zuziaka, piotradamczyk5 and bootstraponline December 10, 2020 11:36

piotradamczyk5 reviewed Dec 10, 2020

View reviewed changes

Update flank_ios.yml

46840bb

jan-goral reviewed Dec 11, 2020

View reviewed changes

pawelpasterz reviewed Dec 11, 2020

View reviewed changes

Fix run time out test

87c4337

Sloox suggested changes Dec 11, 2020

View reviewed changes

Fix styling

1c6c083

Sloox approved these changes Dec 11, 2020

View reviewed changes

jan-goral reviewed Dec 11, 2020

View reviewed changes

jan-goral suggested changes Dec 11, 2020

View reviewed changes

jan-goral approved these changes Dec 11, 2020

View reviewed changes

mergify bot merged commit 354213d into master Dec 11, 2020

mergify bot deleted the 1315-it-verify-xml-results branch December 11, 2020 15:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add logic to verify xml results #1362

feat: Add logic to verify xml results #1362

adamfilipow92 commented Nov 27, 2020 •

edited

Loading

codecov-io commented Nov 27, 2020 •

edited

Loading

github-actions bot commented Nov 27, 2020 •

edited

Loading

piotradamczyk5 Dec 10, 2020

adamfilipow92 Dec 10, 2020

Sloox Dec 11, 2020

piotradamczyk5 Dec 10, 2020

adamfilipow92 Dec 10, 2020

jan-goral left a comment

pawelpasterz Dec 11, 2020

adamfilipow92 Dec 11, 2020

adamfilipow92 Dec 11, 2020

Sloox Dec 11, 2020

Sloox Dec 11, 2020

Sloox Dec 11, 2020

adamfilipow92 Dec 11, 2020

Sloox Dec 11, 2020

jan-goral left a comment •

edited

Loading

jan-goral left a comment •

edited

Loading


		fun TestSuites.assertCountOfSkippedTests(expectedCount: Int) = assertEquals(expectedCount, testSuites.sumBy { it.skipped })

		fun TestSuites.assertCountOfFailedTests(expectedCount: Int) = assertEquals(expectedCount, testSuites.count { it.failures > 0 })

feat: Add logic to verify xml results #1362

feat: Add logic to verify xml results #1362

Conversation

adamfilipow92 commented Nov 27, 2020 • edited Loading

Test Plan

Description

Checklist

codecov-io commented Nov 27, 2020 • edited Loading

Codecov Report

github-actions bot commented Nov 27, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jan-goral left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jan-goral left a comment • edited Loading

Choose a reason for hiding this comment

jan-goral left a comment • edited Loading

Choose a reason for hiding this comment

adamfilipow92 commented Nov 27, 2020 •

edited

Loading

codecov-io commented Nov 27, 2020 •

edited

Loading

github-actions bot commented Nov 27, 2020 •

edited

Loading

jan-goral left a comment •

edited

Loading

jan-goral left a comment •

edited

Loading