Use Firebase Test Lab (PoC) #3929
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
markushi
requested review from
adinauer,
romtsn and
stefanosiano
as code owners
|
Performance metrics 🚀
|
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 6aacee6 | 410.71 ms | 479.32 ms | 68.60 ms |
| d983fe1 | 420.68 ms | 425.39 ms | 4.71 ms |
| f60000a | 425.20 ms | 495.02 ms | 69.82 ms |
| a4e14b9 | 418.86 ms | 435.02 ms | 16.16 ms |
App size
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 6aacee6 | 1.70 MiB | 2.35 MiB | 669.38 KiB |
| d983fe1 | 1.70 MiB | 2.35 MiB | 669.38 KiB |
| f60000a | 1.70 MiB | 2.35 MiB | 669.38 KiB |
| a4e14b9 | 1.70 MiB | 2.35 MiB | 669.38 KiB |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| run: make assembleUiTests | ||
|
|
||
| - name: Install gcloud | ||
| uses: 'google-github-actions/setup-gcloud@6189d56e4096ee891640bb02ac264be376592d6a' # pin 2.1.2 |
There was a problem hiding this comment.
- 🚫 Please pin the action by specifying a commit SHA instead of a tag/branch.
stefanosiano
reviewed
Dec 3, 2024
| @@ -90,7 +90,7 @@ class AutomaticSpansTest : BaseUiTest() { | |||
| // AGP matrix tests have no frames | |||
| Assume.assumeTrue(totalFrames > 0) | |||
| assertNotEquals(totalFrames, 0) | |||
| assertTrue(totalFrames > slowFrames + frozenFrames, "Expected total frames ($totalFrames) to be higher than the sum of slow ($slowFrames) and frozen ($frozenFrames) frames.") | |||
| assertTrue(totalFrames >= slowFrames + frozenFrames, "Expected total frames ($totalFrames) to be higher than the sum of slow ($slowFrames) and frozen ($frozenFrames) frames.") | |||
There was a problem hiding this comment.
Suggested change
| assertTrue(totalFrames >= slowFrames + frozenFrames, "Expected total frames ($totalFrames) to be higher than the sum of slow ($slowFrames) and frozen ($frozenFrames) frames.") | |
| assertTrue(totalFrames > slowFrames + frozenFrames, "Expected total frames ($totalFrames) to be higher than the sum of slow ($slowFrames) and frozen ($frozenFrames) frames.") |
This test checks there is at least 1 normal frame that is summed to slow + frozen frames. That's why the test should be "long" enough to have at least 1 normal frame even on slower devices
Otherwise total frames is always at least slow + frozen, so it wouldn't make sense to have it, and it could be removed directly
stefanosiano
reviewed
Dec 3, 2024
| @@ -126,7 +126,7 @@ class AutomaticSpansTest : BaseUiTest() { | |||
| // AGP matrix tests have no frames | |||
| Assume.assumeTrue(totalFrames > 0) | |||
| assertNotEquals(totalFrames, 0) | |||
| assertTrue(totalFrames > slowFrames + frozenFrames, "Expected total frames ($totalFrames) to be higher than the sum of slow ($slowFrames) and frozen ($frozenFrames) frames.") | |||
| assertTrue(totalFrames >= slowFrames + frozenFrames, "Expected total frames ($totalFrames) to be higher than the sum of slow ($slowFrames) and frozen ($frozenFrames) frames.") | |||
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
📜 Description
We're frequently running into timeouts on Saucelabs, let's see if Firebase Test Lab is any better.
💡 Motivation and Context
Stable CI
💚 How did you test it?
#skip-changelog
📝 Checklist
sendDefaultPIIis enabled.🔮 Next steps