Flank Scripts 1.9.11
Pre-release
Pre-release
·
454 commits
to master
since this release
test: Add 3dmark instrumented test (#1783) Fixes #1771 The instrumented test based on ui-automator should run a benchmark on the tested device. The benchmark must be installed on the tested device. ## Goal Running the benchmark on FTL device and obtaining the benchmark results. ## Config https://github.com/Flank/flank/blob/1771_ftl_benchmarks/test_runner/src/test/kotlin/ftl/fixtures/benchmark.yml ## Details * The benchmark `apk` file is specified using `additional-apks` option * The 3dmark benchmark results will be pulled from device dir `/sdcard/Android/data/com.futuremark.dmandroid.application/files/3DMarkAndroid/` ## Test Plan > How do we know the code works? copy * `Geekbench 5_v5.3.2.apk` * `3dmarkandroid-v2-1-4726.apk` to `test_artifacts/1771_ftl_benchmarks/apk/benchmark/` ```bash . .env buildFlankScripts flankScripts assemble app --artifact benchmark cd test_runner flank android run -c="./src/test/kotlin/ftl/fixtures/benchmark/*.yml" ``` # Devices * Pixel 5e (physical) - API 30 * NexusLowRes (virtual) - API 30 * NexusLowResEmulator (emulator) - API 30 # Results ## 3dmarkandroid-v2-1-4726.apk * Pixel 5e (physical) - API 30 - [passed](https://gist.github.com/jan-gogo/794a01631645f7a1549f82334eb2fba4) https://github.com/Flank/flank/pull/1783/commits/43dfc014fc3f23c309aa81de8ac0af104b104a0c [video](https://console.firebase.google.com/project/flank-open-source/testlab/histories/bh.da0c237aaa33732/matrices/4704820152614748057/executions/bs.458fb96e63f99c51/videos) * NexusLowRes (virtual) - API 30 - [failed](https://gist.github.com/jan-gogo/e3e4e7d80b1b65d550300c1ecb75498b) https://github.com/Flank/flank/pull/1783/commits/180e154e43546958d007417b4ab3bdb879880786 * NexusLowResEmulator (emulator) - API 30 - [freezed](https://gist.github.com/jan-gogo/683f1477b876649416d4366f6d1b005f) https://github.com/Flank/flank/pull/1783/commits/bc1a2f6a13b55e2cb6e3d0e6e234311b6eb2c942 ## Geekbench - [all passed](https://gist.github.com/jan-gogo/64700f34a5f7511aeecffe6da44a7d07) https://github.com/Flank/flank/pull/1783/commits/836e6221df89f996887a1fa6bcdf21d31c2e4a52 [firebase](https://console.firebase.google.com/project/flank-open-source/testlab/histories/bh.da0c237aaa33732/matrices/7399025084907015803) * Pixel 5e (physical) - API 30 - [video](https://console.firebase.google.com/project/flank-open-source/testlab/histories/bh.da0c237aaa33732/matrices/7399025084907015803/executions/bs.e9ea758006f16bee/videos) * NexusLowRes (virtual) - API 30 - [video](https://console.firebase.google.com/project/flank-open-source/testlab/histories/bh.da0c237aaa33732/matrices/7399025084907015803/executions/bs.7ee7ba87833f43ab/videos) * NexusLowResEmulator (emulator) - API 30 - [video](https://console.firebase.google.com/project/flank-open-source/testlab/histories/bh.da0c237aaa33732/matrices/7399025084907015803/executions/bs.2d604e5ac73159aa/videos) ## PassMark * NexusLowRes (virtual) - API 30 * NexusLowResEmulator (emulator) - API 30 - starting but failing during the benchmark - [video](https://console.firebase.google.com/project/flank-open-source/testlab/histories/bh.da0c237aaa33732/matrices/5356711449268917714/executions/bs.a612c1ef9c1df07/videos) https://github.com/Flank/flank/pull/1783/commits/b239766ffa471ab4055522aa992227e44302def9 ## Checklist - [ ] Documented