[Android] Run CoreCLR functional tests on Android #112283

kotlarmilos · 2025-02-07T18:11:34Z

Description

This PR introduces a new job to runtime and runtime-extra-platforms for running CoreCLR functional tests on the Android emulator.

dotnet-policy-service · 2025-02-07T18:12:06Z

Tagging subscribers to this area: @hoyosjs
See info in area-owners.md if you want to be subscribed.

kotlarmilos · 2025-02-07T18:12:19Z

/azp run runtime-androidemulator

azure-pipelines · 2025-02-07T18:12:29Z

Azure Pipelines successfully started running 1 pipeline(s).

kotlarmilos · 2025-02-07T20:52:38Z

/azp run runtime-androidemulator

azure-pipelines · 2025-02-07T20:52:49Z

Azure Pipelines successfully started running 1 pipeline(s).

kotlarmilos · 2025-02-07T21:33:11Z

/azp run runtime-androidemulator

azure-pipelines · 2025-02-07T21:33:19Z

Azure Pipelines successfully started running 1 pipeline(s).

kotlarmilos · 2025-02-10T12:13:14Z

/azp run runtime-androidemulator

azure-pipelines · 2025-02-10T12:13:25Z

Azure Pipelines successfully started running 1 pipeline(s).

kotlarmilos · 2025-02-10T15:40:44Z

/azp run runtime-androidemulator

azure-pipelines · 2025-02-10T15:40:53Z

Azure Pipelines successfully started running 1 pipeline(s).

kotlarmilos · 2025-02-10T16:38:31Z

/azp run runtime-androidemulator

azure-pipelines · 2025-02-10T16:38:41Z

Azure Pipelines successfully started running 1 pipeline(s).

ivanpovazan

LGTM! I left couple of minor notes, thanks!

eng/pipelines/extra-platforms/runtime-extra-platforms-androidemulator.yml

ivanpovazan · 2025-02-10T18:02:40Z

eng/pipelines/runtime.yml

+          jobParameters:
+            testGroup: innerloop
+            nameSuffix: AllSubsets_CoreCLR
+            buildArgs: -s clr.runtime+clr.alljits+clr.corelib+clr.nativecorelib+clr.tools+clr.packages+libs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true /p:RunSmokeTestsOnly=true


Should we also include host+packs subsets?

ivanpovazan · 2025-02-10T18:04:36Z

src/libraries/tests.proj

@@ -723,7 +723,7 @@
                      BuildInParallel="false" />
  </ItemGroup>

-  <ItemGroup Condition="'$(ArchiveTests)' == 'true' and '$(RunSmokeTestsOnly)' != 'true' and '$(RunGrpcTestsOnly)' != 'true' and '$(TargetOS)' == 'android'">
+  <ItemGroup Condition="'$(ArchiveTests)' == 'true' and '$(RunSmokeTestsOnly)' != 'true' and '$(RunGrpcTestsOnly)' != 'true' and '$(TargetOS)' == 'android' and '$(RuntimeFlavor)' == 'Mono'">


Why are we removing this condition?

We are adding the condition to separate mono and coreclr ProjectReference.

ivanpovazan · 2025-02-10T18:06:48Z

src/libraries/tests.proj

@@ -759,6 +759,12 @@
      AdditionalProperties="%(AdditionalProperties);_IsPublishing=true;Configuration=Release;PublishDir=$(XUnitLogCheckerLibrariesOutDir);ROOTFS_DIR=$(ROOTFS_DIR);RuntimeIdentifier=$(OutputRID)" />
  </ItemGroup>

+  <ItemGroup Condition="'$(ArchiveTests)' == 'true' and '$(TargetOS)' == 'android' and '$(RuntimeFlavor)' == 'CoreCLR'">
+    <ProjectReference Remove="@(ProjectReference)" />


I suppose this is needed since we remove the '$(RuntimeFlavor)' == 'Mono'" condition above, if we kept the condition above, could we just include the Android.Device_Emulator.JIT.Test.csproj here?

The tests are included by default in:

<ProjectReference Include="$(MSBuildThisFileDirectory)*\tests\**\*.Tests.csproj" Exclude="@(ProjectExclusions)" Condition="'$(TestAssemblies)' == 'true'" BuildInParallel="$(BuildTestInParallel)" />

To exclude the default set of tests, we set TestAssemblies to false.

…mulator.yml Co-authored-by: Ivan Povazan <55002338+ivanpovazan@users.noreply.github.com>

kotlarmilos · 2025-02-11T21:40:55Z

/azp run runtime-androidemulator

azure-pipelines · 2025-02-11T21:41:04Z

Azure Pipelines successfully started running 1 pipeline(s).

* main: [Android] Run CoreCLR functional tests on Android (dotnet#112283) [LoongArch64] Fix some assertion failures for Debug ILC building Debug NativeAOT testcases. (dotnet#112229) Fix suspicious code fragments (dotnet#112384) `__ComObject` doesn't support dynamic interface map (dotnet#112375) Native DLLs: only load imported DLLs from System32 (dotnet#112359) [main] Update dependencies from dotnet/roslyn (dotnet#112314) Update SVE instructions that writes to GC regs (dotnet#112389) Bring up android+coreclr windows build. (dotnet#112256) Never use heap for return buffers (dotnet#112060) Wait to complete the test before releasing the agile reference. (dotnet#112387) Prevent returning disposed HTTP/1.1 connections to the pool (dotnet#112383) Fingerprint dotnet.js if writing import map to html is enabled (dotnet#112407) Remove duplicate definition of CORECLR_HOSTING_API_LINKAGE (dotnet#112096) Update the exception message to reflect current behavior. (dotnet#112355) Use enum for frametype not v table (dotnet#112166) Enable AltJits build for LoongArch64 and RiscV64 (dotnet#110282) Guard members of MonoType union & fix related bugs (dotnet#111645) Add optional hooks for debugging OpenSSL memory allocations (dotnet#111539) JIT: Optimize struct parameter register accesses in the backend (dotnet#110819) NativeAOT: Cover more opcodes in type preinitializer (dotnet#112073)

matouskozak

I'm very sorry for the late review, I was out of office and missed this PR when I came back.

matouskozak · 2025-02-12T17:40:24Z

eng/pipelines/extra-platforms/runtime-extra-platforms-androidemulator.yml

+    - android_x64
+    - android_arm64


Do we have arm64 emulators setup? I thought that by default these will go to devices, see

runtime/eng/pipelines/libraries/helix-queues-setup.yml

Lines 96 to 97 in a4f9ec2

- ${{ if in(parameters.platform, 'android_arm', 'android_arm64', 'linux_bionic_arm', 'linux_bionic_arm64') }}:

- Windows.11.Amd64.Android.Open

Comparing the logs between x64 and am64, I would say that the arm64 are run on devices https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-112283-merge-8bd807b9687245838c/Android.Device_Emulator.JIT.Test/1/console.36c43a9c.log?helixlogtype=result.

Yes, it seems that arm64 run on devices so we can move it to extra-platforms-android.

I'm also wondering, if we have the same runs in runtime.yml (which runs always), why we need them again in extra-platforms?

I think we can have just runtime.yml for now and when we include full libraries run we should move that to extra-platforms.

matouskozak · 2025-02-12T17:41:34Z

eng/pipelines/extra-platforms/runtime-extra-platforms-androidemulator.yml

+
+#
+# Android emulators
+# Build the whole product using CoreCLR and run libraries tests


Are these library tests? I thought that functional tests are usually under the runtime test suit.

Functional tests are under libraries. There might be smoke tests under the runtime tests.

Ah, I thought it was the reverse. The https://github.com/dotnet/runtime/blob/main/src/tests/FunctionalTests/Android/Device_Emulator/JIT/Android.Device_Emulator.JIT.Test.csproj is placed under src/tests so I got mistaken that it is part of runtime tests.

In case of smoke tests, I think these run functional tests and System.Runtime.Tests which comes from libraries.

matouskozak · 2025-02-12T17:42:28Z

eng/pipelines/runtime.yml

+          runtimeFlavor: coreclr
+          platforms:
+          - android_x64
+          - android_arm64


If these indeed run on devices (see my other comment), we need to make sure we have enough devices so that we can run them on PRs by default and not block other Android jobs.

How can we test the load?

I think we should monitor the Windows.11.Amd64.Android.Open queue in Grafana to see that the waiting times don't grow so that we don't slow down runtime pipeline. I'm not aware of any stress we have for this.

matouskozak · 2025-02-12T17:42:37Z

eng/pipelines/runtime.yml

@@ -964,6 +964,47 @@ extends:
                    eq(variables['monoContainsChange'], true),
                    eq(variables['isRollingBuild'], true))

+      #
+      # Android devices


Devices or emulators?

This should be x64 emulator and arm64 device.

Setup a new job

d1a6018

kotlarmilos added NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) NO-REVIEW Experimental/testing PR, do NOT review it area-Infrastructure-coreclr os-android labels Feb 7, 2025

kotlarmilos added this to the 10.0.0 milestone Feb 7, 2025

kotlarmilos self-assigned this Feb 7, 2025

Remove -cross argument on linux

0ff724c

Add android job to the runtime pipeline

6d38b25

Use android test exclusions

956e00d

Remove included projects

5a8042e

Remove unused attribute

63cd36e

kotlarmilos marked this pull request as ready for review February 10, 2025 16:38

kotlarmilos removed NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) NO-REVIEW Experimental/testing PR, do NOT review it labels Feb 10, 2025

kotlarmilos requested review from steveisok, akoeplinger and ivanpovazan February 10, 2025 16:38

ivanpovazan approved these changes Feb 10, 2025

View reviewed changes

kotlarmilos and others added 3 commits February 11, 2025 22:32

Update eng/pipelines/extra-platforms/runtime-extra-platforms-androide…

388c626

…mulator.yml Co-authored-by: Ivan Povazan <55002338+ivanpovazan@users.noreply.github.com>

Update build command to include host+packs

2cc3984

Don't include test assemblies in coreclr andorid jobs

13dbb57

kotlarmilos merged commit 7be653f into dotnet:main Feb 12, 2025
150 of 158 checks passed

matouskozak reviewed Feb 12, 2025

View reviewed changes

kotlarmilos mentioned this pull request Feb 12, 2025

[Android] Remove CoreCLR functional tests on Android from extra-platforms #112496

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Android] Run CoreCLR functional tests on Android #112283

[Android] Run CoreCLR functional tests on Android #112283

kotlarmilos commented Feb 7, 2025 •

edited

Loading

dotnet-policy-service bot commented Feb 7, 2025

kotlarmilos commented Feb 7, 2025

azure-pipelines bot commented Feb 7, 2025

kotlarmilos commented Feb 7, 2025

azure-pipelines bot commented Feb 7, 2025

kotlarmilos commented Feb 7, 2025

azure-pipelines bot commented Feb 7, 2025

kotlarmilos commented Feb 10, 2025

azure-pipelines bot commented Feb 10, 2025

kotlarmilos commented Feb 10, 2025

azure-pipelines bot commented Feb 10, 2025

kotlarmilos commented Feb 10, 2025

azure-pipelines bot commented Feb 10, 2025

ivanpovazan left a comment

ivanpovazan Feb 10, 2025

ivanpovazan Feb 10, 2025

kotlarmilos Feb 11, 2025

ivanpovazan Feb 10, 2025

kotlarmilos Feb 12, 2025

kotlarmilos commented Feb 11, 2025

azure-pipelines bot commented Feb 11, 2025

matouskozak left a comment

matouskozak Feb 12, 2025

kotlarmilos Feb 12, 2025

matouskozak Feb 12, 2025

matouskozak Feb 12, 2025

kotlarmilos Feb 12, 2025

matouskozak Feb 12, 2025 •

edited

Loading

matouskozak Feb 12, 2025

kotlarmilos Feb 12, 2025

matouskozak Feb 12, 2025

matouskozak Feb 12, 2025

kotlarmilos Feb 12, 2025

	- ${{ if in(parameters.platform, 'android_arm', 'android_arm64', 'linux_bionic_arm', 'linux_bionic_arm64') }}:
	- Windows.11.Amd64.Android.Open

[Android] Run CoreCLR functional tests on Android #112283

[Android] Run CoreCLR functional tests on Android #112283

Conversation

kotlarmilos commented Feb 7, 2025 • edited Loading

Description

dotnet-policy-service bot commented Feb 7, 2025

kotlarmilos commented Feb 7, 2025

azure-pipelines bot commented Feb 7, 2025

kotlarmilos commented Feb 7, 2025

azure-pipelines bot commented Feb 7, 2025

kotlarmilos commented Feb 7, 2025

azure-pipelines bot commented Feb 7, 2025

kotlarmilos commented Feb 10, 2025

azure-pipelines bot commented Feb 10, 2025

kotlarmilos commented Feb 10, 2025

azure-pipelines bot commented Feb 10, 2025

kotlarmilos commented Feb 10, 2025

azure-pipelines bot commented Feb 10, 2025

ivanpovazan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kotlarmilos commented Feb 11, 2025

azure-pipelines bot commented Feb 11, 2025

matouskozak left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matouskozak Feb 12, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kotlarmilos commented Feb 7, 2025 •

edited

Loading

matouskozak Feb 12, 2025 •

edited

Loading