Switch to selfhosted NativeAOT compiler #81205

MichalStrehovsky · 2023-01-26T07:10:59Z

This will compile the NativeAOT compiler with the LKG build of the NativeAOT compiler that the repo is building with.

At this point NativeAOT-compiled compiler is significantly better than the R2R+Trimmed+SingleFile-compiled combo we're shipping right now.

Time to compile hello world before: 2.3 seconds. Time to compile hello world after: 1.3 seconds.
ilc.exe size before: 31 MB. ilc.exe size after: 14 MB.

Cc @dotnet/ilc-contrib

ghost · 2023-01-26T07:11:11Z

Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas
See info in area-owners.md if you want to be subscribed.

Issue Details

This will compile the NativeAOT compiler with the LKG build of the NativeAOT compiler that the repo is building with.

At this point NativeAOT-compiled compiler is significantly better than the R2R+Trimmed+SingleFile-compiled combo we're shipping right now.

Time to compile hello world before: 2.3 seconds. Time to compile hello world after: 1.3 seconds.
ilc.exe size before: 31 MB. ilc.exe size after: 14 MB.

Cc @dotnet/ilc-contrib

Author:	MichalStrehovsky
Assignees:	MichalStrehovsky
Labels:	`area-NativeAOT-coreclr`
Milestone:	-

filipnavara · 2023-01-30T09:05:35Z

src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj

@@ -10,6 +10,7 @@
  <!-- BEGIN: Workaround for https://github.com/dotnet/runtime/issues/67742 -->
  <PropertyGroup Condition="'$(BuildingInsideVisualStudio)' != 'true'">
    <PublishDir>$(RuntimeBinDir)ilc-published/</PublishDir>
+    <NativeAotSupported Condition="'$(RuntimeIdentifier)' != 'win-x64' and '$(RuntimeIdentifier)' != 'linux-x64'">false</NativeAotSupported>


Why exclude macOS (and arm64)?

See the previous failed attempts: Directory.Build.targets(83,5): error : (NETCORE_ENGINEERING_TELEMETRY=Build) The passed-in TargetOS property value 'OSX' must be lowercase.

Directory.Build.targets became opinionated about how to spell OSX. The old ilc probably doesn't spell it right. I don't particularly care about osx in the scope of this PR - this is to speed up testing and we do the majority of testing on Linux and Windows.

I think we can broaden this pretty easily after we get it started and flush out the problems.

agocke

LGTM. An eventual improvement will be moving this to a test dependency so we don't have to rely on BuildingInsideVisualStudio

agocke · 2023-01-31T19:36:49Z

src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj

@@ -10,6 +10,7 @@
  <!-- BEGIN: Workaround for https://github.com/dotnet/runtime/issues/67742 -->
  <PropertyGroup Condition="'$(BuildingInsideVisualStudio)' != 'true'">
    <PublishDir>$(RuntimeBinDir)ilc-published/</PublishDir>
+    <NativeAotSupported Condition="'$(RuntimeIdentifier)' != 'win-x64' and '$(RuntimeIdentifier)' != 'linux-x64'">false</NativeAotSupported>


I think we can broaden this pretty easily after we get it started and flush out the problems.

agocke · 2023-01-31T19:38:35Z

src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj

-    <PublishSingleFile>true</PublishSingleFile>
+    <NativeAotSupported Condition="'$(RuntimeIdentifier)' != 'win-x64' and '$(RuntimeIdentifier)' != 'linux-x64'">false</NativeAotSupported>
+    <PublishAot Condition="'$(NativeAotSupported)' == 'true'">true</PublishAot>
+    <PublishReadyToRun Condition="'$(NativeAotSupported)' != 'true'">true</PublishReadyToRun>


Consider moving this to a separate ItemGroup with a top-level NativeAOTSupported condition.

This is already in a conditioned itemgroup so I would need to repeat the condition. Given this is eventually going away in favor of a live build, maybe it's fine.

test failures

agocke · 2023-01-31T19:41:20Z

Ah, just looked at the test failures. Looks like the configuration is not correct somewhere -- we're passing --target incorrectly for both gcc and freebsd.

MichalStrehovsky · 2023-02-14T03:28:21Z

@am11 FYI - using gcc with PublishAot seems broken for crossbuild. I tried working around in this PR, but that ran into the issue that this is not using the live Microsoft.NETCore.Native.Unix.targets and my workaround is a noop.

I'm going to meditate on this some more because I don't know what the workaround for the gcc legs would be (I guess we could fix the freebsd one by checking TargetOS).

am11 · 2023-02-14T03:53:27Z

@am11 FYI - using gcc with PublishAot seems broken for crossbuild.

Yes, it is a known issue. Cross compilation doesn't work with gcc #78559. Unlike clang, gcc is arch-specific and doesn't support -target arg.

This will compile the NativeAOT compiler with the LKG build of the NativeAOT compiler that the repo is building with. At this point NativeAOT-compiled compiler is significantly better than the R2R+Trimmed+SingleFile-compiled combo we're shipping right now. ilc.exe size before: 31 MB. ilc.exe size after: 14 MB. Time to compile hello world before: 2.3 seconds. Time to compile hello world after: 1.3 seconds.

MichalStrehovsky · 2023-03-08T03:54:31Z

/azp run runtime-extra-platforms

azure-pipelines · 2023-03-08T03:54:57Z

Azure Pipelines successfully started running 1 pipeline(s).

MichalStrehovsky · 2023-03-09T03:05:03Z

@dotnet/ilc-contrib This is ready for review.

src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj

MichalStrehovsky · 2023-03-09T09:14:52Z

src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj

-    <!-- Don't R2R on ARM64 machines because ARM64 crossgen2 that comes with .NET SDK <= 7.0 Preview 7 crashes.-->
-    <PublishReadyToRun Condition="'$(BuildArchitecture)' != 'arm64'">true</PublishReadyToRun>
-    <PublishSingleFile>true</PublishSingleFile>
+    <NativeAotSupported Condition="'$(TargetOS)' != 'windows' and '$(TargetOS)' != 'linux'">false</NativeAotSupported>


#81205 (comment)

src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj

vitek-karas

I can't comment on the viability of osx, but the rest of the change looks good.

MichalStrehovsky · 2023-03-09T12:22:52Z

/azp run runtime-extra-platforms

azure-pipelines · 2023-03-09T12:23:18Z

Azure Pipelines successfully started running 1 pipeline(s).

MichalStrehovsky · 2023-03-09T23:31:01Z

I can't comment on the viability of osx, but the rest of the change looks good.

Looks like the osx issue was also fixed in the latest SDK drop so I enabled it.

eerhardt · 2023-03-10T21:42:59Z

We see a nice drop in our benchmarks "Build Time", likely due to this change:

MichalStrehovsky · 2023-03-11T03:03:08Z

We see a nice drop in our benchmarks "Build Time", likely due to this change:

Went better than expected!

ghost assigned MichalStrehovsky Jan 26, 2023

dotnet-issue-labeler bot added the area-NativeAOT-coreclr label Jan 26, 2023

filipnavara reviewed Jan 30, 2023

View reviewed changes

agocke previously approved these changes Jan 31, 2023

View reviewed changes

MichalStrehovsky added 4 commits February 24, 2023 14:59

We need R2R sometimes

d051626

Update ILCompiler.csproj

3ed990c

try fixing freebsd

b07c648

MichalStrehovsky force-pushed the selfhostilc branch from 58199fe to b07c648 Compare February 24, 2023 06:13

MichalStrehovsky mentioned this pull request Mar 2, 2023

Implement correct handling of new constraint in ilc #82818

Merged

MichalStrehovsky added 2 commits March 7, 2023 17:56

Merge branch 'main' into selfhostilc

7265116

Update ILCompiler.csproj

6e34224

build-analysis bot mentioned this pull request Mar 7, 2023

Tracking issue for CI build timeouts #76454

Closed

Update ILCompiler.csproj

5bbc2b7

vitek-karas reviewed Mar 9, 2023

View reviewed changes

src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj Show resolved Hide resolved

MichalStrehovsky commented Mar 9, 2023

View reviewed changes

src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj Outdated Show resolved Hide resolved

Apply suggestions from code review

3911bb1

vitek-karas approved these changes Mar 9, 2023

View reviewed changes

MichalStrehovsky merged commit 434d664 into dotnet:main Mar 9, 2023

MichalStrehovsky deleted the selfhostilc branch March 9, 2023 23:30

build-analysis bot mentioned this pull request Mar 10, 2023

[release/6.0] Doublelinklist GC failures on Mono #83245

Closed

mthalman mentioned this pull request Mar 20, 2023

Failure when building self-hosted NativeAOT compiler changes with 8.0 Preview 2 SDK #83695

Closed

ghost locked as resolved and limited conversation to collaborators Apr 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch to selfhosted NativeAOT compiler #81205

Switch to selfhosted NativeAOT compiler #81205

MichalStrehovsky commented Jan 26, 2023

ghost commented Jan 26, 2023

filipnavara Jan 30, 2023

MichalStrehovsky Jan 30, 2023

agocke Jan 31, 2023

agocke left a comment

agocke Jan 31, 2023

agocke Jan 31, 2023

MichalStrehovsky Mar 9, 2023

agocke commented Jan 31, 2023

MichalStrehovsky commented Feb 14, 2023

am11 commented Feb 14, 2023

MichalStrehovsky commented Mar 8, 2023

azure-pipelines bot commented Mar 8, 2023

MichalStrehovsky commented Mar 9, 2023

MichalStrehovsky Mar 9, 2023

vitek-karas left a comment

MichalStrehovsky commented Mar 9, 2023

azure-pipelines bot commented Mar 9, 2023

MichalStrehovsky commented Mar 9, 2023

eerhardt commented Mar 10, 2023

MichalStrehovsky commented Mar 11, 2023

Switch to selfhosted NativeAOT compiler #81205

Switch to selfhosted NativeAOT compiler #81205

Conversation

MichalStrehovsky commented Jan 26, 2023

ghost commented Jan 26, 2023

filipnavara Jan 30, 2023

Choose a reason for hiding this comment

MichalStrehovsky Jan 30, 2023

Choose a reason for hiding this comment

agocke Jan 31, 2023

Choose a reason for hiding this comment

agocke left a comment

Choose a reason for hiding this comment

agocke Jan 31, 2023

Choose a reason for hiding this comment

agocke Jan 31, 2023

Choose a reason for hiding this comment

MichalStrehovsky Mar 9, 2023

Choose a reason for hiding this comment

agocke commented Jan 31, 2023

MichalStrehovsky commented Feb 14, 2023

am11 commented Feb 14, 2023

MichalStrehovsky commented Mar 8, 2023

azure-pipelines bot commented Mar 8, 2023

MichalStrehovsky commented Mar 9, 2023

MichalStrehovsky Mar 9, 2023

Choose a reason for hiding this comment

vitek-karas left a comment

Choose a reason for hiding this comment

MichalStrehovsky commented Mar 9, 2023

azure-pipelines bot commented Mar 9, 2023

MichalStrehovsky commented Mar 9, 2023

eerhardt commented Mar 10, 2023

MichalStrehovsky commented Mar 11, 2023