From 97eb01c527e199b9f526e3661095d4a0a051ad34 Mon Sep 17 00:00:00 2001 From: Michael Simons Date: Tue, 6 Aug 2024 20:31:39 +0000 Subject: [PATCH 1/8] Reformat MD line lengths to be less than 80 --- CODE-OF-CONDUCT.md | 3 +- Documentation/VMR-re-bootstrapping.md | 107 +++-- Documentation/boostrap-new-os.md | 144 +++++-- Documentation/bootstrapping-guidelines.md | 126 ++++-- .../ci-platform-coverage-guidelines.md | 30 +- Documentation/eliminating-pre-builts.md | 127 ++++-- Documentation/leak-detection.md | 31 +- Documentation/package-dependency-flow.md | 129 +++--- Documentation/packaging-installation.md | 95 +++-- Documentation/patching-guidelines.md | 75 ++-- .../arcade-powered-source-build/README.md | 21 +- .../intermediate-nupkg.md | 3 +- .../planning/multi-sdk-band-support.md | 403 ++++++++++++++---- Documentation/poison-report-format.md | 54 ++- .../sourcebuild-in-repos/adding-features.md | 66 ++- .../sourcebuild-in-repos/build-info.md | 67 ++- .../sourcebuild-in-repos/new-dependencies.md | 152 +++---- .../sourcebuild-in-repos/new-repo.md | 251 +++++------ .../update-dependencies.md | 45 +- Documentation/system-requirements.md | 29 +- README.md | 120 ++++-- 21 files changed, 1366 insertions(+), 712 deletions(-) diff --git a/CODE-OF-CONDUCT.md b/CODE-OF-CONDUCT.md index 775f221c98..ffded4bff4 100644 --- a/CODE-OF-CONDUCT.md +++ b/CODE-OF-CONDUCT.md @@ -3,4 +3,5 @@ This project has adopted the code of conduct defined by the Contributor Covenant to clarify expected behavior in our community. -For more information, see the [.NET Foundation Code of Conduct](https://dotnetfoundation.org/code-of-conduct). +For more information, see the [.NET Foundation Code of +Conduct](https://dotnetfoundation.org/code-of-conduct). diff --git a/Documentation/VMR-re-bootstrapping.md b/Documentation/VMR-re-bootstrapping.md index a5f0b3234c..b904138024 100644 --- a/Documentation/VMR-re-bootstrapping.md +++ b/Documentation/VMR-re-bootstrapping.md @@ -1,30 +1,31 @@ # How to re-bootstrap the toolset used to build the VMR -.NET utilizes itself to build. Therefore, in order to build .NET from source, you -first need to acquire or build a bootstrapping .NET SDK and other tooling such -as [Arcade](https://github.com/dotnet/arcade). Re-bootstrapping is the term used to describe when the bootstrapped -toolset need to be updated. This document describes the steps to re-bootstrap -the VMR. +.NET utilizes itself to build. Therefore, in order to build .NET from source, +you first need to acquire or build a bootstrapping .NET SDK and other tooling +such as [Arcade](https://github.com/dotnet/arcade). Re-bootstrapping is the term +used to describe when the bootstrapped toolset need to be updated. This document +describes the steps to re-bootstrap the VMR. ## When is it appropriate to re-bootstrap? As part of the release process, the toolset is updated (e.g. PRs are created via -the release automation). Outside of a release, re-bootstrapping is only permitted -during preview releases. It is not allowed during RC, GA, or servicing releases. -The reason it is not allowed during non-preview releases is because of the negative -impact it has on Linux distro maintainers who source build .NET. It is often a long -and time consuming process for them to re-bootstrap. It is likely to cause -significant delays in the release/availability of .NET within the distros that are -source built. +the release automation). Outside of a release, re-bootstrapping is only +permitted during preview releases. It is not allowed during RC, GA, or servicing +releases. The reason it is not allowed during non-preview releases is because of +the negative impact it has on Linux distro maintainers who source build .NET. It +is often a long and time consuming process for them to re-bootstrap. It is +likely to cause significant delays in the release/availability of .NET within +the distros that are source built. ## Why is re-bootstrap necessary? Re-bootstrapping is necessary when .NET takes a dependency on new functionality -added within the bootstrap toolset. For example suppose a new compiler feature is -added. In order for a repo to take a dependency on the new feature, a re-bootstrap -would be necessary. The implication of this, and the restrictions of when -re-bootstrapping is allowed, means that repos should, in general, wait to take a -dependency on a new toolset feature until after that feature has been released. +added within the bootstrap toolset. For example suppose a new compiler feature +is added. In order for a repo to take a dependency on the new feature, a +re-bootstrap would be necessary. The implication of this, and the restrictions +of when re-bootstrapping is allowed, means that repos should, in general, wait +to take a dependency on a new toolset feature until after that feature has been +released. ## Steps to re-bootstrap @@ -32,48 +33,68 @@ dependency on a new toolset feature until after that feature has been released. > [!IMPORTANT] > The re-bootstrap pipeline uploads the artifacts to the official blob storage, -> so do not use this pipeline for testing of any kind. -> To test stage 2 failures, please refer to [this documentation](bootstrapping-guidelines.md#building-on-a-supported-platform-using-rid-known-to-net). +> so do not use this pipeline for testing of any kind. To test stage 2 failures, +> please refer to [this +> documentation](bootstrapping-guidelines.md#building-on-a-supported-platform-using-rid-known-to-net). -You can re-bootstrap the VMR using [this pipeline](https://dev.azure.com/dnceng/internal/_build?definitionId=1371). -The pipeline will upload the artifacts & open the corresponding re-bootstrap PR. +You can re-bootstrap the VMR using [this +pipeline](https://dev.azure.com/dnceng/internal/_build?definitionId=1371). The +pipeline will upload the artifacts & open the corresponding re-bootstrap PR. ### Manual -In case the automated re-bootstrapping pipeline is unavailable, you can manually re-bootstrap the VMR: +In case the automated re-bootstrapping pipeline is unavailable, you can manually +re-bootstrap the VMR: 1. Update previous source-build artifacts - 1. Find a [dotnet-source-build](https://dev.azure.com/dnceng/internal/_build?definitionId=1219) run - with the desired changes. - 1. If a rebootstrap is needed quickly and it is not feasibly to wait for a - [dotnet-source-build](https://dev.azure.com/dnceng/internal/_build?definitionId=1219) run, - you can also use the artifacts from a - [dotnet-source-build-lite](https://dev.azure.com/dnceng/internal/_build?definitionId=1299) run. - 1. Retrieve the built SDKs and private source-built artifacts archives, from the following legs: + 1. Find a + [dotnet-source-build](https://dev.azure.com/dnceng/internal/_build?definitionId=1219) + run with the desired changes. + 1. If a rebootstrap is needed quickly and it is not feasibly to wait for + a + [dotnet-source-build](https://dev.azure.com/dnceng/internal/_build?definitionId=1219) + run, you can also use the artifacts from a + [dotnet-source-build-lite](https://dev.azure.com/dnceng/internal/_build?definitionId=1299) + run. + 1. Retrieve the built SDKs and private source-built artifacts archives, from + the following legs: 1. Alpine\_Online_MsftSdk_x64 1. CentOSStream\_Online_MsftSdk_x64 - 1. Upload the SDKs to the [source build sdk blob storage](https://dotnetcli.blob.core.windows.net/source-built-artifacts/sdks/) - 1. Upload the private source-built artifacts archives to the [source build assets blob storage](https://dotnetcli.blob.core.windows.net/source-built-artifacts/assets/) + 1. Upload the SDKs to the [source build sdk blob + storage](https://dotnetcli.blob.core.windows.net/source-built-artifacts/sdks/) + 1. Upload the private source-built artifacts archives to the [source build + assets blob + storage](https://dotnetcli.blob.core.windows.net/source-built-artifacts/assets/) 1. Update .NET SDK - 1. Find the [dotnet-sdk-official-ci](https://dev.azure.com/dnceng/internal/_build?definitionId=140) build - that best matches the dotnet-source-build. The following is the suggested - order of precedence for finding the best match. + 1. Find the + [dotnet-sdk-official-ci](https://dev.azure.com/dnceng/internal/_build?definitionId=140) + build that best matches the dotnet-source-build. The following is the + suggested order of precedence for finding the best match. 1. A build from the same commit. - 1. From the [dotnet-source-build](https://dev.azure.com/dnceng/internal/_build?definitionId=1219), + 1. From the + [dotnet-source-build](https://dev.azure.com/dnceng/internal/_build?definitionId=1219), look at the build's installer tag. - 1. From a VMR commit, you can find the corresponding installer commit - by looking at the [source-manifest.json](https://github.com/dotnet/dotnet/blob/main/src/source-manifest.json). + 1. From a VMR commit, you can find the corresponding installer + commit by looking at the + [source-manifest.json](https://github.com/dotnet/dotnet/blob/main/src/source-manifest.json). 1. The next passing build after the same commit. 1. In the odd case where the are no passing builds after the commit, you can try using an earlier passing build. 1. Retrieve the built SDK version from the build. - 1. Update the dotnet version in the [global.json](https://github.com/dotnet/installer/blob/main/src/SourceBuild/content/global.json). + 1. Update the dotnet version in the + [global.json](https://github.com/dotnet/installer/blob/main/src/SourceBuild/content/global.json). 1. Update arcade 1. Lookup the arcade commit and version. From a VMR commit, you can find the - corresponding arcade commit/version by looking at the [source-manifest.json](https://github.com/dotnet/dotnet/blob/main/src/source-manifest.json). - 1. Update the arcade SDK version in the [global.json](https://github.com/dotnet/installer/blob/main/src/SourceBuild/content/global.json). - 1. Update the arcade dependency commit and version in the [Version.Details.xml](https://github.com/dotnet/installer/blob/main/src/SourceBuild/content/eng/Version.Details.xml). + corresponding arcade commit/version by looking at the + [source-manifest.json](https://github.com/dotnet/dotnet/blob/main/src/source-manifest.json). + 1. Update the arcade SDK version in the + [global.json](https://github.com/dotnet/installer/blob/main/src/SourceBuild/content/global.json). + 1. Update the arcade dependency commit and version in the + [Version.Details.xml](https://github.com/dotnet/installer/blob/main/src/SourceBuild/content/eng/Version.Details.xml). 1. Update private source-built SDK and artifacts versions - 1. Update `PrivateSourceBuiltSdkVersion` and `PrivateSourceBuiltArtifactsVersion` in the [Versions.props](https://github.com/dotnet/installer/blob/main/src/SourceBuild/content/eng/Versions.props). + 1. Update `PrivateSourceBuiltSdkVersion` and + `PrivateSourceBuiltArtifactsVersion` in the + [Versions.props](https://github.com/dotnet/installer/blob/main/src/SourceBuild/content/eng/Versions.props). -[Tracking issue for automating this process.](https://github.com/dotnet/source-build/issues/4246) +[Tracking issue for automating this +process.](https://github.com/dotnet/source-build/issues/4246) diff --git a/Documentation/boostrap-new-os.md b/Documentation/boostrap-new-os.md index d30d55405b..c01997472f 100644 --- a/Documentation/boostrap-new-os.md +++ b/Documentation/boostrap-new-os.md @@ -2,36 +2,69 @@ ## Bootstrap CLI -When adding support for a new OS, both the native (C++) and managed (C#) code components in the coreclr, corefx and core-setup repos need to be compiled. Making the native components build on a new OS is a quite straightforward process and the tools like CMake, Clang C/C++ compiler, python and awk that are required for the build are available for almost all platforms. -But for the managed code compilation, we have a chicken and egg problem. A .NET CLI toolchain is needed to build managed components in the coreclr, corefx and core-setup repos, but it is not available for the target platform yet. -The way to solve this problem is to create a bootstrap CLI by taking an existing CLI as a "seed" and replacing native components in it by the native components that we build for the new platform. The seed CLI we use has to be for the same processor architecture as our new platform or we have to use a seed cli that we build from sources on another platform that is already supported and opt for not crossgenning the managed components. +When adding support for a new OS, both the native (C++) and managed (C#) code +components in the coreclr, corefx and core-setup repos need to be compiled. +Making the native components build on a new OS is a quite straightforward +process and the tools like CMake, Clang C/C++ compiler, python and awk that are +required for the build are available for almost all platforms. But for the +managed code compilation, we have a chicken and egg problem. A .NET CLI +toolchain is needed to build managed components in the coreclr, corefx and +core-setup repos, but it is not available for the target platform yet. The way +to solve this problem is to create a bootstrap CLI by taking an existing CLI as +a "seed" and replacing native components in it by the native components that we +build for the new platform. The seed CLI we use has to be for the same processor +architecture as our new platform or we have to use a seed cli that we build from +sources on another platform that is already supported and opt for not +crossgenning the managed components. ## When is bootstrapping necessary? Bootstrapping is necessary when: -- Your available version of the SDK cannot build your desired version of the SDK. +- Your available version of the SDK cannot build your desired version of the + SDK. - You have no version of the SDK available for your platform. - You are building the SDK for a non-x64 platform. -In practice, most maintainers will bootstrap once every major version (e.g. 6.0.100, 8.0.100, etc). If you bootstrap these initial versions once you will generally be able to build 6.0.101 with 6.0.100, 6.0.102 with 6.0.101, 8.0.101 with 8.0.100, and so on. The source-build team attempts to avoid requiring bootstrapping more than once per major version but sometimes this is unavoidable - often when we move from one feature band to another, or if a major language feature is added, or if there are issues discovered with a given version of the SDK after release. These types of situations will be announced on the source-build [discussions page](https://github.com/dotnet/source-build/discussions). +In practice, most maintainers will bootstrap once every major version (e.g. +6.0.100, 8.0.100, etc). If you bootstrap these initial versions once you will +generally be able to build 6.0.101 with 6.0.100, 6.0.102 with 6.0.101, 8.0.101 +with 8.0.100, and so on. The source-build team attempts to avoid requiring +bootstrapping more than once per major version but sometimes this is unavoidable +- often when we move from one feature band to another, or if a major language +feature is added, or if there are issues discovered with a given version of the +SDK after release. These types of situations will be announced on the +source-build [discussions +page](https://github.com/dotnet/source-build/discussions). ### Choosing the seed CLI -While the bootstrap CLI can be built for any version of the seed CLI, it is important to pick a version of CLI that can be used to build current corefx, coreclr and core-setup repos. From time to time, a change in the CLI causes it to not to be usable for compiling one of the repos without fixes in the msbuild project files. So to stay on the safe side, the best practice is to pick a version used by one of the three repos. This version can be found in the `DotnetCLIVersion.txt` file in the root of each repo. +While the bootstrap CLI can be built for any version of the seed CLI, it is +important to pick a version of CLI that can be used to build current corefx, +coreclr and core-setup repos. From time to time, a change in the CLI causes it +to not to be usable for compiling one of the repos without fixes in the msbuild +project files. So to stay on the safe side, the best practice is to pick a +version used by one of the three repos. This version can be found in the +`DotnetCLIVersion.txt` file in the root of each repo. ### Getting the seed CLI -After choosing the seed CLI version as described in the previous paragraph, it can be used to construct an URL to download the .tar.gz file with the seed CLI itself. The way to construct the URL can be found in the `init-tools.sh` file in the root of each repo. Currently, it is constructed as follows: +After choosing the seed CLI version as described in the previous paragraph, it +can be used to construct an URL to download the .tar.gz file with the seed CLI +itself. The way to construct the URL can be found in the `init-tools.sh` file in +the root of each repo. Currently, it is constructed as follows: ```bash https://dotnetcli.azureedge.net/dotnet/Sdk/${__DOTNET_TOOLS_VERSION}/dotnet-sdk-${__DOTNET_TOOLS_VERSION}-${__PKG_RID}-${__PKG_ARCH}.tar.gz ``` -The `${__DOTNET_TOOLS_VERSION}` is replaced by the seed CLI version, the `${__PKG_RID}` by the RID of the current platform and `${__PKG_ARCH}` by the architecture of the current platform. -So for tools version `2.0.0` on linux distro with x64 architecture where portable dotnet core can be used, the URL is `https://dotnetcli.azureedge.net/dotnet/Sdk/2.0.0/dotnet-sdk-2.0.0-linux-x64.tar.gz` -To download it, `wget` or `curl` tools can be used. -Once the file is downloaded, create a new folder and untar the file into it. +The `${__DOTNET_TOOLS_VERSION}` is replaced by the seed CLI version, the +`${__PKG_RID}` by the RID of the current platform and `${__PKG_ARCH}` by the +architecture of the current platform. So for tools version `2.0.0` on linux +distro with x64 architecture where portable dotnet core can be used, the URL is +`https://dotnetcli.azureedge.net/dotnet/Sdk/2.0.0/dotnet-sdk-2.0.0-linux-x64.tar.gz` +To download it, `wget` or `curl` tools can be used. Once the file is downloaded, +create a new folder and untar the file into it. So e.g. for the file mentioned above, use: ```bash @@ -40,12 +73,19 @@ tar -xf dotnet-sdk-2.0.0-linux-x64.tar.gz ### Choosing RID for the new OS -The new RID represents your target OS. It is used by developers to target that OS. The RID format is `.`. The name is lower case and should match the target OS. The version is optional and should be used if it is expected that in future versions the libraries that .NET Core depends on will not be binary compatible with the first supported version. So for example for FreeBSD 11, the RID would likely be `freebsd.11`. -For OS that has `/etc/os-release` file, the RID needs to match the `$ID.$VERSION_ID` extracted from that file. +The new RID represents your target OS. It is used by developers to target that +OS. The RID format is `.`. The name is lower case and should +match the target OS. The version is optional and should be used if it is +expected that in future versions the libraries that .NET Core depends on will +not be binary compatible with the first supported version. So for example for +FreeBSD 11, the RID would likely be `freebsd.11`. For OS that has +`/etc/os-release` file, the RID needs to match the `$ID.$VERSION_ID` extracted +from that file. ### Prerequisites -The following libraries and tools need to be installed in order to build the bootstrap CLI. The precise names of the packages are OS specific. +The following libraries and tools need to be installed in order to build the +bootstrap CLI. The precise names of the packages are OS specific. #### Tools @@ -64,20 +104,38 @@ The following libraries and tools need to be installed in order to build the boo - liblldb - icu -For versions earlier than .NET Core 2.1, following dependencies are also required: +For versions earlier than .NET Core 2.1, following dependencies are also +required: - libunwind - libuuid ### Building the bootstrap CLI -There is a bash script file that automatizes most of the process of building the bootstrap CLI end to end. The bash script is located at `dotnet/source-build/scripts/bootstrap/buildbootstrapcli.sh`. It first creates a folder named by the new target RID, clones the coreclr, corefx and core-setup repos into it and checks out the same commit of each of the repos as the one that was used to build the seed CLI. This first step is skipped if the coreclr, corefx and core-setup folders already exist. This is important so that the sources can be modified to fix possible build issues and to target the new RID. - -The script needs to be passed several arguments. The target architecture, the build configuration, the target OS, the new RID and path to the folder with the untared seed CLI. There is also an optional option to specify the version of the Clang compiler to use to compile the native code. There is also an option to pass in a System.Private.CoreLib.dll built elsewhere. This is useful if you are building debug configuration, since the seed CLI that comes from Azure is built for release configuration and the release version of System.Private.CoreLib.dll is not compatible with debug version of libcoreclr.so. - -The build configuration (debug or release) specified here *must* match with the configuration of the seed CLI. Otherwise everything will build fine but you will get a cryptic error as soon as coreclr tries to load a dll. - -Here is the summary of the options that you get from running the script with `--help` option: +There is a bash script file that automatizes most of the process of building the +bootstrap CLI end to end. The bash script is located at +`dotnet/source-build/scripts/bootstrap/buildbootstrapcli.sh`. It first creates a +folder named by the new target RID, clones the coreclr, corefx and core-setup +repos into it and checks out the same commit of each of the repos as the one +that was used to build the seed CLI. This first step is skipped if the coreclr, +corefx and core-setup folders already exist. This is important so that the +sources can be modified to fix possible build issues and to target the new RID. + +The script needs to be passed several arguments. The target architecture, the +build configuration, the target OS, the new RID and path to the folder with the +untared seed CLI. There is also an optional option to specify the version of the +Clang compiler to use to compile the native code. There is also an option to +pass in a System.Private.CoreLib.dll built elsewhere. This is useful if you are +building debug configuration, since the seed CLI that comes from Azure is built +for release configuration and the release version of System.Private.CoreLib.dll +is not compatible with debug version of libcoreclr.so. + +The build configuration (debug or release) specified here *must* match with the +configuration of the seed CLI. Otherwise everything will build fine but you will +get a cryptic error as soon as coreclr tries to load a dll. + +Here is the summary of the options that you get from running the script with +`--help` option: ```text Usage: buildbootstrapcli.sh [BuildType] -rid -seedcli [-os ] [-clang ] [-corelib ] @@ -92,18 +150,30 @@ Options: -outputpath Optional output directory to contain the generated cli and cloned repos, default: ``` -So, for example, when we were creating bootstrap CLI for RHEL / CentOS 6, the command was: +So, for example, when we were creating bootstrap CLI for RHEL / CentOS 6, the +command was: ```bash ./buildbootstrapcli.sh -rid rhel.6-x64 -os Linux -seedcli ~/seed-cli ``` -After running the script, check the console output. If the last line printed is `**** Bootstrap CLI was successfully built ****`, then everything went fine and the bootstrap CLI is ready. You can find it in the `-/dotnetcli` subfolder. So for the example command above, it would be `rhel.6-x64/dotnetcli`. -If there were build errors, they need to be looked into and fixed. After that run the `buildbootstrapcli.sh` with the same arguments again. Repeat until everything builds. +After running the script, check the console output. If the last line printed is +`**** Bootstrap CLI was successfully built ****`, then everything went fine and +the bootstrap CLI is ready. You can find it in the +`-/dotnetcli` subfolder. So for the example command above, it +would be `rhel.6-x64/dotnetcli`. If there were build errors, they need to be +looked into and fixed. After that run the `buildbootstrapcli.sh` with the same +arguments again. Repeat until everything builds. ### Testing the bootstrap CLI -The easiest way to test the bootstrap CLI that was just created is to create, build and run a "hello world" console application. Create a new folder for the application and then run `/your/path/to/bootstrap/dotnetcli/dotnet new console` followed by `/your/path/to/bootstrap/dotnetcli/dotnet run`. If both of these succeed and the second prints "Hello world!", then everything went ok and you have a working bootstrap CLI. If there are issues in either of these steps, they need to be debugged and the culprits figured out. +The easiest way to test the bootstrap CLI that was just created is to create, +build and run a "hello world" console application. Create a new folder for the +application and then run `/your/path/to/bootstrap/dotnetcli/dotnet new console` +followed by `/your/path/to/bootstrap/dotnetcli/dotnet run`. If both of these +succeed and the second prints "Hello world!", then everything went ok and you +have a working bootstrap CLI. If there are issues in either of these steps, they +need to be debugged and the culprits figured out. ### Troubleshooting @@ -111,17 +181,28 @@ TODO: describe how to do debug build and replace System.Private.CoreLib.dll #### Using bootstrap CLI on platforms with different OS or architecture -If the bootstrap CLI was created for OS other than the one the seed cli supports (e.g. FreeBSD) or the target architecture of the target OS differs from the architecture of the bootstrap cli (e.g. ARM64), the managed assemblies in the bootstrap CLI cannot be loaded, since they contain native code for the target OS and architecture. Fortunately, they also contain the original IL code, so they can be re-crossgened for the target OS and architecture. Or it is possible to instruct the CoreCLR runtime to ignore the native code and use the IL by setting environment variables `COMPlus_ZapDisable=1` and `COMPlus_ReadyToRun=0`. That can be useful during the bringup. But ultimately, re-crossgening should be made to improve startup performance. +If the bootstrap CLI was created for OS other than the one the seed cli supports +(e.g. FreeBSD) or the target architecture of the target OS differs from the +architecture of the bootstrap cli (e.g. ARM64), the managed assemblies in the +bootstrap CLI cannot be loaded, since they contain native code for the target OS +and architecture. Fortunately, they also contain the original IL code, so they +can be re-crossgened for the target OS and architecture. Or it is possible to +instruct the CoreCLR runtime to ignore the native code and use the IL by setting +environment variables `COMPlus_ZapDisable=1` and `COMPlus_ReadyToRun=0`. That +can be useful during the bringup. But ultimately, re-crossgening should be made +to improve startup performance. ## Use the bootstrap CLI to build source-build -Now that you have an SDK that works on the new OS, you can build all of the .NET Core source code. To do this, first tar up the boostrap CLI: +Now that you have an SDK that works on the new OS, you can build all of the .NET +Core source code. To do this, first tar up the boostrap CLI: ```bash tar -czf ~/dotnet.tar --directory /your/path/to/bootstrap/dotnetcli "." ``` -Next, the ILASM tool that you built locally needs to be copied out so it can be used. +Next, the ILASM tool that you built locally needs to be copied out so it can be +used. ```bash mkdir ~/ilasm @@ -129,8 +210,9 @@ cp dotnet/source-build/scripts/bootstrap/RID/coreclr/bin/Product/RID.Release/* ~ cp /your/path/to/bootstrap/dotnetcli/shared/Microsoft.NETCore.App/VERSION/System.Private.CoreLib.dll ~/ilasm ``` -TODO: make this next step better - possibly by adding a new build parameter that takes the path to ILASM. -Now, we can initialize our build tools with the bootstrap CLI and the ILASM tool. +TODO: make this next step better - possibly by adding a new build parameter that +takes the path to ILASM. Now, we can initialize our build tools with the +bootstrap CLI and the ILASM tool. ```bash DotNetBootstrapCliTarPath=~/dotnet.tar SOURCE_BUILD_SKIP_SUBMODULE_CHECK=1 ./init-tools.sh diff --git a/Documentation/bootstrapping-guidelines.md b/Documentation/bootstrapping-guidelines.md index 8b2bf92c5b..2956308496 100644 --- a/Documentation/bootstrapping-guidelines.md +++ b/Documentation/bootstrapping-guidelines.md @@ -1,32 +1,50 @@ # Bootstrapping Guidelines to Build .NET from Source -.NET utilizes itself to build therefore in order to build .NET from source, you first need to acquire or build a bootstrapping .NET SDK. This document provides guidance around acquiring and building this bootstrapping .NET SDK. +.NET utilizes itself to build therefore in order to build .NET from source, you +first need to acquire or build a bootstrapping .NET SDK. This document provides +guidance around acquiring and building this bootstrapping .NET SDK. -The version of the SDK used to source build .NET is referred to as "N-1" (e.g. 8.0.100). The version of the SDK produced by source build is referred to as "N" (e.g. 8.0.101). The previous SDK (e.g. N-1) supplies the tools required to build. +The version of the SDK used to source build .NET is referred to as "N-1" (e.g. +8.0.100). The version of the SDK produced by source build is referred to as "N" +(e.g. 8.0.101). The previous SDK (e.g. N-1) supplies the tools required to +build. -For new major versions or new platforms, you need to acquire or build the bootstrapping SDK as you cannot use a previous source-built SDK. This is to say you cannot use a 8.0 version of the SDK to build a 9.0 SDK. +For new major versions or new platforms, you need to acquire or build the +bootstrapping SDK as you cannot use a previous source-built SDK. This is to say +you cannot use a 8.0 version of the SDK to build a 9.0 SDK. -Bootstrapping typically requires an exception in the distro packaging guidelines (e.g. [Fedora Bootstrapping Guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping)). +Bootstrapping typically requires an exception in the distro packaging guidelines +(e.g. [Fedora Bootstrapping +Guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping)). -Refer to the [build instructions](https://github.com/dotnet/installer/blob/main/README.md#build-net-from-source-source-build) to review how to build the .NET SDK from source. +Refer to the [build +instructions](https://github.com/dotnet/installer/blob/main/README.md#build-net-from-source-source-build) +to review how to build the .NET SDK from source. ## Scenarios There are three major scenarios for bootstrapping: -1. [Building on a supported platform (Using RID known to .NET)](#building-on-a-supported-platform-using-rid-known-to-net) -1. [Building for New OS (Using a RID unknown to .NET)](#building-for-new-os-using-a-rid-unknown-to-net) -1. [Building for New Architecture (Using a RID unknown to .NET)](#building-for-new-architecture-using-a-rid-unknown-to-net) +1. [Building on a supported platform (Using RID known to + .NET)](#building-on-a-supported-platform-using-rid-known-to-net) +1. [Building for New OS (Using a RID unknown to + .NET)](#building-for-new-os-using-a-rid-unknown-to-net) +1. [Building for New Architecture (Using a RID unknown to + .NET)](#building-for-new-architecture-using-a-rid-unknown-to-net) ## Building on a supported platform (Using RID known to .NET) -To find out if your platform is supported you must first determine its [RID](https://learn.microsoft.com/en-us/dotnet/core/rid-catalog). You can then check if it's supported by looking at the RID graph in the [runtime.json](https://github.com/dotnet/runtime/blob/main/src/libraries/Microsoft.NETCore.Platforms/src/runtime.json). +To find out if your platform is supported you must first determine its +[RID](https://learn.microsoft.com/en-us/dotnet/core/rid-catalog). You can then +check if it's supported by looking at the RID graph in the +[runtime.json](https://github.com/dotnet/runtime/blob/main/src/libraries/Microsoft.NETCore.Platforms/src/runtime.json). Building .NET for the first time is a two stage process: **Stage 1:** Build bootstrapping .NET SDK for the targeted platform. -1. Download a platform-native portable Microsoft-built version of the dotnet SDK for bootstrapping as well as the previously-source-built package archive. +1. Download a platform-native portable Microsoft-built version of the dotnet SDK + for bootstrapping as well as the previously-source-built package archive. ``` bash ./prep-source-build.sh @@ -38,7 +56,8 @@ Building .NET for the first time is a two stage process: ./build.sh --source-only ``` -**Stage 2:** Use the source-built .NET SDK and source-built artifacts created in stage 1 to build .NET SDK from source. +**Stage 2:** Use the source-built .NET SDK and source-built artifacts created in +stage 1 to build .NET SDK from source. 1. Extract your freshly-built stage 1 SDK to a convenient location. @@ -46,7 +65,8 @@ Building .NET for the first time is a two stage process: tar -ozxf //artifacts/assets/Release/dotnet-sdk---tar.gz -C ``` -1. Extract your freshly-built stage 1 source-built artifacts to a convenient location. +1. Extract your freshly-built stage 1 source-built artifacts to a convenient + location. ```bash tar -ozxf //artifacts/assets/Release/Private.SourceBuilt.Artifacts.--.tar.gz -C @@ -65,60 +85,106 @@ Building .NET for the first time is a two stage process: ## Building for New OS (Using a RID unknown to .NET) -Building for an OS that Microsoft does not currently build the SDK for is possible but requires more work. If [Microsoft produces](https://dotnet.microsoft.com/en-us/download/dotnet) a portable SDK for your platform (e.g. amd64 and arm64), you can follow the two-step process below. +Building for an OS that Microsoft does not currently build the SDK for is +possible but requires more work. If [Microsoft +produces](https://dotnet.microsoft.com/en-us/download/dotnet) a portable SDK for +your platform (e.g. amd64 and arm64), you can follow the two-step process below. **RIDs:** -The RID graph or runtime fallback graph is a list of RIDs that are compatible with each other. You can see the list of supported RIDs and the RID graph in the [runtime.json](https://github.com/dotnet/runtime/blob/main/src/libraries/Microsoft.NETCore.Platforms/src/runtime.json). Learn more about RID catalog [here](https://learn.microsoft.com/en-us/dotnet/core/rid-catalog#linux-rids). +The RID graph or runtime fallback graph is a list of RIDs that are compatible +with each other. You can see the list of supported RIDs and the RID graph in the +[runtime.json](https://github.com/dotnet/runtime/blob/main/src/libraries/Microsoft.NETCore.Platforms/src/runtime.json). +Learn more about RID catalog +[here](https://learn.microsoft.com/en-us/dotnet/core/rid-catalog#linux-rids). -If a compatible RID is found, you can use a compatible supported OS as host to build. Choose a host with same processor architecture as that of the new targeted platform. If you choose this option, the RID of the resulting SDK will be that of the host. If this is acceptable follow the instructions in [Building on a supported platform (Using RID known to .NET)](#building-on-a-supported-platform-using-rid-known-to-net) using a compatible host OS. +If a compatible RID is found, you can use a compatible supported OS as host to +build. Choose a host with same processor architecture as that of the new +targeted platform. If you choose this option, the RID of the resulting SDK will +be that of the host. If this is acceptable follow the instructions in [Building +on a supported platform (Using RID known to +.NET)](#building-on-a-supported-platform-using-rid-known-to-net) using a +compatible host OS. -If no compatible RID is found or you want a RID specific SDK use the folloring the steps (works for .NET 6, but requires validation for .NET 7): +If no compatible RID is found or you want a RID specific SDK use the folloring +the steps (works for .NET 6, but requires validation for .NET 7): **Stage 0:** 1. Get Microsoft portable SDK. -1. Update the RID graph (runtime.json) in the Microsoft-built portable SDK with the same changes you will make below to add your new RID to the RID graph. This should include a fallback to the portable RID (linux-x64 or similar). +1. Update the RID graph (runtime.json) in the Microsoft-built portable SDK with + the same changes you will make below to add your new RID to the RID graph. + This should include a fallback to the portable RID (linux-x64 or similar). **Stage 1:** -1. Update the RID graph in source with the same changes made in Stage 0. For an example, see . -1. Build with Stage 0 SDK using `--with-sdk` with your modified portable SDK. See the Stage 1 instructions in [Building on a supported platform (Using RID known to .NET)](#building-on-a-supported-platform-using-rid-known-to-net). +1. Update the RID graph in source with the same changes made in Stage 0. For an + example, see . +1. Build with Stage 0 SDK using `--with-sdk` with your modified portable SDK. + See the Stage 1 instructions in [Building on a supported platform (Using RID + known to .NET)](#building-on-a-supported-platform-using-rid-known-to-net). **Stage 2:** -1. Now you have a RID-specific SDK that knows about your new RID, build with Stage 1 SDK as done in [Building on a supported platform (Using RID known to .NET)](#building-on-a-supported-platform-using-rid-known-to-net). +1. Now you have a RID-specific SDK that knows about your new RID, build with + Stage 1 SDK as done in [Building on a supported platform (Using RID known to + .NET)](#building-on-a-supported-platform-using-rid-known-to-net). ## Building for New Architecture (Using a RID unknown to .NET) -Building for an architecture that Microsoft does not currently build the SDK for is possible but requires more work. +Building for an architecture that Microsoft does not currently build the SDK for +is possible but requires more work. **RIDs:** -The RID graph or runtime fallback graph is a list of RIDs that are compatible with each other. You can see the list of supported RIDs and the RID graph in the [runtime.json](https://github.com/dotnet/runtime/blob/main/src/libraries/Microsoft.NETCore.Platforms/src/runtime.json). Learn more about RID catalog in [here](https://learn.microsoft.com/en-us/dotnet/core/rid-catalog#linux-rids). +The RID graph or runtime fallback graph is a list of RIDs that are compatible +with each other. You can see the list of supported RIDs and the RID graph in the +[runtime.json](https://github.com/dotnet/runtime/blob/main/src/libraries/Microsoft.NETCore.Platforms/src/runtime.json). +Learn more about RID catalog in +[here](https://learn.microsoft.com/en-us/dotnet/core/rid-catalog#linux-rids). -You will need to update the RID graph to include your new platform and runtime IDs. See or for examples. +You will need to update the RID graph to include your new platform and runtime +IDs. See or + for examples. -Building for unsupported architectures require cross-compilaton on the supported platform. Determine the compatible host to build which provides cross-compilation toolchain. [IBM has published](https://community.ibm.com/community/user/powerdeveloper/blogs/sapana-khemkar/2023/01/13/cross-build-dotnet7-on-x86-ibm-power?CommunityKey=8cc2a1f0-6307-48cb-9178-ace50920244e) a detailed description of how they successfully built .NET 7 for IBM Power. +Building for unsupported architectures require cross-compilaton on the supported +platform. Determine the compatible host to build which provides +cross-compilation toolchain. [IBM has +published](https://community.ibm.com/community/user/powerdeveloper/blogs/sapana-khemkar/2023/01/13/cross-build-dotnet7-on-x86-ibm-power?CommunityKey=8cc2a1f0-6307-48cb-9178-ace50920244e) +a detailed description of how they successfully built .NET 7 for IBM Power. -While this is a more complicated scenario that may differ from platform to platform, the steps will be roughly: +While this is a more complicated scenario that may differ from platform to +platform, the steps will be roughly: **Stage 0:** -1. Cross compile an SDK (using prebuilts) on x64 for target platform (this process may be quite long and involved and include setting up a rootfs for your architecture). -2. Cross compile the runtime repo (on x64 for target platform, generally done as part of previous step) and save the nuget packages, use these to augment the Microsoft-built previously-source-built archive. +1. Cross compile an SDK (using prebuilts) on x64 for target platform (this + process may be quite long and involved and include setting up a rootfs for + your architecture). +2. Cross compile the runtime repo (on x64 for target platform, generally done as + part of previous step) and save the nuget packages, use these to augment the + Microsoft-built previously-source-built archive. **Stage 1:** -1. Use the cross-compiled SDK and augmented previously-source-built-archive to build a stage 1 SDK. See the Stage 1 instructions in [Building on a supported platform (Using RID known to .NET)](#building-on-a-supported-platform-using-rid-known-to-net). +1. Use the cross-compiled SDK and augmented previously-source-built-archive to + build a stage 1 SDK. See the Stage 1 instructions in [Building on a + supported platform (Using RID known to + .NET)](#building-on-a-supported-platform-using-rid-known-to-net). **Stage 2:** -1. Use your stage 1 SDK to build a stage 2 SDK, pointing it to the SDK and previously-source-built archives from stage 1. See the Stage 2 instructions in [Building on a supported platform (Using RID known to .NET)](#building-on-a-supported-platform-using-rid-known-to-net). +1. Use your stage 1 SDK to build a stage 2 SDK, pointing it to the SDK and + previously-source-built archives from stage 1. See the Stage 2 instructions + in [Building on a supported platform (Using RID known to + .NET)](#building-on-a-supported-platform-using-rid-known-to-net). ## Building a Servicing Release of .NET -Building a subsequent or servicing version of .NET requires that you have source built the previous version of .NET available as descibed in one of the [building scenarios](#scenarios). Once you have a previous verion of the .NET SDK available, all you have to do is run the following build command. +Building a subsequent or servicing version of .NET requires that you have source +built the previous version of .NET available as descibed in one of the [building +scenarios](#scenarios). Once you have a previous verion of the .NET SDK +available, all you have to do is run the following build command. ``` bash ./build.sh --source-only --with-sdk --with-packages diff --git a/Documentation/ci-platform-coverage-guidelines.md b/Documentation/ci-platform-coverage-guidelines.md index 56175c2f12..da8fea6e2e 100644 --- a/Documentation/ci-platform-coverage-guidelines.md +++ b/Documentation/ci-platform-coverage-guidelines.md @@ -1,26 +1,28 @@ # Guidelines for Platforms Tested in CI -This document contains the guidelines for which platforms (OS and architectures) to test in -the source build CI. +This document contains the guidelines for which platforms (OS and architectures) +to test in the source build CI. ## Distro Families 1. Prefer testing base distros of families over derivatives. 1. Prioritize testing distros that source build .NET. 1. Use CentOS Stream instead of Red Hat because it is the free alternative. -1. Include a permutation of distros for the supported C standard library implementations -(e.g. glibc and musl). -1. For each C standard library implementation, include at least one distro that uses the minimum -supported version of this library. -1. Only test [distros that are officially supported by .NET](https://github.com/dotnet/core/blob/main/os-lifecycle-policy.md#net-supported-os-policy). +1. Include a permutation of distros for the supported C standard library +implementations (e.g. glibc and musl). +1. For each C standard library implementation, include at least one distro that +uses the minimum supported version of this library. +1. Only test [distros that are officially supported by +.NET](https://github.com/dotnet/core/blob/main/os-lifecycle-policy.md#net-supported-os-policy). Community supported distros will not be covered. ## Distro Versions -1. Prefer testing latest LTS version. If LTS is unsupported, use the latest version instead. +1. Prefer testing latest LTS version. If LTS is unsupported, use the latest + version instead. 1. Drop distro versions that will be or are nearing EOL on the .NET release day. -1. Stop updating the test matrix in the last 6 months of support for a .NET version as new distro -versions are released. +1. Stop updating the test matrix in the last 6 months of support for a .NET +version as new distro versions are released. ## Architectures @@ -29,10 +31,12 @@ versions are released. ## Permutations -It is not the intent to test every permutation of distro family, distro version, and architecture. -Rather, smart decisions should be made to best utilize resources. +It is not the intent to test every permutation of distro family, distro version, +and architecture. Rather, smart decisions should be made to best utilize +resources. -The following distro versions will be included in the [CI matrix](https://github.com/dotnet/sdk/blob/main/eng/pipelines/templates/stages/vmr-build.yml): +The following distro versions will be included in the [CI +matrix](https://github.com/dotnet/sdk/blob/main/eng/pipelines/templates/stages/vmr-build.yml): 1. CentOS Stream - Latest version (amd64) 1. Fedora - Latest version (amd64) diff --git a/Documentation/eliminating-pre-builts.md b/Documentation/eliminating-pre-builts.md index 72ac6c76a7..ed1c23ca35 100644 --- a/Documentation/eliminating-pre-builts.md +++ b/Documentation/eliminating-pre-builts.md @@ -1,7 +1,8 @@ # Eliminating pre-builts in .NET repositories -This is a detailed guide on how to eliminate pre-builts in a source-buildable repository. -It is primarily intended for developers contributing to the `dotnet` organization. +This is a detailed guide on how to eliminate pre-builts in a source-buildable +repository. It is primarily intended for developers contributing to the `dotnet` +organization. ## Table of content @@ -12,13 +13,24 @@ It is primarily intended for developers contributing to the `dotnet` organizatio ## What is a Prebuilt -_Source-build_ is a process of building a given product on a single machine from source with no internet access. - -By definition, _pre-builts_ are dependencies that are not built from source, such as reference packages, nuget packages and built tools. _Build from source_ points to any package produced during the _current source-build_ with the exception of dependencies that are picked up from the host distro such as `cmake`. -In layman terms, this means that packages from `nuget.org`, Microsoft builds or other non-source-built binaries cannot be used for source-building a given repository. - -When onboarding a repository to source-build or adding a new dependency to a source-buildable one, the contributor runs the risk of introducing a new pre-built to the product. -To protect against this and catch any new pre-builts as soon as possible, Arcade source-build infrastructure provides _pre-built detection_ - MSBuild logic responsible for veryfing that no used dependency is a pre-built. In case one is discovered (for example, during a PR pipeline), the build will fail with an appropriate message somewhat similar to the following: +_Source-build_ is a process of building a given product on a single machine from +source with no internet access. + +By definition, _pre-builts_ are dependencies that are not built from source, +such as reference packages, nuget packages and built tools. _Build from source_ +points to any package produced during the _current source-build_ with the +exception of dependencies that are picked up from the host distro such as +`cmake`. In layman terms, this means that packages from `nuget.org`, Microsoft +builds or other non-source-built binaries cannot be used for source-building a +given repository. + +When onboarding a repository to source-build or adding a new dependency to a +source-buildable one, the contributor runs the risk of introducing a new +pre-built to the product. To protect against this and catch any new pre-builts +as soon as possible, Arcade source-build infrastructure provides _pre-built +detection_ - MSBuild logic responsible for veryfing that no used dependency is a +pre-built. In case one is discovered (for example, during a PR pipeline), the +build will fail with an appropriate message somewhat similar to the following: ```text 3 new packages used not in baseline! See report at ./artifacts/source-build/self/prebuilt-report/baseline-comparison.xml for more information. Package IDs are: @@ -27,33 +39,64 @@ To protect against this and catch any new pre-builts as soon as possible, Arcade Microsoft.Build.Framework.14.3.0 ``` -Pre-built detection identifies the source of dependencies used to build the repository. -These dependencies include not only direct dependencies, but also build tooling as well as dangling dependencies - packages downloaded / used by tooling during the build process and not referenced by the project itself. -Dependencies retrieved from external sources that are not explicitly excluded from pre-built detection will be flagged as pre-builts. +Pre-built detection identifies the source of dependencies used to build the +repository. These dependencies include not only direct dependencies, but also +build tooling as well as dangling dependencies - packages downloaded / used by +tooling during the build process and not referenced by the project itself. +Dependencies retrieved from external sources that are not explicitly excluded +from pre-built detection will be flagged as pre-builts. ## Eliminating pre-builts -When altering the dependecy tree of a repository, specifically adding or updating dependencies, there is a posibility that a new pre-built is introduced, failing the build and blocking any merge. -This can be resolved by identifying what exactly is the pre-built and following the approriate steps listed below. +When altering the dependecy tree of a repository, specifically adding or +updating dependencies, there is a posibility that a new pre-built is introduced, +failing the build and blocking any merge. This can be resolved by identifying +what exactly is the pre-built and following the approriate steps listed below. -To check if new pre-builts were introduce, the repository needs to be source-built first. This can be done through the following command: +To check if new pre-builts were introduce, the repository needs to be +source-built first. This can be done through the following command: ```sh ./build.sh --sb ``` -If a new unhandled pre-built is found, the build will fail with a detailed exception pointing to the exact package / version of the dependency that caused the failure. -Additional information can also be found in the generated pre-built detection reports, located in the `./artifacts/source-build/self/prebuild-report` directory, specifically in the `./prebuild-usage.xml` file. The information in question can, for example, be the path to the project that is referencing the dependency. +If a new unhandled pre-built is found, the build will fail with a detailed +exception pointing to the exact package / version of the dependency that caused +the failure. Additional information can also be found in the generated pre-built +detection reports, located in the +`./artifacts/source-build/self/prebuild-report` directory, specifically in the +`./prebuild-usage.xml` file. The information in question can, for example, be +the path to the project that is referencing the dependency. -With this information retrieved, the [adding a new source-build dependency](https://github.com/dotnet/source-build/blob/main/Documentation/sourcebuild-in-repos/new-dependencies.md#adding-dependencies) documentation should be referred to as the main guide for resolving the pre-built. +With this information retrieved, the [adding a new source-build +dependency](https://github.com/dotnet/source-build/blob/main/Documentation/sourcebuild-in-repos/new-dependencies.md#adding-dependencies) +documentation should be referred to as the main guide for resolving the +pre-built. ### Pre-built through a transitive dependency -During a project dependency updated, a new pre-built might be introduced by a new or updated transitive dependency. While the Arcade tooling will highlight the name and version of the pre-built in the build exception as well as the project that restored the dependency in question, it will not point out where exactly in the dependency tree it is. This means that it's up to the developer to identify what type of dependency they are dealing with (direct or transitive) and choose the correct way of handling the issue. In case of a transitive dependency, it might be hard to identify the relationship that is bringing in the pre-built into the project, especially if the developer has limited knowledge of the project or code-base in general. - -Arcade source-build infrastructure helps accomplish this this by pointing out the `project.assets.json` file that is referencing the pre-built in the `./artifacts/source-build/self/prebuild-report/prebuild-usage.xml` report file. A `project.assets.json` file is a NuGet restore process artifact that contains a resolved dependency tree for a specific project. Every package that was restored by a given project is mentioned there with links between the dependencies, allowing the reader to identify transitive dependencies and the direct dependencies referencing them. - -Example of a pre-built caused by a transitive dependency and corresponding entries in files mentioned above: +During a project dependency updated, a new pre-built might be introduced by a +new or updated transitive dependency. While the Arcade tooling will highlight +the name and version of the pre-built in the build exception as well as the +project that restored the dependency in question, it will not point out where +exactly in the dependency tree it is. This means that it's up to the developer +to identify what type of dependency they are dealing with (direct or transitive) +and choose the correct way of handling the issue. In case of a transitive +dependency, it might be hard to identify the relationship that is bringing in +the pre-built into the project, especially if the developer has limited +knowledge of the project or code-base in general. + +Arcade source-build infrastructure helps accomplish this this by pointing out +the `project.assets.json` file that is referencing the pre-built in the +`./artifacts/source-build/self/prebuild-report/prebuild-usage.xml` report file. +A `project.assets.json` file is a NuGet restore process artifact that contains a +resolved dependency tree for a specific project. Every package that was restored +by a given project is mentioned there with links between the dependencies, +allowing the reader to identify transitive dependencies and the direct +dependencies referencing them. + +Example of a pre-built caused by a transitive dependency and corresponding +entries in files mentioned above: Exception identifying the pre-built: @@ -89,16 +132,24 @@ Entry in project.assets.json: } ``` -In this example, `Microsoft.Extensions.DependencyModel` would be the direct dependency causing the `System.Text.Json` pre-built. +In this example, `Microsoft.Extensions.DependencyModel` would be the direct +dependency causing the `System.Text.Json` pre-built. ## Allowed exceptions -The list of permitted pre-builts can be found in the `./eng/SourceBuildPrebuiltBaseline.xml` file in the root of the repository. It contains package information of pre-builts that for one reason or another are allowed in the source-build of the repository. +The list of permitted pre-builts can be found in the +`./eng/SourceBuildPrebuiltBaseline.xml` file in the root of the repository. It +contains package information of pre-builts that for one reason or another are +allowed in the source-build of the repository. -Any new addition to the pre-built exception list must be signed-off by a member of the `@dotnet/source-build-internal` team. +Any new addition to the pre-built exception list must be signed-off by a member +of the `@dotnet/source-build-internal` team. -A common example of a exception that is present in several .NET repositories is an [_intermediate package_](https://github.com/dotnet/source-build/blob/main/Documentation/planning/arcade-powered-source-build/intermediate-nupkg.md). -When a repository utilizes an intermediate package, it will be excluded from pre-built detection with the following declaration in the above-mentioned file: +A common example of a exception that is present in several .NET repositories is +an [_intermediate +package_](https://github.com/dotnet/source-build/blob/main/Documentation/planning/arcade-powered-source-build/intermediate-nupkg.md). +When a repository utilizes an intermediate package, it will be excluded from +pre-built detection with the following declaration in the above-mentioned file: ```xml @@ -108,9 +159,13 @@ When a repository utilizes an intermediate package, it will be excluded from pre ``` -With this ignore pattern in place, pre-built detection will not mark any .NET intermediate package as long as it conforms to the naming in the pattern. +With this ignore pattern in place, pre-built detection will not mark any .NET +intermediate package as long as it conforms to the naming in the pattern. -In cases where a specific package needs to be excluded from pre-built detection (for example, to not block the introduction of changes until a source-build acceptable solution for the pre-built is provided), the developer can directly specify the name / version of the depedency: +In cases where a specific package needs to be excluded from pre-built detection +(for example, to not block the introduction of changes until a source-build +acceptable solution for the pre-built is provided), the developer can directly +specify the name / version of the depedency: ```xml @@ -119,9 +174,15 @@ In cases where a specific package needs to be excluded from pre-built detection ``` -If a new pre-built is encountered, pre-built detection will also generate a new version of the baseline file by adding the dependency that contains the pre-built to the existing baseline. -The new file can be found at `./artifacts/source-build/self/prebuild-report/generated-new-baseline.xml`. +If a new pre-built is encountered, pre-built detection will also generate a new +version of the baseline file by adding the dependency that contains the +pre-built to the existing baseline. The new file can be found at +`./artifacts/source-build/self/prebuild-report/generated-new-baseline.xml`. ## Contacts -For any questions or additional information about this document, pre-builts or source-build in general, please create an [issue](https://github.com/dotnet/source-build/issues) or open a [discussion](https://github.com/dotnet/source-build/discussions) in the [source-build](https://github.com/dotnet/source-build) repository. +For any questions or additional information about this document, pre-builts or +source-build in general, please create an +[issue](https://github.com/dotnet/source-build/issues) or open a +[discussion](https://github.com/dotnet/source-build/discussions) in the +[source-build](https://github.com/dotnet/source-build) repository. diff --git a/Documentation/leak-detection.md b/Documentation/leak-detection.md index c17ce41479..3f6b4da56b 100644 --- a/Documentation/leak-detection.md +++ b/Documentation/leak-detection.md @@ -2,25 +2,42 @@ ## Overview -Source-build includes a mechanism for *poisoning* its input files, and then for checking for that poison in the build output. This allows us to ensure that all output files were built during the build rather than copied directly from the input, which would be an illegal prebuilt usage. +Source-build includes a mechanism for *poisoning* its input files, and then for +checking for that poison in the build output. This allows us to ensure that all +output files were built during the build rather than copied directly from the +input, which would be an illegal prebuilt usage. ## Before the build Before the build, the MarkAndCatalogFiles task runs. This does a few things: -- Record the hash of every file in the source-build binary input directories (prebuilts, previously-source-built, and reference-packages). If the file is a zip, tarball, or nupkg, unpack it and do the same thing recursively. -- For managed DLLs, either bare or in an archive, add a custom attribute that marks the file as poisoned. +- Record the hash of every file in the source-build binary input directories + (prebuilts, previously-source-built, and reference-packages). If the file is + a zip, tarball, or nupkg, unpack it and do the same thing recursively. +- For managed DLLs, either bare or in an archive, add a custom attribute that + marks the file as poisoned. - For nupkgs, drop a `.poisoned` file. -- Repack the poisoned assemblies and extra files, removing nupkg signatures so they don't fail verification. +- Repack the poisoned assemblies and extra files, removing nupkg signatures so + they don't fail verification. - Replace the binary inputs with these new packages and archives. -- Record the hash of each file again, to make sure we will be able to track output files whether they were used poisoned or unpoisoned. +- Record the hash of each file again, to make sure we will be able to track + output files whether they were used poisoned or unpoisoned. ## During the build There's no change in source-build operation in poisoning mode during the build. -**Note**: During the build of the source-build-reference-packages repository (regardless of poisoning mode), reference packages have the `System.Reflection.AssemblyMetadataAttribute("source", "source-build-reference-packages")` attribute injected into their respective reference assemblies. Leak detection flags all assemblies with this attribute. +**Note**: During the build of the source-build-reference-packages repository +(regardless of poisoning mode), reference packages have the +`System.Reflection.AssemblyMetadataAttribute("source", +"source-build-reference-packages")` attribute injected into their respective +reference assemblies. Leak detection flags all assemblies with this attribute. ## After the build -After the build, the CheckForPoison task is run on the source-build output directory. It again unpacks any archives and packages recursively, and checks for the three kinds of markers (AssemblyAttribute, Hash, and NupkgFile) injected before the build and the source-build-reference-packages attribute added during the build. It then writes out a [report](poison-report-format.md) that details everything that was found. +After the build, the CheckForPoison task is run on the source-build output +directory. It again unpacks any archives and packages recursively, and checks +for the three kinds of markers (AssemblyAttribute, Hash, and NupkgFile) injected +before the build and the source-build-reference-packages attribute added during +the build. It then writes out a [report](poison-report-format.md) that details +everything that was found. diff --git a/Documentation/package-dependency-flow.md b/Documentation/package-dependency-flow.md index 16bf5f39ab..165fe31834 100644 --- a/Documentation/package-dependency-flow.md +++ b/Documentation/package-dependency-flow.md @@ -1,50 +1,55 @@ # Package Dependency Flow -This document describes how package dependencies are handled within source build. -It describes the mechanisms that exist to control which package versions are used. +This document describes how package dependencies are handled within source +build. It describes the mechanisms that exist to control which package versions +are used. ## Origins of Packages A source build must be self-contained, meaning the entire product must be built -from source in an offline environment. To achieve this, all packages dependencies must -be satisfied by one of the following: +from source in an offline environment. To achieve this, all packages +dependencies must be satisfied by one of the following: ### Source-Build-Reference-Packages The first repo that is built as part of source build is [source-build-reference-packages](https://github.com/dotnet/source-build-reference-packages). -This repo contains all of the reference packages used to build the product. The repo -contains [tooling to generate new reference packages](https://github.com/dotnet/source-build-reference-packages?tab=readme-ov-file#adding-new-packages). +This repo contains all of the reference packages used to build the product. The +repo contains [tooling to generate new reference +packages](https://github.com/dotnet/source-build-reference-packages?tab=readme-ov-file#adding-new-packages). ### Current Source Built Packages -This refers to all of the packages produced in the current build. The set of packages -available to each repo varies based on its build order. For example the msbuild repo -can take a dependency on the current version of Microsoft.CodeAnalysis from roslyn -because roslyn builds before msbuild. Conversely, since roslyn builds before msbuild, -roslyn cannot take a dependency on the current version of Microsoft.Build; it can only -take a dependency on a previously released version. +This refers to all of the packages produced in the current build. The set of +packages available to each repo varies based on its build order. For example the +msbuild repo can take a dependency on the current version of +Microsoft.CodeAnalysis from roslyn because roslyn builds before msbuild. +Conversely, since roslyn builds before msbuild, roslyn cannot take a dependency +on the current version of Microsoft.Build; it can only take a dependency on a +previously released version. ### Previous Source Built Packages -Because the .NET product uses itself to build, the .NET source build product must be -[bootstrapped](./bootstrapping-guidelines.md). This process allows the packages from the -previous source build release to be used to build the next version of the product. This -provides a means for breaking the product's circular dependencies. For example repos like -[arcade](https://github.com/dotnet/arcade) can self-reference its previous version to -produce the next version. +Because the .NET product uses itself to build, the .NET source build product +must be [bootstrapped](./bootstrapping-guidelines.md). This process allows the +packages from the previous source build release to be used to build the next +version of the product. This provides a means for breaking the product's +circular dependencies. For example repos like +[arcade](https://github.com/dotnet/arcade) can self-reference its previous +version to produce the next version. -When referencing previous source built packages, it is important to not leak these -previously built packages into the resulting packages/product. This is considered a -[poison leak](./leak-detection.md) and is not permitted during a source build as it -breaks the notion of building the product entirely from source. This hinders the -ability to service the product. +When referencing previous source built packages, it is important to not leak +these previously built packages into the resulting packages/product. This is +considered a [poison leak](./leak-detection.md) and is not permitted during a +source build as it breaks the notion of building the product entirely from +source. This hinders the ability to service the product. ## Package Versions -Package dependencies defined using -[Arcade's dependency patterns](https://github.com/dotnet/arcade/blob/main/Documentation/Darc.md) -will get lifted dynamically during a source build if the following conditions are met: +Package dependencies defined using [Arcade's dependency +patterns](https://github.com/dotnet/arcade/blob/main/Documentation/Darc.md) will +get lifted dynamically during a source build if the following conditions are +met: 1. The dependency is declared in the Version.Details.xml file. @@ -69,9 +74,9 @@ will get lifted dynamically during a source build if the following conditions ar ... ``` -1. A repository reference is defined in the - [VMR's project dependency graph](https://github.com/dotnet/dotnet/tree/main/repo-projects). -This reference does not have to be direct, it can be transitive. +1. A repository reference is defined in the [VMR's project dependency + graph](https://github.com/dotnet/dotnet/tree/main/repo-projects). This +reference does not have to be direct, it can be transitive. **{VMR repo project}.proj** @@ -81,14 +86,14 @@ This reference does not have to be direct, it can be transitive. ... ``` -When these conditions are met during a source build, the infrastructure will scan -the Version.Details.xml file and dynamically create two new Versions.props files -containing updated version properties for all non-pinned dependencies. +When these conditions are met during a source build, the infrastructure will +scan the Version.Details.xml file and dynamically create two new Versions.props +files containing updated version properties for all non-pinned dependencies. -**PackageVersions.Previous.props:** This will contain version properties with the -package versions from the [previous release of source build](#previous-source-built-packages). -If a new package exists that has never been released before, it will not have a -version property defined. +**PackageVersions.Previous.props:** This will contain version properties with +the package versions from the [previous release of source +build](#previous-source-built-packages). If a new package exists that has never +been released before, it will not have a version property defined. ```xml ... @@ -97,9 +102,9 @@ version property defined. ``` **PackageVersions.Current.props:** This will contain version properties with the -package versions from the [current source build](#current-source-built-packages). -If a package comes from a repo that has not been built yet, it will not have a version -property defined. +package versions from the [current source +build](#current-source-built-packages). If a package comes from a repo that has +not been built yet, it will not have a version property defined. ```xml ... @@ -107,20 +112,22 @@ property defined. ... ``` -These two version.props files get imported by the arcade source build infrastructure after -the repo's Version.props file. Therefore the repo's Versions.props property versions -get overridden by the source build versions. In the case of the `SystemCommandLineVersion` -example, the current source build version, 2.0.0-beta4, would win. All msbuild references -(e.g. project PackageReferences) to these Versions.props properties pick up the newer -versions. This is known as package version lifting since it lifts the originally defined -package version to the current source built version. This behavior only applies to source -build in the context of the [VMR](https://github.com/dotnet/dotnet) (see also -[Repo Level Source Builds](#repo-level-source-builds)). +These two version.props files get imported by the arcade source build +infrastructure after the repo's Version.props file. Therefore the repo's +Versions.props property versions get overridden by the source build versions. In +the case of the `SystemCommandLineVersion` example, the current source build +version, 2.0.0-beta4, would win. All msbuild references (e.g. project +PackageReferences) to these Versions.props properties pick up the newer +versions. This is known as package version lifting since it lifts the originally +defined package version to the current source built version. This behavior only +applies to source build in the context of the +[VMR](https://github.com/dotnet/dotnet) (see also [Repo Level Source +Builds](#repo-level-source-builds)). ### Transitive Version Properties -Transitive version properties in your Versions.props file may not work as intended with -source build. +Transitive version properties in your Versions.props file may not work as +intended with source build. #### Versions.props @@ -143,18 +150,18 @@ source build. ... ``` -In this case source build will override the `MicrosoftBuildFrameworkVersion` to the -latest version but the `MicrosoftBuildVersion` will remain set to `17.7.0-preview-23217-02` -because of the property evaluation order. If the desired behavior is for -`MicrosoftBuildVersion` to be set to the same value as `MicrosoftBuildFrameworkVersion` -for source build, then you either need to declare the Microsoft.Build dependency -in the Version.Details.xml file or move the `MicrosoftBuildVersion` assignment outside -of the Versions.props file. +In this case source build will override the `MicrosoftBuildFrameworkVersion` to +the latest version but the `MicrosoftBuildVersion` will remain set to +`17.7.0-preview-23217-02` because of the property evaluation order. If the +desired behavior is for `MicrosoftBuildVersion` to be set to the same value as +`MicrosoftBuildFrameworkVersion` for source build, then you either need to +declare the Microsoft.Build dependency in the Version.Details.xml file or move +the `MicrosoftBuildVersion` assignment outside of the Versions.props file. ### Repo Level Source Builds -The source build package lifting mechanism is not applicable when building individual -repos in source build mode because it doesn't have the context of the other product -repos or previous source build release. In repo source build mode, the versions of the -packages declared in the Versions.props are used (see also +The source build package lifting mechanism is not applicable when building +individual repos in source build mode because it doesn't have the context of the +other product repos or previous source build release. In repo source build mode, +the versions of the packages declared in the Versions.props are used (see also [backlog issue](https://github.com/dotnet/source-build/issues/3562)). diff --git a/Documentation/packaging-installation.md b/Documentation/packaging-installation.md index 0955a7f7eb..ec2973c7ee 100644 --- a/Documentation/packaging-installation.md +++ b/Documentation/packaging-installation.md @@ -1,24 +1,38 @@ # .NET Core Packaging and Installation -This document helps you install or package a .NET Core SDK built using source-build. +This document helps you install or package a .NET Core SDK built using +source-build. -The SDK built by source-build is generally not portable. That means it will work on the same operating system where it was built. It will not work on older operating systems. It may work on newer operating systems. +The SDK built by source-build is generally not portable. That means it will work +on the same operating system where it was built. It will not work on older +operating systems. It may work on newer operating systems. -The built SDK is generally located at `artifacts/${ARCHITECTURE}/Release/dotnet-sdk-${SDK_VERSION}-${RUNTIME_ID}.tar.gz`. +The built SDK is generally located at +`artifacts/${ARCHITECTURE}/Release/dotnet-sdk-${SDK_VERSION}-${RUNTIME_ID}.tar.gz`. ## Using the SDK directly (install per-user) -To use the SDK directly, unpack the SDK to any directory and then use the `dotnet` executable from it. +To use the SDK directly, unpack the SDK to any directory and then use the +`dotnet` executable from it. -You can find more details on how to manually install an SDK from a tarball at [manually installing an SDK](https://docs.microsoft.com/en-us/dotnet/core/install/linux-ubuntu#manual-install). The same steps listed should work for all Linux distributions where `bash` (or `sh`) is the default shell. +You can find more details on how to manually install an SDK from a tarball at +[manually installing an +SDK](https://docs.microsoft.com/en-us/dotnet/core/install/linux-ubuntu#manual-install). +The same steps listed should work for all Linux distributions where `bash` (or +`sh`) is the default shell. ## Installing the SDK globally -If you want to install the SDK globally (and not per user), here are some suggestions. +If you want to install the SDK globally (and not per user), here are some +suggestions. -1. Extract the tarball to a distribution-appropriate location such as `/usr/local/lib64/dotnet/`, `/usr/local/lib/dotnet/`, or `/usr/local/lib/x86_64-linux-gnu/dotnet/`. +1. Extract the tarball to a distribution-appropriate location such as + `/usr/local/lib64/dotnet/`, `/usr/local/lib/dotnet/`, or + `/usr/local/lib/x86_64-linux-gnu/dotnet/`. -2. Create a symlink from `/usr/local/bin/dotnet` (or an equivalent location available in `$PATH`) to the `dotnet` binary in the SDK that you installed in the previous step. For example: +2. Create a symlink from `/usr/local/bin/dotnet` (or an equivalent location + available in `$PATH`) to the `dotnet` binary in the SDK that you installed in + the previous step. For example: ```bash ln -s /usr/local/lib64/dotnet/dotnet /usr/local/bin/dotnet @@ -26,15 +40,18 @@ If you want to install the SDK globally (and not per user), here are some sugges Now users can simply run `dotnet` and it will work. -3. Create an `/etc/dotnet/install_location` file and add the path of the SDK directory in there. The file should contain a single line like this: +3. Create an `/etc/dotnet/install_location` file and add the path of the SDK + directory in there. The file should contain a single line like this: ```bash /usr/local/lib64/dotnet/ ``` - This file is used by [.NET Core to find the SDK/Runtime location](https://github.com/dotnet/designs/blob/master/accepted/2020/install-locations.md) + This file is used by [.NET Core to find the SDK/Runtime + location](https://github.com/dotnet/designs/blob/master/accepted/2020/install-locations.md) -4. Define `DOTNET_ROOT` and update `PATH` by saving the following as `/etc/profile.d/dotnet-local.sh` (or equivalent) +4. Define `DOTNET_ROOT` and update `PATH` by saving the following as + `/etc/profile.d/dotnet-local.sh` (or equivalent) ```bash # Set location for AppHost lookup @@ -50,21 +67,29 @@ If you want to install the SDK globally (and not per user), here are some sugges Make sure to adjust the paths to match what you used on your system. - This snippet should work in `sh` (including `dash`) and `bash`. You may need to adapt it, or use something entirely different, for other shells. + This snippet should work in `sh` (including `dash`) and `bash`. You may need + to adapt it, or use something entirely different, for other shells. - This allows apphost-lookup to work via `DOTNET_ROOT` and allows users to easily use dotnet tools directly after a `dotnet tool install`. + This allows apphost-lookup to work via `DOTNET_ROOT` and allows users to + easily use dotnet tools directly after a `dotnet tool install`. ## Creating a Linux distribution package -If you want to create a Linux distribution package (`rpm`, `deb`) out of the source-build SDK here are some suggestions. +If you want to create a Linux distribution package (`rpm`, `deb`) out of the +source-build SDK here are some suggestions. -See [.NET Core distribution packaging](https://docs.microsoft.com/en-us/dotnet/core/distribution-packaging) for information on suggested packages, subpackages, name and contents. +See [.NET Core distribution +packaging](https://docs.microsoft.com/en-us/dotnet/core/distribution-packaging) +for information on suggested packages, subpackages, name and contents. This is the minimal amount of content you need to package up: -1. Extract the tarball to the distribution appropriate location such as `/usr/lib64/dotnet/`, `/usr/lib/dotnet/`, or `/usr/lib/x86_64-linux-gnu/dotnet/`. +1. Extract the tarball to the distribution appropriate location such as + `/usr/lib64/dotnet/`, `/usr/lib/dotnet/`, or + `/usr/lib/x86_64-linux-gnu/dotnet/`. -2. Create a symlink from `/usr/bin/dotnet` (or equivalent) to the `dotnet` binary in the SDK that you installed in the previous step. For example: +2. Create a symlink from `/usr/bin/dotnet` (or equivalent) to the `dotnet` + binary in the SDK that you installed in the previous step. For example: ```bash ln -s /usr/lib64/dotnet/dotnet /usr/bin/dotnet @@ -72,15 +97,18 @@ This is the minimal amount of content you need to package up: Now users can simply run `dotnet` and it will work. -3. Create an `/etc/dotnet/install_location` file and add the path of the SDK directory in there. The file should contain a single line like this: +3. Create an `/etc/dotnet/install_location` file and add the path of the SDK + directory in there. The file should contain a single line like this: ```bash /usr/lib64/dotnet ``` - This file is used by [.NET Core to find the SDK/Runtime location](https://github.com/dotnet/designs/blob/master/accepted/2020/install-locations.md). + This file is used by [.NET Core to find the SDK/Runtime + location](https://github.com/dotnet/designs/blob/master/accepted/2020/install-locations.md). -4. Define `DOTNET_ROOT` and update `PATH` by saving the following as `/etc/profile.d/dotnet.sh` (or equivalent) +4. Define `DOTNET_ROOT` and update `PATH` by saving the following as + `/etc/profile.d/dotnet.sh` (or equivalent) ```bash # Set location for AppHost lookup @@ -96,19 +124,32 @@ This is the minimal amount of content you need to package up: Make sure to adjust the paths to match what the distribution policies. - This snippet should work in `sh` (including `dash`) and `bash`. You may need to adapt it, or use something entirely different, for other shells. + This snippet should work in `sh` (including `dash`) and `bash`. You may need + to adapt it, or use something entirely different, for other shells. - This allows apphost-lookup to work via `DOTNET_ROOT` and allows users to easily use dotnet tools directly after a `dotnet tool install`. + This allows apphost-lookup to work via `DOTNET_ROOT` and allows users to + easily use dotnet tools directly after a `dotnet tool install`. There are other optional things you can do: -- .NET Core source repositories include man pages. You can search for them and package them up: `find -iname '*.1' -exec cp {} /usr/share/man/man1/ \;` +- .NET Core source repositories include man pages. You can search for them and + package them up: `find -iname '*.1' -exec cp {} /usr/share/man/man1/ \;` -- .NET Core includes [bash-completion](https://github.com/dotnet/cli/blob/master/scripts/register-completions.bash) and [zsh-completion](https://github.com/dotnet/cli/blob/master/scripts/register-completions.zsh) scripts. The copies in the source code you used to build the SDK should be the latest version. See [how to enable tab completion for .NET Core cli](https://docs.microsoft.com/en-us/dotnet/core/tools/enable-tab-autocomplete) for more information. +- .NET Core includes + [bash-completion](https://github.com/dotnet/cli/blob/master/scripts/register-completions.bash) + and + [zsh-completion](https://github.com/dotnet/cli/blob/master/scripts/register-completions.zsh) + scripts. The copies in the source code you used to build the SDK should be the + latest version. See [how to enable tab completion for .NET Core + cli](https://docs.microsoft.com/en-us/dotnet/core/tools/enable-tab-autocomplete) + for more information. ## Resources and references -- [.NET Core distribution packaging](https://docs.microsoft.com/en-us/dotnet/core/distribution-packaging) -- [Fedora .NET Core 3.1 package](https://src.fedoraproject.org/rpms/dotnet3.1/tree/master) +- [.NET Core distribution + packaging](https://docs.microsoft.com/en-us/dotnet/core/distribution-packaging) +- [Fedora .NET Core 3.1 + package](https://src.fedoraproject.org/rpms/dotnet3.1/tree/master) - [.NET Core - ArchWiki](https://wiki.archlinux.org/index.php/.NET_Core) -- [State of .NET Core on Arch - a discussion between a few distribution package maintainers](https://www.reddit.com/r/archlinux/comments/cx64r5/the_state_of_net_core_on_arch/) +- [State of .NET Core on Arch - a discussion between a few distribution package + maintainers](https://www.reddit.com/r/archlinux/comments/cx64r5/the_state_of_net_core_on_arch/) diff --git a/Documentation/patching-guidelines.md b/Documentation/patching-guidelines.md index 777689bf1a..698f4a5ac6 100644 --- a/Documentation/patching-guidelines.md +++ b/Documentation/patching-guidelines.md @@ -1,31 +1,33 @@ # Patching Guidelines There are times when the source build product will have build errors, functional -defects, poison leaks, prebuilts, etc. Ideally, the underlying issues -would be addressed at the source of the problem (e.g. within the product repos) -and flow into source build. The reality is that repo fixes can take a long - time to flow into source build. This can block source build development -(e.g. builds, tests, feature development, etc.). To avoid these delays source -patches are applied within source-build to the repo source. These patches are -intended to be ephemeral until the repo fixes flow in. +defects, poison leaks, prebuilts, etc. Ideally, the underlying issues would be +addressed at the source of the problem (e.g. within the product repos) and flow +into source build. The reality is that repo fixes can take a long time to flow + into source build. This can block source build development (e.g. builds, tests, +feature development, etc.). To avoid these delays source patches are applied +within source-build to the repo source. These patches are intended to be +ephemeral until the repo fixes flow in. This document provides guidance for creating and managing source build patches. ## Creating Patches To create a repo patch file, first commit your changes to the source repo (e.g. -runtime, aspnetcore) as normal, then run this command inside the repo to generate -a patch file inside the repo: +runtime, aspnetcore) as normal, then run this command inside the repo to +generate a patch file inside the repo: ```sh git format-patch --zero-commit --no-signature -1 ``` -Then, move the patch file into the `src/SourceBuild/patches/` directory of the -following repo: +Then, move the patch file into the `src/SourceBuild/patches/` directory of +the following repo: -* [.NET 9.0+] [sdk](https://github.com/dotnet/sdk/tree/main/src/SourceBuild/patches) -* [.NET 8.0] [installer](https://github.com/dotnet/installer/tree/main/src/SourceBuild/patches) +* [.NET 9.0+] + [sdk](https://github.com/dotnet/sdk/tree/main/src/SourceBuild/patches) +* [.NET 8.0] + [installer](https://github.com/dotnet/installer/tree/main/src/SourceBuild/patches) If an existing directory for the repo does not already exist, you will need to create one. @@ -50,30 +52,33 @@ git am "$PATCH_DIR/sdk/*" This creates a Git commit with the patch contents, so you can easily amend a patch or create a new commit on top that you can be sure will apply cleanly. -> **Note:** The VMR has all of the `src/SourceBuild/patches` applied. This is done -as part of the [synchronization process](https://github.com/dotnet/arcade/blob/main/Documentation/UnifiedBuild/VMR-Design-And-Operation.md#source-build-patches). +> **Note:** The VMR has all of the `src/SourceBuild/patches` applied. This is +done as part of the [synchronization +process](https://github.com/dotnet/arcade/blob/main/Documentation/UnifiedBuild/VMR-Design-And-Operation.md#source-build-patches). ## Patch Guidelines 1. **Documentation** - The patch file name and `Subject` should indicate the purpose of the patch. It - is often helpful to explain the problem/build issue the patch addresses. + The patch file name and `Subject` should indicate the purpose of the patch. + It is often helpful to explain the problem/build issue the patch addresses. 1. **Naming Conventions** - After generating the patch file, the numeric prefix of the filename may need to - be changed. By convention, new patches should be one number above the largest - number that already exists in the patch file directory. + After generating the patch file, the numeric prefix of the filename may need + to be changed. By convention, new patches should be one number above the + largest number that already exists in the patch file directory. - > **Note:** If there's a gap in the number sequence, do not fix it (generally - speaking), to avoid unnecessary diffs and potential merge conflicts. + > **Note:** If there's a gap in the number sequence, do not fix it + (generally speaking), to avoid unnecessary diffs and potential merge + conflicts. 1. **Backport** - All patches should have an issue or PR opened to address/backport the patch before - openings PRs that add new patches. All patches are required to include a link - in its `Subject` to the backport issue/PR in the following format: + All patches should have an issue or PR opened to address/backport the patch + before openings PRs that add new patches. All patches are required to + include a link in its `Subject` to the backport issue/PR in the following + format: `Backport: ` @@ -98,16 +103,18 @@ as part of the [synchronization process](https://github.com/dotnet/arcade/blob/m Dependency flow PRs into the installer repo can cause patch conflicts (e.g. the patch will fail to apply). These conflicts come in two forms. -1. The patch was backported - when this happen, the backported patch will need to -be deleted. +1. The patch was backported - when this happen, the backported patch will need +to be deleted. -1. The code being changed or surrounding code was changed - when this happens, the -patch will need to be updated. This can be done by manually updating the patch or -by re-applying the changes to the repo and recreating the patch. +1. The code being changed or surrounding code was changed - when this happens, +the patch will need to be updated. This can be done by manually updating the +patch or by re-applying the changes to the repo and recreating the patch. ## Unified Build Plans -The [Unified Build](https://github.com/dotnet/arcade/blob/main/Documentation/UnifiedBuild/README.md) -project will add support for source edits in the [VMR](https://github.com/dotnet/dotnet). -This will eliminate the need for patches as the required changes can be directly made in -the VMR. All changes made to the VMR will automatically flow to the associated repos. +The [Unified +Build](https://github.com/dotnet/arcade/blob/main/Documentation/UnifiedBuild/README.md) +project will add support for source edits in the +[VMR](https://github.com/dotnet/dotnet). This will eliminate the need for +patches as the required changes can be directly made in the VMR. All changes +made to the VMR will automatically flow to the associated repos. diff --git a/Documentation/planning/arcade-powered-source-build/README.md b/Documentation/planning/arcade-powered-source-build/README.md index ee44c4f2da..5f73d472ed 100644 --- a/Documentation/planning/arcade-powered-source-build/README.md +++ b/Documentation/planning/arcade-powered-source-build/README.md @@ -88,15 +88,18 @@ SBRP ultimately won't use the restore-side infrastructure because it has no upstreams, however, it can be developed here as a prototype. * [intermediate-nupkg.md] -* [Create proof of concept: the source-build intermediate nupkg format #1543](https://github.com/dotnet/source-build/issues/1543) -* [Use SBRP intermediate nupkg to build SBRP #1636](https://github.com/dotnet/source-build/issues/1636) +* [Create proof of concept: the source-build intermediate nupkg format + #1543](https://github.com/dotnet/source-build/issues/1543) +* [Use SBRP intermediate nupkg to build SBRP + #1636](https://github.com/dotnet/source-build/issues/1636) ### Prebuilt usage tracking We aren't eliminating instances of prebuilt usage at this stage in the process, but the tooling needs to work in the context of an Arcade-powered build. -* [Include prebuilt reports in intermediate nupkgs #1725](https://github.com/dotnet/source-build/issues/1725) +* [Include prebuilt reports in intermediate nupkgs + #1725](https://github.com/dotnet/source-build/issues/1725) ### Managed-only and RID-specific builds @@ -107,7 +110,8 @@ for all RIDs downstream. All repositories must be buildable under any RID, but for simple managed-only repositories, this is a reasonable assumption. The templates and Arcade SDK need to handle this. -* [Add mechanism to restore RID-specific intermediate nupkgs #1722](https://github.com/dotnet/source-build/issues/1722) +* [Add mechanism to restore RID-specific intermediate nupkgs + #1722](https://github.com/dotnet/source-build/issues/1722) ### Integrating tooling into Arcade @@ -116,8 +120,10 @@ points, initially. Before onboarding any extra repos, it should be integrated into the Arcade SDK proper to reuse the code. * [in-arcade.md] -* [Source code location in Arcade SDK (`Microsoft.DotNet.Arcade.Sdk/tools/SourceBuild`)](https://github.com/dotnet/arcade/tree/master/src/Microsoft.DotNet.Arcade.Sdk/tools/SourceBuild) -* [Source-build-specific MSBuild tool source in Arcade SDK (`Microsoft.DotNet.SourceBuild`)](https://github.com/dotnet/arcade/tree/master/src/Microsoft.DotNet.SourceBuild) +* [Source code location in Arcade SDK + (`Microsoft.DotNet.Arcade.Sdk/tools/SourceBuild`)](https://github.com/dotnet/arcade/tree/master/src/Microsoft.DotNet.Arcade.Sdk/tools/SourceBuild) +* [Source-build-specific MSBuild tool source in Arcade SDK + (`Microsoft.DotNet.SourceBuild`)](https://github.com/dotnet/arcade/tree/master/src/Microsoft.DotNet.SourceBuild) ## Incremental progress @@ -208,7 +214,8 @@ A: There are two reasons to make a patch: [speculative-build.md]: speculative-build.md [intermediate-nupkg.md]: intermediate-nupkg.md [intermediate nupkgs]: intermediate-nupkg.md -[onboarding/local-onboarding.md#patching]: onboarding/local-onboarding.md#patching +[onboarding/local-onboarding.md#patching]: + onboarding/local-onboarding.md#patching --- diff --git a/Documentation/planning/arcade-powered-source-build/intermediate-nupkg.md b/Documentation/planning/arcade-powered-source-build/intermediate-nupkg.md index f5082fea79..cad69f1c75 100644 --- a/Documentation/planning/arcade-powered-source-build/intermediate-nupkg.md +++ b/Documentation/planning/arcade-powered-source-build/intermediate-nupkg.md @@ -45,7 +45,8 @@ RID. * `Microsoft.SourceBuild.Intermediate.runtime.Libraries.linux.x64/6.0.0-foo` * Contains the dotnet/runtime platform extensions libraries. - * E.g. `System.IO.Pipelines.6.0.0-foo.nupkg`, `System.Numerics.Tensors.6.0.0-foo.nupkg`, ... + * E.g. `System.IO.Pipelines.6.0.0-foo.nupkg`, + `System.Numerics.Tensors.6.0.0-foo.nupkg`, ... * `Microsoft.SourceBuild.Intermediate.runtime.Crossgen2Pack.linux.x64/6.0.0-foo` * Contains the crossgen2 framework pack: * `Microsoft.NETCore.App.Crossgen2.linux-x64.6.0.0-foo.nupkg` diff --git a/Documentation/planning/multi-sdk-band-support.md b/Documentation/planning/multi-sdk-band-support.md index 9afa4e3abe..5619d8b1ec 100644 --- a/Documentation/planning/multi-sdk-band-support.md +++ b/Documentation/planning/multi-sdk-band-support.md @@ -1,91 +1,212 @@ # Source-Build Support for Multiple SDKs -This document serves as the design planning document for how .NET source build will support multiple SDK feature bands. More generally, this support could be described as "partial VMR support". This is the implementation plan for [Managing SDK Bands Unified Build feature](https://github.com/dotnet/arcade/blob/main/Documentation/UnifiedBuild/VMR-Managing-SDK-Bands.md). +This document serves as the design planning document for how .NET source build +will support multiple SDK feature bands. More generally, this support could be +described as "partial VMR support". This is the implementation plan for +[Managing SDK Bands Unified Build +feature](https://github.com/dotnet/arcade/blob/main/Documentation/UnifiedBuild/VMR-Managing-SDK-Bands.md). ## Terminology for this document -- **VMR** - Virtual Monolithic Repository - A projection of the sources of a set of input repo and branch combinations onto another repository. -- **SDK Feature Bands** - SDK functionality, especially around tooling, may evolve signficantly during the servicing lifetime of a .NET major release. This evolution is primarily to support new Visual Studio scenarios. To support this, .NET uses a series of SDK "bands" that correspond to VS releases and the .NET runtime GA release. They are denoted by diferences in the first digit of the patch numbers of the .NET SDK. All patch numbers have 3 digits. Fore more information, please see [official documentation](https://learn.microsoft.com/en-us/dotnet/core/releases-and-support#feature-bands-sdk-only). -- **Component** - A logical collection of sources in a VMR, mapped from an input source repo, which build together to produce a set of binaries that relate in functionality. Examples include [runtime](https://github.com/dotnet/dotnet/tree/main/src/runtime), [roslyn](https://github.com/dotnet/dotnet/tree/main/src/roslyn) or [installer](https://github.com/dotnet/dotnet/tree/main/src/installer). -- **Previously source-built (PSB)** - Artifacts from a previous servicing/preview iteration, or from a bootstrap build. These may be used when building the current dotnet VMR branch, but may not be bundled into any output. +- **VMR** - Virtual Monolithic Repository - A projection of the sources of a set + of input repo and branch combinations onto another repository. +- **SDK Feature Bands** - SDK functionality, especially around tooling, may + evolve signficantly during the servicing lifetime of a .NET major release. + This evolution is primarily to support new Visual Studio scenarios. To support + this, .NET uses a series of SDK "bands" that correspond to VS releases and the + .NET runtime GA release. They are denoted by diferences in the first digit of + the patch numbers of the .NET SDK. All patch numbers have 3 digits. Fore more + information, please see [official + documentation](https://learn.microsoft.com/en-us/dotnet/core/releases-and-support#feature-bands-sdk-only). +- **Component** - A logical collection of sources in a VMR, mapped from an input + source repo, which build together to produce a set of binaries that relate in + functionality. Examples include + [runtime](https://github.com/dotnet/dotnet/tree/main/src/runtime), + [roslyn](https://github.com/dotnet/dotnet/tree/main/src/roslyn) or + [installer](https://github.com/dotnet/dotnet/tree/main/src/installer). +- **Previously source-built (PSB)** - Artifacts from a previous + servicing/preview iteration, or from a bootstrap build. These may be used when + building the current dotnet VMR branch, but may not be bundled into any + output. ## Overview and General Approach -The general approach to supporting more than one SDK band is to **not** view the build of additional SDK feature bands as being any different from any other source build. Each build of the product combines a set of inputs (previously source-built artifacts + source) to produce a set of outputs that can ship to customers. When building two completely different major versions of .NET, the set of inputs is different (e.g. different source, 6.0 SDK vs. 8.0 SDK). Some of the previously source-built artifacts come from .NET, some may come from the source-built package ecosystem (e.g. icu or clang/llvm). If a repo were to be eliminated from the input sources, the binaries previously built from those sources would now need to come from previously-source built binaries. - -The primary restriction on building multiple SDK feature bands is that they must not differ in the shared runtimes. For N bands, there must be a single runtime artifact. Thus, we can view a source-build of N SDK feature bands as: - -- As a single build of one runtime, producing a set of source-built outputs and **then** +The general approach to supporting more than one SDK band is to **not** view the +build of additional SDK feature bands as being any different from any other +source build. Each build of the product combines a set of inputs (previously +source-built artifacts + source) to produce a set of outputs that can ship to +customers. When building two completely different major versions of .NET, the +set of inputs is different (e.g. different source, 6.0 SDK vs. 8.0 SDK). Some of +the previously source-built artifacts come from .NET, some may come from the +source-built package ecosystem (e.g. icu or clang/llvm). If a repo were to be +eliminated from the input sources, the binaries previously built from those +sources would now need to come from previously-source built binaries. + +The primary restriction on building multiple SDK feature bands is that they must +not differ in the shared runtimes. For N bands, there must be a single runtime +artifact. Thus, we can view a source-build of N SDK feature bands as: + +- As a single build of one runtime, producing a set of source-built outputs and + **then** - Building N SDK feature bands **using** -- The *outputs* of the runtime build combined with additional previously source-built artifacts (e.g. a matching band SDK, required tools and libraries) and source, to produce N SDK feature bands that meet source-build requirements. - -The runtime build requires a 1xx band SDK to build **and** redistributes functionality from tooling that ships in the 1xx band (e.g. it has dependencies on roslyn and roslyn-analyzers). Furthermore, most changes to a .NET major version occur when only one SDK band exists. Therefore it makes practical sense at this point to include the source of the 1xx SDK and the shared runtime in the same dotnet VMR branch, and to produce all artifacts related to 1xx, including the runtime, in its build. *Newer bands will be branches of the dotnet VMR that contain **only** the components that **differ** from the 1xx band.* Only those components that differ. When building the newer band, the outputs of the 1xx build would be fed into the newer band build, in a similar manner as previously source built artifacts are today. If the source for a component does not exist in the dotnet VMR, it is simply not built, and obviously no outputs would be produced for that component. Thus any downstream dependencies would be forced to use the versions provided by input source built artifacts, typically from the 1xx source build. If no such version exists, then the input would fail to be found or restored from online sources and be reported as a prebuilt. - -This approach fits naturally into the source-build methodology. Generally, additional SDK feature bands may be represented as "subsetted dotnet VMRs", since use of a subsetted dotnet VMR is not restricted to SDKs only. +- The *outputs* of the runtime build combined with additional previously + source-built artifacts (e.g. a matching band SDK, required tools and + libraries) and source, to produce N SDK feature bands that meet source-build + requirements. + +The runtime build requires a 1xx band SDK to build **and** redistributes +functionality from tooling that ships in the 1xx band (e.g. it has dependencies +on roslyn and roslyn-analyzers). Furthermore, most changes to a .NET major +version occur when only one SDK band exists. Therefore it makes practical sense +at this point to include the source of the 1xx SDK and the shared runtime in the +same dotnet VMR branch, and to produce all artifacts related to 1xx, including +the runtime, in its build. *Newer bands will be branches of the dotnet VMR that +contain **only** the components that **differ** from the 1xx band.* Only those +components that differ. When building the newer band, the outputs of the 1xx +build would be fed into the newer band build, in a similar manner as previously +source built artifacts are today. If the source for a component does not exist +in the dotnet VMR, it is simply not built, and obviously no outputs would be +produced for that component. Thus any downstream dependencies would be forced to +use the versions provided by input source built artifacts, typically from the +1xx source build. If no such version exists, then the input would fail to be +found or restored from online sources and be reported as a prebuilt. + +This approach fits naturally into the source-build methodology. Generally, +additional SDK feature bands may be represented as "subsetted dotnet VMRs", +since use of a subsetted dotnet VMR is not restricted to SDKs only. ### Conceptual splitting of 'previously source-built' artifact types -Today, the term "previously source-built artifacts' refers to those artifacts that were built in a previous product build iteration, typically from the previous month. For example, they would refer to the outputs of 8.0.1's build, when building 8.0.2. When building a subsetted dotnet VMR, two sets of input sets of source-built artifacts are required. While similar in some ways, they should be kept distinct. They are referred to as: - -- **Previously source-built (PSB)** - Artifacts from a previous servicing/preview iteration, or from a bootstrap build. These may be used when building the current dotnet VMR branch, but may not be bundled into any output. -- **Current source-built (CSB)** - Artifacts just built, e.g. from a 1xx dotnet VMR build. These may be used **and** bundled into the build outputs, since logically they are from the same source-build iteration. +Today, the term "previously source-built artifacts' refers to those artifacts +that were built in a previous product build iteration, typically from the +previous month. For example, they would refer to the outputs of 8.0.1's build, +when building 8.0.2. When building a subsetted dotnet VMR, two sets of input +sets of source-built artifacts are required. While similar in some ways, they +should be kept distinct. They are referred to as: + +- **Previously source-built (PSB)** - Artifacts from a previous + servicing/preview iteration, or from a bootstrap build. These may be used when + building the current dotnet VMR branch, but may not be bundled into any + output. +- **Current source-built (CSB)** - Artifacts just built, e.g. from a 1xx dotnet + VMR build. These may be used **and** bundled into the build outputs, since + logically they are from the same source-build iteration. ### Supported input artifact combinations and validation thereof -Not all toolsets can be used to build all dotnet VMR branches. SDK feature bands are driven by VS development, which moves forward between .NET releases. Those components of the .NET SDK introduce new functionality, which may require support from the SDK. Prior to the release of the corresponding SDK feature band, components of newer SDKs ocassionally take dependencies on SDK features introduced in the same band, and ocassionally deficiences in older SDK bands may require SDK upgrades. Therefore, it is possible that the 1xx SDK will not successfully build *all* Nxx+ bands from the same .NET major version. This may mean a bootstrap of the initial release of a new non-1xx SDK band. This is an unfortunate reality. - -To set expectations and reduce potential distro maintainer pain, the following statements are made: - -- The N.0.1xx dotnet VMR branch containing the runtime (1xx) is only gauranteed to build with the 1xx SDK feature band. -- A SDK feature band (e.g. 2xx) *may* be buildable with an older band, typically the 1xx band. However this is not gauranteed. E.g. `8.0.200` may be buildable with `8.0.103`. -- A newer SDK feature band (e.g. 2xx) will *always* be able to build itself. E.g. `8.0.200` will be able to build `8.0.200`. `8.0.105` can build `8.0.105`. -- A new release of an SDK feature band will *always* be buildable using the previous release of the same feature band. E.g. `8.0.201` will be buildable with `8.0.200`, and `8.0.102` will be buildable with `8.0.101`. - -From these statements. we can deduce that the initial release of a newer band (n00) may require a bootstrap, or may be buildable using an older band. +Not all toolsets can be used to build all dotnet VMR branches. SDK feature bands +are driven by VS development, which moves forward between .NET releases. Those +components of the .NET SDK introduce new functionality, which may require +support from the SDK. Prior to the release of the corresponding SDK feature +band, components of newer SDKs ocassionally take dependencies on SDK features +introduced in the same band, and ocassionally deficiences in older SDK bands may +require SDK upgrades. Therefore, it is possible that the 1xx SDK will not +successfully build *all* Nxx+ bands from the same .NET major version. This may +mean a bootstrap of the initial release of a new non-1xx SDK band. This is an +unfortunate reality. + +To set expectations and reduce potential distro maintainer pain, the following +statements are made: + +- The N.0.1xx dotnet VMR branch containing the runtime (1xx) is only gauranteed + to build with the 1xx SDK feature band. +- A SDK feature band (e.g. 2xx) *may* be buildable with an older band, typically + the 1xx band. However this is not gauranteed. E.g. `8.0.200` may be buildable + with `8.0.103`. +- A newer SDK feature band (e.g. 2xx) will *always* be able to build itself. + E.g. `8.0.200` will be able to build `8.0.200`. `8.0.105` can build `8.0.105`. +- A new release of an SDK feature band will *always* be buildable using the + previous release of the same feature band. E.g. `8.0.201` will be buildable + with `8.0.200`, and `8.0.102` will be buildable with `8.0.101`. + +From these statements. we can deduce that the initial release of a newer band +(n00) may require a bootstrap, or may be buildable using an older band. To avoid confusion, Microsoft will: -- Document the set of supported build combinations, including required inputs for all active SDK feature bands. This provides a guide for distro maintainers. +- Document the set of supported build combinations, including required inputs + for all active SDK feature bands. This provides a guide for distro + maintainers. - Validate those supported build combinations. - Validate 'stage2' builds (that a new SDK can build itself). ## Changes required for the source-build infrastructure -This section details changes to the source build infrastructure to support building of subsetted dotnet VMRs. +This section details changes to the source build infrastructure to support +building of subsetted dotnet VMRs. ### Scripting changes -Since conceptually CSB and PSB inputs are different, the scripting should reflect this. Furthermore, since NuGet and non-NuGet archives are treated similarly within the source-build infrastructure, we should take this opportunity to make changes to the names of existing switches: - -- `--with-packages` becomes `--with-previous-artifacts`, reflecting that it represents only previously source built artifacts that may be used in the build, but not redistributed with outputs. When poisoning is enabled, this set is poisoned. `--with-previous-artifacts` should be viewed as functionally equivalent to `--with-packages` -- `--with-current-artifacts` - This switch is added. It points to a directory that contains input NuGet and non-NuGet assets that may be redistributed in the current build. Typically this set would come from the 1xx build. - -In addition, we should add a script that prepares the input artifacts for the build of a particular dotnet VMR branch, given a set of input directories from other builds/other previously source built artifacts. See [below](#what-artifacts-should-be-passed-with---with-current-artifacts--with-previous-artifacts) for an explanation of the input artifacts. +Since conceptually CSB and PSB inputs are different, the scripting should +reflect this. Furthermore, since NuGet and non-NuGet archives are treated +similarly within the source-build infrastructure, we should take this +opportunity to make changes to the names of existing switches: + +- `--with-packages` becomes `--with-previous-artifacts`, reflecting that it + represents only previously source built artifacts that may be used in the + build, but not redistributed with outputs. When poisoning is enabled, this set + is poisoned. `--with-previous-artifacts` should be viewed as functionally + equivalent to `--with-packages` +- `--with-current-artifacts` - This switch is added. It points to a directory + that contains input NuGet and non-NuGet assets that may be redistributed in + the current build. Typically this set would come from the 1xx build. + +In addition, we should add a script that prepares the input artifacts for the +build of a particular dotnet VMR branch, given a set of input directories from +other builds/other previously source built artifacts. See +[below](#what-artifacts-should-be-passed-with---with-current-artifacts--with-previous-artifacts) +for an explanation of the input artifacts. #### Improvements to input artifact handling -`--with-previous-packages`/`--with-packages` takes an input directory. With this effort, the infrastructure should be changed to allow `--with-previous-artifacts` and `--with-current-artifacts` to take a tarball as input, in addition to a directory. The build scripting would extract a tarball to a temporary location. +`--with-previous-packages`/`--with-packages` takes an input directory. With this +effort, the infrastructure should be changed to allow +`--with-previous-artifacts` and `--with-current-artifacts` to take a tarball as +input, in addition to a directory. The build scripting would extract a tarball +to a temporary location. ### Changes to PackageVersion.props flow -When determining what version of a package or asset that a component should build against, there are 3 distinct sets in source-build today, listed in order of precedence from highest to lowest: +When determining what version of a package or asset that a component should +build against, there are 3 distinct sets in source-build today, listed in order +of precedence from highest to lowest: -- 1. A version just produced by a component built earlier in the build (if available). +- 1. A version just produced by a component built earlier in the build (if + available). - 2. A version in the previously source-built packages (if available). - 3. The version in the checked in `Versions.props` for a repo. -With the addition of a new input set (CSB), we now have an additional set. Changes should be made such that a PackageVersions.props file is generated for this new set, and is imported in the correct place based on its precedence. +With the addition of a new input set (CSB), we now have an additional set. +Changes should be made such that a PackageVersions.props file is generated for +this new set, and is imported in the correct place based on its precedence. -- 1. A version just produced by a component built earlier in the build (if available). +- 1. A version just produced by a component built earlier in the build (if + available). - 2. A version in the current source-built packages (if available). - 3. A version in the previously source-built packages (if available). - 3. The version in the checked in `Versions.props` for a repo. ### Filtering builds of components -The dotnet VMR defines a set of projects (under `repo-projects`) that correspond to each component of the build. These projects also define a build dependency graph of these components via the `RepositoryReference` ItemGroup. *Note: there are a few projects that do not correspond directly to a component, but instead serve as general join points in the build graph.*. Source build invokes a root build (`dotnet.proj`), which evaluates the dependency graph and invokes the MSBuild task on dependent projects. To avoid maintenance complexity, we should avoid altering this dependency graph or removing project files when a component should not be built. Instead, it should simply be filtered out and ignored as if the node in the graph was inactive. - -When filtering out components, we do not want to build a filtered subproject at all. `AfterTargets=Build`, `BeforeTargets=Build` are common in both component projects as well as the shared targets. We do not want these invoked at all, or else a large variety of special casing per-target will be required. Instead, each individual project should determine whether it should be built by setting a property. By default, this property would be set based on the whether the source for the component exists. Therefore, a user has two options for explicitly excluding or including a project: +The dotnet VMR defines a set of projects (under `repo-projects`) that correspond +to each component of the build. These projects also define a build dependency +graph of these components via the `RepositoryReference` ItemGroup. *Note: there +are a few projects that do not correspond directly to a component, but instead +serve as general join points in the build graph.*. Source build invokes a root +build (`dotnet.proj`), which evaluates the dependency graph and invokes the +MSBuild task on dependent projects. To avoid maintenance complexity, we should +avoid altering this dependency graph or removing project files when a component +should not be built. Instead, it should simply be filtered out and ignored as if +the node in the graph was inactive. + +When filtering out components, we do not want to build a filtered subproject at +all. `AfterTargets=Build`, `BeforeTargets=Build` are common in both component +projects as well as the shared targets. We do not want these invoked at all, or +else a large variety of special casing per-target will be required. Instead, +each individual project should determine whether it should be built by setting a +property. By default, this property would be set based on the whether the source +for the component exists. Therefore, a user has two options for explicitly +excluding or including a project: - Explicitly setting the property value in the project file - Removing the source code altogether from the branch. @@ -101,7 +222,11 @@ When filtering out components, we do not want to build a filtered subproject at false ``` -Given this set of properties, a new target can be added that will determine the active dependency projects (those `RepositoryReference` items that should be built) given the original set of a particular project's dependencies. Then, only active dependency projects would be built. Example code from the Proof-of-Concept is shown below. +Given this set of properties, a new target can be added that will determine the +active dependency projects (those `RepositoryReference` items that should be +built) given the original set of a particular project's dependencies. Then, only +active dependency projects would be built. Example code from the +Proof-of-Concept is shown below. ```xml @@ -141,13 +266,29 @@ Given this set of properties, a new target can be added that will determine the ### Choosing the correct source-built MSBuild SDKs -There is one wrinkle to the simple graph filtering approach. Most .NET projects have dependencies on the arcade toolset and associated MSBuild SDK. This dependency is handled specially by source-build as it helps bootstrap a repo's build process. There are three potential arcade versions available: +There is one wrinkle to the simple graph filtering approach. Most .NET projects +have dependencies on the arcade toolset and associated MSBuild SDK. This +dependency is handled specially by source-build as it helps bootstrap a repo's +build process. There are three potential arcade versions available: -- The bootstrap arcade version coming in via the previously source-built packages + artifacts (PSB). -- The bootstrap arcade version coming in via the current source-built packages + artifacts (CSB). +- The bootstrap arcade version coming in via the previously source-built + packages + artifacts (PSB). +- The bootstrap arcade version coming in via the current source-built packages + + artifacts (CSB). - The arcade built during the current source build invocation. -The 'bootstrap' version of arcade is the version of arcade present in CSB, or if not present, in PSB. Repositories built **before** arcade (and arcade itself) is built must choose the bootstrap arcade and know its location and version. Today, repos explicitly choose this via the `UseBootstrapArcade` property, which causes a series of environment variables to be set up for the Arcade SDK. If arcade is filtered from the graph, however, then a repo must **always** use the bootstrap arcade version. *In this case, the arcade should have been produced by the 1xx band build.* To achieve this, the UseBootstrapArcade property must be dynamically set, since the graph is dynamically filtered, based on whether arcade appears in the filtered input dependency projects. Then, the appropriate environment variables like `_InitializeToolset` would be set in a target based on its value. +The 'bootstrap' version of arcade is the version of arcade present in CSB, or if +not present, in PSB. Repositories built **before** arcade (and arcade itself) is +built must choose the bootstrap arcade and know its location and version. Today, +repos explicitly choose this via the `UseBootstrapArcade` property, which causes +a series of environment variables to be set up for the Arcade SDK. If arcade is +filtered from the graph, however, then a repo must **always** use the bootstrap +arcade version. *In this case, the arcade should have been produced by the 1xx +band build.* To achieve this, the UseBootstrapArcade property must be +dynamically set, since the graph is dynamically filtered, based on whether +arcade appears in the filtered input dependency projects. Then, the appropriate +environment variables like `_InitializeToolset` would be set in a target based +on its value. Example code from the PoC: @@ -168,7 +309,10 @@ Example code from the PoC: ``` -***Note: Today it is necessary to explicitly set UseBootstrapArcade for those projects where arcade is not available (arcade, source-build-reference-packages). It may be possible to remove this requirement, simply relying on the repo dependency graph to give the correct answer.*** +***Note: Today it is necessary to explicitly set UseBootstrapArcade for those +projects where arcade is not available (arcade, +source-build-reference-packages). It may be possible to remove this requirement, +simply relying on the repo dependency graph to give the correct answer.*** ### Input asset locations @@ -177,41 +321,89 @@ The input assets required for the .NET build come in two primary forms: - NuGet assets - Non-NuGet assets (typically `.zip` or `.tar.gz` archives) -Source build handles NuGet assets using a set of local NuGet feeds (which are just directories of NuGet packages). It handles non-Nuget assets by using one of these local NuGet feeds as a shared storage location where repos producing archives (e.g. runtime, sdk, aspnetcore) place their outputs (`blob-feed` dir). Downstream repos that need these archives copy the files from that location. They locate specific file names based on set naming patterns combined with package input versions. For instance, the installer repo might find the runtime archive for linux-x64 by looking for `dotnet-runtime--linux-x64.tar.gz`. When building a subsetted dotnet VMR, just as the shared location may not contain all the input packages, it may not contain all of the input archives. For NuGet packages, this is handled reasonably transparently. Given a set of potential input feeds, NuGet simply checks each one. So if arcade wasn't built and version 1.2.3 isn't available in the shared `blob-feed` dir, NuGet will look in the previously source-built artifacts feed. - -Because CSBs are separated from PSBs when building a subsetted dotnet VMR, source-build needs to be altered to add new CSB location as a NuGet input to each component build. In addition, builds of subsetted dotnet VMRs need a way to locate non-NuGet artifacts from the CSBs, without mixing them into the shared `blob-feed` where component builds currently look. Repositories should be altered to accept an additional location where they may find input artifacts. This has precedent, as it is the same logic used in installer, aspnetcore, windowsdesktop, and others in Microsoft's non-source-build scenarios. These repos will check a public official source for binaries first, then a public build artifacts location, and finally an internal build artifacts location, if the appropriate credentials are available. - -For example, `installer.proj` might add the following logic. dotnet/installer then uses `AdditionalBaseUrl` when available, adding it as a location for downloading assets: +Source build handles NuGet assets using a set of local NuGet feeds (which are +just directories of NuGet packages). It handles non-Nuget assets by using one of +these local NuGet feeds as a shared storage location where repos producing +archives (e.g. runtime, sdk, aspnetcore) place their outputs (`blob-feed` dir). +Downstream repos that need these archives copy the files from that location. +They locate specific file names based on set naming patterns combined with +package input versions. For instance, the installer repo might find the runtime +archive for linux-x64 by looking for `dotnet-runtime--linux-x64.tar.gz`. When building a subsetted dotnet VMR, just as the +shared location may not contain all the input packages, it may not contain all +of the input archives. For NuGet packages, this is handled reasonably +transparently. Given a set of potential input feeds, NuGet simply checks each +one. So if arcade wasn't built and version 1.2.3 isn't available in the shared +`blob-feed` dir, NuGet will look in the previously source-built artifacts feed. + +Because CSBs are separated from PSBs when building a subsetted dotnet VMR, +source-build needs to be altered to add new CSB location as a NuGet input to +each component build. In addition, builds of subsetted dotnet VMRs need a way to +locate non-NuGet artifacts from the CSBs, without mixing them into the shared +`blob-feed` where component builds currently look. Repositories should be +altered to accept an additional location where they may find input artifacts. +This has precedent, as it is the same logic used in installer, aspnetcore, +windowsdesktop, and others in Microsoft's non-source-build scenarios. These +repos will check a public official source for binaries first, then a public +build artifacts location, and finally an internal build artifacts location, if +the appropriate credentials are available. + +For example, `installer.proj` might add the following logic. dotnet/installer +then uses `AdditionalBaseUrl` when available, adding it as a location for +downloading assets: ```xml $(BuildCommandArgs) /p:AdditionalBaseUrl=file:%2F%2F$(AdditionalInputPackagesPath) ``` -In the above example, `AdditionalInputPackagesPath` is the current source-built packages (CSB) path (provided by `--with-current-assets`). This path is also added as a NuGet feed for each component. +In the above example, `AdditionalInputPackagesPath` is the current source-built +packages (CSB) path (provided by `--with-current-assets`). This path is also +added as a NuGet feed for each component. ### Gathering assets for delivery to customers -To gather assets for delivery to customers, the outputs present in each of the `artifacts//` (e.g. `artifacts/x64/Release`) should be used. For each build, the artifacts present in the output directory are only those artifacts that were produced in that build. +To gather assets for delivery to customers, the outputs present in each of the +`artifacts//` (e.g. `artifacts/x64/Release`) should be used. For +each build, the artifacts present in the output directory are only those +artifacts that were produced in that build. ## Changes to distro maintainer workflow -A distro maintainer wishing to support a newer band, either in addition to or in place of an older band, would see some change in workflow. First off, it is important to remember that the 1xx band will still be required. This is for two reasons: +A distro maintainer wishing to support a newer band, either in addition to or in +place of an older band, would see some change in workflow. First off, it is +important to remember that the 1xx band will still be required. This is for two +reasons: -- The 1xx SDK is the only SDK that is supported for building the runtime components. Because tooling differs between the bands, using a newer band to build the runtime is likely to expose new warnings or issues. -- There are components of the 1xx band that are used in the runtime. There are "sdk-like" components present in the runtime (analyzers, generators, etc.) that are dependent upon "sdk" functionality. Substituting in a 2xx aligned roslyn, for instance, would result in a product that differs from Microsoft's build. +- The 1xx SDK is the only SDK that is supported for building the runtime + components. Because tooling differs between the bands, using a newer band to + build the runtime is likely to expose new warnings or issues. +- There are components of the 1xx band that are used in the runtime. There are + "sdk-like" components present in the runtime (analyzers, generators, etc.) + that are dependent upon "sdk" functionality. Substituting in a 2xx aligned + roslyn, for instance, would result in a product that differs from Microsoft's + build. ### What artifacts should be passed with --with-current-artifacts/--with-previous-artifacts? -When building a full dotnet VMR branch (no skipped components), only --with-previous-artifacts should be passed. When building a subsetted dotnet VMR branch (e.g. 2xx, 3xx, etc.), both `--with-current-artifacts` and `--with-previous-artifacts` should be passed. +When building a full dotnet VMR branch (no skipped components), only +--with-previous-artifacts should be passed. When building a subsetted dotnet VMR +branch (e.g. 2xx, 3xx, etc.), both `--with-current-artifacts` and +`--with-previous-artifacts` should be passed. -`--with-current-artifacts` - The NuGet and non-NuGet archive outputs of the 1xx dotnet VMR branch build that is logically part of the same product iteration. -`--with-previous-artifacts` - The NuGet outputs of the previous dotnet VMR branch build(s) that are logically part of the previous product iteration. This set may encompass more than one set of build outputs. +`--with-current-artifacts` - The NuGet and non-NuGet archive outputs of the 1xx +dotnet VMR branch build that is logically part of the same product iteration. +`--with-previous-artifacts` - The NuGet outputs of the previous dotnet VMR +branch build(s) that are logically part of the previous product iteration. This +set may encompass more than one set of build outputs. -For an illustration of artifact flow, please see the [Appendix](#conceptual-diagram-of-source-build-artifact-flow) +For an illustration of artifact flow, please see the +[Appendix](#conceptual-diagram-of-source-build-artifact-flow) ### Previews and major release GA builds -A distro maintainer sees no change here. Until a few months **after** GA, only the 1xx band ships. The workflow remains the same as today. Clone the dotnet VMR branch/tag in question, build using previously source-built artifacts and SDKs. +A distro maintainer sees no change here. Until a few months **after** GA, only +the 1xx band ships. The workflow remains the same as today. Clone the dotnet VMR +branch/tag in question, build using previously source-built artifacts and SDKs. ### A distro maintainer wishing to only support the 1xx SDK @@ -219,67 +411,101 @@ A distro maintainer sees no change here. ### A distro maintainer wishing to build the initial release (n00) of a Nxx+ band SDK -The initial release of a Nxx+ band SDK should be straightforward. It is unlikely (though possible) that the initial Nxx band would depend on a Nxx SDK to build. If this is not the case, then a distro maintainer performs the following actions, and no bootstrap will be required: +The initial release of a Nxx+ band SDK should be straightforward. It is unlikely +(though possible) that the initial Nxx band would depend on a Nxx SDK to build. +If this is not the case, then a distro maintainer performs the following +actions, and no bootstrap will be required: 1. **Checkout** 1xx branch: -2. **Build** using `./build.sh --with-sdk --with-previous-artifacts `. -3. **Gather** 1xx build outputs from `artifacts/x64/Release` = ``. +2. **Build** using `./build.sh --with-sdk + --with-previous-artifacts `. +3. **Gather** 1xx build outputs from `artifacts/x64/Release` = ``. 4. **Checkout** Nxx branch -5. **Build** using `./build.sh --with-sdk --with-previous-artifacts --with-current-artifacts ` +5. **Build** using `./build.sh --with-sdk + --with-previous-artifacts --with-current-artifacts ` -If a Nxx SDK is required to build the initial Nxx SDK, then the following workflow is used: +If a Nxx SDK is required to build the initial Nxx SDK, then the following +workflow is used: 1. **Checkout** 1xx branch: -2. **Build** using `./build.sh --with-sdk --with-previous-artifacts `. -3. **Gather** 1xx build outputs from `artifacts/x64/Release` = ``. +2. **Build** using `./build.sh --with-sdk + --with-previous-artifacts `. +3. **Gather** 1xx build outputs from `artifacts/x64/Release` = ``. 4. **Checkout** Nxx branch 5. **Prep** using `./prep.sh` -6. **Build** with bootstrap SDK using `./build.sh --with-previous-artifacts --with-current-artifacts ` -7. **Gather** Nxx bootstrap inputs - ` + ` = `` -8. **Build** using `./build.sh --with-sdk --with-current-artifacts --with-previous-artifacts ` +6. **Build** with bootstrap SDK using `./build.sh --with-previous-artifacts --with-current-artifacts ` +7. **Gather** Nxx bootstrap inputs - ` + ` + = `` +8. **Build** using `./build.sh --with-sdk --with-current-artifacts + --with-previous-artifacts ` ### A distro maintainer wishing to support non-1xx SDKs (monthly workflow) -A distro maintainer who to support the Nxx (N>1) SDK(s) still needs to build the 1xx branch to obtain the runtime and an updated 1xx SDK for use the next month. Each non 1xx SDK is built with its corresponding SDK +A distro maintainer who to support the Nxx (N>1) SDK(s) still needs to build the +1xx branch to obtain the runtime and an updated 1xx SDK for use the next month. +Each non 1xx SDK is built with its corresponding SDK 1. **Checkout** 1xx branch: -2. **Build** using `./build.sh --with-sdk --with-previous-artifacts `. -3. **Gather** 1xx build outputs from `artifacts/x64/Release` = ``. +2. **Build** using `./build.sh --with-sdk + --with-previous-artifacts `. +3. **Gather** 1xx build outputs from `artifacts/x64/Release` = ``. The following additional steps may be done in parallel. For each SDK band Nxx: -1. **Gather** Previous Nxx inputs - ` + ` = `` +1. **Gather** Previous Nxx inputs - ` + ` + = `` 2. **Checkout** Nxx branch -3. **Build** using `./build.sh --with-sdk --with-previous-artifacts --with-current-artifacts ` +3. **Build** using `./build.sh --with-sdk + --with-previous-artifacts --with-current-artifacts ` ## Poison and Prebuilt detection Poisoning and prebuilt workflows may need slight tweaks. -- The poisoning infrastructure will not poison CSB inputs, but should continue to poison PSBs. +- The poisoning infrastructure will not poison CSB inputs, but should continue + to poison PSBs. - Prebuilt detection should not view CSB inputs as prebuilts. ## Potential issues and mitigations -There are some potential issues that may require tweaks to this approach, source fixes, additional guidance, etc. +There are some potential issues that may require tweaks to this approach, source +fixes, additional guidance, etc. ### A toolset input to a newer band does not work with the PSB from that same band -When preparing a newer band's input artifacts, there would be up to 3 versions of some components. For example: +When preparing a newer band's input artifacts, there would be up to 3 versions +of some components. For example: - PSB 1xx Microsoft.NET.Compilers.Toolset @ 4.7.1 - CSB 1xx Microsoft.NET.Compilers.Toolset @ 4.7.2 - PSB 2xx Microsoft.NET.Compilers.Toolset @ 4.8.0 -In this scenario, the *newest* version of Microsoft.NET.Compilers.Toolset will get used as input to any repo taking a dependency on it, before the 2xx dotnet/roslyn is built. If this particular repo has an explicit dependency on 4.7.x, then transparently updating to 4.8.x might cause a break. In the current source build infrastructure, the only way to avoid that transparent upgrade would be to not use the PSB 2xx inputs with the 2xx SDK build. But of course, the 2xx inputs would be required to provide supporting packages that the SDK depends on. +In this scenario, the *newest* version of Microsoft.NET.Compilers.Toolset will +get used as input to any repo taking a dependency on it, before the 2xx +dotnet/roslyn is built. If this particular repo has an explicit dependency on +4.7.x, then transparently updating to 4.8.x might cause a break. In the current +source build infrastructure, the only way to avoid that transparent upgrade +would be to not use the PSB 2xx inputs with the 2xx SDK build. But of course, +the 2xx inputs would be required to provide supporting packages that the SDK +depends on. -If this scenario ends up becoming a common problem, I think we could alter the source build PVP infrastructure to be smarter about version selection, attempting to match on best "MAJOR.MINOR" when multiple input versions are available. +If this scenario ends up becoming a common problem, I think we could alter the +source build PVP infrastructure to be smarter about version selection, +attempting to match on best "MAJOR.MINOR" when multiple input versions are +available. ## Appendix ### Conceptual diagram of source-build artifact flow -Below is an illustration of source-build currently uses various types of input artifacts: +Below is an illustration of source-build currently uses various types of input +artifacts: ```mermaid flowchart TD @@ -301,7 +527,8 @@ flowchart TD end ``` -What this proposal is doing is extending this concept of another set of "current" artifacts that come from another build invocation: +What this proposal is doing is extending this concept of another set of +"current" artifacts that come from another build invocation: ```mermaid flowchart TD diff --git a/Documentation/poison-report-format.md b/Documentation/poison-report-format.md index fc4248ede7..c6a8149ff6 100644 --- a/Documentation/poison-report-format.md +++ b/Documentation/poison-report-format.md @@ -1,6 +1,8 @@ # Interpreting the poison report -This is an explanation of the poison report and its interpretation. For an overview of poisoning, [see this document](leak-detection.md). The poison report looks something like this: +This is an explanation of the poison report and its interpretation. For an +overview of poisoning, [see this document](leak-detection.md). The poison +report looks something like this: ```xml @@ -24,26 +26,39 @@ This is an explanation of the poison report and its interpretation. For an over ``` -The first thing to notice is the different `Type`s. These can help you track down where the leak came from. One match can have multiple types, e.g. `AssemblyAttribute,Hash`. +The first thing to notice is the different `Type`s. These can help you track +down where the leak came from. One match can have multiple types, e.g. +`AssemblyAttribute,Hash`. ## Failure types ### AssemblyAttribute -This means that we detected a custom attribute that source-build adds to each prebuilt and previously-source-built assembly before the build. This is a *definite* leak - there will not be any false positives. However, this may not be cause for failure - some categories of assemblies may leak without causing problems: +This means that we detected a custom attribute that source-build adds to each +prebuilt and previously-source-built assembly before the build. This is a +*definite* leak - there will not be any false positives. However, this may not +be cause for failure - some categories of assemblies may leak without causing +problems: - Localization assemblies, e.g. .../langcode/Assembly.Resources.dll -- Reference assemblies (these should not ship, but also will very rarely cause problems). +- Reference assemblies (these should not ship, but also will very rarely cause + problems). - Test or resource assemblies - these should be removed. -- N-1 (previously-source-built) assemblies included in shipping packages to break cycles - this is expected behavior. Usually this use case is covered by reference packages but in rare cases we need to actually run an N-1 binary in the build. +- N-1 (previously-source-built) assemblies included in shipping packages to + break cycles - this is expected behavior. Usually this use case is covered by + reference packages but in rare cases we need to actually run an N-1 binary in + the build. ### SourceBuildReferenceAssembly -This means that a reference assembly originating from the source-build-reference-packages repo was leaked. +This means that a reference assembly originating from the +source-build-reference-packages repo was leaked. ### Hash -Hash failures mean that a file matches the catalog of hashes that we take of existing files before the build. These are commonly false positives due to shipping static files not produced by the build such as: +Hash failures mean that a file matches the catalog of hashes that we take of +existing files before the build. These are commonly false positives due to +shipping static files not produced by the build such as: - Text files (licenses, templates, etc) - Empty files (\_.\_) @@ -52,20 +67,33 @@ However, if this is a DLL this is probably a legitimate leak. ### NupkgFile -This type of failure means that a nupkg was copied wholesale to the final output. This is a definite leak, but very uncommon and likely easy to trace in the binlogs. +This type of failure means that a nupkg was copied wholesale to the final +output. This is a definite leak, but very uncommon and likely easy to trace in +the binlogs. ## Other metadata ### \ -This is the location in the output where the leak was detected. The first component of the path is an archive or package name, e.g. `dotnet-sdk-6.0.100` is `dotnet-sdk-6.0.100.tar.gz` in the `artifacts/` directory. Components after that are the file's path within the archive or nupkg. +This is the location in the output where the leak was detected. The first +component of the path is an archive or package name, e.g. `dotnet-sdk-6.0.100` +is `dotnet-sdk-6.0.100.tar.gz` in the `artifacts/` directory. Components +after that are the file's path within the archive or nupkg. ### \ -This is the hash of the failing file. This is useful to track down the correct file if there are multiple with the same name. It is included in all cases, even when the failure type is not `Hash`. +This is the hash of the failing file. This is useful to track down the correct +file if there are multiple with the same name. It is included in all cases, +even when the failure type is not `Hash`. ### \ -When available, this describes the file in the source-build *input* that matches the failing file. For instance: -`` -means that the failing `System.Text.Encoding.CodePages.dll` matches the hash of the file `lib/netcoreapp3.1/System.Text.Encoding.CodePages.dll` in the previously-source-built package System.Text.Encoding version 1.2.3. This can help track down the exact *input* file that made it to the final output. +When available, this describes the file in the source-build *input* that matches +the failing file. For instance: `` means that the +failing `System.Text.Encoding.CodePages.dll` matches the hash of the file +`lib/netcoreapp3.1/System.Text.Encoding.CodePages.dll` in the +previously-source-built package System.Text.Encoding version 1.2.3. This can +help track down the exact *input* file that made it to the final output. diff --git a/Documentation/sourcebuild-in-repos/adding-features.md b/Documentation/sourcebuild-in-repos/adding-features.md index f3f8540008..cb6327e0e7 100644 --- a/Documentation/sourcebuild-in-repos/adding-features.md +++ b/Documentation/sourcebuild-in-repos/adding-features.md @@ -1,43 +1,41 @@ # New features -New features are great! They also have source-build considerations though. -.NET is no longer just multi-platform but also multi-distribution: there's -a Linux SDK produced by Microsoft, but there's also a Linux SDK produced -by Red Hat, one produced by Fedora, and one that anyone in the community -can build himself from source. +New features are great! They also have source-build considerations though. .NET +is no longer just multi-platform but also multi-distribution: there's a Linux +SDK produced by Microsoft, but there's also a Linux SDK produced by Red Hat, one +produced by Fedora, and one that anyone in the community can build himself from +source. ## Things to consider -New features, or expansions of current features, should act sensibly -across Windows, Linux, and OSX. This also involves taking into account -the limitations and conventions of the different platforms - for instance, -on Linux, it's typical for the .NET SDK to be installed by root and -therefore be unchangeable by normal users, so installing global tools -needs to take into account the fact that the user might not be able to -add anything to the SDK directories (see -[this docker global tools issue](https://github.com/dotnet/dotnet-docker/issues/520) -and [a similar installer issue](https://github.com/dotnet/installer/issues/7069)). - -New features also need to be compatible across all distributions of -.NET - for instance, Fedora and Debian cannot distribute *any* -non-open-source code, even samples, tests, or minor additions to -licenses like restricting the field that the code can be used in. -This includes any code or packages that are used to build product -code as well. Microsoft generally prefers the MIT license. - -One example of a licensing issue that source-build found was with -dotnet-check and dotnet-format: these tools were brought into the -product build but had a dependency that was not licensed compatibly - -even without the source-build aspect. We had to scramble to replace -the dependency before the release. Dependencies need to be carefully -checked for license compatibility as well. +New features, or expansions of current features, should act sensibly across +Windows, Linux, and OSX. This also involves taking into account the limitations +and conventions of the different platforms - for instance, on Linux, it's +typical for the .NET SDK to be installed by root and therefore be unchangeable +by normal users, so installing global tools needs to take into account the fact +that the user might not be able to add anything to the SDK directories (see +[this docker global tools +issue](https://github.com/dotnet/dotnet-docker/issues/520) and [a similar +installer issue](https://github.com/dotnet/installer/issues/7069)). + +New features also need to be compatible across all distributions of .NET - for +instance, Fedora and Debian cannot distribute *any* non-open-source code, even +samples, tests, or minor additions to licenses like restricting the field that +the code can be used in. This includes any code or packages that are used to +build product code as well. Microsoft generally prefers the MIT license. + +One example of a licensing issue that source-build found was with dotnet-check +and dotnet-format: these tools were brought into the product build but had a +dependency that was not licensed compatibly - even without the source-build +aspect. We had to scramble to replace the dependency before the release. +Dependencies need to be carefully checked for license compatibility as well. ## Resources -Fedora's approved open-source licenses can be found -[on their wiki](https://fedoraproject.org/wiki/Licensing:Main#Good_Licenses), -or you can check on the [OSI-approved list of licenses](https://opensource.org/licenses/alphabetical). +Fedora's approved open-source licenses can be found [on their +wiki](https://fedoraproject.org/wiki/Licensing:Main#Good_Licenses), or you can +check on the [OSI-approved list of +licenses](https://opensource.org/licenses/alphabetical). -If you would like some of our distribution maintainer partners to -review your new feature you can ping @dotnet/distro-maintainers -on a PR or issue. +If you would like some of our distribution maintainer partners to review your +new feature you can ping @dotnet/distro-maintainers on a PR or issue. diff --git a/Documentation/sourcebuild-in-repos/build-info.md b/Documentation/sourcebuild-in-repos/build-info.md index 12ea53f96e..b8087c3135 100644 --- a/Documentation/sourcebuild-in-repos/build-info.md +++ b/Documentation/sourcebuild-in-repos/build-info.md @@ -1,50 +1,49 @@ # Source-build build info -This is a collection of notes about how source-build can differ in general -from your repo's build and what kind of issues that can create. +This is a collection of notes about how source-build can differ in general from +your repo's build and what kind of issues that can create. ## Single-version and single-RID build -Source-build is required to build on a single machine with no internet -access. This means that we build targeting a single RID, usually the -non-portable RID for the build machines (like rhel.7-x64). We do -support building portable (linux-x64) as well - this is useful for -bootstrapping new distributions. +Source-build is required to build on a single machine with no internet access. +This means that we build targeting a single RID, usually the non-portable RID +for the build machines (like rhel.7-x64). We do support building portable +(linux-x64) as well - this is useful for bootstrapping new distributions. -Source-build cannot build with any *prebuilts*. This is our term for -any package that comes from *outside the current source-build*. This means -that everything that ships out of source-build and everything that is used to -build those shipping products must come from the source-build in progress. -Packages from nuget.org, Microsoft builds, or other unrelated source-builds -cannot be used in source-build except in a limited bootstrapping process. +Source-build cannot build with any *prebuilts*. This is our term for any +package that comes from *outside the current source-build*. This means that +everything that ships out of source-build and everything that is used to build +those shipping products must come from the source-build in progress. Packages +from nuget.org, Microsoft builds, or other unrelated source-builds cannot be +used in source-build except in a limited bootstrapping process. Source-build supplies a *previously-source-built* set of packages for this -bootstrapping process. This is one way we have of breaking cycles in the -build. However, none of these packages can make it to the final build output. -This also means that your repo should be buildable with the immediately -previous version of the SDK than you are building for; i.e., if you are -building for 6.0.103, everything should be buildable with the 6.0.102 SDK. - -We also only build one version of each repo. This means that if your repo -turns production of some packages on and off, for instance, if you only -produce packages if they are changed, source-build will need a workaround -to force all packages to be produced. Additionally, we can only supply -one version of each package to a repo. This is injected into the -`$({PackageName}PackageVersion)` variables, e.g. SystemReflectionMetadataPackageVersion. -One exception is reference-only packages - +bootstrapping process. This is one way we have of breaking cycles in the build. +However, none of these packages can make it to the final build output. This also +means that your repo should be buildable with the immediately previous version +of the SDK than you are building for; i.e., if you are building for 6.0.103, +everything should be buildable with the 6.0.102 SDK. + +We also only build one version of each repo. This means that if your repo turns +production of some packages on and off, for instance, if you only produce +packages if they are changed, source-build will need a workaround to force all +packages to be produced. Additionally, we can only supply one version of each +package to a repo. This is injected into the `$({PackageName}PackageVersion)` +variables, e.g. SystemReflectionMetadataPackageVersion. One exception is +reference-only packages - [dotnet/source-build-reference-packages](https://github.com/dotnet/source-build-reference-packages) produces multiple versions and these can be hard-coded or use a -`$(ReferenceVersion)` property or similar if you don't -need source-build to change them. +`$(ReferenceVersion)` property or similar if you don't need +source-build to change them. ## Platform-specific packages Packages that require components or packages only available on some other -operating system than the building OS cannot be built in source-build. -These should use `true` or -other options to be excluded from the source-build. For instance, if a -project depends on a package that can only be built on Windows, it will need -to be disabled or worked around in source-build. As an example, -[Roslyn removes](https://github.com/dotnet/roslyn/blob/b999a65c8b0feeccb2b58da3d7a6e80e5f08feab/src/Workspaces/Core/Portable/Storage/PersistentStorageExtensions.cs#L23) +operating system than the building OS cannot be built in source-build. These +should use `true` or other +options to be excluded from the source-build. For instance, if a project +depends on a package that can only be built on Windows, it will need to be +disabled or worked around in source-build. As an example, [Roslyn +removes](https://github.com/dotnet/roslyn/blob/b999a65c8b0feeccb2b58da3d7a6e80e5f08feab/src/Workspaces/Core/Portable/Storage/PersistentStorageExtensions.cs#L23) a small performance improvement when building for source-build because it requires a component that isn't available. diff --git a/Documentation/sourcebuild-in-repos/new-dependencies.md b/Documentation/sourcebuild-in-repos/new-dependencies.md index 1a3c4d02ed..fc9de762ae 100644 --- a/Documentation/sourcebuild-in-repos/new-dependencies.md +++ b/Documentation/sourcebuild-in-repos/new-dependencies.md @@ -2,15 +2,15 @@ ## Basics -When adding a new dependency, there are a few steps common between any type -of dependency. If you use darc, `darc add-dependency` will take care of -this process for you. Otherwise: +When adding a new dependency, there are a few steps common between any type of +dependency. If you use darc, `darc add-dependency` will take care of this +process for you. Otherwise: 1. Add the package and a default version to eng/Versions.props. The default - version property should be named `PackageVersion`, - e.g. `SystemCollectionsImmutablePackageVersion`. -1. Add the package and a repo reference to eng/Version.Details.xml. This - should include a SourceBuild metadata entry, looking something like: + version property should be named `PackageVersion`, e.g. + `SystemCollectionsImmutablePackageVersion`. +1. Add the package and a repo reference to eng/Version.Details.xml. This should + include a SourceBuild metadata entry, looking something like: ```xml ` -version and the source-built intermediate package version don't match. -In this case, a direct dependency should be added to the source-build -intermediate package and the CoherentParentDependency attribute should be -set to the repo that consumes the dependency. For an example, -see [installer's F# dependency](https://github.com/dotnet/installer/blob/ba1739a2363b1062f03ea386ec67174c6468d3b2/eng/Version.Details.xml#L128). -You can find the version needed by running `darc get-build` -or using [BAR](https://aka.ms/bar). +version and the source-built intermediate package version don't match. In this +case, a direct dependency should be added to the source-build intermediate +package and the CoherentParentDependency attribute should be set to the repo +that consumes the dependency. For an example, see [installer's F# +dependency](https://github.com/dotnet/installer/blob/ba1739a2363b1062f03ea386ec67174c6468d3b2/eng/Version.Details.xml#L128). +You can find the version needed by running `darc get-build` or using +[BAR](https://aka.ms/bar). ## Adding dependencies Source build classifies dependencies in the following ways 1. .NET - a dependency on a component from the .NET org - e.g. dotnet/runtime -1. Microsoft - a dependency on a component from the Microsoft org - e.g. microsoft/vstest -1. External - a dependency on a component outside of Microsoft/.NET - e.g. JamesNK/Newtonsoft.Json +1. Microsoft - a dependency on a component from the Microsoft org - e.g. + microsoft/vstest +1. External - a dependency on a component outside of Microsoft/.NET - e.g. + JamesNK/Newtonsoft.Json The following checklist can be used to determine how to handle each type of dependency and the nuances it may have. -1. Are you already using a package from the same repo and is the new - dependency already source-built? You can determine this by checking if - the repo has a SourceBuild leg building, e.g. runtime's is - `runtime-dev-innerloop (Build Linux x64 release SourceBuild)`. - - 1. This is the simplest case. You can add new dependencies from repos - you are already using freely. Note that you need to use the same version - for all packages from the repo. -1. Is the repo already built in source-build including the specific - package you want? - 1. Add the dependency using [`darc add-dependency`](https://github.com/dotnet/arcade/blob/main/Documentation/Darc.md#add-dependency) as normal, then - add the [source-build metadata](#basics) as above. +1. Are you already using a package from the same repo and is the new dependency + already source-built? You can determine this by checking if the repo has a + SourceBuild leg building, e.g. runtime's is `runtime-dev-innerloop (Build + Linux x64 release SourceBuild)`. + + 1. This is the simplest case. You can add new dependencies from repos you + are already using freely. Note that you need to use the same version for + all packages from the repo. +1. Is the repo already built in source-build including the specific package you + want? + 1. Add the dependency using [`darc + add-dependency`](https://github.com/dotnet/arcade/blob/main/Documentation/Darc.md#add-dependency) + as normal, then add the [source-build metadata](#basics) as above. 1. Is the repo already built in source-build but the specific package is not? - 1. There's probably an issue with source-building this package. Please - talk to a [source-build team member](https://github.com/orgs/dotnet/teams/source-build-internal) - about why that is and whether we can fix it. + 1. There's probably an issue with source-building this package. Please talk + to a [source-build team + member](https://github.com/orgs/dotnet/teams/source-build-internal) about + why that is and whether we can fix it. 1. Is this a repo that uses Arcade to build? - 1. Does the foreign repo depend on your repo, directly or indirectly? - i.e. would adding the dependency create a cycle? + 1. Does the foreign repo depend on your repo, directly or indirectly? i.e. + would adding the dependency create a cycle? 1. This isn't necessarily a deal-breaker - it can sometimes be worked - around with reference-only packages. Please contact a - [source-build team member](https://github.com/orgs/dotnet/teams/source-build-internal) - to discuss. + around with reference-only packages. Please contact a [source-build + team + member](https://github.com/orgs/dotnet/teams/source-build-internal) to + discuss. 1. Does the foreign repo publish to BAR? - 1. If not, please contact them to get them publishing to BAR - in an appropriate channel. - 1. If neither of these caveats apply you should be in good shape. - Follow the instructions under "Basics" above. + 1. If not, please contact them to get them publishing to BAR in an + appropriate channel. + 1. If neither of these caveats apply you should be in good shape. Follow the + instructions under "Basics" above. 1. Dependencies that have no code (e.g. SDKs with just props and targets) can - usually be added using the [source-build text-only-package process](https://github.com/dotnet/source-build-reference-packages/tree/main/src/textOnlyPackages/src). - If the package is not already included there, you can [open a PR](https://github.com/dotnet/source-build-reference-packages/pulls) - or [file an issue](https://github.com/dotnet/source-build/issues/new/choose) - to include it. -1. Source-build has in the past used Arcade shims to allow non-Arcade repos - to build appropriate packages for source-build. Please - [log an issue](https://github.com/dotnet/source-build/issues/new/choose) - to determine if this is a workable approach for your foreign repo. -1. We build some external dependencies in the [dotnet/source-build-externals](https://github.com/dotnet/source-build-externals) + usually be added using the [source-build text-only-package + process](https://github.com/dotnet/source-build-reference-packages/tree/main/src/textOnlyPackages/src). + If the package is not already included there, you can [open a + PR](https://github.com/dotnet/source-build-reference-packages/pulls) or [file + an issue](https://github.com/dotnet/source-build/issues/new/choose) to include + it. +1. Source-build has in the past used Arcade shims to allow non-Arcade repos to + build appropriate packages for source-build. Please [log an + issue](https://github.com/dotnet/source-build/issues/new/choose) to determine + if this is a workable approach for your foreign repo. +1. We build some external dependencies in the + [dotnet/source-build-externals](https://github.com/dotnet/source-build-externals) repo. Good targets for this generally have very few if any dependencies and - very simple build processes. - [Please log an issue](https://github.com/dotnet/source-build/issues/new/choose) - to get the process started. + very simple build processes. [Please log an + issue](https://github.com/dotnet/source-build/issues/new/choose) to get the + process started. ## Deciding what version to use - If you are using the package as reference-only and want the version to be pinned, use a literal version number in the csprojs that reference the - project. You can also set up a reference-only package version variable - in eng/Versions.props, for instance `1.2.3` - in addition to `4.5.6`. - Also verify that the package is available in + project. You can also set up a reference-only package version variable in + eng/Versions.props, for instance + `1.2.3` in addition to + `4.5.6`. Also verify + that the package is available in [source-build-reference-packages](https://github.com/dotnet/source-build-reference-packages), - and if not, [file a source-build issue](https://github.com/dotnet/source-build/issues). -- If you are using the package in the actual build or want the version - to be updated whenever the foreign repo publishes to your channel, use - the version number property set up in eng/Versions.props. When performing - a source-build, the version number will get updated to the current - source-built version. -- If you are using an external or non-Arcade package, please coordinate as - much as possible with other teams using that package. Each - package-version is essentially maintained as a separate concern, so - something like repo A requiring Newtonsoft.Json 9.0.1 and repo B requiring - 12.0.2 essentially doubles the source-build team's work. + and if not, [file a source-build + issue](https://github.com/dotnet/source-build/issues). +- If you are using the package in the actual build or want the version to be + updated whenever the foreign repo publishes to your channel, use the version + number property set up in eng/Versions.props. When performing a source-build, + the version number will get updated to the current source-built version. +- If you are using an external or non-Arcade package, please coordinate as much + as possible with other teams using that package. Each package-version is + essentially maintained as a separate concern, so something like repo A + requiring Newtonsoft.Json 9.0.1 and repo B requiring 12.0.2 essentially + doubles the source-build team's work. diff --git a/Documentation/sourcebuild-in-repos/new-repo.md b/Documentation/sourcebuild-in-repos/new-repo.md index 1b7a59edb9..2ca49d779e 100644 --- a/Documentation/sourcebuild-in-repos/new-repo.md +++ b/Documentation/sourcebuild-in-repos/new-repo.md @@ -1,28 +1,30 @@ # Adding a new repository to source build -This document describes the overall process of onboarding new repos onto source build. +This document describes the overall process of onboarding new repos onto source +build. 1. [Source Build Configuration](#source-build-configuration) 1. [Setup CI](#setup-ci) -1. [Incorporate the new repo into the source build dependency tree](#incorporate-the-new-repo-into-the-source-build-dependency-tree) +1. [Incorporate the new repo into the source build dependency + tree](#incorporate-the-new-repo-into-the-source-build-dependency-tree) 1. [Source build repos and the VMR](#source-build-repos-and-the-vmr) 1. [Validate](#validate) 1. [Additional resources](#additional-resources) ## Source Build Configuration -Configuring source build involves setting up files in `eng/` that -determine the behavior of source build in the repo. +Configuring source build involves setting up files in `eng/` that determine the +behavior of source build in the repo. These changes are all needed before source build will work: -* [`eng/SourceBuild.props`](#engsourcebuildprops) - - Basic properties, such as repo name. +* [`eng/SourceBuild.props`](#engsourcebuildprops) - Basic properties, such as + repo name. * [`eng/SourceBuildPrebuiltBaseline.xml`](#engsourcebuildprebuiltbaselinexml) - List of allowed prebuilts (approval required). -* [`eng/Version.Details.xml`](#engversiondetailsxml) - - Already exists, but modifications are needed to pull dependencies from - upstream [intermediate nupkgs](planning/arcade-powered-source-build/README.md#intermediate-nupkg-outputsinputs). +* [`eng/Version.Details.xml`](#engversiondetailsxml) - Already exists, but + modifications are needed to pull dependencies from upstream [intermediate + nupkgs](planning/arcade-powered-source-build/README.md#intermediate-nupkg-outputsinputs). See the following sections for details: @@ -39,14 +41,14 @@ See the following sections for details: ``` -This file contains basic configuration used to restore the correct -dependencies (managed-only or not) and produce the right name for -the repo's [intermediate nupkg]. +This file contains basic configuration used to restore the correct dependencies +(managed-only or not) and produce the right name for the repo's [intermediate +nupkg]. -* `this-repo` should be the same as the repo's name on GitHub, - without the GitHub organization name. -* `SourceBuildManagedOnly` defaults to false if omitted. `true` means - that the repo doesn't produce any RID-specific artifacts like +* `this-repo` should be the same as the repo's name on GitHub, without the + GitHub organization name. +* `SourceBuildManagedOnly` defaults to false if omitted. `true` means that the + repo doesn't produce any RID-specific artifacts like `Microsoft.NETCore.App.Runtime.linux-x64`, only managed code. These two properties determine the name of the [intermediate nupkg]: @@ -69,17 +71,18 @@ MSBuild code that can change the way source build behaves. ``` -This defines which prebuilt NuGet packages are allowed to be used during -the build. Initially, only the source build intermediate packages are allowed. -The `*/*` glob means "any intermediate package, any version". -All other prebuilts require approval from the source build team. +This defines which prebuilt NuGet packages are allowed to be used during the +build. Initially, only the source build intermediate packages are allowed. The +`*/*` glob means "any intermediate package, any version". All other prebuilts +require approval from the source build team. When a PR introduces an unexpected prebuilt, PR validation will fail and let us resolve the source-buildability issue before the PR gets merged. ### CODEOWNERS -Add the source build team as the [CODEOWNER](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners) +Add the source build team as the +[CODEOWNER](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners) of the source build infrastructure. ``` text @@ -88,41 +91,44 @@ of the source build infrastructure. ### Trying it out locally -If a repo passes build options through to the common arcade build script, -a source build can be triggered via the following command. +If a repo passes build options through to the common arcade build script, a +source build can be triggered via the following command. ```bash ./build.sh -sb ``` -If a repo does not pass through build options through to the common arcade -build script, then you must specify the `/p:ArcadeBuildFromSource=true` property as follows. +If a repo does not pass through build options through to the common arcade build +script, then you must specify the `/p:ArcadeBuildFromSource=true` property as +follows. ```bash ./build.sh -c Release --restore --build --pack /p:ArcadeBuildFromSource=true -bl ``` -> Note: [source build is not supported on Windows](https://github.com/dotnet/source-build/issues/1190), -only Linux and macOS. +> Note: [source build is not supported on +Windows](https://github.com/dotnet/source-build/issues/1190), only Linux and +macOS. -After running the build, source build artifacts will be in `artifacts/source-build`, -and the [intermediate nupkg] will be something like +After running the build, source build artifacts will be in +`artifacts/source-build`, and the [intermediate nupkg] will be something like `artifacts/packages/*/Microsoft.SourceBuild.Intermediate.*.nupkg`. -The MSBuild binlog will be placed somewhere like: `artifacts/log/Debug/Build.binlog`. -However, this "outer" binlog doesn't contain the meat of the build: the -"inner" build runs inside an `Exec` task. The inner binlog will be written to: +The MSBuild binlog will be placed somewhere like: +`artifacts/log/Debug/Build.binlog`. However, this "outer" binlog doesn't contain +the meat of the build: the "inner" build runs inside an `Exec` task. The inner +binlog will be written to: `artifacts/source-build/self/src/artifacts/sourcebuild.binlog`. ### Excluding components -It is not always necessary or correct to include all repo components in -source build. Components should be excluded if they are not required for -the platform being source-built. Including them expands the source build -graph and may not be possible because of licensing. -Examples include tests, Windows components (remember source build currently -only supports Linux and macOS), etc. To exlcude these components -use the `DotNetBuildFromSource` msbuild property to conditionally exclude. +It is not always necessary or correct to include all repo components in source +build. Components should be excluded if they are not required for the platform +being source-built. Including them expands the source build graph and may not +be possible because of licensing. Examples include tests, Windows components +(remember source build currently only supports Linux and macOS), etc. To exlcude +these components use the `DotNetBuildFromSource` msbuild property to +conditionally exclude. ```code Condition="'$(DotNetBuildFromSource)' != 'true'" @@ -153,22 +159,22 @@ Dependency changes may include adding `SourceBuild` to existing dependency elements, and adding a new `source-build-reference-packages` element. `SourceBuild` causes the source build targets in the Arcade SDK to download -[intermediate nupkg]s from the upstream repo's official build, rather than -using prebuilt binaries to fulfill the dependencies. Note that `RepoName` -is used to calculate the ID of the -[intermediate nupkg](planning/arcade-powered-source-build/README.md#intermediate-nupkg-outputsinputs): +[intermediate nupkg]s from the upstream repo's official build, rather than using +prebuilt binaries to fulfill the dependencies. Note that `RepoName` is used to +calculate the ID of the [intermediate +nupkg](planning/arcade-powered-source-build/README.md#intermediate-nupkg-outputsinputs): the `Dependency` `Name` is ignored by source build. -Building with the source-built versions of your dependencies also means that -any upstream repos will have been built in a source build context, including -things projects that are excluded from the source build. This can help you -find issues where your source build depends on an upstream component that -isn't actually built in source build. +Building with the source-built versions of your dependencies also means that any +upstream repos will have been built in a source build context, including things +projects that are excluded from the source build. This can help you find issues +where your source build depends on an upstream component that isn't actually +built in source build. -`ManagedOnly` determines whether a RID suffix is necessary on the -[intermediate nupkg](planning/arcade-powered-source-build/README.md#intermediate-nupkg-outputsinputs) -ID. For example, running source build on `dotnet/installer` with -`linux-x64` with the above example configuration will restore: +`ManagedOnly` determines whether a RID suffix is necessary on the [intermediate +nupkg](planning/arcade-powered-source-build/README.md#intermediate-nupkg-outputsinputs) +ID. For example, running source build on `dotnet/installer` with `linux-x64` +with the above example configuration will restore: * `Microsoft.SourceBuild.Intermediate.runtime.linux-x64` * `.linux-x64` because `ManagedOnly` is not `true`. @@ -178,23 +184,22 @@ ID. For example, running source build on `dotnet/installer` with ## Setup CI Source build needs to run during official builds to create source build -[intermediate nupkg]s for the downstream repos that will consume them. -Source build should also run in PR validation builds, to prevent regressions. +[intermediate nupkg]s for the downstream repos that will consume them. Source +build should also run in PR validation builds, to prevent regressions. -Source build CI can be activated with a single flag in the -ordinary Arcade SDK jobs template for easy consumption. If a repo can't -simply use the Arcade SDK jobs template, more granular templates are -also available. +Source build CI can be activated with a single flag in the ordinary Arcade SDK +jobs template for easy consumption. If a repo can't simply use the Arcade SDK +jobs template, more granular templates are also available. -See -for the current template source code. The inline comments in the -`parameters:` section in those files are the most up to date docs, maintained -with higher priority than this general onboarding doc. +See for the +current template source code. The inline comments in the `parameters:` section +in those files are the most up to date docs, maintained with higher priority +than this general onboarding doc. ### `eng/common/templates/jobs/jobs.yml` opt-in switch -The simplest way to onboard. This approach applies if the repo already -uses the `eng/common/templates/jobs/jobs.yml` template. +The simplest way to onboard. This approach applies if the repo already uses the +`eng/common/templates/jobs/jobs.yml` template. To opt in: @@ -202,7 +207,8 @@ To opt in: Set `enableSourceBuild: true` in the template parameters. - This should look something like [this sourcelink implementation:](https://github.com/dotnet/sourcelink/blob/dfe619dc722be42d475595c755c958afe6177554/azure-pipelines.yml#L40) + This should look something like [this sourcelink + implementation:](https://github.com/dotnet/sourcelink/blob/dfe619dc722be42d475595c755c958afe6177554/azure-pipelines.yml#L40) ```yaml stages: @@ -225,8 +231,8 @@ To opt in: A repo is managed-only if `eng/SourceBuild.props` contains `true`. If this is true, - this step is not necessary. Otherwise, specify `sourceBuildParameters` - in the `jobs.yml` template's parameters like this: + this step is not necessary. Otherwise, specify `sourceBuildParameters` in + the `jobs.yml` template's parameters like this: ```yaml sourceBuildParameters: @@ -242,18 +248,18 @@ To opt in: Submit the changes above in a PR and include [@source-build-internal](https://github.com/orgs/dotnet/teams/source-build-internal) -as a reviewer. The jobs (or job, if managed-only) are automatically be added -to CI in the existing pipeline alongside existing jobs, -with a name like `Build Source-Build ()`: +as a reviewer. The jobs (or job, if managed-only) are automatically be added to +CI in the existing pipeline alongside existing jobs, with a name like `Build +Source-Build ()`: ![Build jobs image with source build leg](img/ci-job.png) Once this PR works, run a mock official build (AKA "validation build") in your official build definition. The usual workflow is to push a -`dev//` branch to the AzDO repo and then queue a build on that -branch. This makes sure that merging the PR won't immediately break the -official build: `enableSourceBuild: true` does add job(s) to the official -build, not just PR validation. +`dev//` branch to the AzDO repo and then queue a build +on that branch. This makes sure that merging the PR won't immediately break the +official build: `enableSourceBuild: true` does add job(s) to the official build, +not just PR validation. If the PR is green, but merging it produces red PR or Official builds, immediately let the source build team know about the failure and revert the @@ -266,15 +272,15 @@ templating infra built on top of the Arcade jobs template, the simple `enableSourceBuild` flag might not work out. There are a few more granular templates to use in this case. -Look at the documentation in each YAML file itself to figure out how to use -it properly, and if it fits the scenario. This list is only an overview. +Look at the documentation in each YAML file itself to figure out how to use it +properly, and if it fits the scenario. This list is only an overview. #### `eng/common/templates/jobs/source-build.yml` -This is one level deeper than `eng/common/templates/jobs/jobs.yml`. It is -a `jobs` template that produces just the set of source build jobs based -on the specified `platforms`. Or, just one job with the default platform, -if managed-only. +This is one level deeper than `eng/common/templates/jobs/jobs.yml`. It is a +`jobs` template that produces just the set of source build jobs based on the +specified `platforms`. Or, just one job with the default platform, if +managed-only. #### `eng/common/templates/job/source-build.yml` @@ -283,63 +289,64 @@ specifiedplatform. #### `eng/common/templates/steps/source-build.yml` -This template defines the build `steps` for a single source build job. -This is the most granular template, and may be useful if some -repo-specific preamble or cleanup steps are required, or if the repo -already has job matrix templates and this just happens to fit in nicely. +This template defines the build `steps` for a single source build job. This is +the most granular template, and may be useful if some repo-specific preamble or +cleanup steps are required, or if the repo already has job matrix templates and +this just happens to fit in nicely. ### Official build publishing Publishing [intermediate nupkg]s in the official build is handled by the -standard Arcade publish infrastructure, which should already be set up -in the repo. The source build steps handle uploading the -[intermediate nupkg] to the pipeline in the standard way that Arcade -will detect and publish. +standard Arcade publish infrastructure, which should already be set up in the +repo. The source build steps handle uploading the [intermediate nupkg] to the +pipeline in the standard way that Arcade will detect and publish. -[intermediate nupkg]: https://github.com/dotnet/source-build/blob/main/Documentation/planning/arcade-powered-source-build/intermediate-nupkg.md +[intermediate nupkg]: + https://github.com/dotnet/source-build/blob/main/Documentation/planning/arcade-powered-source-build/intermediate-nupkg.md ## Incorporate the new repo into the source build dependency tree -Once your repo can be source-built, it is time to register it into the -source build dependency tree. The graph of the product is defined by the +Once your repo can be source-built, it is time to register it into the source +build dependency tree. The graph of the product is defined by the `eng/Version.Details.xml` files. This dependency graph starts at [dotnet/installer](https://github.com/dotnet/installer/blob/main/eng/Version.Details.xml). -The dependendecies of repos declared in these files are walked and the -first copy/commit of each repo found in the dependency graph is used. +The dependendecies of repos declared in these files are walked and the first +copy/commit of each repo found in the dependency graph is used. -Therefore, when adding new repositories, a dependency must be created within -the `eng/Version.Details.xml` graph. To do this, **go to the repo which depends -on the new repo and add a [new source build dependency](#engversiondetailsxml) -to the new source build repo**. +Therefore, when adding new repositories, a dependency must be created within the +`eng/Version.Details.xml` graph. To do this, **go to the repo which depends on +the new repo and add a [new source build dependency](#engversiondetailsxml) to +the new source build repo**. ## Source build repos and the VMR -Another effect of adding a new source build repository is that its sources -will be synchronized into the [Virtual Monolithic Repository of .NET](https://github.com/dotnet/dotnet). -The VMR is then where the official source build happens from. The sources -are synchronized once the associated commit/package flows into `dotnet/installer`. - -In order for the sources of the new repo to by synchronized into the VMR, -the repo needs to be registered in the -[`source-mappings.json` file](https://github.com/dotnet/dotnet/blob/main/src/source-mappings.json) -which tells the tooling where from and which sources should be synchronized. -Please open a PR in [`dotnet/installer`](https://github.com/dotnet/installer) -and add your repository into `src/VirtualMonoRepo/source-mappings.json`. -The name must match the name declared in the `SourceBuild` tag in -`Version.Details.xml` created in the previous step. +Another effect of adding a new source build repository is that its sources will +be synchronized into the [Virtual Monolithic Repository of +.NET](https://github.com/dotnet/dotnet). The VMR is then where the official +source build happens from. The sources are synchronized once the associated +commit/package flows into `dotnet/installer`. + +In order for the sources of the new repo to by synchronized into the VMR, the +repo needs to be registered in the [`source-mappings.json` +file](https://github.com/dotnet/dotnet/blob/main/src/source-mappings.json) which +tells the tooling where from and which sources should be synchronized. Please +open a PR in [`dotnet/installer`](https://github.com/dotnet/installer) and add +your repository into `src/VirtualMonoRepo/source-mappings.json`. The name must +match the name declared in the `SourceBuild` tag in `Version.Details.xml` +created in the previous step. ### Cloaking (filtering) the repository sources -When creating the `source-mappings.json` record for the new repo, there is -a possibility of filtering which sources get synchronized into the VMR. -The VMR should only really contain plain text-based files as it is consumed -by 3rd party .NET distro builders who do not accept any non-text sources -(e.g. binaries). When registering the repository into the VMR, it is a good -time to consider which files are required for it to build and only synchronize -those. Commonly, repositories contain binaries that are required for testing -or similar purposes. Files like these should not be synchronized into the VMR. -Another common scenario is that the repo has multiple products/ship vehicles -and only a subset is needed for the source-built .NET scenario. +When creating the `source-mappings.json` record for the new repo, there is a +possibility of filtering which sources get synchronized into the VMR. The VMR +should only really contain plain text-based files as it is consumed by 3rd party +.NET distro builders who do not accept any non-text sources (e.g. binaries). +When registering the repository into the VMR, it is a good time to consider +which files are required for it to build and only synchronize those. Commonly, +repositories contain binaries that are required for testing or similar purposes. +Files like these should not be synchronized into the VMR. Another common +scenario is that the repo has multiple products/ship vehicles and only a subset +is needed for the source-built .NET scenario. ## Validate @@ -353,8 +360,8 @@ to be on the lookout for the new repo and they will validate as necessary. ## Additional resources -* For more details about how the build executes, see - [Arcade's build tools](https://github.com/dotnet/arcade/tree/main/src/Microsoft.DotNet.Arcade.Sdk/tools/SourceBuild). +* For more details about how the build executes, see [Arcade's build + tools](https://github.com/dotnet/arcade/tree/main/src/Microsoft.DotNet.Arcade.Sdk/tools/SourceBuild). * The source code for the build tasks that run for prebuilt validation and intermediate nupkg dependency reading are maintained in [Arcade](https://github.com/dotnet/arcade/tree/main/src/Microsoft.DotNet.SourceBuild) diff --git a/Documentation/sourcebuild-in-repos/update-dependencies.md b/Documentation/sourcebuild-in-repos/update-dependencies.md index faf9d082e1..b5a8414c6e 100644 --- a/Documentation/sourcebuild-in-repos/update-dependencies.md +++ b/Documentation/sourcebuild-in-repos/update-dependencies.md @@ -7,13 +7,12 @@ when doing manual updates. If you are manually updating a version, also see ## Internal packages -If you are manually updating a package, please make sure it's from a -compatible branch (e.g. runtime release/6.0 to sdk release/6.0.1xx, -release/6.0.2xx, etc). Package versions that you are updating to should -be source-built in their respective repos. If the version you need is -produced in a branch that is not yet source-build-compatible please let the -[source-build team](https://github.com/orgs/dotnet/teams/source-build-internal) -know. +If you are manually updating a package, please make sure it's from a compatible +branch (e.g. runtime release/6.0 to sdk release/6.0.1xx, release/6.0.2xx, etc). +Package versions that you are updating to should be source-built in their +respective repos. If the version you need is produced in a branch that is not +yet source-build-compatible please let the [source-build +team](https://github.com/orgs/dotnet/teams/source-build-internal) know. Another common error we see is updating eng/Versions.props but not eng/Version.Details.xml. This causes problems for source-build because we @@ -21,17 +20,16 @@ depend on these files being updated in lockstep. Please prefer updating with Darc - it takes care of these issues - or, if a manual update really is necessary, make sure you update both files. -Example Darc command: -`darc update-dependencies --name MyCoolPackage -v 1.2.3` +Example Darc command: `darc update-dependencies --name MyCoolPackage -v 1.2.3` ## External packages -Updating a non-Microsoft (e.g. Newtonsoft.Json 9.0.1 to 13.0.1) or -non-Arcade (e.g. ApplicationInsights) package is typically equivalent to -adding the new version all over again. Please -[log an issue](https://github.com/dotnet/source-build/issues/new/choose) -to discuss this. You can check if the external package you want is already -included in source-build in the +Updating a non-Microsoft (e.g. Newtonsoft.Json 9.0.1 to 13.0.1) or non-Arcade +(e.g. ApplicationInsights) package is typically equivalent to adding the new +version all over again. Please [log an +issue](https://github.com/dotnet/source-build/issues/new/choose) to discuss +this. You can check if the external package you want is already included in +source-build in the [source-build-externals](https://github.com/dotnet/source-build-externals) repo. ## Splitting, combining, and moving packages @@ -43,11 +41,12 @@ what can happen is 1. In version 2, Package A is split into Package A1 and Package A2. 1. A downstream repo is never updated to take the split packages. 1. The version 2 source-build succeeds because Package A exists in the - [previously-source-built archive](build-info.md#Single-version_and_single-RID_build), - but no new version of Package A is produced. -1. Source-build version 3 fails because Package A no longer exists in - the previously-source-built archive either. - -So please keep in mind consumers of your packages and try to keep them -informed of these kinds of changes. If you are the consuming repo, consider -tracking PRs in the repos that produce your dependencies. + [previously-source-built + archive](build-info.md#Single-version_and_single-RID_build), but no new + version of Package A is produced. +1. Source-build version 3 fails because Package A no longer exists in the + previously-source-built archive either. + +So please keep in mind consumers of your packages and try to keep them informed +of these kinds of changes. If you are the consuming repo, consider tracking PRs +in the repos that produce your dependencies. diff --git a/Documentation/system-requirements.md b/Documentation/system-requirements.md index e85ac3e21a..ddde4612fa 100644 --- a/Documentation/system-requirements.md +++ b/Documentation/system-requirements.md @@ -1,28 +1,43 @@ # System Requirements to Source-Build -This document provides the system requirements to source build the .NET SDK for a targeted platform. +This document provides the system requirements to source build the .NET SDK for +a targeted platform. ## Operating System ### Linux -* [Toolchain Setup](https://github.com/dotnet/runtime/blob/main/docs/workflow/requirements/linux-requirements.md#toolchain-setup) -* [Preconfigured Container Images](https://github.com/dotnet/dotnet-buildtools-prereqs-docker) - These images are used by [CI](https://github.com/dotnet/dotnet/blob/main/src/sdk/eng/pipelines/templates/stages/vmr-build.yml) to build and test source-build. -* [Distros Source Building .NET](https://github.com/dotnet/source-build#net-in-linux-distributions) +* [Toolchain + Setup](https://github.com/dotnet/runtime/blob/main/docs/workflow/requirements/linux-requirements.md#toolchain-setup) +* [Preconfigured Container + Images](https://github.com/dotnet/dotnet-buildtools-prereqs-docker) - These + images are used by + [CI](https://github.com/dotnet/dotnet/blob/main/src/sdk/eng/pipelines/templates/stages/vmr-build.yml) + to build and test source-build. +* [Distros Source Building + .NET](https://github.com/dotnet/source-build#net-in-linux-distributions) ### MacOS -MacOS is not currently supported: [Tracking Issue](https://github.com/dotnet/source-build/issues/2909). However, community users have created a [Homebrew project](https://github.com/Homebrew/homebrew-core/blob/master/Formula/dotnet.rb) to build .NET for OSX. Please feel free to open new issues in individual repos or in source-build for OSX issues. +MacOS is not currently supported: [Tracking +Issue](https://github.com/dotnet/source-build/issues/2909). However, community +users have created a [Homebrew +project](https://github.com/Homebrew/homebrew-core/blob/master/Formula/dotnet.rb) +to build .NET for OSX. Please feel free to open new issues in individual repos +or in source-build for OSX issues. ### Windows -Windows is not currently supported. [Tracking Issue](https://github.com/dotnet/source-build/issues/2910) +Windows is not currently supported. [Tracking +Issue](https://github.com/dotnet/source-build/issues/2910) ## Hardware ### Disk Space -80 GB of space is required for a typical build. You can reduced this down to ~30 GB if you build with the `clean-while-building` option. This might increase over time, so consider this to be a minimum bar. +80 GB of space is required for a typical build. You can reduced this down to ~30 +GB if you build with the `clean-while-building` option. This might increase over +time, so consider this to be a minimum bar. ### Memory diff --git a/README.md b/README.md index 589f007a79..69c946e898 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,12 @@ # .NET Source-Build -Please use [GitHub discussions](https://github.com/dotnet/source-build/discussions) to see announcements, ask questions, make suggestions, and share information with other members of the source-build community. +Please use [GitHub +discussions](https://github.com/dotnet/source-build/discussions) to see +announcements, ask questions, make suggestions, and share information with other +members of the source-build community. -This repo is the starting point for building .NET from source. It contains documentation, tools, and is used for issue tracking. +This repo is the starting point for building .NET from source. It contains +documentation, tools, and is used for issue tracking. ## Prerequisites @@ -10,18 +14,25 @@ This repo is the starting point for building .NET from source. It contains docum ## Building .NET 8.0+ -.NET 8.0 and newer will be built from the [dotnet/dotnet](https://github.com/dotnet/dotnet) repo. -Clone the dotnet/dotnet repo and check out the tag for the desired release. -Then, follow the instructions in [dotnet/dotnet's README](https://github.com/dotnet/dotnet/blob/main/README.md#dev-instructions) to build .NET from source. +.NET 8.0 and newer will be built from the +[dotnet/dotnet](https://github.com/dotnet/dotnet) repo. Clone the dotnet/dotnet +repo and check out the tag for the desired release. Then, follow the +instructions in [dotnet/dotnet's +README](https://github.com/dotnet/dotnet/blob/main/README.md#dev-instructions) +to build .NET from source. ## Building .NET 6.0 -.NET 6.0 is built from source using the [dotnet/installer](https://github.com/dotnet/installer/tree/release/6.0.1xx) repo. -Clone the dotnet/installer repo and check out the tag for the desired release. -Then, follow the instructions in [dotnet/installer's README](https://github.com/dotnet/installer/tree/release/6.0.1xx?tab=readme-ov-file#build-net-from-source-source-build) to build .NET from source. -Please see the [support](#support) section below to see which feature branches are currently supported. +.NET 6.0 is built from source using the +[dotnet/installer](https://github.com/dotnet/installer/tree/release/6.0.1xx) +repo. Clone the dotnet/installer repo and check out the tag for the desired +release. Then, follow the instructions in [dotnet/installer's +README](https://github.com/dotnet/installer/tree/release/6.0.1xx?tab=readme-ov-file#build-net-from-source-source-build) +to build .NET from source. Please see the [support](#support) section below to +see which feature branches are currently supported. -> The source-build repository doesn't currently support Windows. See [source-build#1190](https://github.com/dotnet/source-build/issues/1190). +> The source-build repository doesn't currently support Windows. See +> [source-build#1190](https://github.com/dotnet/source-build/issues/1190). ## Source-build goals @@ -29,9 +40,20 @@ There are two primary goals of the source-build effort: 1. Increase .NET adoption by focusing on making .NET available everywhere - If .NET was available everywhere - including Linux distributions and package managers like Homebrew - as a first class citizen, it would make .NET a more attractive option for developers who might otherwise look at other languages or runtimes. Users would be more likely to start using and keep using .NET if .NET is available on their development and release platforms of choice. - - To achieve this, we try to make it easier for community and partner maintainers to build and release .NET for their platforms. We need to make sure source-build satisfies the official packaging rules of commonly used Linux distributions, such as [Fedora](https://fedoraproject.org/wiki/Packaging:Guidelines) and [Debian](https://www.debian.org/doc/manuals/maint-guide/build.en.html). Many Linux distributions have similar rules. These rules tend to have three main principles: + If .NET was available everywhere - including Linux distributions and package + managers like Homebrew - as a first class citizen, it would make .NET a more + attractive option for developers who might otherwise look at other languages + or runtimes. Users would be more likely to start using and keep using .NET if + .NET is available on their development and release platforms of choice. + + To achieve this, we try to make it easier for community and partner + maintainers to build and release .NET for their platforms. We need to make + sure source-build satisfies the official packaging rules of commonly used + Linux distributions, such as + [Fedora](https://fedoraproject.org/wiki/Packaging:Guidelines) and + [Debian](https://www.debian.org/doc/manuals/maint-guide/build.en.html). Many + Linux distributions have similar rules. These rules tend to have three main + principles: * Limited or no network access * Consistent reproducibility @@ -39,30 +61,59 @@ There are two primary goals of the source-build effort: 2. Make maintenance of the .NET product easier - The current way of making changes to .NET during a servicing release is to make changes to individual product repositories and then adjust the dependency versions to flow the changes to the next set of repositories. This is repeated until all the repositories are updated. If there's an issue discovered late in the release cycle, the fixes and the dependency updates need to be re-done quickly, which becomes difficult. It's also difficult to verify that the issue is fixed in the final product. It would be much easier to make and test product-wide changes if we could make atomic changes to one repository and be able to build the whole product based on that source code at once. + The current way of making changes to .NET during a servicing release is to + make changes to individual product repositories and then adjust the + dependency versions to flow the changes to the next set of repositories. This + is repeated until all the repositories are updated. If there's an issue + discovered late in the release cycle, the fixes and the dependency updates + need to be re-done quickly, which becomes difficult. It's also difficult to + verify that the issue is fixed in the final product. It would be much easier + to make and test product-wide changes if we could make atomic changes to one + repository and be able to build the whole product based on that source code + at once. - In addition, getting source-build fully functional would provide everyone with a place to try changes that would otherwise require a lot of coordination between multiple repositories - such as landing features that require changes to both the runtime and the SDK. + In addition, getting source-build fully functional would provide everyone + with a place to try changes that would otherwise require a lot of + coordination between multiple repositories - such as landing features that + require changes to both the runtime and the SDK. - For more details about this Unified Build, see [this overview](https://github.com/dotnet/arcade/blob/main/Documentation/UnifiedBuild/Overview.md). + For more details about this Unified Build, see [this + overview](https://github.com/dotnet/arcade/blob/main/Documentation/UnifiedBuild/Overview.md). -Source-build can help achieve both these goals by making it easier for everyone to build and release the entire .NET product end-to-end. +Source-build can help achieve both these goals by making it easier for everyone +to build and release the entire .NET product end-to-end. ## What does the source-build infrastructure do? -Source-build solves common challenges that most developers encounter when trying to build the whole .NET SDK from source. +Source-build solves common challenges that most developers encounter when trying +to build the whole .NET SDK from source. -* .NET is composed of many repositories that need to be built at a specific combination of commits. +* .NET is composed of many repositories that need to be built at a specific + combination of commits. * Each repository's build output needs to flow into the next repository's build. -* By default, most .NET repositories download prebuilt binary dependencies from online sources. These are forbidden by typical Linux distribution rules, and interfere with build output flow. -* Some of the binary build dependencies are under a proprietary license, making it difficult to build everything without taking accidental dependencies on non-Open Source code. Many of our community members and partners want to build an Open Source .NET only. -* Flowing the build output automatically also means we can make and test changes that require coordination across a number of repositories. -* Nearly all .NET repositories require the .NET SDK to build. This is a circular dependency, which presents a bootstrapping problem, whether that's bringing up new architectures (eg, riscv), or support new operating systems (eg, FreeBSD). -* Microsoft controls the SDKs used by `dotnet-install.sh` scripts and `Microsoft.*` and `runtime.*` nuget packages at nuget.org; source-build makes it possible for the community to bring up the platforms without Microsoft having to accept/bless things. +* By default, most .NET repositories download prebuilt binary dependencies from + online sources. These are forbidden by typical Linux distribution rules, and + interfere with build output flow. +* Some of the binary build dependencies are under a proprietary license, making + it difficult to build everything without taking accidental dependencies on + non-Open Source code. Many of our community members and partners want to build + an Open Source .NET only. +* Flowing the build output automatically also means we can make and test changes + that require coordination across a number of repositories. +* Nearly all .NET repositories require the .NET SDK to build. This is a circular + dependency, which presents a bootstrapping problem, whether that's bringing up + new architectures (eg, riscv), or support new operating systems (eg, FreeBSD). +* Microsoft controls the SDKs used by `dotnet-install.sh` scripts and + `Microsoft.*` and `runtime.*` nuget packages at nuget.org; source-build makes + it possible for the community to bring up the platforms without Microsoft + having to accept/bless things. ## Comprehensive Guidelines -* [Bootstrapping new distro and architecture guidelines](Documentation/bootstrapping-guidelines.md) -* [Distribution packaging guidelines](https://learn.microsoft.com/dotnet/core/distribution-packaging) +* [Bootstrapping new distro and architecture + guidelines](Documentation/bootstrapping-guidelines.md) +* [Distribution packaging + guidelines](https://learn.microsoft.com/dotnet/core/distribution-packaging) ## .NET in Linux Distributions @@ -78,13 +129,20 @@ Source-build solves common challenges that most developers encounter when trying ## Support -.NET Source-Build is supported on the oldest available .NET SDK feature update for each major release, and on Linux only. -For example, if .NET `6.0.1xx`, `6.0.2xx`, `8.0.1xx`, and `8.0.2xx` feature updates are available from [dotnet.microsoft.com](https://dotnet.microsoft.com/en-us/download/dotnet/6.0), Source-Build will support `6.0.1xx` and `8.0.1xx`. +.NET Source-Build is supported on the oldest available .NET SDK feature update +for each major release, and on Linux only. For example, if .NET `6.0.1xx`, +`6.0.2xx`, `8.0.1xx`, and `8.0.2xx` feature updates are available from +[dotnet.microsoft.com](https://dotnet.microsoft.com/en-us/download/dotnet/6.0), +Source-Build will support `6.0.1xx` and `8.0.1xx`. -For the latest information about Source-Build support for new .NET versions, please check our [GitHub Discussions page](https://github.com/dotnet/source-build/discussions) for announcements. +For the latest information about Source-Build support for new .NET versions, +please check our [GitHub Discussions +page](https://github.com/dotnet/source-build/discussions) for announcements. -* [More information about .NET Versioning](https://docs.microsoft.com/en-us/dotnet/core/versions/) -* [More information about .NET Support Policies](https://dotnet.microsoft.com/en-us/platform/support/policy/dotnet-core) +* [More information about .NET + Versioning](https://docs.microsoft.com/en-us/dotnet/core/versions/) +* [More information about .NET Support + Policies](https://dotnet.microsoft.com/en-us/platform/support/policy/dotnet-core) ## License From f38f6a97d467058e0c37270beeb2dd717fdd8f36 Mon Sep 17 00:00:00 2001 From: Michael Simons Date: Tue, 6 Aug 2024 20:38:37 +0000 Subject: [PATCH 2/8] Remove MD line length rule suppression --- .markdownlint.json | 1 - 1 file changed, 1 deletion(-) diff --git a/.markdownlint.json b/.markdownlint.json index 619547d742..7144a657c0 100644 --- a/.markdownlint.json +++ b/.markdownlint.json @@ -1,5 +1,4 @@ { "default": true, - "MD013": false, // 80 char line length "MD033": false, // no inline HTML } From 2d58e08436d708a73ffd319af3e42310065a4fec Mon Sep 17 00:00:00 2001 From: Michael Simons Date: Tue, 6 Aug 2024 20:41:55 +0000 Subject: [PATCH 3/8] Update node-version --- .github/workflows/markdownlint.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/markdownlint.yml b/.github/workflows/markdownlint.yml index ecc9121bcd..5fbfec943e 100644 --- a/.github/workflows/markdownlint.yml +++ b/.github/workflows/markdownlint.yml @@ -19,7 +19,7 @@ jobs: steps: - uses: actions/checkout@v2 - name: Use Node.js - uses: actions/setup-node@v1 + uses: actions/setup-node@v4 with: node-version: lts/* - name: Run Markdownlint From 84dfec7bc9d2511e617924e64cf11350325a0023 Mon Sep 17 00:00:00 2001 From: Michael Simons Date: Tue, 6 Aug 2024 20:57:09 +0000 Subject: [PATCH 4/8] re-enable line length rule --- .markdownlint.json | 1 + 1 file changed, 1 insertion(+) diff --git a/.markdownlint.json b/.markdownlint.json index 7144a657c0..d1f7a68daa 100644 --- a/.markdownlint.json +++ b/.markdownlint.json @@ -1,4 +1,5 @@ { "default": true, + "MD013": false, // 80 char line length - produces undesireable results within code blocks "MD033": false, // no inline HTML } From 354dd6b28715a9867ccbb8d9244825d1159b4e7b Mon Sep 17 00:00:00 2001 From: Michael Simons Date: Tue, 6 Aug 2024 21:04:59 +0000 Subject: [PATCH 5/8] reword to avoid violation --- Documentation/boostrap-new-os.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Documentation/boostrap-new-os.md b/Documentation/boostrap-new-os.md index c01997472f..f731219913 100644 --- a/Documentation/boostrap-new-os.md +++ b/Documentation/boostrap-new-os.md @@ -30,8 +30,8 @@ In practice, most maintainers will bootstrap once every major version (e.g. 6.0.100, 8.0.100, etc). If you bootstrap these initial versions once you will generally be able to build 6.0.101 with 6.0.100, 6.0.102 with 6.0.101, 8.0.101 with 8.0.100, and so on. The source-build team attempts to avoid requiring -bootstrapping more than once per major version but sometimes this is unavoidable -- often when we move from one feature band to another, or if a major language +bootstrapping more than once per major version but sometimes this is unavoidable, +often when we move from one feature band to another, or if a major language feature is added, or if there are issues discovered with a given version of the SDK after release. These types of situations will be announced on the source-build [discussions From 1d0e1208d24b5b73b864cf2fbf86a1a3be2386d7 Mon Sep 17 00:00:00 2001 From: Michael Simons Date: Tue, 6 Aug 2024 22:10:00 +0000 Subject: [PATCH 6/8] Re-enable line length rule but allow long code block lines --- .markdownlint.json | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/.markdownlint.json b/.markdownlint.json index d1f7a68daa..144763adeb 100644 --- a/.markdownlint.json +++ b/.markdownlint.json @@ -1,5 +1,8 @@ { "default": true, - "MD013": false, // 80 char line length - produces undesireable results within code blocks + // line length + "MD013": { + "code_block_line_length": 256 + }, "MD033": false, // no inline HTML } From 7aebecb1bdc785b749963998815a9f95c9afbce9 Mon Sep 17 00:00:00 2001 From: Michael Simons Date: Tue, 6 Aug 2024 22:11:38 +0000 Subject: [PATCH 7/8] fix code block syntax --- Documentation/bootstrapping-guidelines.md | 1 + 1 file changed, 1 insertion(+) diff --git a/Documentation/bootstrapping-guidelines.md b/Documentation/bootstrapping-guidelines.md index 2956308496..f2a1424007 100644 --- a/Documentation/bootstrapping-guidelines.md +++ b/Documentation/bootstrapping-guidelines.md @@ -76,6 +76,7 @@ stage 1 to build .NET SDK from source. ```bash ./prep-source-build.sh --no-sdk --no-artifacts --with-sdk --with-packages + ``` 1. Build the source built .NET SDK. From f44e153d1016740cf44b8bb954bcff0982cf7895 Mon Sep 17 00:00:00 2001 From: Michael Simons Date: Tue, 6 Aug 2024 22:13:17 +0000 Subject: [PATCH 8/8] Tweaks --- .markdownlint.json | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.markdownlint.json b/.markdownlint.json index 144763adeb..cd8da2c680 100644 --- a/.markdownlint.json +++ b/.markdownlint.json @@ -2,7 +2,8 @@ "default": true, // line length "MD013": { - "code_block_line_length": 256 + "code_block_line_length": 256, + "heading_line_length": 96 }, "MD033": false, // no inline HTML }