Skip to content

Commit 17c6409

Browse files
dagooddagood
authored
Add doc describing arcade sb CI onboarding (#1793)
1 parent 0202230 commit 17c6409

File tree

3 files changed

+243
-0
lines changed

3 files changed

+243
-0
lines changed

‎Documentation/planning/arcade-powered-source-build/onboarding/README.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# Onboarding to arcade-powered source-build
2+
3+
Onboarding to arcade-powered source-build takes a few steps:
4+
5+
1. [Local onboarding: basic configuration in `eng/`.](local-onboarding.md)
6+
* Implemented by source-build team.
7+
2. [CI onboarding: adding PR validation and official build jobs.](ci-onboarding.md)
8+
* Implemented by repo source-build champion.
9+
10+
See [arcade-powered source-build](..) for more general information about the
11+
arcade-powered source-build plan.

‎Documentation/planning/arcade-powered-source-build/onboarding/ci-onboarding.md

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
# CI onboarding
2+
3+
Source-build needs to run during official builds to create source-build
4+
[intermediate nupkg]s for the downstream repos that will consume them.
5+
Source-build should also run in PR validation builds, to prevent regression in
6+
the source-build flow and (when enabled) catch unintended prebuilt usage.
7+
8+
The source-build job implementation is provided in the ordinary Arcade jobs
9+
template for easy consumption, but more detailed templates are also available in
10+
case the Arcade jobs template isn't suitable for certain repos.
11+
12+
See <https://github.com/dotnet/arcade/tree/master/eng/common/templates> for
13+
current implementation of the templates. The `parameters:` inline comments in
14+
those files should be the most up to date docs and maintained with higher
15+
priority than this doc.
16+
17+
## `eng/common/templates/jobs/jobs.yml` opt-in switch
18+
19+
This is the simplest way to onboard. So far we think this should only require a
20+
two line change if the repo already uses the arcade
21+
`eng/common/templates/jobs/jobs.yml` template and it's managed-only.
22+
("Managed-only" meaning it doesn't produce any NuGet packages that are specific
23+
to certain RIDs, like `Microsoft.NETCore.App.Runtime.linux-x64`.)
24+
25+
To opt in:
26+
* Set `runSourceBuild: true`
27+
* If the repo is managed-only, set:
28+
```yaml
29+
sourceBuildParameters:
30+
includeDefaultManagedPlatform: true
31+
```
32+
* If the repo is not managed-only, set `platforms:` to a list of objects.
33+
34+
A managed-only repo using this implementation should look something like
35+
[this source-build-reference-packages implementation:](https://github.com/dotnet/source-build-reference-packages/blob/0ee4e822dad9cc624b67f7486c2902fcbee05312/azure-pipelines/builds/ci.yml#L16-L31)
36+
37+
```yaml
38+
stages:
39+
- stage: build
40+
displayName: Build
41+
jobs:
42+
- template: /eng/common/templates/jobs/jobs.yml
43+
parameters:
44+
enablePublishUsingPipelines: true
45+
enablePublishBuildArtifacts: true
46+
enablePublishBuildAssets: true
47+
artifacts:
48+
publish:
49+
artifacts: true
50+
manifests: true
51+
runSourceBuild: true
52+
sourceBuildParameters:
53+
includeDefaultManagedPlatform: true
54+
```
55+
56+
A non-managed-only repo would instead specify `sourceBuildParameters` like this:
57+
58+
```yaml
59+
sourceBuildParameters:
60+
platforms:
61+
- name: 'Centos71_Portable'
62+
container: 'mcr.microsoft.com/dotnet-buildtools/prereqs:centos-7-3e800f1-20190501005343'
63+
- name: 'MacOS_Portable'
64+
pool: { vmImage: 'macOS-10.14' }
65+
- name: 'Centos71'
66+
nonPortable: true
67+
container: 'mcr.microsoft.com/dotnet-buildtools/prereqs:centos-7-3e800f1-20190501005343'
68+
- name: 'Centos8'
69+
nonPortable: true
70+
container: 'mcr.microsoft.com/dotnet-buildtools/prereqs:centos-8-daa5116-20200325130212'
71+
- name: 'Debian9'
72+
nonPortable: true
73+
container: 'mcr.microsoft.com/dotnet-buildtools/prereqs:debian-stretch-20200918130533-047508b'
74+
- name: 'Fedora30'
75+
nonPortable: true
76+
container: 'mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-30-38e0f29-20191126135223'
77+
- name: 'Fedora32'
78+
nonPortable: true
79+
container: 'mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-20200512010618-163ed2a'
80+
- name: 'MacOS'
81+
nonPortable: true
82+
pool: { vmImage: 'macOS-10.14' }
83+
- name: 'Ubuntu1804'
84+
nonPortable: true
85+
container: 'mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-20200918145614-047508b'
86+
```
87+
88+
## `eng/common/templates/jobs/source-build.yml`
89+
90+
This is one level deeper than `eng/common/templates/jobs/jobs.yml`. It is a
91+
`jobs` template that produces just the set of source-build jobs.
92+
93+
This template defines the platform used for `includeDefaultManagedPlatform`.
94+
95+
## `eng/common/templates/job/source-build.yml`
96+
97+
This template defines a single `job` that runs source-build on a specified
98+
platform.
99+
100+
## `eng/common/templates/steps/source-build.yml`
101+
102+
This template defines the `steps` for a single source-build job. This is the
103+
most granular template, and may be useful if some repo-specific preamble or
104+
cleanup steps are required, or if the repo already has job matrix templates and
105+
this just fits in nicely.
106+
107+
108+
# Official build publishing
109+
110+
Publishing [intermediate nupkg]s in the official build is handled by the
111+
standard Arcade publish infrastructure, which should already be set up in the
112+
repo. The source-build steps handle uploading the [intermediate nupkg] to the
113+
pipeline in the correct way for Arcade to detect and publish.
114+
115+
116+
[intermediate nupkg]: https://github.com/dotnet/source-build/blob/release/3.1/Documentation/planning/arcade-powered-source-build/intermediate-nupkg.md

‎Documentation/planning/arcade-powered-source-build/onboarding/local-onboarding.md

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
# Local onboarding
2+
3+
Local onboarding involves setting up files in `eng/` that determine the behavior
4+
of source-build in the repo.
5+
6+
7+
## Source-build configuration overview
8+
9+
### `eng/SourceBuild.props`
10+
11+
```xml
12+
<Project>
13+
14+
<PropertyGroup>
15+
<GitHubRepositoryName>this-repo</GitHubRepositoryName>
16+
<SourceBuildManagedOnly>true</SourceBuildManagedOnly>
17+
</PropertyGroup>
18+
19+
</Project>
20+
```
21+
22+
This file contains basic configuration used to restore the correct dependencies
23+
(managed-only or not) and produce the right name for the repo's [intermediate
24+
nupkg].
25+
26+
* `this-repo` should be the same as the repo's name on GitHub, without the
27+
GitHub organization name.
28+
* `SourceBuildManagedOnly` defaults to false if omitted. `true` means that the
29+
repo doesn't produce any RID-specific artifacts like
30+
`Microsoft.NETCore.App.Runtime.linux-x64`, only managed code.
31+
32+
These two properties determine the name of the [intermediate nupkg]:
33+
`Microsoft.SourceBuild.Intermediate.$(GitHubRepositoryName)[.$(RidSuffix)]`.
34+
35+
It's possible more configuration will be required for specific repos.
36+
`eng/SourceBuild.props`, similar to `eng/Build.props`, is a place to add extra
37+
MSBuild code that can change the way source-build behaves.
38+
39+
### `eng/SourceBuildPrebuiltBaseline.xml`
40+
41+
```xml
42+
<UsageData>
43+
<IgnorePatterns>
44+
<UsagePattern IdentityGlob="*/*" />
45+
</IgnorePatterns>
46+
</UsageData>
47+
```
48+
49+
This defines which prebuilt NuGet packages are allowed to be used during the
50+
build. Initially, all package IDs and versions are permitted (`*/*`).
51+
52+
The `*/*` glob means "any nupkg id, any version". It will be replaced with more
53+
specific rules at a later phase of the implementation plan. The goal is to make
54+
it specific, so when a PR introduces an unexpected prebuilt, PR validation will
55+
fail and let us resolve the source-buildability issue before the PR gets merged.
56+
57+
### `eng/Version.Details.xml`
58+
59+
```xml
60+
...
61+
<Dependency Name="Microsoft.NETCore.App.Runtime.win-x64" Version="6.0.0-alpha.1.20468.7" CoherentParentDependency="Microsoft.NET.Sdk">
62+
<Uri>https://github.com/dotnet/runtime</Uri>
63+
<Sha>a820ca1c4f9cb5892331e2624d3999c39161fe2a</Sha>
64+
<SourceBuild RepoName="runtime" />
65+
</Dependency>
66+
...
67+
<Dependency Name="Microsoft.SourceBuild.Intermediate.source-build-reference-packages" Version="5.0.0-alpha.1.20473.1">
68+
<Uri>https://github.com/dotnet/source-build-reference-packages</Uri>
69+
<Sha>def2e2c6dc5064319250e2868a041a3dc07f9579</Sha>
70+
<SourceBuild RepoName="source-build-reference-packages" ManagedOnly="true" />
71+
</Dependency>
72+
...
73+
```
74+
75+
Dependency changes may include adding `SourceBuild` to existing dependency
76+
elements, and adding a new `source-build-reference-packages` element.
77+
78+
`SourceBuild` causes the source-build targets in the Arcade SDK to download
79+
[intermediate nupkg]s from the upstream repo's official build, rather than using
80+
prebuilt binaries to fulfill the dependencies. Note that `RepoName` is used to
81+
calculate the ID of the [intermediate nupkg]: the `Dependency` `Name` is
82+
ignored by source-build.
83+
84+
`ManagedOnly` determines whether a RID suffix is necessary on the [intermediate
85+
nupkg] ID. For example, running source-build on `dotnet/installer` with
86+
`linux-x64` with the above example configuration will restore:
87+
88+
* `Microsoft.SourceBuild.Intermediate.runtime.linux-x64`
89+
* `.linux-x64` because `ManagedOnly` is not `true`.
90+
* `Microsoft.SourceBuild.Intermediate.source-build-reference-packages`
91+
* Ends with the `RepoName` without a suffix because `ManagedOnly="true"`.
92+
93+
94+
# Trying it out locally
95+
96+
Running source-build locally is done by passing `/p:ArcadeBuildFromSource=true`
97+
at the end of the usual arcade-based build command for the repo. For example:
98+
99+
```
100+
./build.sh -c Release --restore --build --pack /p:ArcadeBuildFromSource=true
101+
```
102+
103+
> Note: [source-build is not supported on
104+
> Windows](https://github.com/dotnet/source-build/issues/1190), only Linux and
105+
> macOS.
106+
107+
After running the build, source-build artifacts will be in
108+
`artifacts/source-build`, and the [intermediate nupkg] will be something like
109+
`artifacts/packages/*/Microsoft.SourceBuild.Intermediate.*.nupkg`.
110+
111+
It isn't strictly necessary to try it out locally to proceed with CI onboarding.
112+
The source-build contributor who submits the initial configuration PR to the
113+
repo will have tested out the build locally themselves.
114+
115+
116+
[intermediate nupkg]: https://github.com/dotnet/source-build/blob/release/3.1/Documentation/planning/arcade-powered-source-build/intermediate-nupkg.md

0 commit comments

Comments
 (0)