Create internal CI & PR builds

[WilliamXieMSFT]

Summary

This change introduces internal builds that mirror our public builds in dev.azure.com/ms so that we can remove a cross-org PAT token (scoped to build-artifact-read, used by our test infrastructure) that was getting flagged.

Change Log

• Consolidate stages into build_and_package.yaml template that all top-level pipeline yaml can consume
• Introduce pr_internal.yaml and ci_internal.yaml that rely on internal templates
• Fixed some binskim issues that popped up (disabled in CI/PR builds as we have a dedicated internal analysis pipeline)

Test Methodology

Internal builds

• PR: https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=9314294&view=results
• CI: https://devdiv.visualstudio.com/DevDiv/_build?definitionId=11310&_a=summary
• StaticAnalysis: https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=9314357&view=results

[delmyers]

pool

refresh my memory: the reason that we have to have two of these is because the internal builds don't have outputs that are available to the public, right?

Are there other teams that are doing this in the same way? For example, Roslyn?

I can imagine a world where these builds aren't in this repo at all, instead, we have builds that are in a separate repo that check out the source code from here. But, I don't think that those can easily be triggered from changes in github, right?

[WilliamXieMSFT]

the internal builds don't have outputs that are available to the public, right?

That's correct, and yes, Roslyn has internal builds pulling from their public GitHub repo (eg. https://github.com/dotnet/roslyn/blob/main/azure-pipelines-pr-validation.yml)

We have builds that are in a separate repo that check out the source code from here

Moving the builds to a separate repo would help reduce churn in this repo but might make it more difficult to maintain build hygiene. Having an internal clone of the repo with builds is another option; Perfview has a process like that.

I don't think that those can easily be triggered from changes in github, right?

I think we can do something with GitHub webhooks or actions - maybe something we do in the near future

[delmyers]

[WilliamXieMSFT]

IndexSources: true

The only difference between CI and PR is whether we inex sources or not. However, the task itself doesn't support GitHub source provider so it doesn't do anything anyways.