Add Azure Linux to regular CI #115415

richlander · 2025-05-08T22:38:04Z

Plan:

Upgrade Azure Linux to regular libraries CI.
Ensure Azure Linux and Ubuntu are both present in inner and outerloop
Ensure one raw VM is present in both inner and outerloop (but not the same one; partially blocked on Re-enable Azure Linux 3 testing #114276)
Debian can be moved to outer loop since it is no longer our default container OS
Outer loop should bias to latest OS version, like Ubuntu interim releases or Debian pre-release (which may require updates in servicing, for example Ubuntu 25.x to 26.04).
Will repeat on other branches (once this merges).

High-level summary:

Establishes clear tiers for functional tests for Linux.
Tier 1: Azure Linux, Debian, Ubuntu (due to being in inner and outer loop)
Tier 2: Alpine and CentOS Stream (due to being in inner loop)
Tier 3: Fedora and OpenSUSE (due to being in extra platforms)
Tiers represent risk/responsibility for the project as a whole.
Partners are encouraged to augment testing in their own infra, as needed.

Notes:

Ubuntu could be argued to be Tier 0 because we're testing (in effect) two LTS versions across inner and outer loop and it is also tested by CoreCLR test legs, resulting in the highest coverage.
Debian could be argued to be Tier 1.5 since it is in inner loop via CoreCLR and outer loop by Libraries resulting in lesser coverage.

Relevant test legs:

@jkotas

Copilot

Pull Request Overview

This PR adds Azure Linux support to the regular CI pipelines by updating the job configuration in the libraries pipeline.

Added Azure Linux configuration to the conditional extra platforms build block.
Ensured consistency with the upgrade requirements per the referenced issue.

eng/pipelines/libraries/helix-queues-setup.yml

richlander · 2025-05-09T00:22:39Z

/azp run runtime-libraries-coreclr outerloop-linux

azure-pipelines · 2025-05-09T00:22:48Z

Azure Pipelines successfully started running 1 pipeline(s).

richlander · 2025-05-09T00:33:37Z

@akoeplinger @ViktorHofer

dotnet-policy-service · 2025-05-09T01:07:39Z

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

jkotas

LGTM

jeffhandley

I'm good with this if @bartonjs and @vcsjones are.

bartonjs

CentOS Stream 10 or Debian 13 are important to keep on in inner loop because those two distros have OpenSSL 3.5 (latest and greatest). Deb-13 was taken out, but since CentOS Stream 10 is still there, and Ubuntu is still there, it's fine by me.

richlander · 2025-05-09T21:30:40Z

@jeffhandley -- can you validate that the XSLT failure is OK? Sounds like it is not crypto related but it could be something else. Have we seen that failure elsewhere?

jkotas · 2025-05-09T21:45:31Z

can you validate that the XSLT failure is OK?

The test is complaining about unexpected data and time formatting on Azure Linux:

e.g. 1:45 PM vs. 13:45

This failure needs to be addressed before this PR gets merged. We have not seen this failure anywhere else.

vcsjones · 2025-05-09T21:57:34Z

I can consistently reproduce it on AZL 3.

It appears fixed by #114200 which is not yet merged.

richlander · 2025-05-09T21:58:53Z

This appears to be the relevant code: https://github.com/dotnet/runtime/blob/main/src/libraries/System.Private.Xml/tests/Xslt/TestFiles/TestData/XsltApiV2/bug93189.xsl

We need to validate that this works correctly on Azure Linux 3.

@jeffhandley

tarekgh · 2025-05-09T23:39:17Z

I disabled the failing test case to unblock the PR. The issue is the date day is formatted with dd instead of d and time formatted with HH instead of hh. I'll wait to see if the test pass and will log issue track fixing the issue.

jkotas · 2025-05-10T01:51:03Z

Set environment variables
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8

Do you see it as a temporary workaround or as an actual fix? I thought that we want the libraries tests to be robust against environment locale settings.

tarekgh · 2025-05-10T01:52:56Z

Do you see it as a temporary workaround or as an actual fix? I thought that we want the libraries tests to be robust against environment locale settings.

Agree. I want first to validate my understanding, but the test needs to change too to work reliably regardless of the machine configuration.

jkotas · 2025-05-10T01:53:48Z

So dotnet/dotnet-buildtools-prereqs-docker#1427 should be reverted, and this test disabled and eventually fixed instead.

tarekgh · 2025-05-10T01:57:43Z

So dotnet/dotnet-buildtools-prereqs-docker#1427 should be reverted, and this test disabled and eventually fixed instead.

Looks the test not failing on other images which indicate the default locale was set there. So, I don't think reverting dotnet/dotnet-buildtools-prereqs-docker#1427 is needed. The test needs to change anyway though.

richlander · 2025-05-10T02:07:12Z

Relevant ENVs, for reference:

Set environment variables
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8

Here's how the images are configured (that recent PR has not yet come into effect):

rich@vancouver:~$ docker run --rm mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-helix-amd64 locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8
rich@vancouver:~$ docker run --rm mcr.microsoft.com/dotnet-buildtools/prereqs:debian-13-helix-amd64 locale
LANG=en_US.utf8
LANGUAGE=
LC_CTYPE="en_US.utf8"
LC_NUMERIC="en_US.utf8"
LC_TIME="en_US.utf8"
LC_COLLATE="en_US.utf8"
LC_MONETARY="en_US.utf8"
LC_MESSAGES="en_US.utf8"
LC_PAPER="en_US.utf8"
LC_NAME="en_US.utf8"
LC_ADDRESS="en_US.utf8"
LC_TELEPHONE="en_US.utf8"
LC_MEASUREMENT="en_US.utf8"
LC_IDENTIFICATION="en_US.utf8"
LC_ALL=
rich@vancouver:~$ docker run --rm mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-25.04-helix-amd64 locale
LANG=en_US.utf8
LANGUAGE=
LC_CTYPE="en_US.utf8"
LC_NUMERIC="en_US.utf8"
LC_TIME="en_US.utf8"
LC_COLLATE="en_US.utf8"
LC_MONETARY="en_US.utf8"
LC_MESSAGES="en_US.utf8"
LC_PAPER="en_US.utf8"
LC_NAME="en_US.utf8"
LC_ADDRESS="en_US.utf8"
LC_TELEPHONE="en_US.utf8"
LC_MEASUREMENT="en_US.utf8"
LC_IDENTIFICATION="en_US.utf8"
LC_ALL=
$ docker run --rm mcr.microsoft.com/dotnet-buildtools/prereqs:centos-stream-10-helix-amd64 locale
LANG=en_US.utf8
LC_CTYPE="en_US.utf8"
LC_NUMERIC="en_US.utf8"
LC_TIME="en_US.utf8"
LC_COLLATE="en_US.utf8"
LC_MONETARY="en_US.utf8"
LC_MESSAGES="en_US.utf8"
LC_PAPER="en_US.utf8"
LC_NAME="en_US.utf8"
LC_ADDRESS="en_US.utf8"
LC_TELEPHONE="en_US.utf8"
LC_MEASUREMENT="en_US.utf8"
LC_IDENTIFICATION="en_US.utf8"
LC_ALL=

Upon seeing this (which I should have done initially), It doesn't seem like the PR we made is correct. The only interesting difference is that LC_ALL isn't set to a value on Debian and Ubuntu.

Revert: dotnet/dotnet-buildtools-prereqs-docker#1428

jkotas · 2025-05-10T02:07:23Z

I don't think reverting dotnet/dotnet-buildtools-prereqs-docker#1427 is needed.

I disagree. We want our default test environment to be as close as possible to the default environment used by customers.

We know that Azure Linux is non-standard in number of ways. We want to cover as much of these differences as possible in our tests. It is the reason why we are beefing up the Azure Linux testing.

tarekgh · 2025-05-10T02:12:49Z

Upon seeing this (which I should have done initially), It doesn't seem like the PR we made is correct.

Not really, setting the environment variables I sent is enough and not necessary setting all the rest.

I disagree. We want our default test environment to be as close as possible to the default environment used by customers.

I don't mind either way. The failing test is checking against hardcoded precalculated values depending on specific settings. If we think testing with default image settings is beneficiary for whole tests, that will be fine.

richlander · 2025-05-10T02:16:04Z

Slightly wrong data. The images got published just as I pulled them.

Before and after PR1427:

rich@vancouver:~$ docker run --rm mcr.microsoft.com/dotnet-buildtools/prereqs@sha256:5ded7a1ff99093aa48ecea8649aa75aadb007c98105d664d79878fa0a7020e10 locale
LANG=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=
rich@vancouver:~$ docker run --rm mcr.microsoft.com/dotnet-buildtools/prereqs@sha256:a9b9de612f54c6e34ad273ab85aee01ff0cdb456bd6c7a4d3475c26b2068f4ac locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8

This aligns with the initial investigation by @tarekgh that language info is not set.

The philosophy that we take the images as-is sounds good. We can make a tracking issue for AL4 if we think this should be changed.

I'll re-kickoff CI when the new image gets published to ensure we get a clean run.

tarekgh · 2025-05-10T02:22:06Z

The philosophy that we take the images as-is sounds good. We can make a tracking issue for AL4 if we think this should be changed.

Sounds good to me. We need two issues one for the image and one for the test :-)

I'll go ahead and revert the disabling test case too here.

richlander · 2025-05-10T02:24:20Z

Sorry ... what are the issues?

I thought that keeping the test disabled would be the correct action. We cannot merge this PR otherwise.

tarekgh · 2025-05-10T02:35:04Z

ok:

we'll not change the locale setting on the image. will continue having the default posix settings
We'll disable the failing test to unblock this PR
We'll create issue track fixing and enabling the test.

richlander · 2025-05-10T02:46:24Z

More context. Looks like AL3 closely matches Fedora, which we also support (obviously).

rich@vancouver:~$ docker run --rm mcr.microsoft.com/azurelinux/base/core:3.0 locale
LANG=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=
rich@vancouver:~$ docker run --rm fedora locale
LANG=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=

This locks in the idea that we should be testing the distros unmodified.

richlander · 2025-05-10T05:10:27Z

/azp run runtime-libraries-coreclr outerloop-linux

azure-pipelines · 2025-05-10T05:10:35Z

Azure Pipelines successfully started running 1 pipeline(s).

richlander · 2025-05-10T05:12:07Z

Respin per rebuild: dotnet/versions@911f03a

richlander · 2025-05-10T16:01:53Z

Looks like the tests are clean.

BA is green with no unique failures reported
Azure Linux outerloop failures appear to match failures in CentOS and Alpine

This is a big change with significant implications. I'll wait for someone else to "OK" merging (or just merge).

I'll port this change to release branches next.

jkotas · 2025-05-11T01:40:07Z

src/libraries/System.Private.Xml/tests/Xslt/XslCompiledTransformApi/XslCompiledTransform.cs

@@ -3165,7 +3165,7 @@ public void ValidCases_ExternalURI(object param0, object param1, object param2,
        //[Variation("No Import/Include, NullResolver", Pri = 0, Params = new object[] { "Bug382198.xsl", "fruits.xml", "bug382198.txt", "NullResolver", true })]
        [InlineData("Bug382198.xsl", "fruits.xml", "bug382198.txt", "NullResolver", true, "IXPathNavigable")]
        [InlineData("Bug382198.xsl", "fruits.xml", "bug382198.txt", "NullResolver", true, "XmlReader")]
-        [InlineData("bug93189.xsl", "bug93189.xml", "bug93189.xml", "NullResolver", true, "XmlReader")]
+        // [InlineData("bug93189.xsl", "bug93189.xml", "bug93189.xml", "NullResolver", true, "XmlReader")]


This should have a link to an issue that tracks fixing the test.

I have created #115455

src/libraries/System.Private.Xml/tests/Xslt/XslCompiledTransformApi/XslCompiledTransform.cs

…rmApi/XslCompiledTransform.cs

src/libraries/System.Private.Xml/tests/Xslt/XslCompiledTransformApi/XslCompiledTransform.cs

…rmApi/XslCompiledTransform.cs

Add Azure Linux to regular CI

fb5e5d0

Copilot AI review requested due to automatic review settings May 8, 2025 22:38

github-actions bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label May 8, 2025

dotnet-policy-service bot assigned richlander May 8, 2025

Copilot AI reviewed May 8, 2025

View reviewed changes

eng/pipelines/libraries/helix-queues-setup.yml Show resolved Hide resolved

Remove duplication

e5f9bca

richlander mentioned this pull request May 9, 2025

Consider removing SLES.15.Amd64.Open VM dotnet/dnceng#5604

Open

jkotas added the area-Infrastructure-coreclr label May 9, 2025

github-project-automation bot added this to Runtime Infra May 9, 2025

jkotas added area-Infrastructure and removed area-Infrastructure-coreclr needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels May 9, 2025

jkotas approved these changes May 9, 2025

View reviewed changes

akoeplinger approved these changes May 9, 2025

View reviewed changes

jeffhandley approved these changes May 9, 2025

View reviewed changes

bartonjs approved these changes May 9, 2025

View reviewed changes

Disable failing test case to unblock the PR

e51d26e

Undo disabling the test case

c04bf3b

disable the test again.

e72c632

build-analysis bot mentioned this pull request May 10, 2025

/root/helix/work/correlation/scripts/<hash>/execute.sh: Permission denied dotnet/dnceng#3412

Open

3 tasks

Merge branch 'main' into al3-ci

1cb1388

jkotas reviewed May 11, 2025

View reviewed changes

src/libraries/System.Private.Xml/tests/Xslt/XslCompiledTransformApi/XslCompiledTransform.cs Show resolved Hide resolved

Update src/libraries/System.Private.Xml/tests/Xslt/XslCompiledTransfo…

9e6b099

…rmApi/XslCompiledTransform.cs

build-analysis bot mentioned this pull request May 11, 2025

OSX failure on StringTests.StartsWithNoMatch_StringComparison #112195

Open

tarekgh mentioned this pull request May 11, 2025

Track enable XslCompiledTransform Date/Time XSL test #115455

Open

jkotas reviewed May 12, 2025

View reviewed changes

src/libraries/System.Private.Xml/tests/Xslt/XslCompiledTransformApi/XslCompiledTransform.cs Outdated Show resolved Hide resolved

Update src/libraries/System.Private.Xml/tests/Xslt/XslCompiledTransfo…

60e0be3

…rmApi/XslCompiledTransform.cs

richlander merged commit d2c5618 into dotnet:main May 12, 2025
83 of 85 checks passed

github-project-automation bot moved this to Done in Runtime Infra May 12, 2025

richlander deleted the al3-ci branch May 12, 2025 19:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Azure Linux to regular CI #115415

Add Azure Linux to regular CI #115415

richlander commented May 8, 2025 •

edited

Loading

Copilot AI left a comment

richlander commented May 9, 2025

azure-pipelines bot commented May 9, 2025

richlander commented May 9, 2025

dotnet-policy-service bot commented May 9, 2025

jkotas left a comment

jeffhandley left a comment

bartonjs left a comment

richlander commented May 9, 2025

jkotas commented May 9, 2025 •

edited

Loading

vcsjones commented May 9, 2025

richlander commented May 9, 2025

tarekgh commented May 9, 2025

jkotas commented May 10, 2025 •

edited

Loading

tarekgh commented May 10, 2025

jkotas commented May 10, 2025

tarekgh commented May 10, 2025

richlander commented May 10, 2025 •

edited

Loading

jkotas commented May 10, 2025

tarekgh commented May 10, 2025

richlander commented May 10, 2025 •

edited

Loading

tarekgh commented May 10, 2025

richlander commented May 10, 2025

tarekgh commented May 10, 2025

richlander commented May 10, 2025 •

edited

Loading

richlander commented May 10, 2025

azure-pipelines bot commented May 10, 2025

richlander commented May 10, 2025

richlander commented May 10, 2025 •

edited

Loading

jkotas May 11, 2025

richlander May 11, 2025

tarekgh May 11, 2025

Add Azure Linux to regular CI #115415

Add Azure Linux to regular CI #115415

Conversation

richlander commented May 8, 2025 • edited Loading

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

richlander commented May 9, 2025

azure-pipelines bot commented May 9, 2025

richlander commented May 9, 2025

dotnet-policy-service bot commented May 9, 2025

jkotas left a comment

Choose a reason for hiding this comment

jeffhandley left a comment

Choose a reason for hiding this comment

bartonjs left a comment

Choose a reason for hiding this comment

richlander commented May 9, 2025

jkotas commented May 9, 2025 • edited Loading

vcsjones commented May 9, 2025

richlander commented May 9, 2025

tarekgh commented May 9, 2025

jkotas commented May 10, 2025 • edited Loading

tarekgh commented May 10, 2025

jkotas commented May 10, 2025

tarekgh commented May 10, 2025

richlander commented May 10, 2025 • edited Loading

jkotas commented May 10, 2025

tarekgh commented May 10, 2025

richlander commented May 10, 2025 • edited Loading

tarekgh commented May 10, 2025

richlander commented May 10, 2025

tarekgh commented May 10, 2025

richlander commented May 10, 2025 • edited Loading

richlander commented May 10, 2025

azure-pipelines bot commented May 10, 2025

richlander commented May 10, 2025

richlander commented May 10, 2025 • edited Loading

jkotas May 11, 2025

Choose a reason for hiding this comment

richlander May 11, 2025

Choose a reason for hiding this comment

tarekgh May 11, 2025

Choose a reason for hiding this comment

richlander commented May 8, 2025 •

edited

Loading

jkotas commented May 9, 2025 •

edited

Loading

jkotas commented May 10, 2025 •

edited

Loading

richlander commented May 10, 2025 •

edited

Loading

richlander commented May 10, 2025 •

edited

Loading

richlander commented May 10, 2025 •

edited

Loading

richlander commented May 10, 2025 •

edited

Loading