Skip to content

Adding Offload Test into DXC build pipeline #8584

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
joaosaffran merged 5 commits into from
Jun 26, 2026
+193 −1
Merged

Adding Offload Test into DXC build pipeline #8584

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
1cab50c
add vk to release pipeline
joaosaffran-zz Jun 23, 2026
fad2cb9
clean up
joaosaffran-zz Jun 24, 2026
aba9720
address comments, test changes
joaosaffran-zz Jun 25, 2026
418b47c
remove release-dxil from test
joaosaffran-zz Jun 25, 2026
5c25c24
add parameters
joaosaffran-zz Jun 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
194 changes: 193 additions & 1 deletion azure-pipelines.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,13 @@
parameters:
- name: lavapipeVersion
displayName: 'lavapipe (Mesa) version'
type: string
default: '26.1.2'
- name: warpVersion
displayName: 'WARP version'
type: string
default: '1.0.19'

trigger:
- main
- release*
Expand Down Expand Up @@ -28,12 +38,15 @@ stages:
VS2022_Release:
configuration: Release
spirvBuildFlag: -spirvtest
artifactName: dxc_spirv_release
VS2022_Debug:
configuration: Debug
spirvBuildFlag: -spirvtest
artifactName: ''
VS2022_Release_NoSPIRV:
configuration: Release
spirvBuildFlag: ''
artifactName: ''

steps:
- checkout: self
Expand Down Expand Up @@ -76,7 +89,27 @@ stages:
call utils\hct\hcttest.cmd -$(configuration) compat-suite v1.8.2505.1
displayName: 'DXIL Compat Suite Tests (1.8 point release)'
condition: succeededOrFailed()

- script: |

@damyanp damyanp Jun 24, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For PR builds, I'm wondering if we should only do this if the PR has a particular label on it? I don't think we'll want to gate all PRs on this since it'll add significant time to the builds.

call utils\hct\hctstart.cmd %HLSL_SRC_DIR% %HLSL_BLD_DIR%
call utils\hct\hcttest.cmd -$(configuration) spirv
displayName: 'SPIR-V Tests'
condition: and(succeeded(), ne(variables['spirvBuildFlag'], ''))
- pwsh: |
$bin = "$(HLSL_BLD_DIR)\$(configuration)\bin"
$dst = "$(Build.ArtifactStagingDirectory)\dxc"
New-Item -ItemType Directory -Force -Path $dst | Out-Null
foreach ($f in @("dxc.exe","dxv.exe","dxcompiler.dll","dxil.dll")) {
if (Test-Path "$bin\$f") { Copy-Item "$bin\$f" $dst } else { throw "Required file not found: $bin\$f" }
}
displayName: 'Stage DXC binaries'
condition: and(succeeded(), ne(variables['artifactName'], ''))
- task: PublishPipelineArtifact@1
displayName: 'Publish DXC binaries'
condition: and(succeeded(), ne(variables['artifactName'], ''))
inputs:
targetPath: '$(Build.ArtifactStagingDirectory)\dxc'
artifact: '$(artifactName)'


- job: Nix
timeoutInMinutes: 165
Expand Down Expand Up @@ -177,3 +210,162 @@ stages:
testResultsFormat: 'JUnit'
testResultsFiles: '**/testresults.xunit.xml'
condition: succeededOrFailed()

- stage: OffloadGate

@alsepkow alsepkow Jun 25, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this stage neccessary? Seems wasteful to spin up a VM to check labels. I think we could just do that at the end of the build job instead? Or is there a reason you chose to do this in its own stage.

@joaosaffran joaosaffran Jun 25, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ADO doesn't have access to github labels, the way I came up to do that was to use github api. This is required to block the offload-test execution unless a PR opts in to it by explicitly setting the run-offload-tests in GitHub. This happen before the tests and in parallel with build stage, It must be before testing in order to block it

@Icohedron Icohedron Jun 26, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is it a requirement that the label checking is done in parallel with the build stage instead of as part of the build stage? The concern alex raised is about spinning an entire VM just to check labels and nothing else, when it could be merged with the build stage.

@joaosaffran joaosaffran Jun 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding the check to the previous stage would make the check run for all the builds we do. Which is not ideal, since it would hit the GitHub API multiple times. I personally prefer to spawn a new VM for such a check; in my tests, this didn't took long to do compared to being rate limited by github

dependsOn: []
jobs:
- job: gate
pool:
vmImage: ubuntu-latest
steps:
- checkout: none
- bash: |
run=true
if [ "$(Build.Reason)" = "PullRequest" ]; then
run=false
labels=$(curl -sf "https://api.github.com/repos/$(Build.Repository.Name)/issues/$(System.PullRequest.PullRequestNumber)/labels" | python3 -c "import sys, json; print(' '.join(l['name'] for l in json.load(sys.stdin)))")
for l in $labels; do [ "$l" = "run-offload-tests" ] && run=true; done
fi
echo "Build.Reason=$(Build.Reason) run=$run"
echo "##vso[task.setvariable variable=run;isOutput=true]$run"
name: decide

- stage: OffloadTests
displayName: 'Offload tests'
dependsOn:
- Build
- OffloadGate
condition: and(succeededOrFailed(), eq(dependencies.OffloadGate.outputs['gate.decide.run'], 'true'))

@alsepkow alsepkow Jun 25, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was the succeedOrFailed() intentional? Wondering if it would make more sense to have this as succeeded.
Is there value in running the tests if the build failed?

@joaosaffran joaosaffran Jun 25, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We only need one stage in the build to pass and emit binaries to run the offload-tests. Adding succeededOrFailed() makes sure tests are run even if an unrelated stage fails.

@Icohedron Icohedron Jun 26, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do DXC PR builds often fail due to "unrelated" things? I know the offload test suite often has unrelated fails because of constant updates to llvm and the machines' drivers and such, but what about DXC?

@joaosaffran joaosaffran Jun 27, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"unrelated" in that scenario is a failure in a build that doesn't produce a binary, currently, that is the case for all of them except one. So if any build that doesn't produce a binary fails, we will not test this. This pipeline will be part of VK Release, therefore I don't want to block a VK release because a MacOS or Linux build is failling.


jobs:
- job: Offload
timeoutInMinutes: 180
pool:
vmImage: windows-2025
workspace:
clean: all
strategy:
matrix:
spirv_release:
artifact: 'dxc_spirv_release'

@alsepkow alsepkow Jun 25, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whats the reasoning for this? If we just default this to true all the time then we could remove the branhcing logic and just do that every run instead?

@joaosaffran joaosaffran Jun 25, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not all emitted binaries support SPIR-V codegen; in those cases, we don't want to test on lavapipe.

@Icohedron Icohedron Jun 26, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This matrix only has one element: spirv_release. Is it planned to add more configurations to this matrix in the future? If not, we could fold that runLavapipe check into the steps instead of as part of the matrix.

@joaosaffran joaosaffran Jun 27, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't have current plans to add more configurations, although I personally think we should. I personally think that leaving the matrix with one element helps clarify that this is a testing job and that we could add more binaries to be tests easily to that stage

runLavapipe: 'true'
steps:
- checkout: none

- task: DownloadPipelineArtifact@2
displayName: 'Download DXC binaries'
inputs:
artifact: '$(artifact)'
path: '$(Agent.TempDirectory)\dxc'

- pwsh: |
$dxc = Get-ChildItem "$(Agent.TempDirectory)\dxc" -Recurse -Filter dxc.exe | Select-Object -First 1
if (-not $dxc) { throw "dxc.exe not found in artifact $(artifact)" }
Write-Host "##vso[task.setvariable variable=DXC_DIR]$($dxc.Directory.FullName)"
Write-Host "DXC_DIR = $($dxc.Directory.FullName)"
displayName: 'Locate DXC_DIR'

- task: UsePythonVersion@0
inputs:
versionSpec: '3.11'
- script: choco install ninja -y

@Icohedron Icohedron Jun 25, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We install ninja using chocolatey as opposed to winget?

@joaosaffran joaosaffran Jun 25, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In previous jobs, we used hctstart, which is only available inside DXC repo. I am avoiding cloning that in this stage of the pipeline. Not sure how hctstart gets ninja installed

@Icohedron Icohedron Jun 26, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is Chocolatey a standard piece of software that's available on all the Windows VMs?
I was thinking WinGet would be more preferrable than Chocolatey, since Chocolatey is a third-party piece of software whereas WinGet is included with Windows.

@joaosaffran joaosaffran Jun 27, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Chocolatey is available as part of the 1ES VM images, which we use in this build

displayName: 'Install Ninja'
- pwsh: |
$vswhere = "${env:ProgramFiles(x86)}\Microsoft Visual Studio\Installer\vswhere.exe"
$vs = & $vswhere -latest -products * -requires Microsoft.VisualStudio.Component.VC.Tools.x86.x64 -property installationPath
if (-not $vs) { throw "Visual Studio with the VC toolset was not found" }
Write-Host "##vso[task.setvariable variable=VCVARS]$vs\VC\Auxiliary\Build\vcvars64.bat"
displayName: 'Find vcvars'
- task: BatchScript@1
displayName: 'Set up MSVC environment'
inputs:
filename: '$(VCVARS)'
modifyEnvironment: true

- pwsh: |
$exe = "$(Agent.TempDirectory)\VulkanSDK.exe"
Invoke-WebRequest "https://sdk.lunarg.com/sdk/download/latest/windows/vulkan_sdk.exe?Human=true" -OutFile $exe
Start-Process $exe -ArgumentList "--accept-licenses","--default-answer","--confirm-command","install" -Wait
$sdk = (Get-ChildItem "C:\VulkanSDK" -Directory | Sort-Object Name -Descending | Select-Object -First 1).FullName
if (-not $sdk) { throw "Vulkan SDK install dir not found under C:\VulkanSDK" }
Write-Host "##vso[task.setvariable variable=VULKAN_SDK]$sdk"
Write-Host "##vso[task.prependpath]$sdk\Bin"
displayName: 'Install Vulkan SDK'
condition: and(succeeded(), eq(variables['runLavapipe'], 'true'))

- pwsh: |
$ver = "${{ parameters.lavapipeVersion }}"
$url = "https://github.com/pal1000/mesa-dist-win/releases/download/$ver/mesa3d-$ver-release-msvc.7z"
Invoke-WebRequest $url -OutFile "$(Agent.TempDirectory)\mesa.7z"
7z x "$(Agent.TempDirectory)\mesa.7z" -o"$(Agent.TempDirectory)\mesa" -y | Out-Null
$lvp = (Get-ChildItem "$(Agent.TempDirectory)\mesa" -Recurse -Filter "lvp_icd.x86_64.json" | Select-Object -First 1).FullName
if (-not $lvp) { throw "lavapipe ICD (lvp_icd.x86_64.json) not found in Mesa package" }
reg add "HKLM\SOFTWARE\Khronos\Vulkan\Drivers" /v "$lvp" /t REG_DWORD /d 0 /f /reg:64
Write-Host "##vso[task.setvariable variable=MESA_BIN]$(Split-Path $lvp)"
Write-Host "Registered lavapipe ICD: $lvp"
displayName: 'Install lavapipe (Mesa) and register the ICD'
condition: and(succeeded(), eq(variables['runLavapipe'], 'true'))

- pwsh: |
git clone --depth 1 https://github.com/llvm/offload-test-suite OffloadTest
git clone --depth 1 https://github.com/llvm/llvm-project llvm-project
git clone --depth 1 https://github.com/llvm/offload-golden-images golden-images
python -m pip install -r OffloadTest/test/requirements.txt
$py = (python -c "import sys; print(sys.executable)").Trim() -replace '\\','/'
Write-Host "##vso[task.setvariable variable=PYTHON_EXE]$py"
displayName: 'Checkout offload-test-suite + LLVM'
workingDirectory: '$(Agent.TempDirectory)'

- script: >
cmake -G Ninja -B llvm-project/build
-DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=On
-DLLVM_EXTERNAL_PROJECTS=OffloadTest
-C llvm-project/clang/cmake/caches/HLSL.cmake
-DPython3_EXECUTABLE="$(PYTHON_EXE)"
-DDXC_DIR="$(DXC_DIR)"
-DLLVM_EXTERNAL_OFFLOADTEST_SOURCE_DIR=$(Agent.TempDirectory)/OffloadTest
-DGOLDENIMAGE_DIR=$(Agent.TempDirectory)/golden-images
-DOFFLOADTEST_TEST_CLANG=Off
-DOFFLOADTEST_ENABLE_DEBUG=Off
-DWARP_VERSION=${{ parameters.warpVersion }}
-DLLVM_LIT_ARGS="--xunit-xml-output=testresults.xunit.xml -v"
llvm-project/llvm
displayName: 'Configure LLVM + OffloadTest against DXC'
workingDirectory: '$(Agent.TempDirectory)'

- pwsh: |
$line = (Select-String -Path "llvm-project/build/CMakeCache.txt" -Pattern '^Python3_EXECUTABLE:').Line
$py = ($line -split '=', 2)[1]
if (-not $py) { throw "Python3_EXECUTABLE not found in CMakeCache.txt" }
Write-Host "lit Python: $py"
& "$py" -m pip install -r OffloadTest/test/requirements.txt
displayName: 'Install offload requirements into lit Python'
workingDirectory: '$(Agent.TempDirectory)'

- script: cmake --build llvm-project/build --target hlsl-test-depends
displayName: 'Build offload test dependencies'
workingDirectory: '$(Agent.TempDirectory)'

- pwsh: |
$out = "$(Agent.TempDirectory)/results-d3d12-warp.xml" -replace '\\','/'
$env:LIT_OPTS = "--xunit-xml-output=$out"
cmake --build llvm-project/build --target check-hlsl-warp-d3d12
displayName: 'DirectX offload tests on WARP'
condition: succeededOrFailed()
workingDirectory: '$(Agent.TempDirectory)'

- pwsh: |
$env:PATH = "$(MESA_BIN);$env:PATH"
$env:VK_LOADER_DEBUG = "all"
$out = "$(Agent.TempDirectory)/results-vk-lavapipe.xml" -replace '\\','/'
$env:LIT_OPTS = "--xunit-xml-output=$out"
cmake --build llvm-project/build --target check-hlsl-vk
displayName: 'Vulkan offload tests on lavapipe'
condition: and(succeededOrFailed(), eq(variables['runLavapipe'], 'true'))
workingDirectory: '$(Agent.TempDirectory)'

- task: PublishTestResults@2
condition: succeededOrFailed()
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '$(Agent.TempDirectory)/results-*.xml'
Loading