Skip to content

[CI] Enable long paths on Windows #1297

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
RUIJIEZHONG66166 wants to merge 4 commits into from
Closed

[CI] Enable long paths on Windows #1297

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
8d84e53
Enable long paths on Windows
RUIJIEZHONG66166 Jan 17, 2025
b6c8526
modify the runner path
RUIJIEZHONG66166 Jan 20, 2025
74381c5
Remove redundant ./ from generated file path
CuiYifeng Jan 20, 2025
10bae41
Reduce generated file name
CuiYifeng Jan 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions .github/workflows/_windows_ut.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,9 @@ jobs:
call conda install -y libuv
call conda install -y rust
git config --system core.longpaths true
git config --global core.symlinks true
git config --global core.fsmonitor false
powershell -Command "Set-ItemProperty -Path "HKLM:\\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1"
Copy link
Contributor

@dvrogozh dvrogozh Jan 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Long path issues in XXI century :). Yeah, that's known problem. However, setting long paths as you are doing above might not fully help. It did not help for me in one of the projects I was doing before. The full solution which ultimately resolved the issue was twofold with the setting you applied being one of the fixes. There is a chance that you will still see issues with MSVC toolchain and to resolve that it is required to patch the toolchain executables with long paths support.

I hope you won't need that, but if you will still see the issues, below is a second part of the solution which helped us on that other project. Below is a snapshot of commit message and a relevant script (I can't just post a link since it was inhouse project):

We've stepped into Windows long files limitation which is that by default Windows
does not allow file and path name lenght to be more than 260. We step into this
threefold: 1) with git clone, 2) with meson subproject archives fetch and 3) with MSVC
toolchain. First 2 issues are getting fixed by Windows setting to enable long files.
Toolchain issue requires a workaround to apply manifest to toolchain executables to
enable long files support. Since it's not good to modify system binaries, this commit
copies toolchain executables and patches them with required manifest.

The script:

$ErrorActionPreference = "Stop"

function PatchToolset  {
    mkdir _bin >$null
    Set-Location _bin

    $cvtres = $(Get-Command cvtres -TotalCount 1).Source
    $link = $(Get-Command link -TotalCount 1).Source
    $rc = $(Get-Command rc -TotalCount 1).Source

    # Windows does not allow symlinks for generic users by default
    # so we will just copy some DLLs which link, rc, etc. tools require.
    cp "$(Split-Path $link -Parent)\mspdb140.dll" .\
    cp "$(Split-Path $link -Parent)\mspdbcore.dll" .\
    cp "$(Split-Path $link -Parent)\tbbmalloc.dll" .\

    cp $cvtres .\
    cp $rc .\
    cp $link .\

    # Apply longPathAware manifest to copied executables
    mt -nologo -manifest "$PSScriptRoot\longPathAware.manifest" -outputresource:"rc.exe;#1"
    if( -not $? ) { exit 1 }
    mt -nologo -manifest "$PSScriptRoot\longPathAware.manifest" -outputresource:"cvtres.exe;#1"
    if( -not $? ) { exit 1 }
    mt -nologo -manifest "$PSScriptRoot\longPathAware.manifest" -outputresource:"link.exe;#1"
    if( -not $? ) { exit 1 }

    # Export patched toolset location to github environment
    $env:Path = "$(Get-Location);$env:Path"
    Write-Output "Path=$env:Path" >> $env:GITHUB_ENV

    # Print out search queries for the tools we patched. First line
    # should point to .\_bin, second line - to original executable.
    Write-Output "Where cvtres?"
    $(Get-Command cvtres -TotalCount 2).Source
    Write-Output "Where rc?"
    $(Get-Command rc -TotalCount 2).Source
    Write-Output "Where link?"
    $(Get-Command link -TotalCount 2).Source
}

PatchToolset

With the longPathAware.manifest being:

<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
<application  xmlns="urn:schemas-microsoft-com:asm.v3">
    <windowsSettings xmlns:ws2="http://schemas.microsoft.com/SMI/2016/WindowsSettings">
        <ws2:longPathAware>true</ws2:longPathAware>
    </windowsSettings>
</application>
</assembly>

Copy link
Contributor

@Stonepia Stonepia Jan 21, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the suggestion! The usage of using mt to add a manifest to the application is quite interesting!
I also noticed the manifest change from https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry#application-manifest-updates-to-declare-long-path-capability but didn't know how to use that before.

Currently, we still encounter long path issues during torch.compile, I am trying to reduce the path length from the Python side (for generating shorter path). We will try to patch the corresponding exe if we still have that issue.

On the other side, the document also suggests using "\?" prefix. That could also be handled in the Python side I think.

git status
git show -s
git submodule sync && git submodule update --init --recursive
Expand Down Expand Up @@ -106,7 +109,7 @@ jobs:
set MAX_JOBS=4
python setup.py bdist_wheel > build_torch_wheel_log.log
echo "[INFO] begin to install torch whls"
for /r C:\runner\actions-runner\_work\torch-xpu-ops\pytorch\dist %%i in (torch*.whl) do (
for /r C:\actions-runner\_work\torch-xpu-ops\pytorch\dist %%i in (torch*.whl) do (
set TORCH_WHL=%%i
)
echo "[INFO] the torch version is %TORCH_WHL%"
Expand All @@ -127,14 +130,14 @@ jobs:
uses: actions/upload-artifact@v4
with:
name: Torch-XPU-Windows-Log-${{ github.event.pull_request.number || github.sha }}
path: 'C:\runner\actions-runner\_work\torch-xpu-ops\pytorch\build_torch_wheel_log.log'
path: 'C:\actions-runner\_work\torch-xpu-ops\pytorch\build_torch_wheel_log.log'

- name: Upload Windows binary
if: ${{ ! cancelled() }}
uses: actions/upload-artifact@v4
with:
name: Torch-XPU-Windows-Binary-${{ github.event.pull_request.number || github.sha }}
path: 'C:\runner\actions-runner\_work\torch-xpu-ops\pytorch\dist'
path: 'C:\actions-runner\_work\torch-xpu-ops\pytorch\dist'

- name: Run XPU OP Extended UT
if: contains(inputs.ut, 'op_extended') || github.event_name == 'schedule'
Expand Down
8 changes: 6 additions & 2 deletions cmake/Modules/FindSYCL.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -263,8 +263,12 @@ macro(SYCL_WRAP_SRCS sycl_target generated_files)
set(SYCL_compile_output_dir "${SYCL_compile_intermediate_directory}")

get_filename_component( basename ${file} NAME )
set(generated_file_path "${SYCL_compile_output_dir}/${CMAKE_CFG_INTDIR}")
set(generated_file_basename "${sycl_target}_gen_${basename}${generated_extension}")
if(CMAKE_CFG_INTDIR STREQUAL ".")
set(generated_file_path "${SYCL_compile_output_dir}")
else()
set(generated_file_path "${SYCL_compile_output_dir}/${CMAKE_CFG_INTDIR}")
endif()
set(generated_file_basename "${sycl_target}_${basename}${generated_extension}")
set(generated_file "${generated_file_path}/${generated_file_basename}")
set(SYCL_generated_dependency_file "${SYCL_compile_intermediate_directory}/${generated_file_basename}.SYCL-depend") # generate by compiler options -M -MF
set(cmake_dependency_file "${SYCL_compile_intermediate_directory}/${generated_file_basename}.depend") # parse and convert SYCL_generated_dependency_file(compiler format) to cmake format
Expand Down
Loading