Bazel 8.0.0 crashes on windows: Unrecoverable error while evaluating node, java.nio.file.AccessDeniedException

[peakschris]

Description of the bug:

We are seeing this error approx 10-15 times per day in our CI system with bazel 8.0.0. It is for a different DLL each time. Every time it is the same java.nio.file.AccessDeniedException. Rerunning the build command typically succeeds.

[391,431 / 405,569] checking cached actions
FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.RuntimeException: Unrecoverable error while evaluating node 'ActionLookupDataN{actionLookupKey=ConfiguredTargetKey{label=//src/path/to:target, config=BuildConfigurationKey[9cfe96a447c6024399e3d957765b4a9c50e92c7d30e8113fa769583f44a41f7f]}, actionIndex=1989}' (requested by nodes 'ArtifactNestedSetKey[210]@660341467')
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:547)
        at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:435)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: java.lang.IllegalStateException: java.nio.file.AccessDeniedException: D:\udu\b\mor466oi\execroot\_main\bazel-out\x64_windows-opt\bin\src\path\to\libxy.dll
        at com.google.devtools.build.lib.vfs.JavaIoFileSystem.statIfFound(JavaIoFileSystem.java:519)
        at com.google.devtools.build.lib.vfs.Path.statIfFound(Path.java:320)
        at com.google.devtools.build.lib.remote.RemoteActionFileSystem.statInternal(RemoteActionFileSystem.java:658)
        at com.google.devtools.build.lib.remote.RemoteActionFileSystem.statIfFound(RemoteActionFileSystem.java:598)
        at com.google.devtools.build.lib.vfs.Path.statIfFound(Path.java:320)
        at com.google.devtools.build.lib.skyframe.ActionOutputMetadataStore.fileArtifactValueFromArtifact(ActionOutputMetadataStore.java:588)
        at com.google.devtools.build.lib.skyframe.ActionOutputMetadataStore.constructFileArtifactValue(ActionOutputMetadataStore.java:450)
        at com.google.devtools.build.lib.skyframe.ActionOutputMetadataStore.constructFileArtifactValueFromFilesystem(ActionOutputMetadataStore.java:438)
        at com.google.devtools.build.lib.skyframe.ActionOutputMetadataStore.getOutputMetadata(ActionOutputMetadataStore.java:235)
        at com.google.devtools.build.lib.actions.ActionCacheChecker.getOutputMetadataOrConstant(ActionCacheChecker.java:667)
        at com.google.devtools.build.lib.actions.ActionCacheChecker.getOutputMetadataMaybe(ActionCacheChecker.java:694)
        at com.google.devtools.build.lib.actions.ActionCacheChecker.validateArtifacts(ActionCacheChecker.java:223)
        at com.google.devtools.build.lib.actions.ActionCacheChecker.mustExecute(ActionCacheChecker.java:621)
        at com.google.devtools.build.lib.actions.ActionCacheChecker.getTokenIfNeedToExecute(ActionCacheChecker.java:554)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.checkActionCache(SkyframeActionExecutor.java:677)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:854)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.computeInternal(ActionExecutionFunction.java:375)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:216)
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:467)
        ... 7 more
Caused by: java.nio.file.AccessDeniedException: D:\udu\b\mor466oi\execroot\_main\bazel-out\x64_windows-opt\bin\src\path\to\libxy.dll
        at com.google.devtools.build.lib.windows.WindowsFileOperations.getLastChangeTime(WindowsFileOperations.java:166)
        at com.google.devtools.build.lib.windows.WindowsFileSystem.stat(WindowsFileSystem.java:155)
        at com.google.devtools.build.lib.vfs.JavaIoFileSystem.statIfFound(JavaIoFileSystem.java:509)

Which category does this issue belong to?

Core

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

No response

Which operating system are you running Bazel on?

Windows Server 2022 Standard

What is the output of bazel info release?

8.0.0

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

No response

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

[fmeum]

@bazel-io fork 8.0.1

[fmeum]

Could you try to bisect this down to the breaking commit? My best guess would be the switch over to the toolchain in rules_cc, which may be missing a fix.

[peakschris]

It's going to be challenging to bisect as it only fails once every 20 builds or so. If you give me a few commits to manually bisect I can leave each one running for a few hours and see how they get on.

[fmeum]

Could you test 263ad1a? That's the parent of the commit that moves the toolchain to rules_cc.

[peakschris]

I'm hitting build errors with most past commits that I am trying. 263ad1a results in the following. I do not have compatibility_proxy referenced anywhere in my build files.

ERROR: Failed to load Starlark extension '@@compatibility_proxy//:proxy.bzl'.
Cycle in the workspace file detected. This indicates that a repository is used prior to being defined.
The following chain of repository dependencies lead to the missing definition.
 - @@compatibility_proxy
This could either mean you have to add the '@@compatibility_proxy' repository with a statement like `http_archive` in your WORKSPACE file (note that transitive dependencies are not added automatically), or move an existing definition earlier in your WORKSPACE file.
ERROR: Error computing the main repository mapping: cycles detected during computation of main repo mapping

[fmeum]

You probably need to prepend the full rules_java release snippet for a recent version to your WORKSPACE file: https://github.com/bazelbuild/rules_java/releases/tag/8.6.3

[peakschris]

I removed the one last import in my workspace file, enabled common --noenable_workspace, updated to rules_java 8.6.3 in module.bazel. Still getting the same compatibility_proxy error.

[fmeum]

I'm pretty sure that this error can only arise when WORKSPACE is enabled. Are you sure it's disabled on your repo? Does this reproduce if you create an empty WORKSPACE.bazel file?

[peakschris]

Found an errant .bazelrc enable_workspace command, now removed. Now I'm seeing this issue. I have tried rules_java 8.6.1, 8.6.2, 8.6.3

bazel build //src/... --noenable_workspace
WARNING: Couldn't auto load rules or symbols, because no dependency on module/repository 'rules_android' found. This will result in a failure if there's a reference to those rules or symbols.
ERROR: Traceback (most recent call last):
        File "D:/udu/b/mor466oi/external/rules_java+/java/private/java_common.bzl", line 323, column 32, in <toplevel>
                java_common = _make_java_common()
        File "D:/udu/b/mor466oi/external/rules_java+/java/private/java_common.bzl", line 313, column 34, in _make_java_common
                if get_internal_java_common().google_legacy_api_enabled():
Error: 'struct' value has no field or method 'google_legacy_api_enabled'
Available attributes: JavaCompilationInfo, JavaPluginDataInfo, collect_native_deps_dirs, compile, expand_java_opts, get_runtime_classpath_for_archive, incompatible_disable_non_executable_java_binary, run_ijar_private_for_builtins, target_kind, to_java_binary_info
WARNING: Target pattern parsing failed.
ERROR: Skipping '@rules_go//go/config:pure': error loading package '@@rules_go+//go/config': Internal error while loading Starlark builtins: Failed to autoload external symbols: at D:/udu/b/mor466oi/external/rules_java+/java/java_import.bzl:16:6: at D:/udu/b/mor466oi/external/rules_java++compatibility_proxy+compatibility_proxy/proxy.bzl:10:6: initialization of module 'java/private/java_common.bzl' failed
ERROR: @rules_go//go/config:pure :: Error loading option @rules_go//go/config:pure: error loading package '@@rules_go+//go/config': Internal error while loading Starlark builtins: Failed to autoload external symbols: at D:/udu/b/mor466oi/external/rules_java+/java/java_import.bzl:16:6: at D:/udu/b/mor466oi/external/rules_java++compatibility_proxy+compatibility_proxy/proxy.bzl:10:6: initialization of module 'java/private/java_common.bzl' failed

[fmeum]

Can you test with a custom Bazel binary? Reverting https://cs.opensource.google/bazel/bazel/+/a0a72265ea37270c110630f528127e18939f4016 on 8.0.0 would be an interesting experiment as that's the only change in 8.0.0 that actually touches the method that now fails.

If the revert isn't clean, I can help with it.

[peakschris]

I got set up for a custom build, but unless I'm missing something, that commit isn't part of 8.0.0

[peakschris]

as that commit isn't in 8.0.0, I can't back it out. Do you have any other suggestions?

[jayakasadev]

will the commit be in the next release? im seeing this issue on ubuntu and mac as well

[fmeum]

You're right, it's not in 8. I don't have any guesses then. Will take another look. Trying some rolling releases between the two prod releases would help.

Maybe @hvadehra can help with the rules_java failures during the bisect (in the new year)?