[Bazel CI] Windows: Bazel crashes with Internal error: Unrecoverable error while evaluating node

[sgowroji]

CI: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/4170#019255cd-e000-4a35-aed5-07e872874c02

Platform: Windows

Logs:

FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.RuntimeException: Unrecoverable error while evaluating node 'Key{moduleKey=rules_python@_, override=LocalPathOverride{path=../../..}}' (requested by nodes 'com.google.devtools.build.lib.bazel.bzlmod.BazelModuleResolutionValue$$Lambda/0x0000000800522060@702d1dec')
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:547)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:435)
	at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: java.lang.IllegalStateException
	at com.google.devtools.build.lib.rules.repository.RepositoryDirectoryValue$NoRepositoryDirectoryValue.getPath(RepositoryDirectoryValue.java:164)
	at com.google.devtools.build.lib.bazel.bzlmod.ModuleFileFunction.getModuleFile(ModuleFileFunction.java:562)
	at com.google.devtools.build.lib.bazel.bzlmod.ModuleFileFunction.compute(ModuleFileFunction.java:164)
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:467)
	... 7 more
bazel build failed with exit code 37

Culprit:

Steps:

git clone https://github.com/bazelbuild/rules_python
git reset 9c3d303418a4eec0f701a8a46fb06b1aa7f7ea61  --hard
export USE_BAZEL_VERSION=166eaa09f9cd31836df898ea236e1e0633a73b12
bazel build -- ... 

CC Greenteam @mai93 @Wyverald

[xansec]

Is it possible this is the same issue as reported here? I see the colon in the deps path a few times, not sure if this is the same:
#23729

[meteorcloudy]

@xansec Looks very possible, I'll take a look now, thanks for the repro!

Spoke too fast, the places where Bazel fails are actually different, so probably different reasons.

[meteorcloudy]

The exact error in this issue is not reproducible, and the latest downstream shows another error seemingly caused by other incompatible changes in Bazel, so closing this one for now.

https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/4187#01926f7a-3631-476e-b588-dc3b3dc6f742

[peakschris]

Although this issue was closed as non-reproducable, I am seeing it in our own product CI builds repeatedly with bazel 8.0.0:

[183,495 / 404,843] checking cached actions
FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.RuntimeException: Unrecoverable error while evaluating node 'ActionLookupDataN{actionLookupKey=ConfiguredTargetKey{label=//src/core/textsrv/test/main:test_textserver_concurrency, config=BuildConfigurationKey[9cfe96a447c6024399e3d957765b4a9c50e92c7d30e8113fa769583f44a41f7f]}, actionIndex=306}' (requested by nodes 'ArtifactNestedSetKey[207]@86907955')
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:547)
        at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:435)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: java.lang.IllegalStateException: java.nio.file.AccessDeniedException: D:\udu\b\mor466oi\execroot\_main\bazel-out\x64_windows-opt\bin\src\path\to\libxy.dll
        at com.google.devtools.build.lib.vfs.JavaIoFileSystem.statIfFound(JavaIoFileSystem.java:519)
        at com.google.devtools.build.lib.vfs.Path.statIfFound(Path.java:320)
        at com.google.devtools.build.lib.remote.RemoteActionFileSystem.statInternal(RemoteActionFileSystem.java:658)
        at com.google.devtools.build.lib.remote.RemoteActionFileSystem.statIfFound(RemoteActionFileSystem.java:598)
        at com.google.devtools.build.lib.vfs.Path.statIfFound(Path.java:320)
        at com.google.devtools.build.lib.skyframe.ActionOutputMetadataStore.fileArtifactValueFromArtifact(ActionOutputMetadataStore.java:588)
        at com.google.devtools.build.lib.skyframe.ActionOutputMetadataStore.constructFileArtifactValue(ActionOutputMetadataStore.java:450)
        at com.google.devtools.build.lib.skyframe.ActionOutputMetadataStore.constructFileArtifactValueFromFilesystem(ActionOutputMetadataStore.java:438)
        at com.google.devtools.build.lib.skyframe.ActionOutputMetadataStore.getOutputMetadata(ActionOutputMetadataStore.java:235)
        at com.google.devtools.build.lib.actions.ActionCacheChecker.getOutputMetadataOrConstant(ActionCacheChecker.java:667)
        at com.google.devtools.build.lib.actions.ActionCacheChecker.getOutputMetadataMaybe(ActionCacheChecker.java:694)
        at com.google.devtools.build.lib.actions.ActionCacheChecker.validateArtifacts(ActionCacheChecker.java:223)
        at com.google.devtools.build.lib.actions.ActionCacheChecker.mustExecute(ActionCacheChecker.java:621)
        at com.google.devtools.build.lib.actions.ActionCacheChecker.getTokenIfNeedToExecute(ActionCacheChecker.java:554)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.checkActionCache(SkyframeActionExecutor.java:677)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:854)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.computeInternal(ActionExecutionFunction.java:375)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:216)
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:467)
        ... 7 more
Caused by: java.nio.file.AccessDeniedException: D:\udu\b\mor466oi\execroot\_main\bazel-out\x64_windows-opt\bin\src\core\path\to\libxy.dll
        at com.google.devtools.build.lib.windows.WindowsFileOperations.getLastChangeTime(WindowsFileOperations.java:166)
        at com.google.devtools.build.lib.windows.WindowsFileSystem.stat(WindowsFileSystem.java:155)
        at com.google.devtools.build.lib.vfs.JavaIoFileSystem.statIfFound(JavaIoFileSystem.java:509)
        ... 25 more
Compile failed result: 37

Can we reopen this one or should I open a new issue?