Bazel 8 + rules_java 8 compatibility updates

[mbland]

Description

Bumps dependencies to versions that are compatible with both Bazel 7.5.0 and 8.0.0. Part of #1482. Closes #1652.

• ScalaPB jars: 0.11.17 => 1.0.0-alpha.1
• rules_python: 0.38.0 => 1.2.0
• rules_cc: 0.0.9 => 0.1.1
• abseil-cpp: 20220623.1 => 20250127.0
• rules_java: 7.12.4 => 8.9.0
• protobuf: 21.7 => 29.3
• rules_proto: 6.0.2 => 7.1.0
• google-common-protos: 2.52.0 => 2.53.0

This precipitated the following updates also included in this commit:

• Loads java_proto_library from com_google_protobuf.
• Bumps .bazelversion to 7.5.0.
• Sets .bazelci/presubmit.yml to use Bazel 7.5.0 instead of 6.5.0, and last_rc in place of 7.x.
• Sets common --enable_workspace --noenable_bzlmod in .bazelrc and tools/bazel\.rc\.buildkite.
• Adds allow_empty = True to a glob expression in //test/semanticdb:lib_with_tempsrc.
• Removes Scala 2.11 test cases.

Bazel 6 is officially unsupported as of this change and the upcoming rules_scala 7.0.0 release. WORKSPACE builds succeed under Bazel 6.5.0 (with C++ compiler flags) for the time being, but are not guaranteed to continue working. (Bazel 6 Bzlmod builds would break anyway, because Bazel 6 doesn't provide use_repo_rule, required by rules_jvm_external 6.3, which is required by protobuf v29.)

The README now documents that scala_proto or any rules otherwise depending on protobuf are no longer supported out of the box for Scala 2.11. Such users will have to ensure they register their own downgraded versions, at their own risk of future rules_scala 7.x incompatibility.

Version bump related updates to .bazelversion,WORKSPACE, and third_party/repositories comprise the bulk of this change. Other small, yet important changes (other than those due to the version bumps noted above) include:

• Adding a new examples/overridden_artifacts repository and the overridden_artifacts_example test case in test/shell/test_examples.sh.

• Making _validate_scalac_srcjar() and dt_patched_compiler_setup() in scala/private/macros/scala_repositories.bzl more tolerant of dictionaries containing keys mapped to None. The new overridden_artifacts_example test covers this.

Motivation

We're no longer planning to support Bazel 6 in the next major release per @simuons's decision in:

• Support Bzlmod and add rules_scala to bazel-central-registry #1482 (comment)

The plan is now to land the Bazel 7 and 8 compatibility updates first, then land the Bzlmod change. This enables us to make only one new major version release, instead of two (the first supporting with Bazel 6, the second dropping support).

Bazel 8 and rules_java 8 require protobuf >= v29. After the protobuf v29 bump, and before the ScalaPB 1.0.0-alpha.1 bump, scala_proto targets would fail with the following error:

ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14:
  ProtoScalaPBRule
  external/com_google_protobuf/src/google/protobuf/any_proto_jvm_extra_protobuf_generator_scalapb.srcjar
  failed: (Exit 1): scalapb_worker failed:
  error executing ProtoScalaPBRule command
  (from target @@com_google_protobuf//src/google/protobuf:any_proto)
  bazel-out/.../bin/src/scala/scripts/scalapb_worker
    ... (remaining 2 arguments skipped)

--jvm_extra_protobuf_generator_out:
  java.lang.NoSuchMethodError:
  'java.lang.Object com.google.protobuf.DescriptorProtos$FieldOptions.getExtension(com.google.protobuf.GeneratedMessage$GeneratedExtension)'
    at scalapb.compiler.DescriptorImplicits$ExtendedFieldDescriptor.fieldOptions(DescriptorImplicits.scala:329)
  [ ...snip... ]

java.lang.RuntimeException: Exit with code 1
    at scala.sys.package$.error(package.scala:30)
    at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44)
    at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96)
    at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49)
    at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39)
    at scripts.ScalaPBWorker.main(ScalaPBWorker.scala)

ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14
  Building source jar external/com_google_protobuf/src/google/protobuf/any_proto_scalapb-src.jar
  failed: (Exit 1): scalapb_worker failed:
  error executing ProtoScalaPBRule command
  (from target @@com_google_protobuf//src/google/protobuf:any_proto)
  bazel-out/darwin_arm64-opt-exec-ST-a828a81199fe/bin/src/scala/scripts/scalapb_worker
    ... (remaining 2 arguments skipped)

Here's why the other changes were necessary in light of the version bumps:

• java_proto_library from rules_java is now officially deprecated, hence loading it from com_google_protobuf.

• Setting common --enable_workspace --noenable_bzlmod (instead of build) fixes test_semanticdb_handles_removed_sourcefiles. This test relies on bazel query, which is also affected by these flags, hence common instead of build. Bazel 8 defaults to --enable_bzlmod --noenable_workspace, causing the WORKSPACE run of this test to fail.

• glob requires an explicit allow_empty = True parameter now that --incompatible_disallow_empty_glob defaults to True in Bazel 8.

• ScalaPB 0.9.8, the last version compatible with Scala 2.11, does not support protobuf >= 26.0. For this reason, we must remove the Scala 2.11 test cases.

We should consider dropping Scala 2.11 support at this point, since there's no ScalaPB release for it that supports later versions of protobuf. That, and we could remove some of the special case code added in the following changes, amongst other 2.11 support details:

• Update to Scalafmt 3.8.3 #1631
• Bump ScalaPB, protoc-bridge, gRPC, and Guava deps, and add ProtobufAdapters and ScalaPBCodeGenerator wrappers #1648
• Toolchainize //scala_proto:{,deps_}toolchain_type #1687
• Bump protoc-bridge to 0.9.8 for Scala >= 2.12 #1688

Finally, the motivation for the non-version bump changes:

• The design bug in the upcoming Bzlmod API for overridden_artifacts reported by @dmivankov precipitated the examples/overriden_artifacts repo and test. See: Support Bzlmod and add rules_scala to bazel-central-registry #1482 (comment) Support Bzlmod and add rules_scala to bazel-central-registry #1482 (comment)

• The _validate_scalac_srcjar() update maintains the strict checks for mutually exclusive values, while preventing client code from having to explicitly filter out None entries. This pairs with the change in dt_patched_compiler_setup() that uses the compiler_srcjar dictionary. These changes helps keep the upcoming module extension a bit cleaner.

[mbland]

Hmm, an odd failure on the Windows build:

ERROR: C:/tools/msys64/home/b/_bazel_b/xknd5zlq/external/com_google_protobuf/src/google/protobuf/compiler/java/BUILD.bazel:87:11: Compiling src/google/protobuf/compiler/java/java_features.pb.cc [for tool] failed: (Exit 2): cl.exe failed: error executing CppCompile command (from target @@com_google_protobuf//src/google/protobuf/compiler/java:java_features_bootstrap) C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\bin\HostX64\x64\cl.exe ... (remaining 1 argument skipped)
--
  | external/com_google_protobuf/src/google/protobuf/compiler/java/java_features.pb.cc(6): fatal error C1083: Cannot open include file: 'google/protobuf/compiler/java/java_features.pb.h': No such file or directory

src/google/protobuf/compiler/java/java_features.pb.h most definitely exists in protobuf v29.3.

All the other builds seem to pass. If they do, I'll kick this with an empty commit to see if we can get Windows to play along, in case it was some weird transient error.

[mbland]

Yep, just windows. And my empty commit bump just failed in the same way. Hmm... why can't it find the #include file right next to the source?

[crt-31]

I looked at the windows issue... the issue is that the full include path for that file is over the char limit (260 chars).
I think its a cc_library issue, but still investigating when it got broken... I think it has to do with the param file limitations, but I'm still looking into it.

Also, I don't remember having to compile those protobuf tools locally before... I feel like they came precompiled before?

[crt-31]

OK, I found the related issue here: protocolbuffers/protobuf#12947. (I'm going to look into it more cause I use MSVC all the time and haven't run into these issues.)

I wonder if we can go back to using the prebuilt protobuf libraries, then we don't need to require the c++ compiler in order to use rules_scala.

[mbland]

@crt-31 Heh, you beat me to it...I snuck a look on my phone while out this evening and found the same issue. Got online to post it, but was too late. 😛 What's more, my next Bzlmod blog post is going to talk about how I had to patch rules_rust to work around Windows path length breakages.

And I saw the paragraph that you just responded to:

Since we don't have any good way to mitigate this issue, and because bazel+msvc usage is low compared to other ways to build on Windows, we are planning to drop support for bazel+msvc in Protobuf 34 (the Q1 2026 release). Bazel 30 already disables support for this combination, with a flag to reenable support for it which gives time to gather feedback on this as well as give folks a longer horizon to migrate their builds to either make+msvc or bazel+clang-cl which don't suffer from this artificial limitation without being stuck on Protobuf 29 or lower:

https://protobuf.dev/news/v30/#poison-msvc--bazel

The linked announcement makes reference to protocolbuffers/protobuf#20085: "Breaking Change: Dropping support for Bazel+MSVC".

But to your point above:

I wonder if we can go back to using the prebuilt protobuf libraries, then we don't need to require the c++ compiler in order to use rules_scala.

I did experiment with using prebuilt Protobuf toolchains in my proto-toolchainization branch in mbland/rules_scala back in October, when I was first experimenting with Bzlmodification. I think I may dust it off and play with it again in light of this development; it may also provide a performance win, since protobuf wants to recompile constantly. (I'm also not aware of what "going back to using prebuilt protobuf libraries" would involve. We're already using the prebuilt protobuf-java artifact; I think protobuf keeps wanting to recompile protoc.)

[mbland]

UPDATE: Please skip this comment and read my new comment about the more recent fully working and polished protoc toolchainization.

The good news is, I went ahead and implemented protoc toolchainization in the latest commit, and it works really well. Please check out what I did in 7b7ab2a and let me know what you think of the README update.

The bad news is, the Windows build is still broken in the same way. I'm pretty sure it's because protobuf v29.3 still uses native.java_proto_library, which still depends on @com_google_protobuf//:protoc: Update: Nope, it's not. It's because the precompiled toolchain wasn't registered early enough.

• https://github.com/protocolbuffers/protobuf/blob/v29.3/bazel/java_proto_library.bzl#L11-L16

def java_proto_library(**kwattrs):
    # Only use Starlark rules when they are removed from Bazel
    if not hasattr(native, "java_proto_library"):
        _java_proto_library(**kwattrs)
    else:
        native.java_proto_library(**kwattrs)

I believe this is what @fmeum is trying to fix in protocolbuffers/protobuf#19679 (which I borrowed from in scala_proto/scala_proto_toolchain.bzl).

@simuons Until this gets resolved one way or another, what do you think of making Windows builds optional for now? Then when a fix comes, we can make it required again. Or maybe I can hack something to downgrade the Windows build down to protobuf v21.7 again, as it was before this pull request (i.e., currently remains on master).

@crt-31 I know you really don't want that to happen, but it's effectively blocking the next rules_scala release at this point.

@fmeum If you've any better ideas, or my understanding is incorrect in any way, I'm all ears.

At any rate, this gives me time to think about maybe adding more tests for this latest protoc toolchainization, if we need them.

[mbland]

Hold the presses... I just pushed an experiment that fixed the Windows build. I created a patch from protocolbuffers/protobuf#19679, then added a line to WORKSPACE to register the prebuilt toolchain early. Success!

Everything else is failing right now (all the tests for all the other WORKSPACE repos I haven't modified), but I'll fix that up later. The main question is whether to actually commit this patch.

[mbland]

OK, please ignore my update from yesterday. I've learned a lot, including that the protoc failure was due to the new toolchain not getting registered early enough, not that protobuf was selecting native rules. The latest commit polishes the protoc toolchainization implementation, and the Windows build is passing again (along with all the others).

The majority of the changes in the PR are still the boilerplate updates to WORKSPACE and .bazelversion files, and automated updates to third_party/repositories/scala_*.bzl files. The most substantial changes are in:

• Updating Bazel versions in .bazelci/presubmit.yml
• Adding common --incompatible_enable_proto_toolchain_resolution to .bazelrc and tools/bazel.rc.buildkite
• README.md
• The new examples/overridden_artifacts repo and new test in test/shell/test_examples.sh
• The new //protoc package with the protoc toolchainization logic
• The protobuf patch in scala/deps.bzl from bazel: Remove hardcoded dependency on //:protoc from language runtimes protocolbuffers/protobuf#19679
• The slight update to scala/private/macros/scala_repositories.bzl
• Updates to scala/toolchains.bzl and scala/toolchains_repo.bzl for toolchainization
• Minor tweaks to scala_config.bzl
• scala_proto/scala_proto_toolchain.bzl updates based on bazel: Remove hardcoded dependency on //:protoc from language runtimes protocolbuffers/protobuf#19679
• Version bumps in scripts/create_repository.py
• The new scripts/update_protoc_integrity.py
• Adding allow_empty = True to a glob() in test/semanticdb/BUILD
• Dropping Scala 2.11 tests from test_thirdparty_version.sh and test_version.sh

Still, this is a lot more than I'd originally planned to include in this pull request. Some of these updates can probably be teased out into separate PRs.

And/or, we could go back to the original 7.x and 8.x release plan. This would mean backing out the protobuf upgrade from v21.7 to v29.3, and other updates along with it, most notably protoc toolchainization.

In either case, Bzlmod could land in 7.0 or 7.1. I'll prepare my Bzlmod branch, but it would remain a relatively light update on top of whatever changes we decide to include, and when.

Anywho, as usual, options... @simuons Let me know what you'd prefer to do here.