checklist for 2.10.x (+ request for comment, feature requests, etc.)

[ngam]

Comment:

I am interested in the following (minor) changes to the build. None of these is important and if we don't manage to figure them out, they can wait.

• ensuring gcp support (likely enabled) (update: I believe this is working as of 2.9.1)
• ensuring hdfs support (likely disabled)
• ensuring aws support (likely disabled)

An example of our current configs:

Configuration finished
+ bazel build --crosstool_top=//custom_toolchain:toolchain --logging=6 --verbose_failures --config=opt --define=PREFIX=/home/conda/feedstock_root/build_artifacts/tensorflow-split_1659720782575/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac --define=PROTOBUF_INCLUDE_PATH=/home/conda/feedstock_root/build_artifacts/tensorflow-split_1659720782575/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/include --config=noaws --cpu=k8 --local_cpu_resources=2 //tensorflow/tools/pip_package:build_pip_package //tensorflow/tools/lib_package:libtensorflow //tensorflow:libtensorflow_cc.so
Starting local Bazel server and connecting to it...
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=0 --terminal_columns=80
INFO: Reading rc options for 'build' from /home/conda/feedstock_root/build_artifacts/tensorflow-split_1659720782575/work/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'build' from /home/conda/feedstock_root/build_artifacts/tensorflow-split_1659720782575/work/.bazelrc:
  'build' options: --define framework_shared_object=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true --experimental_cc_shared_library --experimental_link_static_libraries_once=false
INFO: Reading rc options for 'build' from /home/conda/feedstock_root/build_artifacts/tensorflow-split_1659720782575/work/.tf_configure.bazelrc:
  'build' options: --action_env PYTHON_BIN_PATH=/home/conda/feedstock_root/build_artifacts/tensorflow-split_1659720782575/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/bin/python --action_env PYTHON_LIB_PATH=/home/conda/feedstock_root/build_artifacts/tensorflow-split_1659720782575/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/python3.9/site-packages --python_path=/home/conda/feedstock_root/build_artifacts/tensorflow-split_1659720782575/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/bin/python --define=with_xla_support=true --action_env TF_SYSTEM_LIBS=absl_py,astor_archive,astunparse_archive,boringssl,com_github_googlecloudplatform_google_cloud_cpp,com_github_grpc_grpc,com_google_protobuf,curl,cython,dill_archive,flatbuffers,gast_archive,gif,icu,libjpeg_turbo,org_sqlite,png,pybind11,snappy,zlib --define=PREFIX=/home/conda/feedstock_root/build_artifacts/tensorflow-split_1659720782575/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac --define=LIBDIR=/home/conda/feedstock_root/build_artifacts/tensorflow-split_1659720782575/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib --define=INCLUDEDIR=/home/conda/feedstock_root/build_artifacts/tensorflow-split_1659720782575/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/include
INFO: Reading rc options for 'build' from /home/conda/feedstock_root/build_artifacts/tensorflow-split_1659720782575/work/.bazelrc:
  'build' options: --deleted_packages=tensorflow/compiler/mlir/tfrt,tensorflow/compiler/mlir/tfrt/benchmarks,tensorflow/compiler/mlir/tfrt/jit/python_binding,tensorflow/compiler/mlir/tfrt/jit/transforms,tensorflow/compiler/mlir/tfrt/python_tests,tensorflow/compiler/mlir/tfrt/tests,tensorflow/compiler/mlir/tfrt/tests/ir,tensorflow/compiler/mlir/tfrt/tests/analysis,tensorflow/compiler/mlir/tfrt/tests/jit,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_tfrt,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_jitrt,tensorflow/compiler/mlir/tfrt/tests/tf_to_corert,tensorflow/compiler/mlir/tfrt/tests/tf_to_tfrt_data,tensorflow/compiler/mlir/tfrt/tests/saved_model,tensorflow/compiler/mlir/tfrt/transforms/lhlo_gpu_to_tfrt_gpu,tensorflow/core/runtime_fallback,tensorflow/core/runtime_fallback/conversion,tensorflow/core/runtime_fallback/kernel,tensorflow/core/runtime_fallback/opdefs,tensorflow/core/runtime_fallback/runtime,tensorflow/core/runtime_fallback/util,tensorflow/core/tfrt/common,tensorflow/core/tfrt/eager,tensorflow/core/tfrt/eager/backends/cpu,tensorflow/core/tfrt/eager/backends/gpu,tensorflow/core/tfrt/eager/core_runtime,tensorflow/core/tfrt/eager/cpp_tests/core_runtime,tensorflow/core/tfrt/gpu,tensorflow/core/tfrt/run_handler_thread_pool,tensorflow/core/tfrt/runtime,tensorflow/core/tfrt/saved_model,tensorflow/core/tfrt/graph_executor,tensorflow/core/tfrt/saved_model/tests,tensorflow/core/tfrt/tpu,tensorflow/core/tfrt/utils
INFO: Found applicable config definition build:short_logs in file /home/conda/feedstock_root/build_artifacts/tensorflow-split_1659720782575/work/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:v2 in file /home/conda/feedstock_root/build_artifacts/tensorflow-split_1659720782575/work/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
INFO: Found applicable config definition build:opt in file /home/conda/feedstock_root/build_artifacts/tensorflow-split_1659720782575/work/.tf_configure.bazelrc: --copt=-O2 --host_copt=-O2
INFO: Found applicable config definition build:noaws in file /home/conda/feedstock_root/build_artifacts/tensorflow-split_1659720782575/work/.bazelrc: --define=no_aws_support=true
INFO: Found applicable config definition build:linux in file /home/conda/feedstock_root/build_artifacts/tensorflow-split_1659720782575/work/.bazelrc: --copt=-w --host_copt=-w --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --config=dynamic_kernels --distinct_host_configuration=false --experimental_guard_against_concurrent_changes
INFO: Found applicable config definition build:dynamic_kernels in file /home/conda/feedstock_root/build_artifacts/tensorflow-split_1659720782575/work/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS
Loading: 

See this on filesystems: tensorflow/tensorflow@3437098

CUDA specific:

• drop all but 11.2 (or at least, I won't personally bother with debugging anything other than 11.2 going forward). The rationale for this is twofold: burden on maintainers and likely lack of support (e.g. JAX no longer supports anything but 11.2+)
• investigate new problems with jax's cuda build, likely related to protobuf, abseil, and grpc and the interaction thereof. It is important to investigate these issues in tandem (i.e. between tensorflow and jax) because we can benefit from jax's completion in time, but also make decisions for the interest of the two packages, rather than one. Current jaxlib release (0.3.15) is blocked by elusive cuda issues; the error is clearly a protobuf one, but I suspect it is related to abseil and grpc underneath.

Copying @h-vetinari and @hmaarrfk, please feel free to add to (or take away from) this. None of these is important imminently. My goal is to get this to be as close to "default" (e.g. tensorflow's images) and "optimized" (e.g. ngc's images) as possible. Continuing from the XLA additions.

[h-vetinari]

Thanks for opening.

Speaking for myself:

• Fine with the cloud-additions (though it makes me wonder why that's not part of the IO package - what's the status on that?)
• Fine with dropping CUDA<11.2. It's been long enough, and if this precludes progress on this feedstock, I'm all for it.
• Hope we can fix the jax+protobuf+grpc etc. situation with the new abseil builds. I'm a bit annoyed with myself that I gave so much time for comments re: Add static builds per cxx_version abseil-cpp-feedstock#35, but in any case, it'll go in in less than 24h.

[h-vetinari]

What's the role of protobuf in this? It doesn't depend on either abseil or grpc AFAICT...

[ngam]

What's the role of protobuf in this? It doesn't depend on either abseil or grpc AFAICT...

Honestly, no idea. I was just guessing. I will want to look deeper into this... Things tend to change slightly if I (un)vendor abseil, grpc, and protobuf. I will start systematically looking into this once we move abseil and grpc.

Just to be clear: The problem currently doesn't affect tensorflow directly. It only affects jax and sometimes if both jax and tensorflow use the same shared libraries (e.g. abseil)

[ngam]

• ... (though it makes me wonder why that's not part of the IO package - what's the status on that?)

So... I think you mean this (#206)? Well, no matter, because there is tf.io (https://www.tensorflow.org/api_docs/python/tf/io) and then there is tfio (https://www.tensorflow.org/io/api_docs/python/tfio):

• https://www.tensorflow.org/api_docs/python/tf/io: internal to tf itself (here; my main concern for now)
• https://www.tensorflow.org/io/api_docs/python/tfio: the separate tensorflow-io package

Basically, my principal motivation is to allow someone to to use something like tf.data.Dataset.list_files("gs://...") in their code, where ... is some bucket on GCP; and for that GCP support should be enabled, which it is currently. I am yet to go through the other two (HDFS and AWS) mainly because I do not use them. Unless enabling them is a dependency mess (i.e. requiring extra complicated packages), then we can enable them.

I should submit a revised version of #206 straight to staged-recipes soon, though honestly not sure if it is worth the effort given their confusing split of packages (tfio vis-a-vis tf.io). That PR (#206) is not ready for prime time btw; I edited and built it locally a while ago, but it doesn't work as intended (it only superficially works, but it will segfaults pretty quickly upon actual usage because one needs to build the components carefully and make sure they align with how tensorflow is being built: the custom toolchain, etc.)

[ngam]

@h-vetinari could you help me check the pin for (lib)protobuf?

• tensorflow upstream still has a hard upper pin (<3.20)
• tensorboard (both upstream and in conda-forge) has the same hard <3.20 pin
• tensorflow in conda-forge doesn't have that pin (we went with protobuf >=3.9.2, but we have libprotobuf pinned at >=3.20.1,<3.21.0a0)

Our tensorflow builds are excellent and we haven't had any issues. I suspect from reading upstream that the protobuf issues are mainly windows (but we don't build for windows).

My questions:

• Are protobuf and libprotobuf coupled or decoupled? Can we have libprotobuf 3.20 and protobuf 3.19 in the same env?
• What should we do?
  • option A: blindly follow upstream pin everywhere (need to carefully pin libprotobuf here then)
  • option B: just roll the dice and go with our global pinning (because I actually think it works totally fine for non-windows machines) and for that we have to relax tensorboard

Thanks!

[h-vetinari]

• Are protobuf and libprotobuf coupled or decoupled?

Tightly coupled, even though upstream recently changed the major number for the python bindings (but not the C++ ones), see https://github.com/conda-forge/protobuf-feedstock/blob/main/recipe/meta.yaml.

Can we have libprotobuf 3.20 and protobuf 3.19 in the same env?

No, we need to migrate to a consistent state for all packages (there are caveats where this becomes kinda possible with static libs, but it's fraught and painful and potentially a lot of work/churn and so you shouldn't consider it an option).

What should we do?

Upstream pins protobuf because they cannot handle distribution of non-python artefacts like the C++ protobuf libs, much less distribute different builds for different protobufs. We can. Since protobuf has a very stable API overall, there's basically zero risk of breakage if we use a different protobuf version (as long as we use it consistently).

Currently, we're running a migration for (lib)protobuf 3.21, but it's intertwined in a very messy pile-up with the abseil & grpc migrations, and so the conda-forge ecosystem is not yet consistent for all those versions. Given the effort needed to build tensorflow, it might be worth considering to wait until these migrations have concluded (or at least progressed to a consistent state), and then rebuild for whatever versions are in the global pinning then.

The alternative is to build against the current global pinning, but that means the tensorflow builds will "age" quite fast (as they'll become impossible to co-install with packages built for newer abseil/grpc/protobuf, etc., once the migrations have moved on).

[ngam]

Okay, I will relax tensorboard pinning and proceed with libprotobuf 3.20 here for now. We can rebuild once 3.21 is ready

[h-vetinari]

I note that #266 doesn't take care of some of the cloud bindings. We can open a new issue to tackle that later (e.g. there'll be a grpc 1.48 migration soonish).

[ngam]

Fwiw, they seem to require more work than anticipated. For now, I am satisfied with gcp support as the other two will likely need some smithing...