Compatibility with Tensorflow Text

[hellodanylo]

Background

Tensorflow Text (TFT) is an add-on package for Tensorflow Core (TFC) that implements various NLP-related operators.
TFT is a dependency for KerasNLP, which is an add-on package for Keras.
https://www.tensorflow.org/text
https://keras.io/api/keras_nlp/

Challenge

Currently, TFT requires ABI-level compatibility with the Tensorflow Core package.
For example, installing TFC from conda-forge, but TFT from pip results in errors such as:

ImportError: /root/miniconda3/envs/env/lib/python3.10/site-packages/tensorflow_text/core/pybinds/tflite_registrar.so: undefined symbol: _ZN4absl12lts_2021032420raw_logging_internal21internal_log_functionE

This has been reported in TFT's tracker: tensorflow/text#644
TFT documentation recommends building from source: https://www.tensorflow.org/text/guide/tf_text_intro#build_from_source

Steps to reproduce:

conda create -n text python
conda activate text
conda install -c conda-forge tensorflow=2.10
pip install tensorflow_text==2.10
python -c 'import tensorflow_text'

Discussion Questions

1. Is there an easy way to make TFT compatible with conda-forge's Tensorflow Core without rebuilding?
2. If not, what would be the steps build TFT in conda-forge?

[hmaarrfk]

There isn't good cross compatibility between conda and pip. They are built with different compilers, and thus potential differences in the library versions they use.

This is a "known issue". The only resolution is express to upstream that compatibility with "conda-forge" is important to you as a user, and that it is worth their time to help us here.

Given the breadth of libraries and pacakges we target, I don't think we can guarantee to be compatible with "pip" packages.

All that said, you can consider contributing the tensorflow-text pacakge. Follow: https://conda-forge.org/docs/maintainer/adding_pkgs.html#contributing-packages

Note that we don't have any windows builds for tensorflow at the moment.

[ngam]

xref conda-forge/staged-recipes#19578

If you decide to work on this, please tag us (@conda-forge/tensorflow). The likely productive approach is likely going to be similar to this PR: #206

[ngam]

Also, to give you an idea, this points to an issue with absl (... ZN4absl12lts_2021032420 ... in the error you provided); you may be able to reconcile the two versions if you find some build of tensorflow-text that utilizes the same toolchain we use here (perhaps unlikely)

[Anselmoo]

In order to implement TensorFlow-Text, it looks like that it depends on TensorFlow-Hub.

    install_requires=[
        'tensorflow>=2.11.0, <2.12; platform_machine != "arm64" or platform_system != "Darwin"',
        'tensorflow-macos>=2.11.0, <2.12; platform_machine == "arm64" and platform_system == "Darwin"',
        'tensorflow_hub>=0.8.0',

https://github.com/tensorflow/text/blob/3069d02044df98c5a96fee69bb9d3753dfbf4a50/oss_scripts/pip_package/setup.py#L75-L78

Most likely, this hast to be implemented first.

[AIWintermuteAI]

Is there any resolution to this?

[hmaarrfk]

The solution would be submit your version of a package to conda-forge at
https://github.com/conda-forge/staged-recipes/