document when conda support breaks down (aka GLIBC errors)

[jr3cermak]

The pinned versions of various pythonic items for gridtools still works. Conda support begins to break down when authors recompile prior versions with newer underlying operating systems. Pinned versions of compiled python modules will begin to fail on compute clusters with very old GLIBC libraries. Documentation to follow.

[jr3cermak]

Reloading an environment that was previously stated as working may yield a result like this:

(gridtest) jrcermakiii@chinook03:~/src/gridtools/examples$ python mkGridsExample01.py 
Traceback (most recent call last):
  File "mkGridsExample01.py", line 12, in <module>
    from gridtools.gridutils import GridUtils
  File "/import/AKWATERS/jrcermakiii/src/gridtools/gridtools/gridutils.py", line 5, in <module>
    import xarray as xr
  File "/import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/xarray/__init__.py", line 1, in <module>
    from . import testing, tutorial, ufuncs
  File "/import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/xarray/testing.py", line 8, in <module>
    from xarray.core import duck_array_ops, formatting, utils
  File "/import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/xarray/core/duck_array_ops.py", line 13, in <module>
    import pandas as pd
  File "/import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/__init__.py", line 22, in <module>
    from pandas.compat import (
  File "/import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/compat/__init__.py", line 15, in <module>
    from pandas.compat.numpy import (
  File "/import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/compat/numpy/__init__.py", line 7, in <module>
    from pandas.util.version import Version
  File "/import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/util/__init__.py", line 1, in <module>
    from pandas.util._decorators import (  # noqa
  File "/import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/util/_decorators.py", line 14, in <module>
    from pandas._libs.properties import cache_readonly  # noqa
  File "/import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/__init__.py", line 13, in <module>
    from pandas._libs.interval import Interval
ImportError: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/interval.cpython-37m-x86_64-linux-gnu.so)

This is an indication that the underlying system (or cluster) you are using is falling out of date. The supported python versions of modules managed by conda are under constant integration (CI) are are rebuilt using newer systems and hence are linked to more recent versions of GLIBC. On my particular system, the underlying GLIBC is version 2.12. One or more of the loaded python modules now requires GLIBC 2.14. Unless you have a backup of the conda environment, there is no way to recover the original operating environment. You will likely have to resort to conventional manual installation of modules to restore the environment.

To see what your conda installation is using for GLIBC, use conda info. The version of glibc should be noted in the virtual packages.

$ conda info

     active environment : gridtest
    active env location : /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest
            shell level : 1
       user config file : /home/jrcermakiii/.condarc
 populated config files : /home/jrcermakiii/.condarc
          conda version : 4.9.2
    conda-build version : not installed
         python version : 3.9.1.final.0
       virtual packages : __glibc=2.12=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /import/AKWATERS/jrcermakiii/local/miniconda3  (writable)
           channel URLs : https://conda.anaconda.org/pyviz/linux-64
                          https://conda.anaconda.org/pyviz/noarch
                          https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /import/AKWATERS/jrcermakiii/local/miniconda3/pkgs
                          /home/jrcermakiii/.conda/pkgs
       envs directories : /import/AKWATERS/jrcermakiii/local/miniconda3/envs
                          /home/jrcermakiii/.conda/envs
               platform : linux-64
             user-agent : conda/4.9.2 requests/2.25.0 CPython/3.9.1 Linux/2.6.32-754.35.1.el6.61015g0000.x86_64 centos/6.10 glibc/2.12
                UID:GID : 3739:2161
             netrc file : None
           offline mode : False

[jr3cermak]

The second failure mode is updating the base environment for conda. This message is typically seen when a new conda version is available.

==> WARNING: A newer version of conda exists. <==
  current version: 4.9.2
  latest version: 4.12.0

Please update conda by running

    $ conda update -n base -c defaults conda

However, due to the age of your operating system, doing the above can render your conda unusable even though the above command will succeed! You may see the following error messages after updating for one or more of the conda commands:

$ conda create -n gridtest python==3.7.10
Collecting package metadata (current_repodata.json): failed

CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://conda.anaconda.org/pyviz/linux-64/current_repodata.json>
Elapsed: -

An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.
'https://conda.anaconda.org/pyviz/linux-64'

Again, there is no easy way to recover from this incident unless you have a backup of your conda installation directory. The only way to recover is to reinstall conda with a last known working version and reload your individual conda environments.

If conda is broken beyond repair, there may be a way to salvage your environments. If you
were fortunate to keep the miniconda shell installer archive (Miniconda3-py39_4.9.2-Linux-x86_64.sh),
then you may be in luck.

Process:

• Deactivate any exiting environments
• Rename the existing miniconda root to a backup
• Remove any .bashrc references
• Re-login to clear any lingering shell enviroment references
• Re-install miniconda to create a new miniconda root
• Re-login to initialize the new shell environment
• Change directory into the backup root envs directory
• Copy (cp -a) environments you wish to salvage into the newly created miniconda root envs directory
• Attempt to activate the copied environment
• If all checks out, erase the backup miniconda root

[jr3cermak]

We have also discovered that this GLIBC issue is currently confined to one particular package: pandas. The fix is to load the conda environment as usual for gridtools. Manually uninstall and reinstall pandas via pip.

$ pip uninstall pandas
Found existing installation: pandas 1.3.5
Uninstalling pandas-1.3.5:
  Would remove:
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas-1.3.5-py3.7.egg-info
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/*
  Would not remove (might be manually added):
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/algos.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/arrays.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/groupby.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/hashing.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/hashtable.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/index.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/indexing.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/internals.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/interval.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/join.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/json.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/lib.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/missing.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/ops.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/ops_dispatch.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/parsers.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/properties.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/reduction.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/reshape.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/sparse.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/testing.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/tslib.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/tslibs/base.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/tslibs/ccalendar.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/tslibs/conversion.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/tslibs/dtypes.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/tslibs/fields.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/tslibs/nattype.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/tslibs/np_datetime.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/tslibs/offsets.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/tslibs/parsing.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/tslibs/period.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/tslibs/strptime.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/tslibs/timedeltas.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/tslibs/timestamps.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/tslibs/timezones.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/tslibs/tzconversion.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/tslibs/vectorized.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/window/aggregations.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/window/indexers.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/_libs/writers.cpython-37m-x86_64-linux-gnu.so
    /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages/pandas/io/sas/_sas.cpython-37m-x86_64-linux-gnu.so
Proceed (Y/n)? y
  Successfully uninstalled pandas-1.3.5


(gridtest) jrcermakiii@chinook03:~/src/gridtools/docs/conda$ pip install pandas==1.3.5
Collecting pandas==1.3.5
  Downloading pandas-1.3.5.tar.gz (4.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.7/4.7 MB 7.6 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: python-dateutil>=2.7.3 in /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages (from pandas==1.3.5) (2.8.2)
Requirement already satisfied: numpy>=1.17.3 in /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages (from pandas==1.3.5) (1.21.6)
Requirement already satisfied: pytz>=2017.3 in /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages (from pandas==1.3.5) (2022.1)
Requirement already satisfied: six>=1.5 in /import/AKWATERS/jrcermakiii/local/miniconda3/envs/gridtest/lib/python3.7/site-packages (from python-dateutil>=2.7.3->pandas==1.3.5) (1.16.0)
Building wheels for collected packages: pandas
  Building wheel for pandas (pyproject.toml) ... done
  Created wheel for pandas: filename=pandas-1.3.5-cp37-cp37m-linux_x86_64.whl size=29515586 sha256=f5ace4f1391d0a3f8d58490453859e86cdd194c47247b879cade9e05bebbb4e2
  Stored in directory: /import/home/jrcermakiii/.cache/pip/wheels/8b/a0/9b/00c6a08ec7b87bf8d7bf4d9ddde7dd7da7c6e822041c580287
Successfully built pandas
Installing collected packages: pandas
Successfully installed pandas-1.3.5