Chaining existing spack-stack installation is not working as expected on Derecho

[uturuncoglu]

I am trying to use chaining on Hercules to extend the existing installation for UFS Coastal. So, I did following,

git clone --recursive https://github.com/JCSDA/spack-stack.git
cd spack-stack
# edit packages slightly in configs/common/ and configs/sites/tier1/derecho/ to match with the exiting space-stack installation
. setup.sh
spack stack create env --name cop --template empty --site derecho --compiler intel --upstream /glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.6.0/envs/unified-env/install
cd envs/cop
spack env activate .
spack add conduit@develop+python+adios ^python@3.10.13 ^py-numpy@1.22.3 ^zstd@1.5.2
spack add esmf@=8.8.0
spack concretize --force --deprecated

After that, I check the modules with the spack find -c and it looks like following,

==> In environment /glade/work/turuncu/COP/spack-stack/envs/cop
==> 2 root specs
 -  conduit@develop +adios+python   -  esmf@8.8.0

-- linux-sles15-zen3 / intel@2021.10.0 --------------------------
 -   adios@1.13.1          -   conduit@develop     -   gettext@0.21.1              -   libffi@3.4.6        -   ncurses@6.5              -   py-mpi4py@3.1.6        -   py-wheel@0.41.2   -   util-linux-uuid@2.40.2
[e]  autoconf@2.69        [e]  cray-mpich@8.1.25  [e]  glibc@2.31                  -   libiconv@1.17       -   netcdf-c@4.9.2           -   py-mpi4py@3.1.6        -   python@3.10.13    -   util-macros@1.20.1
[e]  automake@1.15.1      [e]  curl@8.1.2         [e]  gmake@4.2.1                 -   libpciaccess@0.17   -   netcdf-fortran@4.6.1     -   py-numpy@1.22.3        -   python@3.10.13    -   xz@5.4.6
 -   berkeley-db@18.1.40  [e]  diffutils@3.6       -   hdf5@1.14.0                 -   libtool@2.4.7       -   parallel-netcdf@1.12.2   -   py-numpy@1.22.3        -   python-venv@1.0   -   zfp@0.5.5
 -   blt@0.6.2             -   esmf@8.8.0          -   hwloc@2.11.1                -   libxml2@2.13.4      -   parallelio@2.5.10        -   py-pip@23.1.2          -   python-venv@1.0   -   zlib-ng@2.2.1
 -   bzip2@1.0.8           -   expat@2.6.4         -   hwloc@2.11.1                -   libxml2@2.13.4      -   parmetis@4.0.3           -   py-pip@23.1.2          -   readline@8.2      -   zlib-ng@2.2.1
 -   c-blosc@1.21.5       [e]  findutils@4.8.0     -   intel-oneapi-mkl@2023.2.0   -   libyaml@0.2.5       -   perl@5.40.0              -   py-pyyaml@6.0.2        -   snappy@1.2.1      -   zstd@1.5.2
 -   c-blosc@1.21.5        -   gdbm@1.23           -   intel-oneapi-mkl@2023.2.0   -   lz4@1.10.0         [e]  pkg-config@0.29.2        -   py-setuptools@63.4.3   -   sqlite@3.46.0
 -   cmake@3.26.6          -   gdbm@1.24           -   intel-tbb@2021.12.0        [e]  m4@1.4.18           -   py-cython@0.29.36        -   py-setuptools@63.4.3   -   sqlite@3.46.0
 -   cmake@3.26.6          -   gettext@0.21.1      -   intel-tbb@2021.12.0         -   metis@5.1.0         -   py-cython@0.29.36        -   py-wheel@0.41.2       [e]  tar@1.34
==> 8 installed packages
==> 69 concretized packages to be installed

There are couple of issues with the output,

• Some modules are listed twice such as intel-oneapi-mkl@2023.2.0. I also checked with the spack find -c -f -vto see those have different compiler, variances but they look identical. So, I am not sure why spack is trying to install same package twice (maybe one of each new package - esmf vs. conduit). I also tried by passing--fresh-roots --reuse-depsto thespack concretize` but it did not helped.

• The local installation does not seem to see the upstream packages and modules. The spack find -c command shows that they will be still installed. I also tried to add following to the upstream section of my local spack.yaml but again it is still same.

  upstreams:
    spack-stack-1.6.0-unified-env:
      install_tree: /glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.6.0/envs/unified-env/install
      modules:
        lmod: /glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.6.0/envs/unified-env/install/modulefiles

At this point, I just wonder if anyone tried to chain exiting spack-stack installation before. Any suggestion would be helpful at this point.

[climbfuji]

If you have duplicates for basic packages (one from the original environment, one from your chained env), then that's an indication that everything that depends on these will be rebuilt if uses these. Look at the diffs between the duplicates, e.g. spack diff /hash1 /hash2.

[uturuncoglu]

@climbfuji Thanks for the tip. I am not seeing any particular difference between two hashes (except new_strategies variant for zlib-ng).

$ spack diff /x2dna55 /m5suhmz
==> Warning: This interface is subject to change.

--- intel-oneapi-mkl@2023.2.0/x2dna55nczunusjqxjeveuqw7a4tikht
+++ intel-oneapi-mkl@2023.2.0/m5suhmz4xokkqnw56arlpnthacftytxz
@@ hash @@
-  cmake doogtyd4au3iurzj4hmzyztn7rtzxrqz
+  cmake pl3j7xwea5ti5boxm3ytbbylhlvvcp4i
-  hwloc 42tu5qejscecjfxjq6sjy2rolqfkfaqe
+  hwloc rntkrzx4body4ahjvutolfifkxy2zja6
-  intel-oneapi-mkl x2dna55nczunusjqxjeveuqw7a4tikht
+  intel-oneapi-mkl m5suhmz4xokkqnw56arlpnthacftytxz
-  intel-tbb zr5oehbydozrll3k6qpikeelg2zd4mhk
+  intel-tbb b7wy6brovlmqzmjtejj7lppojmaq4kwn
-  libxml2 jbzklgflc7cp6jv67ldbcgexlo2v46kh
+  libxml2 f5tstilfsyxuwboxqh7kmfu427o2m73c
-  zlib-ng hf24kknkomts76iqfomafoxe6axgt7kr
+  zlib-ng g5szkxqbp6mlqpij6sgozhuh65d6vkqr
@@ variant_value @@
-  zlib-ng new_strategies True
+  zlib-ng new_strategies False

[uturuncoglu]

Maybe if I fix the variant issue I would have only one copy of intel-oneapi-mkl. Let me try.

[uturuncoglu]

@climbfuji Okay. After adding following to conduit package ^gdbm@1.23 ^zlib-ng+compat~new_strategies+opt+pic+shared. I could able to get single version of packages. I am not sure why spack not trying to use same variants for different packages but that is fine at this point. The only issue left at this point accessing to the modules provided by the upstream but I could still go ahead and install packages locally even if they are available in the upstream.

[climbfuji]

The concretizer should mark packages available in upstream (i.e. identical hash) as [^]. If it doesn't, then the hash is different and it can't reuse them.

[uturuncoglu]

@climbfuji Is there any way to compare the package in the upstream with the local one like we did with spack diff. Then , I could find the underlying difference and trying to fix those too.

[climbfuji]

The hash must be different, thus spack diff /hash1 /hash2 should work.

[climbfuji]

Closing due to a lack of activity