[feature/update_to_spack_v1] Update to spack version 1.0.1 #563
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
CMake has released its first major version in a decade! This major version officially breaks support for multiple features deprecated in 3.5, and will no longer be compatible with projects specifying a required version below 3.5. This PR: - [x] Adds CMake 4.0 without making it preferred (yet) Co-authored-by: Todd Gamblin <[email protected]>
Co-authored-by: Harmen Stoppels <[email protected]>
* Refactor OSError handling in filesystem.py for Python 3 I changed `e.winerror` to `e.errno` in `lib/spack/llnl/util/filesystem.py` for Python 3 compatibility. This change specifically addresses an OSError handling block for Windows where `e.winerror == 5` (ACCESS_DENIED) was checked. The code now uses `e.errno == errno.EACCES` which is the standard way to check for this error in Python 3, while preserving the Windows-specific logic. I ran the existing tests, and they passed, confirming no regressions were introduced by this change. * Refactor OSError handling in filesystem.py for Python 3 (round 2) I simplified a Windows-specific OSError handling case in `lib/spack/llnl/util/filesystem.py` during symlink creation. The check for `e.winerror == 183` (Windows-specific "file exists") was removed in favor of the standard `e.errno == errno.EEXIST`, as Python 3 correctly maps this error code on Windows. This change further removes Python 2 era checks and relies on Python 3's improved error handling consistency across platforms. I ran the existing tests and they passed, confirming no regressions were introduced by this change. * Refactor loop in traverse.py to use `yield from` I replaced a `for item in iterable: yield item` loop with the more idiomatic `yield from iterable` in the `traverse_breadth_first_tree_edges` function within `lib/spack/spack/traverse.py`. This is a standard Python 3.3+ improvement for delegating to a sub-generator. Spack's minimum Python version is 3.6, making this change appropriate and safe. I also updated the associated comment. * I've removed some unnecessary comments from the code. Specifically, I took out the recently added explanatory comments in: - lib/spack/spack/test/llnl/util/lock.py (regarding Barrier exceptions) - lib/spack/spack/traverse.py (regarding `yield from` introduction) You indicated these comments weren't needed. The underlying code functionality, which was refactored previously, remains unchanged. * Remove useless statement * [@spackbot] updating style on behalf of haampie --------- Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> Co-authored-by: haampie <[email protected]>
`spack pkg grep` can construct command lines that are too long for Windows, i.e. command lines that are longer than 32768 characters. This makes `spack pkg grep` respect the Windows limit by default, and gets the unix limit from `sysconfig`. - [x] Add a new `spack.cmd.group_arguments` function to create CLI-safe arg groups - [x] Default to max 500 elements or 32768 chars, whichever comes first - [x] If sysconfig is available, get `SC_ARG_MAX` and use that for max chars - [x] Make `Executable` always return an error if it's not successful - [x] Add test for `group_arguments` --------- Signed-off-by: Todd Gamblin <[email protected]>
* py-pycocotools: numpy 2.3 not yet supported * Update homepage * py-pycocotools: add v2.0.9
Adds support for Spack package repositories from external git repos.
```yaml
repos:
my_repo:
git: https://example.com/example/example.git
destination: ~/example # optional
paths: # optional
- subdir/spack_repo/example/x
- subdir/spack_repo/example/y
```
If `destination` is not configured, Spack clones the git repo to `~/.spack/git_repos/{hash(repository)}`.
Package repositories can put a file `spack-repo-index.yaml` in their root with relative paths to roots of package repositories (i.e. directories that contain `repo.yaml`):
```yaml
repo_index:
paths:
- subdir/spack_repo/example/x
- subdir/spack_repo/example/y
```
so users don't have to put that under `paths` in config. The `spack-repo-index.yaml` is simply a list of paths under `repo_index`. The idea is to avoid duplicating data such as "namespace" already specified in `<git repo>/<repo path>/repo.yaml`, to avoid that there are two sources of truth that go out of sync.
Further, `paths` in user config takes precedence, which allows users to enable specific package repositories in case the git monorepo provides multiple.
Remote package repositories are cloned/initialized in:
* `spack.main`
* `spack repo add`
This is process safe due to the lock in `$SPACK_USER_CACHE_PATH/package-repository.lock`; only one process can clone at a time.
The `spack repo add` command has a few new flags and a new positional arg:
```
spack repo add [-h] [--name NAME] [--path PATH] [--scope ...] path_or_repo [destination]
```
The signature is similar to the familiar `git clone <repository> <directory>`.
The `path_or_repo` argument is detected as a remote git repo if it contains a `:` not preceded by a `/`, which is what git does as well. If in the future we would support package repositories other than local file paths and remote git repos, we can resolve ambiguities with a new flag `[--git | --path | --<other-type>]`, but this is currently unnecessary.
The positional `destination` argument allows users to pick their own clone path, and only applies in case of git repos.
The flag `--path` corresponds to `repos:<name>:paths` and can be repeated to select specific package repositories in a git monorepo, and is also required if the git repo does not provide a `spack-repo-index.yaml` file in its root. Spack will never scan for `repo.yaml` files recursively, it only relies on `spack-repo-index.yaml` and `--repo-path` for package repository roots inside a git repo.
The flag `--name` applies to the *config* name/key in `repos.yaml` under `repos:<name>`. This flag is optional in the common case of adding just one package repository: it is set to the package repository's namespace. In case of monorepos with multiple package repositories, the `--name` flag is required.
spack#50656) intel-oneapi-compilers added to optim_opts Transfer cflags, cxxflags from Spack to COPTS, CXXOPTS. --------- Co-authored-by: teabagk7 <[email protected]>
Using the -d flag leads to unreadable, and possibly truncated, output. See: - https://gitlab.spack.io/spack/spack/-/jobs/16901136 Signed-off-by: Massimiliano Culpo <[email protected]>
Move the functions responsible for retrieving buildcache manifests into the url_buildcache.py module, and generalize the API so that manifests for any type of buildcache component (specs, indices, keys, etc) can be fetched the same way.
spack.repo.PATH is constructed either in: 1. spack.main.setup_main_options (entrypoint of Spack) 2. or spack.repo.PATH.__getattr__ (singleton, in sub-processes that don't run main) The latter was using an old named constructor of RepoPath, which had regressed.
* Bump Trilinos/Kokkos dependency Signed-off-by: Samuel E. Browne <[email protected]> * Pin Kokkos/KokkosKernels more specifically for releases Signed-off-by: Samuel E. Browne <[email protected]> --------- Signed-off-by: Samuel E. Browne <[email protected]>
* set release channel for non nightly builds * style * use .satisfies Co-authored-by: Alec Scott <[email protected]> --------- Co-authored-by: Alec Scott <[email protected]>
Tracks unsupported targets explicitly to avoid clingo having to figure out a negative statement. This should improve solver performance. Signed-off-by: Massimiliano Culpo <[email protected]>
Signed-off-by: Harmen Stoppels <[email protected]>
22 tasks
… feature/update_to_spack_v1p0p0
In this way trying to solve more input specs is at higher priority than trying to satisfy strong preferences or requirements. Fixes a regression introduced by spack#44373 Signed-off-by: Massimiliano Culpo <[email protected]>
…ts.yaml to use new script REMOVE_SPACK_STACK_MODS_FOR_CI.sh to revert spack-stack extensions
climbfuji
changed the title
climbfuji
changed the base branch from
spack-stack-dev
to
feature/update_to_spack_v1
climbfuji
changed the title
…/spack into feature/update_to_spack_v1p0p0
climbfuji
force-pushed
the
feature/update_to_spack_v1p0p0
branch
from
cbc71de to
79385e5
Compare
climbfuji
requested review from
mathomp4,
eap,
AlexanderRichert-NOAA and
rickgrubin-noaa
3 tasks
|
@AlexanderRichert-NOAA Would you be able to also approve this so that I can merge the v1 updates into the staging branch? Thanks! |
mathomp4
approved these changes
Sep 25, 2025
AlexanderRichert-NOAA
approved these changes
Sep 25, 2025
github-project-automation
bot
moved this from In Progress
to Done
in spack-stack-2.0.x (2025 Q2-Q3)
climbfuji
added a commit
to JCSDA/spack-stack
that referenced
this pull request
Sep 25, 2025
This PR updates the spack submodule to version 1.0.1 with a few extra commits (see JCSDA/spack#563). The two most important changes are the separation of the spack packages (builtin repo) into its own repository https://github.com/spack/spack-packages, and that compilers are now "nodes" (regular dependencies with a few extra attributes). Please refer to the updated spack documentation and release pages for more details, I cannot cover this here. The separation of the repository is handled as follows: Our fork of the spack-packages repo https://github.com/jcsda/spack-packages is cloned as a submodule into repos/builtin (and spack is modified to find it there). The spack-stack repo lives under repos/spack_stack as part of the spack-stack GitHub repository. The spack_stack packages were migrated to the new API. The handling of compilers as nodes means that compilers.yaml no longer exists; instead, compilers are configured under their new names in packages.yaml. The GitHub CI runs on the self-hosted runner illustrate how to do this for a new site config. The spack-generated modules differ slightly from beforehand: - For lmod, modules for packages that do not depend on a compiler are now in modulefiles/Core - For tcl/ modules for packages are now in modulefiles/none/none and modulefiles/mpi-name/mpi-version/none/none - The module hierarchy names use the new compiler names instead of the legacy names (see spack.aliases) - The compiler changes and the associated updates to the spack engine provided an opportunity to update and simplify the spack stack setup-meta-modules extension significantly. With this, the 7-digit hash for the mpi module names in lmod are removed, and there is no longer a stack-python module. The spack python module is loaded as part of the module dependencies. We still have stack-compiler and stack-mpi. This PR updates only the Ubuntu self-hosted runner and a few selected site configs (see below, systems affected). I would like to merge this PR without waiting for all other site configs and container builds being updated to make this more manageable. We should create an issue to document that spack-stack develop will be broken for the sites that are not yet updated. Similarly, only selected templates were updated, all others moved to configs/templates_NOT_YET_CONVERTED_TO_SPACK_v1. In configs/common, the following compiler packages.yaml are updated: clang, gcc, oneapi. I suggest removing aocc, apple-clang, intel if nobody else is willing to fix them in a follow-up PR. Site config updates: I encourage spack-stack developers to use the tier1 atlantis or tier2 bounty/blackpearl site configs as blueprints for updating your site configs. Specifically, note one important change for the --compiler= argument for spack stack create env: spack stack create env --name=ne-oneapi-2025.1.0 --site=blackpearl --template=neptune-dev --compiler=oneapi-2025.1.0 2>&1 | tee log.create.ne-oneapi-2025.1.0.001 if the site contains configs/sites/tierX/sitename/packages_oneapi-2025.1.0.yaml. If the site contains configs/sites/tierX/sitename/packages_oneapi.yaml instead (the versioned and non-versioned files are mutually exclusive), then the command is spack stack create env --name=ne-oneapi-2025.1.0 --site=blackpearl --template=neptune-dev --compiler=oneapi 2>&1 | tee log.create.ne-oneapi-2025.1.0.001 This is to support multiple versions of the same compiler for a given site. (Side note: Because filter_compilers is no longer needed, it is removed.) Note that spack stack create env will not fail if the file packages_compiler.yaml or packages_compiler-version.yaml doesn't exist - it will print a big and fat warning, though. MISSING DOCUMENTATION UPDATES - REFER TO A FOLLOW UP PR BY SOMEONE ELSE
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR updates the spack code to release 1.0.1 https://github.com/spack/spack/releases/tag/v1.0.1 from August 11, 2025.
In addition, the following pull requests were cherry-picked from develop to speed up the concretizer (all thanks to @alalazo):
Finally, the following changes were made by @climbfuji:
spack/etc/spack/defaults/{config,repos.yaml}lib/spack/spack/modules/lmod.py: don't add hash to module hierarchy.github/workflows/prechecks.ymland.github/workflows/unit_test.yml(modify how to remove spack-stack extension)The two most important changes for this PR are:
Testing
CI tests for spack are all failing because of a missing import in a test file
or a similar module import error. I spent a little bit of time and concluded that this must be a problem with the spack 1.0.1 release branch itself and that there is nothing I can do unless I move from 1.0.1 to a later hash of develop that includes spack#51119. I cannot pull this PR into 1.0.1 cleanly either. Therefore,
let's just live with the fact that the CI tests don't pass until we make the next update from spack (hopefully 1.0.2 fixes this).I turned off most of the CI tests in commit 076c239.For testing in spack-stack, see JCSDA/spack-stack#1741