Add developer docs

[aidanheerdegen]

It would be helpful to those who are trying to develop a spack package if there were developer docs that stepped through the process of

1. Creating a spack package
2. Modifying an existing package

Docs should include

• style and design recommendations
• how to test and testing requirements
• communication requirements: who should be consulted when modifying existing packages
• review requirements: who should be asked to review PRs

[harshula]

[WIP] Create or modify Spack Package Recipes (SPR) or Spack Package Bundles (SBR)

Introduction

Start by familiarising yourself with the section headings in the Spack Packaging Guide. Remember to read the relevant sections as needed.

Style and Design

There are thousands of builtin Spack Package Recipes in the Spack Git repository in the directory var/spack/repos/builtin/packages/. Refer to them for clarity.

We have many SPRs and SBRs in https://github.com/ACCESS-NRI/spack-packages/tree/main/packages . Refer to them for style and design recommendations.

Procedures

1. The SPR/SBR contains a list of maintainers. Please communicate with the maintainer about why you want to change the SPR/SBR. Add the maintainer as a reviewer when you want to merge the changes.

2. Before setting a Pull Request as Ready for Review, make sure you do a test build of the package via Spack and add the output to the Pull Request.

Layers of abstraction

A Spack Package Recipe (SPR) uses the build system of the underlying source package to build the binary package. A Model Deployment Repository (MDR) contains a Spack environment file (spack.yaml) that uses the SPR to build one or more source packages, including dependencies.

We prefer to define the exact version of the source package(s) and dependencies only in the spack.yaml file. Therefore, we avoid defining an exact version of dependencies in the SPR. If a dependency needs to be limited to a range of versions starting at a minimum version and/or finishing at a maximum version, that is acceptable.

Decisions regarding changing the source package's build system are subjective and the Model Release team can provide advice. A useful rule is to first decide whether you want to modify the source package and whether ACCESS-NRI will need to take responsibility for maintaining it. If you do not want to modify a source package, then any compiler flag modifications and/or dependency version changes should be made from the SPR. Spack provides functions to allow SPRs to modify the source package's build system.

Compiler flags

Compiler flags can be set in the Spack environment file, the SPR and/or the source package's build system.

Generally, compiler flags should not be set in the Spack environment file (spack.yaml), unless it is for debugging or experimental purposes. This functionality should only be used by advanced Spack users because it can be complicated to verify that the compiler flags have been applied. For more details, please read: https://github.com/ACCESS-NRI/dev-docs/wiki/Spack#compiler-flag-override . The inject_flags option is the default flag_handler: https://spack.readthedocs.io/en/latest/packaging_guide.html#compiler-flags . This means that CMake and Autotools based SPRs need to set the flag_handler to build_system_flags in-order for compiler flag overrides to appear in the build logs. Otherwise, they only appear in the Spack debug build logs when spack -d install is run.

SPRs can contain compiler flags that are applied to the source package. In particular, an SPR may have a variant that modifies the source package's compiler flags. e.g. A variant for deterministic builds. For more details on variants, please read: https://spack.readthedocs.io/en/latest/packaging_guide.html#id6 . The way to use variants with CMake are documented with: https://spack.readthedocs.io/en/latest/build_systems/cmakepackage.html . The Model Release team can provide advice on appropriate defaults for variants.

All compiler flags that are required to build the package should be in the source package. The source package can contain compiler flags that are optional and can be enabled/disabled by the user. These compiler flags can be manipulated from the SPR. Consider grouping the compiler flags as required, optional and optimisation flags. Allow the user to override the optional and optimisation compiler flags.

spack-config

The spack-config repository contains ACCESS-NRI's preferred Spack configuration. These configuration files also allow the HPC system's unique customisations to be hidden from the SPRs and users. e.g.

    - spec: openmpi@4.1.5 %intel
      prefix: /apps/openmpi/4.1.5
      modules: [openmpi/4.1.5]
      extra_attributes:
        environment:
          prepend_path:
            CMAKE_PREFIX_PATH: /apps/openmpi/4.1.5/include/Intel

[harshula]

Hi @micaeljtoliveira , Is the above outline roughly the direction you want the document to take?

[micaeljtoliveira]

Hi @harshula, I'm not quite sure what kind of feedback you're looking for here. Could you elaborate a bit?

[harshula]

Hi @micaeljtoliveira , What did you want this document to cover?

[aidanheerdegen]

@harshula and discussed your points about the confusion/lack of clarity about what should go in build systems, spack packages and deployment repos/environments. I believe that is what he is referring to.

Related comment for that: #171 (comment)

[micaeljtoliveira]

@aidanheerdegen Thanks for providing the context!

Hi @micaeljtoliveira , Is the above outline roughly the direction you want the document to take?

@harshula No, I was not looking for detailed step-by-step instructions for creating SPRs, although that's what the description of this issue seems to indicate. Instead, I would like some documentation on some design choices for SPRs regarding:

• compiler flags: what type of flags should go into the SPRs, which ones should go into the package build system and which ones should go into the spack environments (see Reviewing and setting build flags ACCESS-OM3#73 for related OM3 discussion)
• package versions: in which circumstances should package versions be restricted in the SPRs (@aidanheerdegen reply in Spack Components for OM3 #171 (comment) is a good starting point).
• in general, what should be in the build system, what should be implemented in the SPR and what should be left to the spack environments.

Regarding the last point, this has a big impact on who is responsible for what and on how flexible/restrictive our deployment machinery is. So I think it's important this is properly discussed and documented.

[harshula]

Hi @micaeljtoliveira , Can you please review #224 (comment) and see if it provides sufficient guidance?

[micaeljtoliveira]

Hi @harshula, that's exactly the type of information I wanted and it's a great starting point for further discussion. Thanks!

Give me a bit of time to read it again more thoroughly and I'll get back to you with some more detailed feedback ASAP.

[micaeljtoliveira]

Hi @harshula , sorry for taking this long to get back to you. Here are some thoughts, not focusing too much on the details at this point.

Start by reading the Spack Packaging Guide

This might be too much too ask in some cases. For example, if someone only wants to add/change a compiler flag, having to read a document that is a few thousand lines long for such a trivial task seems unreasonable to me. I would say we can't (and shouldn't) all become Spack experts.

There are several places in the draft that call for decisions to be made, but there are not clear responsably people for those decisions. For example:

Decisions regarding changing the source package's build system are subjective. A useful rule is to first decide whether we want to modify the source package and whether we will need to take responsibility for maintaining it. If we do not want to modify a source package, then any compiler flag modifications and/or dependency versions should be mde from the SPR.

I would be good to be more explicit here and say who "we" is. Also, who would have the final say about this decision?

Generally, compiler flags should not be set in the Spack environment file (spack.yaml), unless it is for debugging purposes. This functionality should only be used by advanced Spack users because it is complicated to verify that the compiler flags have been applied.

Why is this complicated? It's a bit concerning that we can't trust Spack to do such a simple thing as setting a compiler flag and honour it.

SPRs can contain compiler flags that are applied to the source package. In particular, an SPR may have a variant that modifies the source package's compiler flags. e.g. a variant for deterministic builds.

Would be nice to have more information about variants. For example, mention build-types and which build-types we should aim at supporting.

All compiler flags that are required to build the package should be in the source package. The source package can contain compiler flags that are optional and can be enabled/disabled by the user. These compiler flags can be manipulated from the SPR.

I'm not sure if this will work in practice. There's a very large number of possible compiler flags and possible combinations and there's usually not a "one size fits all" set of flags for a given package. It's quite hard to predict the needs of users, so having more freedom/flexibility in tuning the environment would probably outweigh possible downsides.

In general, I would favour an approach were we try to minimize the parts that package developers/users need to interact with. From my point of view, the ideal situation would be one where only the model-release team and the corresponding
maintainers interact with the SPRs. Allowing users to set flags in the Spack environments would go towards this goal.

Hope this helps. Let me know if would like to meet to discuss some of these points more in detail.

[harshula]

Hi @micaeljtoliveira , Thanks! I've updated the text according to your feedback. PR #239 and PR #240 should make it easier to use the Spack environment to set compiler flags and then verify that the compiler flags were applied. Furthermore, PR ACCESS-NRI/build-cd#271 will make the Spack debug log files available for Pre-release builds.