Skip to content
Open
Show file tree
Hide file tree
Changes from 37 commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
4dbac71
#240 Added new base class for Fab scripts.
hiker Jan 23, 2026
2254b31
#240 Removed support for transmute step to keep this PR smaller.
hiker Jan 23, 2026
c1e87d1
#240 Add all skeleton fab script, helper classes and site-configurati…
hiker Jan 23, 2026
a350ef3
#240 Updated tests to pass all linters, fixed licence statement.
hiker Jan 27, 2026
962ce0a
#240 Updated skeleton build script.
hiker Jan 27, 2026
54491da
#240 Fixed incorrect typing in test.
hiker Jan 27, 2026
22c2ef9
#240 Added README file.
hiker Jan 27, 2026
25c26fb
#240 Removed fab and vernier support from NCI site.
hiker Jan 27, 2026
72192df
#240 Fixed typos.
hiker Jan 27, 2026
f7a839c
#240 Allow import of psyclone_tools without setting PYTHONPATH.
hiker Jan 27, 2026
170cab3
#240 Fixed flake8 errors.
hiker Jan 27, 2026
d7aa9f1
#240 Added myself to contributors list.
hiker Jan 27, 2026
2315007
Merge branch 'main' into 240_add_skeleton_fab_script
hiker Jan 27, 2026
9ee0b64
Fixed rose picker etc issues raised in review.
hiker Feb 8, 2026
6dc4bd8
Renamed test directory to tests (to avoid clash with .gitignore in ro…
hiker Feb 8, 2026
00b7ec9
#240 Renamed rose_picker_tool.py to rose_picker.py etc.
hiker Feb 9, 2026
6a25195
#240 Addressed issues raised in review.
hiker Feb 9, 2026
1d4c878
Merge remote-tracking branch 'upstream/main' into 240_add_skeleton_fa…
hiker Feb 9, 2026
4872ec9
#240 Removed get_additional_psyclone_options.
hiker Feb 10, 2026
495d651
#240 Fixed configurator to use the new names of the tools.
hiker Feb 10, 2026
fa84662
#240 Fixed building current LFRic skeleton due to changes in build sy…
hiker Feb 10, 2026
bf87398
#240 Updated tests to work with changes to LFRic build system.
hiker Feb 10, 2026
463faae
#240 Removed unnecessary path for rose picker.
hiker Feb 10, 2026
0c9a599
#240 Set a default as project name.
hiker Feb 10, 2026
ac0999c
#240 Fixed incorrect names in templaterator.
hiker Feb 11, 2026
8421891
Merge remote-tracking branch 'upstream/main' into 240_add_skeleton_fa…
hiker Feb 11, 2026
0f42ad1
Merge remote-tracking branch 'upstream/main' into 240_add_skeleton_fa…
hiker Feb 11, 2026
c0fc24f
#240 Removed support for old-style environment variables for precision.
hiker Feb 19, 2026
434c394
#240 Removed unnecessary import.
hiker Feb 19, 2026
890a725
#240 Marked the new steps as steps so they get measured.
hiker Feb 19, 2026
d98699b
#240 Clarified docstring.
hiker Feb 19, 2026
5f2c892
Updated comments.
hiker Feb 20, 2026
3c3ba13
#240 Handle warning in new steps.
hiker Feb 20, 2026
d16a46b
#240 Moved build files into lfric_build directory.
hiker Feb 20, 2026
d3d4673
#240 Renamed default compiler setup scripts to 'setup_script_XXX.py'.
hiker Feb 23, 2026
a5309c1
#240 Fixed failing tests, handled warning.
hiker Feb 23, 2026
40b1863
Merge branch 'main' into 240_add_skeleton_fab_script
MatthewHambley Feb 26, 2026
9731fa1
#240 Remove unit-test as compilation profile.
hiker Mar 2, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 23 additions & 22 deletions CONTRIBUTORS.md
Original file line number Diff line number Diff line change
@@ -1,24 +1,25 @@
# Contributors

| GitHub user | Real Name | Affiliation | Date |
| ---------------- | ---------------------- | ----------- | ---------- |
| andrewcoughtrie | Andrew Coughtrie | Met Office | 2025.12.12 |
| james-bruten-mo | James Bruten | Met Office | 2025-12-09 |
| jedbakerMO | Jed Baker | Met Office | 2025-12-29 |
| jennyhickson | Jenny Hickson | Met Office | 2025-12-10 |
| mo-marqh | Mark Hedley | Met Office | 2025-12-11 |
| mo-rickywong | Ricky Wong | Met Office | 2025-01-30 |
| mike-hobson | Mike Hobson | Met Office | 2025-12-17 |
| MatthewHambley | Matthew Hambley | Met Office | 2025-12-15 |
| mo-lottieturner | Lottie Turner | Met Office | 2025-12-16 |
| tommbendall | Thomas Bendall | Met Office | 2026-01-23 |
| yaswant | Yaswant Pradhan | Met Office | 2025-12-16 |
| stevemullerworth | Steve Mullerworth | Met Office | 2026-01-08 |
| harry-shepherd | Harry Shepherd | Met Office | 2026-01-08 |
| EdHone | Ed Hone | Met Office | 2026-01-09 |
| tom-j-h | Tom Hill | Met Office | 2026-01-19 |
| mo-alistairp | Alistair Pirrie | Met Office | 2026-01-12 |
| t00sa | Sam Clarke-Green | Met Office | 2026-01-27 |
| MetBenjaminWent | Benjamin Went | Met Office | 2026-01-30 |
| jcsmeto | James Cunningham-Smith | Met Office | 2026-02-06 |
| thomasmelvin | Thomas Melvin | Met Office | 2026-01-15 |
| GitHub user | Real Name | Affiliation | Date |
| ---------------- | ----------------- | --------------------- | ---------- |
| andrewcoughtrie | Andrew Coughtrie | Met Office | 2025.12.12 |
| james-bruten-mo | James Bruten | Met Office | 2025-12-09 |
| jedbakerMO | Jed Baker | Met Office | 2025-12-29 |
| jennyhickson | Jenny Hickson | Met Office | 2025-12-10 |
| mo-marqh | Mark Hedley | Met Office | 2025-12-11 |
| mo-rickywong | Ricky Wong | Met Office | 2025-01-30 |
| mike-hobson | Mike Hobson | Met Office | 2025-12-17 |
| MatthewHambley | Matthew Hambley | Met Office | 2025-12-15 |
| mo-lottieturner | Lottie Turner | Met Office | 2025-12-16 |
| tommbendall | Thomas Bendall | Met Office | 2026-01-23 |
| yaswant | Yaswant Pradhan | Met Office | 2025-12-16 |
| stevemullerworth | Steve Mullerworth | Met Office | 2026-01-08 |
| harry-shepherd | Harry Shepherd | Met Office | 2026-01-08 |
| EdHone | Ed Hone | Met Office | 2026-01-09 |
| tom-j-h | Tom Hill | Met Office | 2026-01-19 |
| t00sa | Sam Clarke-Green | Met Office | 2026-01-27 |
| MetBenjaminWent | Benjamin Went | Met Office | 2026-01-30 |
| jcsmeto | James Cunningham-Smith | Met Office | 2026-02-06 |
| thomasmelvin | Thomas Melvin | Met Office | 2026-01-15 |
| hiker | Joerg Henrichs | Bureau of Meteorology | 2026-02-11 |
| mo-alistairp | Alistair Pirrie | Met Office | 2026-01-12 |
73 changes: 73 additions & 0 deletions applications/skeleton/fab_skeleton.py
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Part of the user experience we want to have is that building is always achieved through the same procedure. Specifically, changing into a project directory and issuing a command. The same command. So please rename all build scripts to build. Note that we do not add the .py extension to executables.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yaswant , @t00sa , @Pierre-siddall , @cameronbateman-mo

I did not see this mentioning in any coding style. IMHO, the fafct that typical coding filename extensions are not used in LFRic is pretty ... confusing. You can tell from the extension which kind of script it is, it makes it easier for tools to do the proper highlighting (and avoids the issue that you try to view a binary :) ). I always find it confusing if I see a script and can't even be certain if it is a script in the first place. If this is indeed LFRic coding style, it should be documented to become explicit.

Additionally, esp. if you work on a base class which will affect several other derived classes, having individual names for each applications make it much easier to know where you are in your editor (since an editor tab will usually not have enough space to display the whole path). E.g. you add or modify a method in LFRicBase, and then open one derived class after another (and then need to go back to fix something else up). I often end up with half a dozen fab* scripts. If they were all called build, I would not be able to see which tab is which build script (am happy to share a screenshot of my editor).

Also, using build might give the wrong impression to users, since there is no indication that this script requires Fab to be installed. An alternative would be to use .fav as suffix, but that then also confuses editors :)

Furthermore, we are still experimenting about the best design of scripts. For example, I often use PSyclone's kernel extraction, which requires a new step to be run before PSyclone (and files to be added to the build system etc). ATM, I am using a mixin class to add the new step, and than for any apps that I want to use kernel extraction with, I just create a derived class form the Fab build of this class, add the mixin, and it all works. But I have two different build scripts then (e.g. fab_lfric_atm.py, and fab_lfric_atm_extract.py. Obviously, other designs are possible (use extraction as a command line option and have if-tests to decide in the script what steps to do). For now, I prefer the design that allows me to create a derived script (since kernel extraction is pretty niche, so I don't want other people to get confused by code that in all likelihood they will never need). Additionally, we also use this in lfric inputs, to have separate scripts that build part of the tools (and one script that builds all). Using OO design here makes it easy to avoid code duplication. But we obviously can't do that with just one name.

IMHO, the condition of 'just one name' feels like a left-over from using Makefiles :) We have the opportunity with Fab to allow to design code that's easier to understand and manage, and given that I don't see the 'one name' as an explicit requirement, I would argue that this is the better approach.

Feedback welcome.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is common practice to drop the extension for executables, for instance executable shell scripts do not usually have the .sh extension. It is not in any code style document because it is common practice. The extension is still used when the file is a module in a package. It doesn't cause confusion because from the user's point of view it isn't a Python (or anything else) file, it is an executable which they will execute.

Most editors I have come across which provide syntax highlighting will also interrogate the "shebang" line to inform highlighting mode. Any executable implemented using Python should have a shebang.

Yes, having multiple identically named files open can be confusing, but this will be an infrequent occurrence. Particularly after initial development. However, the key thing to bare in mind here is that this is user facing and so should favour ease of use for the user, even when that is at the expense of development ease.

The requirement of the Fab framework is a project level one and will be documented at that level. A try/except block around the import could be used to provide a fuller error message but the user will still get a "missing module" error without it.

If there is a requirement to support exotic usage such as kernel extraction then projects which require this can have a .py file somewhere within them, possibly a directory fab which can then be imported into the build script and anywhere else it is needed.

Rather than being a "left-over" we looked at the way Make has a consistent user experience and said "Yes! That's what we want for our new system too." Furthermore, it is a common approach among build, and related, tooling. See CMake or Ninja which both use a standard command and a commonly named configuration file. An advantage of this approach is to clearly signal to the user when they can build (they see the configuration file) and how to do it, the use the build command.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While I might not agree with the reasoning for using the same name, I am happy to do this. Do we really want to call it 'build.py', and not 'fab.py'? The latter would make it more obvious to the unaware users :)

But, I have to insist on maintaining the .py endings. Without this, we can't use inheritance in fab application scripts, and that is a big plus of using python as language. The kernel extraction is not an 'exotic' usage, it has been used in NG-ARCH, and other sites are using it ATM to evaluate GPU ports of LFRic. The current, OO design is nice, changes for extraction easily self-contained.

While I am aware that you could of course replace inheritance with if-statements (if extraction then ...), but from a software design point of view this makes here no sense at all. ATM, if you want to know how extraction works, there is one file which contains everything (as opposed to searching through a significantly larger file with if conditions).

Other use cases might include adding run-time checks (which requires different scripts and config files).

I can't of course promise that there will be significant use, we need to get experience. But I am strongly against crippling the Fab build system by restricting very common OO python features.

@yaswant , @t00sa , @Pierre-siddall , @cameronbateman-mo, @MatthewHambley , I fear we might need to have a discussion about this.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fine, fab.py for the time being.

Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
#!/usr/bin/env python3

##############################################################################
# (c) Crown copyright Met Office. All rights reserved.
# The file LICENCE, distributed with this code, contains details of the terms
# under which the code may be used.
##############################################################################
# Author: J. Henrichs, Bureau of Meteorology
# Author: J. Lyu, Bureau of Meteorology

"""
A FAB build script for applications/skeleton. It relies on
the LFRicBase class contained in the infrastructure directory.
"""

import logging
from pathlib import Path
import sys

from fab.steps.grab.folder import grab_folder

# We need to import the base class:
sys.path.insert(0, str(Path(__file__).parents[2] / "lfric_build"))

from lfric_base import LFRicBase # noqa: E402


class FabSkeleton(LFRicBase):
"""
A Fab-based build script for skeleton. It relies on the LFRicBase class
to implement the actual functionality, and only provides the required
source files.

:param name: The name of the application.
"""

def __init__(self, name: str = "skeleton") -> None:
super().__init__(name=name)
# Store the root of this apps for later
this_file = Path(__file__).resolve()
self._this_root = this_file.parent

def grab_files_step(self) -> None:
"""
Grabs the required source files and optimisation scripts.
"""
super().grab_files_step()
dirs = ['applications/skeleton/source/']

# pylint: disable=redefined-builtin
for dir in dirs:
grab_folder(self.config, src=self.lfric_core_root / dir,
dst_label='')

# Copy the optimisation scripts into a separate directory
grab_folder(self.config, src=self._this_root / "optimisation",
dst_label='optimisation')

def get_rose_meta(self) -> Path:
"""
:returns: the rose-meta.conf path.
"""
return (self._this_root / 'rose-meta' / 'lfric-skeleton' / 'HEAD' /
'rose-meta.conf')


# -----------------------------------------------------------------------------
if __name__ == '__main__':

logger = logging.getLogger('fab')
logger.setLevel(logging.DEBUG)
fab_skeleton = FabSkeleton()
fab_skeleton.build()
54 changes: 54 additions & 0 deletions lfric_build/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# LFRic Core Fab Build Scripts

Make sure you have Fab version 2.0.1 or later installed (in addition to all
LFRic core requirements of course).

## Setting up Site- and Platform-specific Settings
Site- and platform-specific settings are contained in
```$LFRIC_CORE/infrastructure/build/site-specific/${SITE}-${PLATFORM}```
The default settings are in ```.../site-specific/default``` (and at this
stage each other site-specific setup inherits the values set in the default,
and then adds or modifies settings). The Fab build system provides various
callbacks to the ```config.py``` file in the corresponding directory (details
are in the [Fab documentation](https://metoffice.github.io/fab/fab_base/config.html).

If there is no existing site-specific setup, it is recommended to copy an existing
configuration file (e.g. from ```nci_gadi/config.py```). This act as a template
to indicate where you can specify linker information, select a default compiler
suite etc.

The default setup contains compiler flags for Cray, GNU, Intel-classic (ifort),
Intel-LLVM (ifx), and NVIDIA. For modularity's sake (and to keep the file length
shorter), the default configuration will get the settings from the corresponding
```setup_...py``` script. There is no need for a site to replicate this structure,
existing ```config.py``` scripts show how this can be done.


## Building the Skeleton Apps

In order to build the skeleton apps, change into the directory
```$LFRIC_CORE/applications/skeleton```,
and use the following command:

```
./fab_skeleton.py --nprocs 4 --site nci --platform gadi --suite intel-classic
```
Select an appropriate number of processes to run in parallel, and your site and platform.
If you don't have a default compiler suite in your site-specific setup (or
want to use a non-default suite), use the ``--suite`` option. Once the process is finished,
you should have a binary in the directory
```./fab-workspace/skeleton-full-debug-COMPILER``` (where ```COMPILER``` is the compiler
used, e.g. ```mpif90-gfortran```).

Using ```./fab_skeleton.py -h``` will show a help message with all supported command line
options (and their default value). If a default value is listed using an environment
variables (```(default: $SITE or 'default')```), the corresponding environment variable
is used if no command line option has been specified.

A different compilation profile can be specified using ```--profile``` option. Note
that the available compilation profiles can vary from site to site (see
[Fab documentation](https://metoffice.github.io/fab/fab_base/config.html) for details).

If Fab has issues finding a compiler, you can use the Fab debug option
```--available-compilers```, which will list all compilers and linkers Fab has
identified as being available.
103 changes: 103 additions & 0 deletions lfric_build/configurator.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
##############################################################################
# (c) Crown copyright Met Office. All rights reserved.
# The file LICENCE, distributed with this code, contains details of the terms
# under which the code may be used.
##############################################################################
# Author J. Henrichs, Bureau of Meteorology
# Author J. Lyu, Bureau of Meteorology

"""
This file defines the configurator script sequence for LFRic.
"""

import logging
from pathlib import Path
from typing import cast, Optional

from fab.api import BuildConfig, find_source_files, Category
from fab.tools.shell import Shell

from rose_picker import RosePicker

logger = logging.getLogger('fab')


def configurator(config: BuildConfig,
lfric_core_source: Path,
rose_meta_conf: Path,
include_paths: Optional[list[Path]] = None,
config_dir: Optional[Path] = None) -> None:
"""
This method implements the LFRic configurator tool.

:param config: the Fab build config instance
:param lfric_core_source: the path to the LFRic core directory
:param rose_meta_conf: the path to the rose-meta configuration file
:param include_paths: additional include paths (each path will be added,
as well as the path with /'rose-meta')
:param config_dir: the directory for the generated configuration files
"""

tools = lfric_core_source / 'infrastructure' / 'build' / 'tools'
config_dir = config_dir or config.build_output / 'configuration'
config_dir.mkdir(parents=True, exist_ok=True)

# rose picker
# -----------
# creates rose-meta.json and config_namelists.txt in
# gungho/build
logger.info('rose_picker')

include_dirs = [lfric_core_source / 'rose-meta']
if include_paths:
for path in include_paths:
include_dirs.append(path / 'rose-meta')

rose_picker = RosePicker()
rose_picker.execute(rose_meta_conf, config_dir,
include_paths=include_dirs)
rose_meta = config_dir / 'rose-meta.json'

shell = config.tool_box.get_tool(Category.SHELL)
shell = cast(Shell, shell)

# build_config_loaders
# --------------------
# builds a bunch of f90s from the json
logger.info('GenerateNamelistLoader')
shell.exec(f"{tools / 'GenerateNamelistLoader'} -verbose {rose_meta} "
f"-directory {config_dir}")

# create configuration_mod.f90 in source root
# -------------------------------------------
logger.info('GenerateConfigLoader')
with open(config_dir / 'config_namelists.txt', encoding="utf8") as f_in:
names = [name.strip() for name in f_in.readlines()]

shell.exec(f"{tools / 'GenerateConfigLoader'} "
f"{' '.join(names)} "
f"-o {config_dir}")

logger.info('GenerateExtendedNamelistType')
shell.exec(f"{tools / 'GenerateExtendedNamelistType'} {rose_meta} "
f"-directory {config_dir}")

duplicates: list[str] = []
with open(config_dir / 'duplicate_namelists.txt', encoding="utf8") as f_in:
for name in f_in.readlines():
duplicates.extend(["-duplicate", name.strip()])

logger.info('GenerateConfigType')
shell.exec(f"{tools / 'GenerateConfigType'} "
f"{' '.join(names)} "
f"{' '.join(duplicates)} "
f"-o {config_dir}")

# create feign_config_mod.f90 in source root
# ------------------------------------------
logger.info('GenerateFeigns')
feign_config_mod_fpath = config_dir / 'feign_config_mod.f90'
shell.exec(f"{tools / 'GenerateFeigns'} {rose_meta} "
f"-output {feign_config_mod_fpath}")

find_source_files(config, source_root=config_dir)
Loading
Loading