Skip to content

Test rocotoboot dryrun #4245

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
TerrenceMcGuinness-NOAA wants to merge 13 commits into from
Closed

Test rocotoboot dryrun #4245

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
6f912c8
Forcing the GitLab pipeline to use the Rocoto dryrun build over the s…
TerrenceMcGuinness-NOAA Nov 13, 2025
efedf2d
added UTIL path on ci config for platform specifics to custom rocoto …
TerrenceMcGuinness-NOAA Nov 13, 2025
53d995e
updated path to Rocoto used in pipeline
TerrenceMcGuinness-NOAA Nov 17, 2025
49541a2
Merge branch 'NOAA-EMC:develop' into test_rocotoboot_dryrun
TerrenceMcGuinness-NOAA Nov 17, 2025
de087c3
added paths to rocotoboot dryrun uddate for Ursa in ci platforms conf…
TerrenceMcGuinness-NOAA Nov 17, 2025
cc5470f
Merge branch 'NOAA-EMC:develop' into test_rocotoboot_dryrun
TerrenceMcGuinness-NOAA Nov 18, 2025
85b1d44
Merge branch 'NOAA-EMC:develop' into test_rocotoboot_dryrun
TerrenceMcGuinness-NOAA Dec 1, 2025
683baf0
Merge branch 'NOAA-EMC:develop' into test_rocotoboot_dryrun
TerrenceMcGuinness-NOAA Dec 3, 2025
9928ed1
needed to update PATH to pickup the rocotoboot we are testing
TerrenceMcGuinness-NOAA Dec 3, 2025
198b249
forgot to get MACHINE_ID to soruce the correct platform for getting t…
TerrenceMcGuinness-NOAA Dec 3, 2025
f93a587
needed to drop the GW on HOMEgfs in execure ctest template
TerrenceMcGuinness-NOAA Dec 3, 2025
df973aa
Merge branch 'NOAA-EMC:develop' into test_rocotoboot_dryrun
TerrenceMcGuinness-NOAA Dec 5, 2025
5196edd
Merge branch 'NOAA-EMC:develop' into test_rocotoboot_dryrun
TerrenceMcGuinness-NOAA Dec 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .gitlab-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,18 @@ include:

echo "Setting up build environment for ${machine^}..."
echo "Using build directory ${GW_HOMEgfs}"

# TEMPORARY: Use custom Rocoto build with dryrun feature for testing
# TODO: Remove this when upstream Rocoto PR is merged
# Load platform-specific configuration first
source "${HOMEgfs}/dev/ci/platforms/config.${MACHINE_ID}"
if [[ -n "${ROCOTO_TEST_PATH:-}" && -d "${ROCOTO_TEST_PATH}" ]]; then
export PATH="${ROCOTO_TEST_PATH}:$PATH"
echo "Using custom Rocoto build at: $(which rocotorun 2>/dev/null || echo 'NOT FOUND')"
else
echo "ROCOTO_TEST_PATH not set or directory not found, using system Rocoto"
fi

git submodule status
# When GFS_CI_RUN_TYPE is set to "nightly", we use the develop branch
# It should be set when configuring the scheculed nightly pipeline
Expand Down
12 changes: 12 additions & 0 deletions dev/ci/gitlab-ci-cases.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,18 @@
script:
- |
echo "Using build directory ${GW_HOMEgfs} (dated ${DATE})"

# TEMPORARY: Use custom Rocoto build with dryrun feature for testing
# TODO: Remove this when upstream Rocoto PR is merged
# Load platform-specific configuration first
source "${GW_HOMEgfs}/dev/ci/platforms/config.${machine}"
if [[ -n "${ROCOTO_TEST_PATH:-}" && -d "${ROCOTO_TEST_PATH}" ]]; then
export PATH="${ROCOTO_TEST_PATH}:$PATH"
echo "Using custom Rocoto build at: $(which rocotorun 2>/dev/null || echo 'NOT FOUND')"
else
echo "ROCOTO_TEST_PATH not set or directory not found, using system Rocoto"
fi

pslot=$("${GW_HOMEgfs}/dev/ci/scripts/utils/ci_utils.sh" get_pslot "${RUNTESTS_DIR}" "${caseName}")
"${GW_HOMEgfs}/dev/ci/scripts/run_check_gitlab_ci.sh" "${GW_RUN_PATH}" "${pslot}" global-workflow
exit $?
Expand Down
22 changes: 22 additions & 0 deletions dev/ci/gitlab-ci-ctests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,18 @@
stage: setup_tests
script: |
echo "Setting up for ctests pipeline building in ${GW_HOMEgfs} on ${machine}"

# TEMPORARY: Use custom Rocoto build with dryrun feature for testing
# TODO: Remove this when upstream Rocoto PR is merged
# Load platform-specific configuration first
source "${GW_HOMEgfs}/dev/ci/platforms/config.${machine}"
if [[ -n "${ROCOTO_TEST_PATH:-}" && -d "${ROCOTO_TEST_PATH}" ]]; then
export PATH="${ROCOTO_TEST_PATH}:$PATH"
echo "Using custom Rocoto build at: $(which rocotorun 2>/dev/null || echo 'NOT FOUND')"
else
echo "ROCOTO_TEST_PATH not set or directory not found, using system Rocoto"
fi

source "${GW_HOMEgfs}/dev/ush/gw_setup.sh"
source "${GW_HOMEgfs}/dev/ci/platforms/config.${machine}"
cd "${GW_HOMEgfs}/dev/ctests" || exit 1
Expand All @@ -52,7 +64,17 @@
stage: run_tests
script: |
echo "Using build directory ${GW_HOMEgfs} for running CTests"

source "${GW_HOMEgfs}/dev/ush/gw_setup.sh"
# Platform config loaded by gw_setup.sh, now check for custom Rocoto
# TEMPORARY: Use custom Rocoto build with dryrun feature for testing
# TODO: Remove this when upstream Rocoto PR is merged
if [[ -n "${ROCOTO_TEST_PATH:-}" && -d "${ROCOTO_TEST_PATH}" ]]; then
export PATH="${ROCOTO_TEST_PATH}:$PATH"
echo "Using custom Rocoto build at: $(which rocotorun 2>/dev/null || echo 'NOT FOUND')"
else
echo "ROCOTO_TEST_PATH not set or directory not found, using system Rocoto"
fi
cd "${GW_HOMEgfs}/dev/ctests/build" || exit 1
echo "Running CTests with label: ${CTEST_NAME} in ${PWD}"
num=$(ctest -N | grep "Total Tests" | awk "{print \$3}")
Expand Down
13 changes: 13 additions & 0 deletions dev/ci/platforms/config.gaeac6
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,3 +61,16 @@ export max_concurrent_pr=4 # number of concurrent pull requests

# HPC account information - For job submission
export HPC_ACCOUNT=${HPC_ACCOUNT:-drsa-precip3}

#########################################################################
# System-specific settings
# Platform-specific paths and utilities for CI/CD operations
#########################################################################

# Utilities path for custom builds and role account tools
export UTIL_PATH="${GFS_CI_ROOT}/utils"

# Custom Rocoto build path for testing (temporary)
# TODO: Remove when upstream Rocoto PR is merged
# and Rocoto with dry_run support is available in Gaea C6 modules
export ROCOTO_TEST_PATH="${UTIL_PATH}/rocoto_forked/bin"
13 changes: 13 additions & 0 deletions dev/ci/platforms/config.hera
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,3 +67,16 @@ export max_concurrent_pr=4

# HPC account which overides the default account
export HPC_ACCOUNT=${HPC_ACCOUNT:-fv3-cpu}

#########################################################################
# System-specific settings
# Platform-specific paths and utilities for CI/CD operations
#########################################################################

# Utilities path for custom builds and role account tools
# TODO: Populate with actual role account path for this machine
export UTIL_PATH="/path/to/role/account/utils"

# Custom Rocoto build path for testing (temporary)
# TODO: Remove when upstream Rocoto PR is merged
export ROCOTO_TEST_PATH="${UTIL_PATH}/Rocoto/bin"
15 changes: 14 additions & 1 deletion dev/ci/platforms/config.hercules
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,4 +49,17 @@ export GITLAB_BUILDS_DIR=${GFS_CI_ROOT}/BUILDS/GITLAB
export GITLAB_RUNNER_DIR="${GFS_CI_ROOT}/GitLab/Runner"

# HPC account which overides the default account
export HPC_ACCOUNT=${HPC_ACCOUNT:-fv3-cpu}
export HPC_ACCOUNT=${HPC_ACCOUNT:-epic-ps}
Copy link
Contributor

@DavidHuber-NOAA DavidHuber-NOAA Nov 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably want to revert this to fv3-cpu before merging. Fine for testing.


#########################################################################
# System-specific settings
# Platform-specific paths and utilities for CI/CD operations
#########################################################################

# Utilities path for custom builds and role account tools
# TODO: Populate with actual role account path for this machine
export UTIL_PATH="/path/to/role/account/utils"

# Custom Rocoto build path for testing (temporary)
# TODO: Remove when upstream Rocoto PR is merged
export ROCOTO_TEST_PATH="${UTIL_PATH}/Rocoto/bin"
13 changes: 13 additions & 0 deletions dev/ci/platforms/config.orion
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,3 +50,16 @@ export GITLAB_RUNNER_DIR="${GFS_CI_ROOT}/GitLab/Runner"

# HPC account which overides the default account
export HPC_ACCOUNT=${HPC_ACCOUNT:-fv3-cpu}

#########################################################################
# System-specific settings
# Platform-specific paths and utilities for CI/CD operations
#########################################################################

# Utilities path for custom builds and role account tools
# TODO: Populate with actual role account path for this machine
export UTIL_PATH="/path/to/role/account/utils"

# Custom Rocoto build path for testing (temporary)
# TODO: Remove when upstream Rocoto PR is merged
export ROCOTO_TEST_PATH="${UTIL_PATH}/Rocoto/bin"
11 changes: 11 additions & 0 deletions dev/ci/platforms/config.ursa
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,3 +67,14 @@ export max_concurrent_pr=4

# HPC account which overides the default account
export HPC_ACCOUNT=${HPC_ACCOUNT:-fv3-cpu}

#########################################################################
# System-specific settings
# Platform-specific paths and utilities for CI/CD operations
#########################################################################

# Utilities path for custom builds and role account tools
export UTIL_PATH="/scratch3/NCEPDEV/global/role.glopara/utils"

# Custom Rocoto build path for testing (temporary)
export ROCOTO_TEST_PATH="${UTIL_PATH}/src/rocoto-1.3.7-dryrun_update/bin"
14 changes: 14 additions & 0 deletions dev/ci/platforms/config.wcoss2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,20 @@

export GFS_CI_ROOT=/lfs/h2/emc/global/noscrub/globalworkflow.ci/GFS_CI_ROOT
export ICSDIR_ROOT=/lfs/h2/emc/global/noscrub/emc.global/data/ICSDIR
# HPC account which overides the default account
export HPC_ACCOUNT=${HPC_ACCOUNT:-GFS-DEV}

#########################################################################
# System-specific settings
# Platform-specific paths and utilities for CI/CD operations
#########################################################################

# Utilities path for custom builds and role account tools
# TODO: Populate with actual role account path for this machine
export UTIL_PATH="/path/to/role/account/utils"

# Custom Rocoto build path for testing (temporary)
# TODO: Remove when upstream Rocoto PR is merged
export ROCOTO_TEST_PATH="${UTIL_PATH}/Rocoto/bin"
export max_concurrent_cases=5
export max_concurrent_pr=4
19 changes: 16 additions & 3 deletions dev/ctests/scripts/execute.sh.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,26 @@
JOB=${2:?"Job name is required"}
idate=$3

rocotoboot_dryrun=$(command -v rocotoboot_dryrun) || true
# Use standard rocotoboot with --dryrun flag
CASEDIR="@CMAKE_CURRENT_BINARY_DIR@/RUNTESTS/EXPDIR/${TEST_NAME}"
cd "${CASEDIR}"
rm -f ./*.db
rm -f ./jobcard
rm -f ./jobcard

yes | "${rocotoboot_dryrun}" --dryrun -d "${TEST_NAME}.db" -w "${TEST_NAME}.xml" -v 10 -c "${idate}00" -t "${JOB}" 2> jobcard || true
# TEMPORARY: Use custom Rocoto build with dryrun feature for testing
# TODO: Remove this when upstream Rocoto PR is merged
# Load platform-specific configuration first

source "${HOMEgfs}/ush/detect_machine.sh"
source "${HOMEgfs}/dev/ci/platforms/config.${MACHINE_ID}"
if [[ -n "${ROCOTO_TEST_PATH:-}" && -d "${ROCOTO_TEST_PATH}" ]]; then
export PATH="${ROCOTO_TEST_PATH}:$PATH"

Check warning on line 24 in dev/ctests/scripts/execute.sh.in

View workflow job for this annotation

GitHub Actions / differential_shellcheck 

[shellcheck] reported by reviewdog 🐶
Prefer putting braces around variable references even when not strictly required.

Raw Output:
./dev/ctests/scripts/execute.sh.in:24:36: info: Prefer putting braces around variable references even when not strictly required. (ShellCheck.SC2250)
echo "Using custom Rocoto build at: $(which rocotorun 2>/dev/null || echo 'NOT FOUND')"

Check warning on line 25 in dev/ctests/scripts/execute.sh.in

View workflow job for this annotation

GitHub Actions / differential_shellcheck 

[shellcheck] reported by reviewdog 🐶
Consider invoking this command separately to avoid masking its return value (or use '|| true' to ignore).

Raw Output:
./dev/ctests/scripts/execute.sh.in:25:41: info: Consider invoking this command separately to avoid masking its return value (or use '|| true' to ignore). (ShellCheck.SC2312)

Check warning on line 25 in dev/ctests/scripts/execute.sh.in

View workflow job for this annotation

GitHub Actions / differential_shellcheck 

[shellcheck] reported by reviewdog 🐶
'which' is non-standard. Use builtin 'command -v' instead.

Raw Output:
./dev/ctests/scripts/execute.sh.in:25:41: info: 'which' is non-standard. Use builtin 'command -v' instead. (ShellCheck.SC2230)
else
echo "ROCOTO_TEST_PATH not set or directory not found, using system Rocoto"
fi

yes | rocotoboot --dryrun -d "${TEST_NAME}.db" -w "${TEST_NAME}.xml" -v 10 -c "${idate}00" -t "${JOB}" 2> jobcard || true
# Rocoto outputs the job card inside of {{}}. Strip all other output to get just the job card.
sed -n '/^{{/,/^}}/p' < jobcard | sed '1d;$d' > "${TEST_NAME}.sub" || true

Expand Down