ci: run nightly e2e against release-1.10 and other test branches

[NiallTwomey2]

rhdh-must-gather

Description

Nightly E2E previously only ran against main within the rhdh-must-gather repo.
Updates .github/workflows/nightly.yaml are made so it also tests release-1.10, similar to what's existing in rhdh-operator nightly.

Nightly tests current tests only main and release-1.10.
Those are the only release branches in this repository.
Note: nightly tests are only run on the release branches that are defined under the matrix of the e2e-test job and the e2e-tests-heap-dumps job within .github/workflows/nightly.yaml.

Scheduled workflows only run from the default branch, so this uses a job matrix on main rather than a on.push branch list:

TARGET_BRANCH plus image tag latest (main) or latest-1.10 (release-1.10), matching .github/workflows/container-build.yaml

Further release lines can be added to the matrix (in side the two defined e2e jobs of nightly.yaml later when those branches exist on this repo.

Which issue(s) does this PR fix or relate to

• Fixes RHDHBUGS-3560

PR acceptance criteria

• Tests (nightly workflow matrix; verified via workflow_dispatch)
• Documentation

Manual run: https://github.com/redhat-developer/rhdh-must-gather/actions/runs/34117117252

[rm3l]

@NiallTwomey2 Could you please update your branch and fix the conflicts?

[rm3l]

/agentic_review

[rhdh-qodo-merge]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 🔗 Cross-repo conflicts (2) 📜 Skill insights (0)

1. Nightly fails daily on a missing branch ✗ Dismissed

Description

Both matrices include release-2.1, which the PR description says does not exist in this repo, so
actions/checkout with ref: release-2.1 fails. That is four failing jobs every night (one
standard, three heap-dump), which marks the whole workflow red; even once the branch exists, the
expected latest-2.1 image tag is only published after that branch receives a push.

Code

.github/workflows/nightly.yaml[26]

+          - release-2.1

Relevance

●●● Strong

The PR explicitly says release-2.1 does not exist, making checkout and image resolution fail
nightly.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The PR description states main and release-1.10 are the only release branches. container-build.yaml
only publishes the latest-<version> image tag when a release branch is pushed.

.github/workflows/container-build.yaml[79-83]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The matrix references a release-2.1 branch that does not exist, so checkout fails every night.

## Fix Focus Areas
- .github/workflows/nightly.yaml[23-26]
- .github/workflows/nightly.yaml[95-98]

## Recommended Fix
Remove `release-2.1` from both matrices; add it back once the branch exists and its image tag has been published.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Heap-dump auto runs test main's image on releases ✓ Resolved

Description

In the auto heap-dump branch, the make test-e2e line ends with a backslash followed by a space,
so the shell does not join the intended variable assignments to the command. For every branch in
that matrix entry, Make receives a stray argument and cannot reach the E2E script; TARGET_BRANCH,
IMAGE_TAG, and HELM_TIMEOUT run separately as assignments.

Code

.github/workflows/nightly.yaml[R135-137]

+            make test-e2e LOCAL=false WITH_HEAP_DUMPS=true \ 
+              TARGET_BRANCH="${{ env.TARGET_BRANCH }}" \
+              IMAGE_TAG="${{ env.IMAGE_TAG }}" \

Relevance

●●● Strong

Whitespace after the continuation backslash breaks the shell command, a deterministic workflow
failure.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The workflow’s auto command has \  at the end of line 135, with the intended Make arguments on
subsequent lines, while the inspector/sigusr2 branch at lines 140–144 has no trailing space after
its continuation backslash. The Makefile’s test-e2e recipe launches the E2E script, so the
malformed Make invocation cannot reach it; the Makefile and script defaults of IMAGE_TAG=latest
and TARGET_BRANCH=main also show why the omitted assignments cannot select the matrix branch and
image.

Makefile[9-10]
tests/e2e/run-e2e-tests.sh[90-90]
.github/workflows/nightly.yaml[132-145]
Makefile[50-67]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The `auto` heap-dump command has a space after its line-continuation backslash. This separates the intended variable assignments from `make test-e2e` and prevents the E2E script from running.

## Fix Focus Areas
- .github/workflows/nightly.yaml[135-138]

## Recommended Fix
Delete the space after the backslash on line 135 so the line ends exactly with `\` and the following variable assignments remain part of the `make test-e2e` command.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Standard nightly tests run in local mode ✓ Resolved

Description

The run: > block preserves newlines before the more-indented arguments, so make test-e2e runs
without LOCAL=false, TARGET_BRANCH, IMAGE_TAG, or HELM_TIMEOUT; the following lines are
separate shell assignments. Both branches of the standard E2E matrix therefore use the Makefile’s
local-mode default instead of testing their selected container images, and a successful local test
can make the job appear to pass.

Code

.github/workflows/nightly.yaml[R54-58]

+        run: >
+          make test-e2e
+            LOCAL=false
+            TARGET_BRANCH=${{ env.TARGET_BRANCH }}
+            IMAGE_TAG=${{ env.IMAGE_TAG }}

Relevance

●●● Strong

Clear YAML indentation bug prevents required arguments from reaching make; this undermines the PR’s
stated nightly image-testing intent.

PR-#223
PR-#385

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The added command has extra indentation on every argument line, which prevents YAML’s folded scalar
from joining those lines to make test-e2e. The Makefile defaults LOCAL to true and selects the
--local recipe unless LOCAL=false is passed; the image and target-branch arguments exist only in
its non-local recipe.

.github/workflows/nightly.yaml[53-59]
Makefile[130-153]
tests/e2e/run-e2e-tests.sh[159-175]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The standard nightly job’s folded YAML scalar retains newlines before its more-indented arguments, causing `make test-e2e` to run without the intended overrides.
## Fix Focus Areas
- .github/workflows/nightly.yaml[54-59]
## Recommended Fix
Use a literal `run: |` block with shell line-continuation backslashes, as in the heap-dump job, or put the entire make invocation on one line. Ensure all four overrides are arguments to `make test-e2e`.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

4. Release tests cannot find the chart 🔗 Cross-repo conflict ≡ Correctness ⭐ New

Description

The new release-1.10 matrix entry selects that branch of rhdh-chart, but the checked-in
run-e2e-tests.sh sets the chart path to charts/rhdh while that repository’s release-1.10
application chart is at charts/backstage. If the release-1.10 checkout uses this E2E script, `helm
dependency build` fails before either the standard or heap-dump job can run its tests.

Code

.github/workflows/nightly.yaml[21]

+          - release-1.10

Relevance

●●● Strong

Branch-specific E2E chart paths are a recognized correctness issue; prior migration explicitly
addressed target chart compatibility.

PR-#381

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The workflow newly selects release-1.10; the checked-in E2E script clones the selected chart branch
but builds a different chart directory from the one present in the pinned release-1.10 chart
repository.

rhdh-must-gather -> rhdh-chart
.github/workflows/nightly.yaml[19-27]
tests/e2e/run-e2e-tests.sh[246-251]
External repo: redhat-developer/rhdh-chart, charts/backstage/Chart.yaml [39-49]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The release-1.10 chart lives at `charts/backstage`, but the checked-in E2E script builds dependencies from `charts/rhdh`.
## Fix Focus Areas
- .github/workflows/nightly.yaml[19-22]
- tests/e2e/run-e2e-tests.sh[246-251]
## Recommended Fix
Confirm which E2E script the release-1.10 checkout runs, then make its chart-path selection match the selected chart branch before enabling these matrix jobs.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗

---

5. Manual target-branch input is silently ignored ✓ Resolved

Description

The run commands now pass only the matrix-derived env.TARGET_BRANCH and drop the previous
inputs.target-branch fallback, while the workflow_dispatch input target-branch is still
declared. A manual run that sets this input still tests the matrix branches, with no warning.

Code

.github/workflows/nightly.yaml[R59-62]

+        run: |
+          make test-e2e LOCAL=false \
+            TARGET_BRANCH="${{ env.TARGET_BRANCH }}" \
+            IMAGE_TAG="${{ env.IMAGE_TAG }}" \

Relevance

●●● Strong

The declared manual input is no longer consumed, so workflow_dispatch silently tests matrix branches
instead.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The input is still declared under workflow_dispatch, but no step reads it any more.

.github/workflows/nightly.yaml[6-10]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The workflow_dispatch input target-branch is declared but no longer used by the test commands.

## Fix Focus Areas
- .github/workflows/nightly.yaml[6-10]
- .github/workflows/nightly.yaml[59-63]
- .github/workflows/nightly.yaml[140-144]

## Recommended Fix
Either remove the `target-branch` input, or use it as an override: `TARGET_BRANCH="${{ inputs.target-branch || env.TARGET_BRANCH }}"`.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

6. Release-2.1 tests may lack dependencies 🔗 Cross-repo conflict ≡ Correctness

Description

The new release-2.1 matrix entry passes that branch to the E2E setup, but neither rhdh-chart nor
rhdh-operator includes it in its nightly branch list. If either repository lacks the branch, the
chart clone or operator manifest fetch fails before the job can test that release.

Code

.github/workflows/nightly.yaml[26]

+          - release-2.1

Relevance

●● Moderate

The dependent repositories may lack release-2.1, but external branch availability is not
conclusively demonstrated.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The PR adds release-2.1 and forwards the selected branch to E2E. The E2E script uses it to clone the
chart repository and fetch the operator manifest, while both related repositories’ nightly workflows
list other branches but not release-2.1.

rhdh-must-gather -> rhdh-chart
rhdh-must-gather -> rhdh-operator
.github/workflows/nightly.yaml[23-26]
.github/workflows/nightly.yaml[95-98]
tests/e2e/run-e2e-tests.sh[221-245]
tests/e2e/run-e2e-tests.sh[402-415]
External repo: redhat-developer/rhdh-chart, .github/workflows/nightly.yaml [20-38]
External repo: redhat-developer/rhdh-operator, .github/workflows/nightly.yaml [13-17]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The new release-2.1 nightly jobs require matching rhdh-chart and rhdh-operator branches, but neither repository includes that branch in its nightly branch list.
## Fix Focus Areas
- .github/workflows/nightly.yaml[26-26]
- .github/workflows/nightly.yaml[98-98]
- tests/e2e/run-e2e-tests.sh[221-245]
- tests/e2e/run-e2e-tests.sh[402-415]
## Recommended Fix
Confirm that release-2.1 exists and provides the required chart and operator manifest in both repositories before scheduling these jobs. Otherwise remove the matrix entries until those branches are available, or explicitly select supported chart and operator branches for the tests.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗

---

Context sources

⚠️ Tickets: not configured — ticket URL found in PR but could not be fetched — check ticket provider credentials

✅ Compliance rules (platform): 4 rules

✅ Cross-repo context — repo relationships

  Explored: repo: redhat-developer/rhdh-chart (branch: release-1.10, sha: df3182b0) — View relationship

  Explored: repo: redhat-developer/rhdh-operator (branch: release-1.10, sha: ceeaebf9) — View relationship

Review mode: Auto: ⚖️ Balanced: Behavioral CI matrix changes affect multiple nightly test paths.

Tip of the day

💡 Did you know, you can turn on the rule miner and Qodo learns your standards from review history

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Previous reviews

Review updated until commit 626c53a

Results up to commit 9485377 ⚖️ Balanced

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Standard nightly tests run in local mode ✓ Resolved

Description

The run: > block preserves newlines before the more-indented arguments, so make test-e2e runs
without LOCAL=false, TARGET_BRANCH, IMAGE_TAG, or HELM_TIMEOUT; the following lines are
separate shell assignments. Both branches of the standard E2E matrix therefore use the Makefile’s
local-mode default instead of testing their selected container images, and a successful local test
can make the job appear to pass.

Code

.github/workflows/nightly.yaml[R54-58]

+        run: >
+          make test-e2e
+            LOCAL=false
+            TARGET_BRANCH=${{ env.TARGET_BRANCH }}
+            IMAGE_TAG=${{ env.IMAGE_TAG }}

Relevance

●●● Strong

Clear YAML indentation bug prevents required arguments from reaching make; this undermines the PR’s
stated nightly image-testing intent.

PR-#223
PR-#385

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The added command has extra indentation on every argument line, which prevents YAML’s folded scalar
from joining those lines to make test-e2e. The Makefile defaults LOCAL to true and selects the
--local recipe unless LOCAL=false is passed; the image and target-branch arguments exist only in
its non-local recipe.

.github/workflows/nightly.yaml[53-59]
Makefile[130-153]
tests/e2e/run-e2e-tests.sh[159-175]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The standard nightly job’s folded YAML scalar retains newlines before its more-indented arguments, causing `make test-e2e` to run without the intended overrides.
## Fix Focus Areas
- .github/workflows/nightly.yaml[54-59]
## Recommended Fix
Use a literal `run: |` block with shell line-continuation backslashes, as in the heap-dump job, or put the entire make invocation on one line. Ensure all four overrides are arguments to `make test-e2e`.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Results up to commit ca75d84 🧠 Deep

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (1) 📜 Skill insights (0)

1. Nightly fails daily on a missing branch ✗ Dismissed

Description

Both matrices include release-2.1, which the PR description says does not exist in this repo, so
actions/checkout with ref: release-2.1 fails. That is four failing jobs every night (one
standard, three heap-dump), which marks the whole workflow red; even once the branch exists, the
expected latest-2.1 image tag is only published after that branch receives a push.

Code

.github/workflows/nightly.yaml[26]

+          - release-2.1

Relevance

●●● Strong

The PR explicitly says release-2.1 does not exist, making checkout and image resolution fail
nightly.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The PR description states main and release-1.10 are the only release branches. container-build.yaml
only publishes the latest-<version> image tag when a release branch is pushed.

.github/workflows/container-build.yaml[79-83]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The matrix references a release-2.1 branch that does not exist, so checkout fails every night.

## Fix Focus Areas
- .github/workflows/nightly.yaml[23-26]
- .github/workflows/nightly.yaml[95-98]

## Recommended Fix
Remove `release-2.1` from both matrices; add it back once the branch exists and its image tag has been published.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Heap-dump auto runs test main's image on releases ✓ Resolved

Description

In the auto heap-dump branch, the make test-e2e line ends with a backslash followed by a space,
so the shell does not join the intended variable assignments to the command. For every branch in
that matrix entry, Make receives a stray argument and cannot reach the E2E script; TARGET_BRANCH,
IMAGE_TAG, and HELM_TIMEOUT run separately as assignments.

Code

.github/workflows/nightly.yaml[R135-137]

+            make test-e2e LOCAL=false WITH_HEAP_DUMPS=true \ 
+              TARGET_BRANCH="${{ env.TARGET_BRANCH }}" \
+              IMAGE_TAG="${{ env.IMAGE_TAG }}" \

Relevance

●●● Strong

Whitespace after the continuation backslash breaks the shell command, a deterministic workflow
failure.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The workflow’s auto command has \  at the end of line 135, with the intended Make arguments on
subsequent lines, while the inspector/sigusr2 branch at lines 140–144 has no trailing space after
its continuation backslash. The Makefile’s test-e2e recipe launches the E2E script, so the
malformed Make invocation cannot reach it; the Makefile and script defaults of IMAGE_TAG=latest
and TARGET_BRANCH=main also show why the omitted assignments cannot select the matrix branch and
image.

Makefile[9-10]
tests/e2e/run-e2e-tests.sh[90-90]
.github/workflows/nightly.yaml[132-145]
Makefile[50-67]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The `auto` heap-dump command has a space after its line-continuation backslash. This separates the intended variable assignments from `make test-e2e` and prevents the E2E script from running.

## Fix Focus Areas
- .github/workflows/nightly.yaml[135-138]

## Recommended Fix
Delete the space after the backslash on line 135 so the line ends exactly with `\` and the following variable assignments remain part of the `make test-e2e` command.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Manual target-branch input is silently ignored ✓ Resolved

Description

The run commands now pass only the matrix-derived env.TARGET_BRANCH and drop the previous
inputs.target-branch fallback, while the workflow_dispatch input target-branch is still
declared. A manual run that sets this input still tests the matrix branches, with no warning.

Code

.github/workflows/nightly.yaml[R59-62]

+        run: |
+          make test-e2e LOCAL=false \
+            TARGET_BRANCH="${{ env.TARGET_BRANCH }}" \
+            IMAGE_TAG="${{ env.IMAGE_TAG }}" \

Relevance

●●● Strong

The declared manual input is no longer consumed, so workflow_dispatch silently tests matrix branches
instead.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The input is still declared under workflow_dispatch, but no step reads it any more.

.github/workflows/nightly.yaml[6-10]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The workflow_dispatch input target-branch is declared but no longer used by the test commands.

## Fix Focus Areas
- .github/workflows/nightly.yaml[6-10]
- .github/workflows/nightly.yaml[59-63]
- .github/workflows/nightly.yaml[140-144]

## Recommended Fix
Either remove the `target-branch` input, or use it as an override: `TARGET_BRANCH="${{ inputs.target-branch || env.TARGET_BRANCH }}"`.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

4. Release-2.1 tests may lack dependencies 🔗 Cross-repo conflict ≡ Correctness

Description

The new release-2.1 matrix entry passes that branch to the E2E setup, but neither rhdh-chart nor
rhdh-operator includes it in its nightly branch list. If either repository lacks the branch, the
chart clone or operator manifest fetch fails before the job can test that release.

Code

.github/workflows/nightly.yaml[26]

+          - release-2.1

Relevance

●● Moderate

The dependent repositories may lack release-2.1, but external branch availability is not
conclusively demonstrated.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The PR adds release-2.1 and forwards the selected branch to E2E. The E2E script uses it to clone the
chart repository and fetch the operator manifest, while both related repositories’ nightly workflows
list other branches but not release-2.1.

rhdh-must-gather -> rhdh-chart
rhdh-must-gather -> rhdh-operator
.github/workflows/nightly.yaml[23-26]
.github/workflows/nightly.yaml[95-98]
tests/e2e/run-e2e-tests.sh[221-245]
tests/e2e/run-e2e-tests.sh[402-415]
External repo: redhat-developer/rhdh-chart, .github/workflows/nightly.yaml [20-38]
External repo: redhat-developer/rhdh-operator, .github/workflows/nightly.yaml [13-17]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The new release-2.1 nightly jobs require matching rhdh-chart and rhdh-operator branches, but neither repository includes that branch in its nightly branch list.
## Fix Focus Areas
- .github/workflows/nightly.yaml[26-26]
- .github/workflows/nightly.yaml[98-98]
- tests/e2e/run-e2e-tests.sh[221-245]
- tests/e2e/run-e2e-tests.sh[402-415]
## Recommended Fix
Confirm that release-2.1 exists and provides the required chart and operator manifest in both repositories before scheduling these jobs. Otherwise remove the matrix entries until those branches are available, or explicitly select supported chart and operator branches for the tests.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗

---

[rm3l]

/agentic_review

[rhdh-qodo-merge]

Code review by qodo was updated up to the latest commit ca75d84

[rm3l]

/agentic_review

[rhdh-qodo-merge]

4. Release tests cannot find the chart 🔗 Cross-repo conflict ≡ Correctness

The new release-1.10 matrix entry selects that branch of rhdh-chart, but the checked-in
run-e2e-tests.sh sets the chart path to charts/rhdh while that repository’s release-1.10
application chart is at charts/backstage. If the release-1.10 checkout uses this E2E script, `helm
dependency build` fails before either the standard or heap-dump job can run its tests.

Agent Prompt

## Issue description
The release-1.10 chart lives at `charts/backstage`, but the checked-in E2E script builds dependencies from `charts/rhdh`.
## Fix Focus Areas
- .github/workflows/nightly.yaml[19-22]
- tests/e2e/run-e2e-tests.sh[246-251]
## Recommended Fix
Confirm which E2E script the release-1.10 checkout runs, then make its chart-path selection match the selected chart branch before enabling these matrix jobs.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗

[rhdh-qodo-merge]

Code review by qodo was updated up to the latest commit 626c53a