You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
feat(gencloud): build the ppc64le images under QEMU TCG on an x86_64 runner
There are no POWER runners, so the ppc64le GenericCloud images (gencloud
and gencloud_ext4) can now be built on the x86_64 metal runner with QEMU
full-system emulation (TCG), as a leg of gencloud-build-test.yml (job
build-ppc64le-tcg; input ppc64le, on by default so the scheduled builds
include it).
The Packer sources are the same ones Jenkins runs on a POWER host: boot
ISO, GRUB boot command, kickstart, SSH and the ansible provisioning.
Their KVM-specific settings become variables whose defaults keep the
Jenkins build unchanged, and shared-steps passes the TCG values for
arch=ppc64le:
ppc64le_accelerator none -> tcg
ppc64le_machine_type pseries,accel=kvm,kvm-type=HV -> pseries
ppc64le_cpu_model (QEMU default) -> POWER9
ppc64le_console_log (unset) -> file streamed into the job log
ppc64le_extra_kernel_args (empty) -> console=hvc0
ppc64le_grub_hold false -> true
gencloud_boot_wait_ppc64le 8s -> 3s, ssh_timeout 3600s -> 4h
What the runs taught, and how the leg deals with it:
- The emulator is not Ubuntu 24.04's QEMU 8.2.2: under TCG it
mis-emulates POWER9 vector loads/stores and the EL9/EL10 installer's
Python dies at "Starting installer" (segfaults, corrupted objects;
QEMU issue 1769). Reproduced on a test host; the same guest under
QEMU 10.1.5 works. shared-steps builds a small Fedora 43 image with
qemu-system-ppc-core and installs /usr/local/bin/qemu-system-ppc64-tcg,
a wrapper that runs it with docker --network host (Packer's VNC and
SSH-forward ports stay on the host loopback) and the current directory
and root's Packer ISO cache mounted at their own paths; Packer gets it
as qemu_binary. qemu-img stays the host's.
- SLOF makes the VGA display the primary console when one exists
(Packer needs it for the VNC keyboard), so anaconda would draw on an
uncaptured screen. ppc64le_extra_kernel_args types console=hvc0 into
GRUB; the ppc64le kickstarts' %post removes it again from
/etc/default/grub and the boot entries, since anaconda copies the
installer's console= arguments into the installed configuration, so
the image's kernel command line is the same as from a POWER host.
- The ISO's GRUB menu auto-boots after 5 s, about 8 s after the VM
starts, so typing at a fixed delay is a race. ppc64le_grub_hold starts
the boot command after 3 s with 30 s of once-a-second spacebar presses
GRUB's menu ignores: the first one that reaches the menu stops the
countdown. It then moves from the ISO's default entry ("Test this
media & install", whose rd.live.check hashes the 1.5 GB ISO for the
better part of an hour under emulation) to the plain "Install" entry
before the usual edit sequence; the 9/10/Kitten boot commands become
locals so they can reference the prefix and the extra arguments. SLOF
keeps auto-booting, so the installed system comes up on its own.
shared-steps also installs qemu-utils, tails the guest console during
the build and dumps it on failure (killing a container Packer's kill of
the wrapper would leave behind), and mounts partition 3 (PReP boot,
/boot, /) for the offline validation. There is no in-job boot test:
gencloud-test-steps needs KVM. The AWS CLI is now downloaded for the
runner's own architecture (uname -m) instead of the image's.
Measured on the c7i/c7a metal runners: 9.8 in 35 min, 10.2 in 31 min,
Kitten in 30 min, 8.10 in 46 min. The images were compared with the
Jenkins-built ones with libguestfs: same packages (newer errata only),
same layout and configuration.
|`self-hosted`|`true`| If `false`, skip the aarch64 matrix entirely. |
54
54
|`s390x`|`true`| Build s390x images under QEMU TCG emulation on an x86_64 self-hosted runner (40 to 90 minutes per run; offline validation only). See [s390x under TCG](#s390x-under-tcg-experimental). |
55
+
|`ppc64le`|`true`| Build ppc64le images (`gencloud` and `gencloud_ext4`) under QEMU TCG emulation on an x86_64 self-hosted runner (30 to 50 minutes per image; offline validation only). See [ppc64le under TCG](#ppc64le-under-tcg-experimental). |
55
56
|`store_as_artifact`|`false`| Upload images as workflow artifacts. |
56
57
|`upload_to_s3`|`true`| Upload to S3 in parallel. The test no longer depends on it; when true, the job summary / Mattermost message link the public S3 URL, otherwise they show the filename only. |
57
58
|`notify_mattermost`|`true`| Post per-image build and test notifications to Mattermost. |
@@ -65,8 +66,10 @@ init-data
65
66
|- build-gh-hosted (x86_64 matrix: subtype x variant) -. shared-steps build,
66
67
|- start-self-hosted-runner (fork EC2) | then gencloud-test-steps
67
68
|- build-self-hosted (aarch64 matrix: subtype) -' in-job on the local qcow2
68
-
'- build-s390x-tcg (s390x, default on) - shared-steps build under
69
+
|- build-s390x-tcg (s390x, default on) - shared-steps build under
69
70
TCG, offline validation only
71
+
'- build-ppc64le-tcg (default on; ppc64le matrix: subtype) shared-steps build under
72
+
TCG + offline validation only
70
73
```
71
74
72
75
There is no collect / publish stage: because the test runs in-job, each
@@ -77,6 +80,7 @@ build matrix leg reports its own build+test result directly. The matrix:
77
80
|`build-gh-hosted`| x86_64 |`subtype` in {`gencloud`, `gencloud_ext4`} x `variant` ({`10`,`10-v2`} for AL10/Kitten, else just the major) |
78
81
|`build-self-hosted`| aarch64 |`subtype` in {`gencloud`, `gencloud_ext4`} |
79
82
|`build-s390x-tcg`| s390x |`gencloud` only (no ext4 kickstart for s390x); runs only with `s390x=true`|
83
+
|`build-ppc64le-tcg`| ppc64le (emulated) |`subtype` in {`gencloud`, `gencloud_ext4`}; on by default, skipped with `ppc64le=false`|
80
84
81
85
### Stage composite actions
82
86
@@ -99,6 +103,7 @@ The change is backward-compatible - `gencloud-test.yml` keeps passing
99
103
|`build-gh-hosted`|`c7i.metal-24xl+c7a.metal-48xl+*8gd.metal*`, `image=ubuntu24-full-x64`|`ubuntu-24.04` (GitHub-hosted, has nested `/dev/kvm`) |
|`ppc64le_console_log`| empty |`<workspace>/ppc64le-console.log`, streamed into the job log as `[ppc64le console]` lines |
174
+
|`ppc64le_extra_kernel_args`| empty |`console=hvc0`, typed at the end of the GRUB boot command. SLOF makes the VGA display the primary console when a VGA adapter exists (Packer needs one for the VNC keyboard), so without it anaconda draws its text UI on the uncaptured VGA console and hvc0 only shows a shell banner |
175
+
|`ppc64le_grub_hold`|`false`|`true`: the boot command starts with 30 s of once-a-second keypresses GRUB's menu ignores, then moves up from the ISO's default entry ("Test this media & install", whose `rd.live.check` hashes the 1.5 GB ISO for the better part of an hour under emulation) to the plain "Install" entry before the usual edit sequence. The ISO's GRUB menu auto-boots after only 5 s and appears about 8 s after the VM starts, so typing at one fixed delay is a race (the first runs lost it and installed without the kickstart); the presses stop the countdown as soon as the menu is up. SLOF keeps auto-booting, so the reboot after the install comes up on its own |
176
+
|`gencloud_boot_wait_ppc64le`|`8s`|`3s` (start the keypress hold before GRUB can appear) |
177
+
|`ssh_timeout`|`3600s`|`4h` (the whole emulated install runs before SSH is up) |
178
+
179
+
The emulator is **not** Ubuntu 24.04's QEMU 8.2.2: under TCG it miscompiles
180
+
POWER9 vector loads/stores, and the EL9/EL10 installer's Python crashes at
181
+
"Starting installer" with segfaults or corrupted objects (reproduced on a
182
+
test host; [QEMU issue 1769](https://gitlab.com/qemu-project/qemu/-/issues/1769)).
183
+
shared-steps builds a small Fedora 43 container image with QEMU 10.x and
184
+
installs `/usr/local/bin/qemu-system-ppc64-tcg`, a wrapper that runs
185
+
`qemu-system-ppc64` in that container with host networking (Packer's VNC and
186
+
SSH-forward ports stay on the host loopback) and the workspace and Packer's
187
+
ISO cache mounted at their own paths; Packer gets it as `qemu_binary`. Disk
188
+
images are still created by the host's `qemu-img`.
189
+
190
+
What the job does and does not do:
191
+
192
+
- Offline validation runs as for every arch (release string, `almalinux-release`
193
+
arch, package list from the RPM database; root is partition 3: PReP boot,
0 commit comments