image-download-rhel: new image-download replacement by allisonkarlitskaya · Pull Request #9071 · cockpit-project/bots

allisonkarlitskaya · 2026-05-22T08:56:11Z

This is capable of downloading RHEL images from AWS.

We identify URLs that should be STS-authenticated by using a list (currently 1) of fnmatches on URLs. This list is only consulted if you don't have a normal static key in ~/.config/cockpit-dev/s3-keys/.

Once we've identified that we want to issue a transient key, the process goes like this:

you get a Kerberos ticket with kinit
we use gssapi to get a SPNEGO token for auth.redhat.com
we post a login attempt using the token and hopefully get back a SAML POST binding HTML page which we can extract the SAML assertion from
with the SAML assertion in hand we can use AWS STS to generate a transient token for a particular IAM Role. If you're in the it-cloud-aws-727920394381-cockpit-ci-images-download rover group this will allow you to download RHEL images from AWS
we use the STS token to do the S3 signing operation

This should be more or less transparent to anyone who is in the correct group and logged in with kerberos. There's no need for us to maintain a list of users and tokens anymore and human developers can delete their old s3-keys.

The new downloader is intended as an eventual replacement for image-download. It's a rewrite from the ground up and has a couple of other niceties:

urllib instead of curl (which was getting complicated with all the extra headers)
checksum verification based on filename
improved locking vs. the old downloader: no more poll loop or subtle race conditions with deleted lockfiles
🌈 rainbows!

Some features not implemented:

--state and modification times (which seems to be dead code)
--force

With S3 keys for EU Linode mirror (light mode):

Without any statically-configured S3 keys (dark mode):

s3: various cleanups #9070

gssapi is already installed in the container so this just brings a very thin wrapper for it. This is basically just for type checking: the gssapi import is lazy and won't run in CI but mypy gets unhappy about it. We could ignore it, but it produces different errors about unused ignores (due to a workaround for another issue). See cockpit-project/bots#9071 and pythongssapi/python-gssapi#338

We're slowly moving away from Linode and this is the next step: this mirror is the one that lets anyone download non-RHEL images, but the AWS mirror now fills that role. The eu-central-1 Linode mirror will remain online for now because of the large number of people who have an S3 key issued there which allows them to download the RHEL images. We're going to slowly migrate to #9071 for that, but it's not there yet. Dropping us-east-1 also lets us get rid of our special "public secret" key.

martinpitt

I had a quick look only -- this is rather overwhelming to review in fine detail. It feels like it ought not to be that complex, especially as your initial PoC was like 100 lines? Why did that blow up so much, apart from accidental complexity like cute and the webkit script?

martinpitt · 2026-05-27T06:43:24Z

+        idp_url='https://auth.redhat.com/auth/realms/EmployeeIDP/protocol/saml/clients/itaws',
+        role_arn='arn:aws:iam::727920394381:role/727920394381-cockpit-ci-images-download',
+        provider_arn='arn:aws:iam::727920394381:saml-provider/RedHatInternal',


Can you please add comments/URLs to where these magic values come from?

martinpitt · 2026-05-27T06:47:04Z

@@ -0,0 +1,155 @@
+# Copyright (C) 2026 Red Hat, Inc.


OMG, can this grow some pwd.getpwuid(os.getuid()).pw_name != 'martin'? This is really distracting..

NO_COLOR=1? :D

no seriously though, I figured this might be controversial. See the name of the branch ;)

martinpitt · 2026-05-27T06:49:40Z

@@ -0,0 +1,113 @@
+#!/usr/bin/env python3


This new saml-login isn't referenced anywhere yet? Do we actually need this for uploading images?

Note: we should eliminate the static AWS_KEY_LOGS secret and move to OIDC/JWT based auth with temporary tokens. For that we can hopefully re-use some machinery that this PR introduces already. But it wouldn't start with a Kerberos token. Let's chat about this today.

Indeed I already played around with OIDC a bit when trying to implement this stuff and it would fit well into this file.

That's definitely "next step" though.

+    key, expiration = _parse_sts_response(xml)
+    path = cache_path(role_name)
+    path.parent.mkdir(parents=True, exist_ok=True)
+    with open(os.open(path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600), 'w') as fp:


Venefilyn

Quickly went over but feel like it needs more docs in general and a bit concerned about the storing of credentials. How long are the credentials valid for?

+            return False
+        url, key, total = result
+
+        with open(os.open(temp, os.O_CREAT | os.O_RDWR, 0o666), 'r+b') as fp:


+            return False
+        url, key, total = result
+
+        with open(os.open(temp, os.O_CREAT | os.O_RDWR, 0o666), 'r+b') as fp:


allisonkarlitskaya · 2026-06-08T11:02:29Z

/image-refresh rhel-8-10

allisonkarlitskaya · 2026-06-08T11:02:35Z

/image-refresh arch

cockpituous · 2026-06-08T11:02:50Z

Task scheduled: issue-9071 image-refresh/rhel-8-10

Testing Farm link: https://artifacts.osci.redhat.com/testing-farm/58f156c1-e1ae-40f7-b17b-a75bbd90c7e1

Job JSON

{
  "repo": "cockpit-project/bots",
  "sha": "4901e1caf922b24c8eaae8b1fc2c4739a58cb78a",
  "pull": 9071,
  "slug": "image-refresh-rhel-8-10-4901e1ca-20260608-110242",
  "context": "image-refresh/rhel-8-10",
  "command": [
    "./image-refresh",
    "--verbose",
    "--issue=9071",
    "rhel-8-10"
  ],
  "secrets": [
    "github-token",
    "image-upload"
  ]
}

cockpituous · 2026-06-08T11:02:54Z

Task scheduled: issue-9071 image-refresh/arch

Testing Farm link: https://artifacts.osci.redhat.com/testing-farm/18db7dd5-bf04-45b7-bb4d-f0257c576174

Job JSON

{
  "repo": "cockpit-project/bots",
  "sha": "4901e1caf922b24c8eaae8b1fc2c4739a58cb78a",
  "pull": 9071,
  "slug": "image-refresh-arch-4901e1ca-20260608-110246",
  "context": "image-refresh/arch",
  "command": [
    "./image-refresh",
    "--verbose",
    "--issue=9071",
    "arch"
  ],
  "secrets": [
    "github-token",
    "image-upload"
  ]
}

cockpituous · 2026-06-08T11:09:04Z

image-refresh rhel-8-10 started on 2fba6b109fbc. Log: https://cockpit-ci-logs.s3.us-east-1.amazonaws.com/image-refresh-rhel-8-10-4901e1ca-20260608-110242/log.html

cockpituous · 2026-06-08T11:11:31Z

image-refresh arch started on 9f771127b2db. Log: https://cockpit-ci-logs.s3.us-east-1.amazonaws.com/image-refresh-arch-4901e1ca-20260608-110246/log.html

cockpituous · 2026-06-08T11:21:53Z

image-refresh arch succeeded: https://github.com/cockpit-project/bots/commits/image-refresh-arch-20260608-112150

cockpituous · 2026-06-08T11:37:40Z

image-refresh rhel-8-10 succeeded: https://github.com/cockpit-project/bots/commits/image-refresh-rhel-8-10-20260608-113737

Removed: yajl (2.1.0-7) Added: hwloc (2.13.0-1) libblake3 (1.8.4-1) libntfs-3g (2026.2.25-1) libtool (2.6.1-1) onetbb (2023.0.0-1) Changed: bash (5.3.9-1 -> 5.3.12-1) capstone (5.0.7-2 -> 5.0.9-1) cockpit (362-1 -> 363.2-1) crun (1.27.1-1 -> 1.28-1) dnsmasq (2.92.rel2-2 -> 2.93-1) hwdata (0.407-1 -> 0.408-1) iana-etc (20260511-1 -> 20260530-1) kbd (2.9.0-1 -> 2.10.0-1) libdrm (2.4.133-1 -> 2.4.134-1) libnet (2:1.3-1 -> 2:1.3-2) libnghttp3 (1.15.0-1 -> 1.16.0-1) libngtcp2 (1.22.1-1 -> 1.23.0-1) libvirt (1:12.3.0-1 -> 1:12.4.0-1) libvirt-python (1:12.3.0-1 -> 1:12.4.0-1) libxkbcommon (1.13.1-1 -> 1.13.2-1) linux (7.0.10.arch1-1 -> 7.0.11.arch1-1) linux-api-headers (6.19-1 -> 7.0-1) llvm-libs (22.1.5-1 -> 22.1.6-1) mesa (1:26.1.1-2 -> 1:26.1.2-1) mkinitcpio (41-3 -> 41-4) ntfs-3g (2022.10.3-2 -> 2026.2.25-1) passt (2026_05_07.1afd4ed-1 -> 2026_05_26.038c51e-1) pcp (7.1.4-1 -> 7.1.5-1) pcsclite (2.4.1-1 -> 2.5.0-1) perf (7.0.9-1 -> 7.0.10-1) protobuf (34.1-1 -> 35.0-1) protobuf-c (1.5.2-9 -> 1.5.2-10) python-debian (1.1.0-2 -> 1.1.1-1) python-idna (3.16-1 -> 3.18-1) python-protobuf (34.1-1 -> 35.0-1) qemu-audio-spice (11.0.0-1 -> 11.0.1-1) qemu-base (11.0.0-1 -> 11.0.1-1) qemu-block-curl (11.0.0-1 -> 11.0.1-1) qemu-chardev-spice (11.0.0-1 -> 11.0.1-1) qemu-common (11.0.0-1 -> 11.0.1-1) qemu-guest-agent (11.0.0-1 -> 11.0.1-1) qemu-hw-usb-host (11.0.0-1 -> 11.0.1-1) qemu-hw-usb-redirect (11.0.0-1 -> 11.0.1-1) qemu-img (11.0.0-1 -> 11.0.1-1) qemu-system-x86 (11.0.0-1 -> 11.0.1-1) qemu-system-x86-firmware (11.0.0-1 -> 11.0.1-1) qemu-tools (11.0.0-1 -> 11.0.1-1) qemu-ui-opengl (11.0.0-1 -> 11.0.1-1) qemu-ui-spice-app (11.0.0-1 -> 11.0.1-1) qemu-ui-spice-core (11.0.0-1 -> 11.0.1-1) rpcbind (1.2.8-1 -> 1.2.9-1) sqlite (3.53.1-1 -> 3.53.2-1) systemd (260.1-2 -> 260.2-2) systemd-libs (260.1-2 -> 260.2-2) systemd-sysvcompat (260.1-2 -> 260.2-2) vim (9.2.0511-1 -> 9.2.0600-1) vim-runtime (9.2.0511-1 -> 9.2.0600-1) which (2.23-1 -> 2.25-1) Closes #9071

Removed: Added: Changed: Closes #9071

Venefilyn

Overall LGTM if it works.

Venefilyn · 2026-06-08T14:24:03Z

+# Red Hat employee IdP (SAML via Kerberos) → AWS IAM role for cockpit-ci-images
+# Rover group: https://rover.redhat.com/groups/group/it-cloud-aws-727920394381-cockpit-ci-images-download
+TARGETS = {
+    'https://cockpit-ci-images*.s3.*.amazonaws.com/rhel-*': SAMLTarget(
+        idp_url='https://auth.redhat.com/auth/realms/EmployeeIDP/protocol/saml/clients/itaws',
+        provider_arn='arn:aws:iam::727920394381:saml-provider/RedHatInternal',
+        role_arn='arn:aws:iam::727920394381:role/727920394381-cockpit-ci-images-download',
+    ),
+}


How did you find the arn?

I created it... this could probably use a writeup in the cockpituous repo.

This is capable of downloading RHEL images from AWS. We identify URLs that should be STS-authenticated by using a list (currently 1) of fnmatches on URLs. This list is only consulted if you don't have a normal static key in ~/.config/cockpit-dev/s3-keys/. Once we've identified that we want to issue a transient key, the process goes like this: - you get a Kerberos ticket with kinit - we use gssapi to get a SPNEGO token for auth.redhat.com - we post a login attempt using the token and hopefully get back a SAML POST binding HTML page which we can extract the SAML assertion from - with the SAML assertion in hand we can use AWS STS to generate a transient token for a particular IAM Role. If you're in the it-cloud-aws-727920394381-cockpit-ci-images-download rover group this will allow you to download RHEL images from AWS - we use the STS token to do the S3 signing operation This should be more or less transparent to anyone who is in the correct group and logged in with kerberos. There's no need for us to maintain a list of users and tokens anymore and human developers can delete their old s3-keys. The new downloader is intended as an eventual replacement for image-download. It's a rewrite from the ground up and has a couple of other niceties: - urllib instead of curl (which was getting complicated with all the extra headers) - checksum verification based on filename - improved locking vs. the old downloader: no more poll loop or subtle race conditions with deleted lockfiles - 🌈 rainbows! Some features not implemented: - `--state` and modification times (which seems to be dead code) - `--force` For users who don't want to use Kerberos (which requires connecting to VPN) there's also a saml-login script which will do a SAML interaction in a simple WebKit view and save the credential to the cache (for a validity of one hour).

allisonkarlitskaya added the blocked label May 22, 2026

github-advanced-security AI found potential problems May 22, 2026

View reviewed changes

Comment thread image-download-rhel Fixed

Comment thread lib/cute.py Fixed

Comment thread lib/cute.py Fixed

Comment thread lib/cute.py Fixed

Comment thread image-download-rhel Fixed

Comment thread lib/cute.py Fixed

Comment thread lib/gssapi_saml_sts.py Fixed

Comment thread lib/locking.py Fixed

allisonkarlitskaya requested review from jelly and martinpitt May 22, 2026 09:08

allisonkarlitskaya force-pushed the unicorn-barf branch from a8d4d12 to 3fe02b7 Compare May 26, 2026 07:22

github-advanced-security AI found potential problems May 26, 2026

View reviewed changes

Comment thread lib/gssapi_saml_sts.py Fixed

Comment thread saml-login Fixed

allisonkarlitskaya force-pushed the unicorn-barf branch from 3fe02b7 to c1edcd3 Compare May 26, 2026 08:01

allisonkarlitskaya mentioned this pull request May 26, 2026

lib/stores.py: retire us-east-1.linodeobjects.com #9078

Merged

allisonkarlitskaya force-pushed the unicorn-barf branch 2 times, most recently from ebc362d to 700e9e4 Compare May 26, 2026 09:06

allisonkarlitskaya mentioned this pull request May 26, 2026

tasks: add python3-gssapi cockpit-project/cockpituous#704

Merged

1 task

allisonkarlitskaya force-pushed the unicorn-barf branch 2 times, most recently from dc8dbdf to e54f025 Compare May 26, 2026 09:45

allisonkarlitskaya requested a review from Venefilyn May 26, 2026 11:32

allisonkarlitskaya force-pushed the unicorn-barf branch from e54f025 to 440478f Compare May 26, 2026 19:53

martinpitt reviewed May 27, 2026

View reviewed changes

allisonkarlitskaya force-pushed the unicorn-barf branch from 440478f to 36f18cc Compare May 27, 2026 07:24

github-advanced-security AI found potential problems May 27, 2026

View reviewed changes

Comment thread lib/gssapi_saml_sts.py

key, expiration = _parse_sts_response(xml)

path = cache_path(role_name)

path.parent.mkdir(parents=True, exist_ok=True)

with open(os.open(path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600), 'w') as fp:

Venefilyn reviewed May 27, 2026

View reviewed changes

allisonkarlitskaya removed the blocked label Jun 2, 2026

allisonkarlitskaya force-pushed the unicorn-barf branch 3 times, most recently from 85f64c2 to 4901e1c Compare June 8, 2026 09:47

github-advanced-security AI found potential problems Jun 8, 2026

View reviewed changes

Comment thread image-download

return False

url, key, total = result

with open(os.open(temp, os.O_CREAT | os.O_RDWR, 0o666), 'r+b') as fp:

Comment thread image-download

return False

url, key, total = result

with open(os.open(temp, os.O_CREAT | os.O_RDWR, 0o666), 'r+b') as fp:

allisonkarlitskaya pushed a commit that referenced this pull request Jun 8, 2026

images: Update rhel-8-10 image

6e6112c

Removed: Added: Changed: Closes #9071

Venefilyn previously approved these changes Jun 8, 2026

View reviewed changes

allisonkarlitskaya dismissed Venefilyn’s stale review via 70c7436 June 8, 2026 15:12

allisonkarlitskaya force-pushed the unicorn-barf branch from b867b7e to 70c7436 Compare June 8, 2026 15:12

allisonkarlitskaya force-pushed the unicorn-barf branch from 70c7436 to 63c2d25 Compare June 8, 2026 15:55

allisonkarlitskaya requested a review from Venefilyn June 8, 2026 16:02

Venefilyn approved these changes Jun 10, 2026

View reviewed changes

allisonkarlitskaya merged commit e635e8d into main Jun 10, 2026
5 of 6 checks passed

allisonkarlitskaya deleted the unicorn-barf branch June 10, 2026 16:51

allisonkarlitskaya mentioned this pull request Jun 10, 2026

hihi #9177

Closed

Conversation

allisonkarlitskaya commented May 22, 2026 • edited by martinpitt Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

martinpitt left a comment

Choose a reason for hiding this comment

Uh oh!

martinpitt May 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

martinpitt May 27, 2026

Choose a reason for hiding this comment

Uh oh!

allisonkarlitskaya May 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

martinpitt May 27, 2026

Choose a reason for hiding this comment

Uh oh!

allisonkarlitskaya May 27, 2026

Choose a reason for hiding this comment

Uh oh!

Venefilyn left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

allisonkarlitskaya commented Jun 8, 2026

Uh oh!

allisonkarlitskaya commented Jun 8, 2026

Uh oh!

cockpituous commented Jun 8, 2026

Uh oh!

cockpituous commented Jun 8, 2026

Uh oh!

cockpituous commented Jun 8, 2026

Uh oh!

cockpituous commented Jun 8, 2026

Uh oh!

cockpituous commented Jun 8, 2026

Uh oh!

cockpituous commented Jun 8, 2026

Uh oh!

Venefilyn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Venefilyn Jun 8, 2026

Choose a reason for hiding this comment

Uh oh!

allisonkarlitskaya Jun 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

allisonkarlitskaya commented May 22, 2026 •

edited by martinpitt

Loading

Venefilyn left a comment •

edited

Loading