Make Python release process more robust

[edmorley]

Bug report

Bug description:

We've encountered a number of issues with the published Python assets (either the source archives or GPG signatures) recently:

• Today's Python 3.10.14 source release signed by wrong GPG key #117053
• Python 3.8.20 and 3.9.20 releases missing GPG signatures #123807
• Python 3.12.5 gzipped source tarball download 404s pythondotorg#2499
• Python 3.10.16 release is missing GPG signatures #127601

Is there any way the automation (or else any manual playbooks shorter term) could be improved to prevent these from occurring?

(I realise GPG signatures will likely be removed starting Python 3.14 due to PEP-761, but they are supported for now)

CPython versions tested on:

3.9, 3.10, 3.11, 3.12, 3.13, 3.14, CPython main branch

Operating systems tested on:

Linux

[Yhg1s]

Well, the 3.12.5 source tarball issue was, if I recall correctly, a transient one because of maintenance on the download server or Fastly configuration. It wasn't caused by, nor could it be solved by, the release automation. The three GPG-related issues are solved by PEP 761 (Deprecating PGP signatures for CPython artifacts). There's certainly ongoing work to improve the release automation as we encounter issues like these (that work is done over in https://github.com/python/release-tools/) but most of the issues we run into now are not visible to the user. For the kinds of issues you mention, I think we've done all we can do at the moment.

[edmorley]

@Yhg1s Hi! Thank you for the reply, however, I feel this issue has been closed a bit prematurely.

I phrased the issue as "release process" in the title, since the issue was intended to cover both the automation and playbooks (as mentioned in the description).

Even if there is not much that can be done via the automation, could the playbooks at least be updated to reduce the chance of issues in the meantime?

I realise GPG signatures will be going away in Python 3.14+ (hence mentioning PEP-761 in the OP). However, GPG signatures are still supported for for Python 3.13 and older (and 3.13 won't be EOL until October 2029), so it would be helpful if we could reduce the regressions in the meantime.

[hugovk]

I'm not sure yet what happened with the 3.10 signatures this time, but yes, we are continuously improving the release automation.

I realise GPG signatures will be going away in Python 3.14+ (hence mentioning PEP-761 in the OP). However, GPG signatures are still supported for for Python 3.13 and older (and 3.13 won't be EOL until October 2029), so it would be helpful if we could reduce the regressions in the meantime.

Indeed, 3.14 alphas already don't have GPG signatures, and 3.9-3.13 are not dropping them either.

However, the full set of 3.9-3.14 do have Sigstore signatures (and have done since October 2022), so I recommend switching over to Sigstore verification for all releases.

[edmorley]

However, the full set of 3.9-3.14 do have Sigstore signatures (and have done since October 2022), so I recommend switching over to Sigstore verification for all releases.

Yeah it's definitely on the list :-)

Though I am expecting it will require some trial and error given using sigstore is not yet the norm. For example, the python.org docs only cover GPG and not sigstore at the moment (is there an issue filed for this?) on:
https://www.python.org/downloads/

(My starting point was going to be the suggestions in https://discuss.python.org/t/pep-761-deprecating-pgp-signatures-for-cpython-artifacts/67180/30)

[hugovk]

However, the full set of 3.9-3.14 do have Sigstore signatures (and have done since October 2022), so I recommend switching over to Sigstore verification for all releases.

Yeah it's definitely on the list :-)

🎉

Though I am expecting it will require some trial and error given using sigstore is not yet the norm. For example, the python.org docs only cover GPG and not sigstore at the moment (is there an issue filed for this?) on: python.org/downloads

If you go to a release page (for example, https://www.python.org/downloads/release/python-3131/) and click "Sigstore" in the table header, it'll take you to a Sigstore guide at https://www.python.org/downloads/metadata/sigstore/

But good point, there's nothing at https://www.python.org/downloads/ so let's add it. I've opened python/pythondotorg#2671.

[edmorley]

Ah thank you - I'd not seen https://www.python.org/downloads/metadata/sigstore/ before. Thank you for filing an issue to improve discovery of it :-)