vuln-scan-smoke: revert s3api put-object workaround once tracker.ceph.com#76963 lands

[mmgaggle]

Status

• Workaround landed in PR #88 (merged as 52f8219): the four producer-artefact uploads in hack/e2e/assert-vuln-scan-smoke.sh switched from aws s3 cp to aws s3api put-object --body, forcing a single PUT and sidestepping multipart entirely. vuln-scan-smoke is back out of E2E_SKIP and passing.
• Upstream bug filed: https://tracker.ceph.com/issues/76963. This issue stays open as the tracking issue to revert the workaround once the upstream fix lands.

Real root cause (diagnosed)

The earlier "classic boto3 multipart-completion bug" framing in this issue (from a session handoff) was wrong. The actual failure is in the zgw-posix POSIX backend driver's CreateMultipartUpload handler:

For any object key containing a ., the returned UploadId is the substring after the last . of the key, instead of a proper random token.

Reproducer (against quay.io/dparkes/zgw-posix:latest, ceph 20.3.0-5348-gce1ebc9e tentacle dev):

key=foo.bin          UploadId=bin
key=foo.tar.zst      UploadId=zst
key=baz.txt          UploadId=txt
key=barnodot         UploadId=2~FCIHXlYJxGgyXYWQzqqYbR1XJJrKLMP   ← keys without `.` get a proper random token

Subsequent UploadPart calls against the bogus UploadId 404 because the driver constructs a path .multipart_<KEY>.<UPLOADID> (e.g. .multipart_vulnerability.db.tar.zst.zst) which was never created. Server logs:

s3:init_multipart  bucket=...  http_status=200   UploadId=zst
s3:put_obj ERROR: could not open dir .multipart_vulnerability.db.tar.zst.zst: (2) No such file or directory

Any S3 client that uses multipart by default for >=8 MiB objects (aws-cli, s3cmd, rclone) cannot upload anything with a file extension to RGW running the POSIX backend until upstream lands a fix.

aws-cli secondary bug (informational)

The error surface clients see is a misleading aws-cli traceback:

upload failed: ... argument of type 'NoneType' is not iterable

This is a secondary bug in awscli/customizations/s3errormsg.py:55 — _is_sigv4_error_message does parsed.get('Error', {}).get('Message', ''), but botocore parses zgw-posix's empty <Message></Message> as None (not ""), so '…' in None crashes the error formatter. The underlying S3 error is a 404 NoSuchKey on UploadPart; aws-cli's crash just hides it. Not worth filing separately unless we see other servers emit empty <Message> elements.

Acceptance

• Upstream https://tracker.ceph.com/issues/76963 resolved and a zgw-posix image cut with the fix.
• Revert PR vuln-scan-smoke: producer s3api put-object workaround + busybox-tar unpack fix #88's s3api put-object workaround in hack/e2e/assert-vuln-scan-smoke.sh back to aws s3 cp (exercises the multipart path again).
• e2e green on the revert PR.

References

• Workaround: PR vuln-scan-smoke: producer s3api put-object workaround + busybox-tar unpack fix #88 (merged 52f8219) — 204f5bb swaps s3 cp → s3api put-object; 4c0d8c9 and 012b75f are the in-cluster fixes the workaround uncovered (busybox tar pipe form, PVC for sboms workspace).
• Upstream: https://tracker.ceph.com/issues/76963
• Image used for diagnosis: quay.io/dparkes/zgw-posix:latest (ceph 20.3.0-5348-gce1ebc9e tentacle dev).
• Failing call site (now using put-object): hack/e2e/assert-vuln-scan-smoke.sh ~line 261.