chore(docker): ca-certificates + h2o binary in benchmarks image - #2251
Conversation
Additive changes needed for S3-backed benchmark runs and image parity with the h2o binary that shipped in apache#2230. - ballista-scheduler + ballista-executor: install `ca-certificates`. Without them, TLS handshakes for S3 (via object_store) and STS (AssumeRoleWithWebIdentity for IRSA-authenticated S3 reads) fail with `InvalidCertificate(UnknownIssuer)`. - ballista-benchmarks: same ca-certificates rationale, plus `COPY target/release/h2o /root/h2o` so the image bundles both tpch and h2o binaries. The h2o binary was added in apache#2230 but never wired into the Dockerfile. - .dockerignore: whitelist `target/release{,-nonlto}/h2o` so the benchmarks build context sees the h2o binary alongside tpch/etc. Whitelist is where the other binaries already live; without it, docker build errors on the new COPY line. Andy's TPC-H bench workflow is unaffected — every change is additive. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
milenkovicm
left a comment
There was a problem hiding this comment.
LGTM, thanks @avantgardnerio
Damn, you're fast today 😄 |
|
I want to be seen as helpful while avoiding other major PR changes that require deep engagement and focus. 😀 |
Well, this is the big one - it gets parallel windows working end-to-end with decent benchmark results. So I'll just keep extracting smaller ones out of there, as long as you're willing to review, I can keep you busy 😄 . Not required, but very much appreciated! |
|
will have a look over the weekend hopefully |
Summary
Additive Docker image changes for S3-backed EKS benchmark runs, plus wiring the
h2obinary (added in #2230) into the benchmarks image.ballista-scheduler+ballista-executor: installca-certificates.Without them, DataFusion's
object_storeS3 client fails TLS verification withInvalidCertificate(UnknownIssuer), and STSAssumeRoleWithWebIdentity(IRSA-authenticated S3 reads on EKS) fails the same way. Ubuntu's minimal
image doesn't populate
/etc/ssl/certs.ballista-benchmarks: sameca-certificatesrationale, plusCOPY target/release/h2o /root/h2oso the image bundles both the tpch andh2o binaries. The h2o binary shipped in ci: add h2o window suite CI job #2230 but was never wired into the
Dockerfile.
.dockerignore: whitelisttarget/release{,-nonlto}/h2oso thebenchmarks build context sees the h2o binary alongside tpch/etc. Missing
whitelist entry means the
COPYline above errors out during build.TPC-H bench workflow is unaffected — every change is additive.
Test plan
docker build -f dev/docker/ballista-scheduler.Dockerfile .succeedsdocker build -f dev/docker/ballista-executor.Dockerfile .succeedsdocker build -f dev/docker/ballista-benchmarks.Dockerfile .succeedsdocker run --rm <benchmarks-image> ls /root/h2oprints the binary pathdocker run --rm <executor-image> ls /etc/ssl/certs | wc -l> 0s3://...path without TLS errors