Skip to content

chore(docker): ca-certificates + h2o binary in benchmarks image - #2251

Merged
avantgardnerio merged 2 commits into
apache:mainfrom
avantgardnerio:brent/docker-h2o-and-ca-certs
Aug 7, 2026
Merged

chore(docker): ca-certificates + h2o binary in benchmarks image#2251
avantgardnerio merged 2 commits into
apache:mainfrom
avantgardnerio:brent/docker-h2o-and-ca-certs

Conversation

@avantgardnerio

@avantgardnerio avantgardnerio commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Summary

Additive Docker image changes for S3-backed EKS benchmark runs, plus wiring the
h2o binary (added in #2230) into the benchmarks image.

  • ballista-scheduler + ballista-executor: install ca-certificates.
    Without them, DataFusion's object_store S3 client fails TLS verification with
    InvalidCertificate(UnknownIssuer), and STS AssumeRoleWithWebIdentity
    (IRSA-authenticated S3 reads on EKS) fails the same way. Ubuntu's minimal
    image doesn't populate /etc/ssl/certs.
  • ballista-benchmarks: same ca-certificates rationale, plus
    COPY target/release/h2o /root/h2o so the image bundles both the tpch and
    h2o binaries. The h2o binary shipped in ci: add h2o window suite CI job #2230 but was never wired into the
    Dockerfile.
  • .dockerignore: whitelist target/release{,-nonlto}/h2o so the
    benchmarks build context sees the h2o binary alongside tpch/etc. Missing
    whitelist entry means the COPY line above errors out during build.

TPC-H bench workflow is unaffected — every change is additive.

Test plan

  • docker build -f dev/docker/ballista-scheduler.Dockerfile . succeeds
  • docker build -f dev/docker/ballista-executor.Dockerfile . succeeds
  • docker build -f dev/docker/ballista-benchmarks.Dockerfile . succeeds
  • docker run --rm <benchmarks-image> ls /root/h2o prints the binary path
  • docker run --rm <executor-image> ls /etc/ssl/certs | wc -l > 0
  • EKS run: h2o Q1 reads from an s3://... path without TLS errors

avantgardnerio and others added 2 commits August 7, 2026 13:06
Additive changes needed for S3-backed benchmark runs and image parity
with the h2o binary that shipped in apache#2230.

- ballista-scheduler + ballista-executor: install `ca-certificates`.
  Without them, TLS handshakes for S3 (via object_store) and STS
  (AssumeRoleWithWebIdentity for IRSA-authenticated S3 reads) fail
  with `InvalidCertificate(UnknownIssuer)`.
- ballista-benchmarks: same ca-certificates rationale, plus
  `COPY target/release/h2o /root/h2o` so the image bundles both tpch
  and h2o binaries. The h2o binary was added in apache#2230 but never wired
  into the Dockerfile.
- .dockerignore: whitelist `target/release{,-nonlto}/h2o` so the
  benchmarks build context sees the h2o binary alongside tpch/etc.
  Whitelist is where the other binaries already live; without it,
  docker build errors on the new COPY line.

Andy's TPC-H bench workflow is unaffected — every change is additive.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

@milenkovicm milenkovicm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @avantgardnerio

@avantgardnerio

Copy link
Copy Markdown
Contributor Author

LGTM, thanks @avantgardnerio

Damn, you're fast today 😄

@milenkovicm

Copy link
Copy Markdown
Contributor

I want to be seen as helpful while avoiding other major PR changes that require deep engagement and focus. 😀
I finally got some time to spend on ballista, hope to help more, and im trying to catch up with all the changes i've missed

@avantgardnerio

Copy link
Copy Markdown
Contributor Author

I want to be seen as helpful while avoiding other major PR changes that require deep engagement and focus. 😀 I finally got some time to spend on ballista, hope to help more, and im trying to catch up with all the changes i've missed

Well, this is the big one - it gets parallel windows working end-to-end with decent benchmark results.

So I'll just keep extracting smaller ones out of there, as long as you're willing to review, I can keep you busy 😄 . Not required, but very much appreciated!

@milenkovicm

milenkovicm commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

will have a look over the weekend hopefully

@avantgardnerio
avantgardnerio merged commit 1b10233 into apache:main Aug 7, 2026
5 checks passed
@avantgardnerio
avantgardnerio deleted the brent/docker-h2o-and-ca-certs branch August 7, 2026 19:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants