Compatibility-tests SDK versions are not aligned (boto3 1.37.1 undercounts SES v2 API)

[okinaka]

Summary

The SDK/tool versions pinned across compatibility-tests/ span a wide release window (~early 2024 → mid 2026) and are not aligned. Some are exact-pinned, others float. Most notably boto3/botocore is pinned to 1.37.1 (~Feb 2025), old enough to omit newer SES v2 API models, which undercounts coverage checks that read the local model.

Current versions

Suite	SDK / tool	Version	Pin	Approx. release
sdk-test-python	boto3 / botocore	1.37.1	exact	~2025-02
sdk-test-java	AWS SDK for Java v2	2.44.14	exact	~2026-05
sdk-test-go	aws-sdk-go-v2 (core)	v1.41.5 (s3 1.97.1)	go.sum	~2026 Q1
sdk-test-node	@aws-sdk/client-*	^3.500.0	floating (floor ~2024-01)	latest 3.x at CI
sdk-test-awscli	amazon/aws-cli	:latest	floating	build-time latest
compat-cdk	aws-cdk / aws-cdk-lib	2.171.1	exact	~2024-12
compat-terraform	hashicorp/aws	~> 6.0	floating (6.x)	provider v6 GA ~2025-06
compat-opentofu	hashicorp/aws	~> 5.0	floating (5.x)	one major behind Terraform

Impact

• boto3/botocore 1.37.1 is missing ~15 SES v2 operations (e.g. the Tenant* / ReputationEntity* families): the local model lists 97 v2 ops while AWS docs list ~112, so any coverage/enumeration read from the pinned model undercounts.
• Terraform pins the aws provider at v6.x while OpenTofu pins v5.x — a full major-version gap between two suites intended to cross-check the same resources.
• The exact-pinned suites alone (Python/Java/Go/CDK) span ~18 months (boto3 ~2025-02 ↔ Java 2.44.14 ~2026-05), so "the SDKs" is not a single point in time.
• The floating pins (Node caret, CLI :latest, TF/tofu ~>) resolve to whatever CI installs, so runs aren't reproducible over time.

Proposal

• Bump the exact-pinned SDKs (Python/Java/Go/CDK) to a common recent baseline.
• Decide whether Terraform and OpenTofu should track the same aws provider major (align, or document the split as intentional).
• Consider pinning the currently-floating deps (Node exact, CLI tag, provider = x.y) for reproducibility — or explicitly document them as "always latest" by design.

[okinaka]

Update from attempting the alignment

I went through the exact-pinned suites and have two findings to add:

Python + Java — aligned. Bumped to the current baseline:

• sdk-test-python: boto3 / botocore 1.37.1 → 1.43.69 (this also restores the full SES v2 model — the pinned botocore now lists all 112 v2 operations, incl. the Tenant* / ReputationEntity* families that 1.37.1 was missing).
• sdk-test-java: AWS SDK for Java v2 2.44.14 → 2.52.0.

Go — not actually pinned (correcting my earlier note). sdk-test-go has no committed go.mod; the Dockerfile generates it at build with go mod init && go mod tidy, which resolves every import to the latest. So Go already floats to the newest SDK on each run (the committed go.sum is just a stale artifact that gets regenerated). Nothing to bump — it belongs with the "float by design" group, not the exact-pinned one.

CDK — blocked, left out of scope. Aligning compat-cdk to a recent aws-cdk-lib forces the aws-cdk 2.1000+ CLI (the CLI and library version lines are now decoupled), which in turn needs aws-cdk-local v3. But cdklocal v3's S3 path-style handling doesn't work against Floci:

• 3.0.3 doesn't apply any path-style patch on the new-CLI code path.
• 3.0.4 adds one, but its AWS_S3_FORCE_PATH_STYLE toggle is broken — shouldForcePathStyle passes the env value to isEnvTrue, which expects the key (AWS_S3_FORCE_PATH_STYLE environment variable is not correctly recognized localstack/aws-cdk-local#127), so only .sandbox. endpoints get path-style.

The result is that CDK asset uploads fall back to virtual-hosted URLs (<bucket>.floci), which have no DNS entry on the Docker network → getaddrinfo ENOTFOUND. So compat-cdk stays on aws-cdk 2.171.1 / aws-cdk-local ^2.0.0 for now. A proper CDK bump needs one of: the upstream cdklocal fix (#127) landing, a .sandbox. endpoint-alias workaround, or Floci-side support for virtual-hosted-style S3. Happy to open a separate issue to track that.

[okinaka]

Correction on CDK — it can be aligned after all, via virtual-hosted-style S3.

My earlier note blamed cdklocal's path-style toggle, but the real cause was simpler: I'd pointed AWS_ENDPOINT_URL_S3 at http://floci:4566, so the new CLI built <bucket>.floci for the template asset — an unresolvable virtual-hosted host (getaddrinfo ENOTFOUND).

The fix is to use Floci's own virtual-hosted S3 domain instead of forcing path-style: AWS_ENDPOINT_URL_S3=http://s3.localhost.floci.io:4566. The SDK then builds <bucket>.s3.localhost.floci.io, which resolves via the public *.s3.localhost.floci.io wildcard locally and via Floci's embedded DNS (already injected with --dns in the compat workflow) inside the Docker network. No path-style forcing, and no dependency on the cdklocal AWS_S3_FORCE_PATH_STYLE bug (localstack/aws-cdk-local#127).

So compat-cdk can move to aws-cdk-lib 2.264 / aws-cdk 2.1135 / aws-cdk-local ^3.0.4 with just that S3 endpoint change.

[okinaka]

Opened #2253 to implement this. It aligns the exact-pinned suites — Python (boto3/botocore 1.43.69), Java (AWS SDK v2 2.52.0), and CDK (aws-cdk-lib 2.264 / aws-cdk 2.1135 / aws-cdk-local ^3.0.4, using the virtual-hosted S3 endpoint discussed above) — and all compatibility suites pass in CI. Go and the other float-by-design suites are left unchanged.