Regression (0.6.5+): Iceberg REST create+write fails with "unsupported catalog capability ... remote signing, vended credentials" (worked in 0.6.4)

[Robin-Everaars]

Summary

Creating and writing an Iceberg table through a REST catalog (Lakekeeper) regressed between Sail 0.6.4 and 0.6.5. On 0.6.4 the create+write succeeds; on 0.6.5 and 0.6.6 it fails with:

pyspark.errors.exceptions.connect.AnalysisException: unsupported catalog capability: Iceberg REST access session requirements returned by create_table are not supported for create+write yet: remote signing, vended credentials

Reads from the same catalog still work on 0.6.6 — only create+write is affected. It looks like the 0.6.5 Iceberg work now parses the catalog's create_table session requirements (vended credentials / remote signing) and fails closed, whereas 0.6.4 fell back to the ambient environment credentials (which reads still do). Per #2214, Sail bootstraps S3 credentials from the environment for all access, so falling back to those for create+write (as 0.6.4 did) would restore the previous behavior.

Environment

• Sail server: 0.6.5 and 0.6.6 fail; 0.6.4 works. Run as spark server, SAIL_MODE=local.
• Client: pyspark-client==4.1.2.
• Catalog: Lakekeeper (iceberg-rest), registered via SAIL_CATALOG__LIST.
• Object store: S3-compatible (RustFS), path-style.
• The server has ambient S3 credentials in its environment (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_ENDPOINT), which are the same credentials the catalog vends.

Catalog / warehouse configuration

Sail catalog registration:

SAIL_CATALOG__LIST=[{name="lake",type="iceberg-rest",uri="http://lakekeeper:8181/catalog",warehouse="demo"}]

Lakekeeper warehouse (vends a static access key, STS disabled):

{
  "warehouse-name": "demo",
  "storage-profile": {"type": "s3", "bucket": "lakesail", "endpoint": "http://rustfs:9000", "region": "us-east-1", "path-style-access": true, "flavor": "s3-compat", "sts-enabled": false},
  "storage-credential": {"type": "s3", "credential-type": "access-key", "access-key-id": "…", "secret-access-key": "…"}
}

Reproduction

from pyspark.sql import SparkSession

spark = SparkSession.builder.remote("sc://localhost:50051").getOrCreate()

spark.sql("CREATE NAMESPACE IF NOT EXISTS lake.demo_ns")
df = spark.createDataFrame([(1, "a"), (2, "b")], ["id", "name"])
df.writeTo("lake.demo_ns.t").using("iceberg").create()

• 0.6.4: succeeds (table created, 2 rows).
• 0.6.5 / 0.6.6: raises the AnalysisException above.
• Reading a table that was created on 0.6.4 succeeds on 0.6.6, confirming the failure is specific to create+write.

Expected

Create+write through the REST catalog succeeds, as it did on 0.6.4 — either by honoring the vended credentials, or (at minimum) by falling back to the ambient environment credentials when the vended-credential / remote-signing session requirements can't be honored, matching the read path and the documented "configure via environment variables" approach.

Version bisect

Works on 0.6.4; fails on 0.6.5 and 0.6.6. Likely introduced by the 0.6.5 "Improved Iceberg integration" (#1929, #2097, #2078, #2119, #2134).

Related

• Per Read Ops Credential for S3 #2214 — per-read-op credential vending (a maintainer comment there notes S3 credentials are bootstrapped from the environment for all access)
• Question : does sail support credential vending from an iceberg rest catalog #1491 — credential vending from an Iceberg REST catalog
• [Epic] Full Iceberg Integration #172 — Iceberg integration epic

[linhr]

Thanks for reporting this! Yeah, indeed, we could be more lenient during table creation if the catalog service declares a capability that we do not support yet.

Supporting the credentials (even static per table) returned by the catalog service may be more work for now (as it's an overlapping scope with credential vending). But it should likely be straightforward to recover the 0.6.4 behavior that uses the environment variable credentials.

@lonless9 will have more insights into how we can fix this.

[Robin-Everaars]

Thank you! Let me know if I can help in any way by means of testing for example!