Iceberg writer ignores absolute `write.data.path`, breaking writes to Unity Catalog managed tables

[carlosjourdan]

Describe the bug

The Iceberg writer reads the write.data.path table property but only honors it when the value is a sub-path of the table location. An absolute path outside the table location — which is exactly what Databricks Unity Catalog sets on managed Iceberg tables — is silently ignored, and data files are written under <table location>/data/ instead.

Against a local filesystem table this only misplaces the data files. Against Databricks Unity Catalog managed Iceberg tables (via the Iceberg REST catalog), UC enforces that committed data files live under the managed prefix it advertises in write.data.path (abfss://…/tables/<uuid>/…), so every Sail write fails at commit time with 500 ServiceFailureException, ErrorCode 2012. The identical parquet bytes placed under write.data.path and committed via PyIceberg ≥ 0.9.0 (which implements LocationProvider semantics for this property) succeed, confirming the rejection is caused purely by the data file location.

To reproduce

No Unity Catalog needed — the property handling reproduces on a local filesystem table:

# pip install "pysail[spark]==0.6.6" "pyspark-client==4.1.2" "pyiceberg[sql-sqlite]" pyarrow "pandas<3"
import glob
import os
import tempfile

import pyarrow as pa
from pyiceberg.catalog.sql import SqlCatalog
from pysail.spark import SparkConnectServer
from pyspark.sql import SparkSession


def parquet_files(root):
    return set(glob.glob(os.path.join(root, "**", "*.parquet"), recursive=True))


base = tempfile.mkdtemp(prefix="sail-wdp-repro-")
table_location = os.path.join(base, "table")
data_path = os.path.join(base, "managed-data")  # absolute, OUTSIDE the table location

# Create an Iceberg table whose `write.data.path` points outside the table
# location — what Unity Catalog does for managed Iceberg tables — and seed it
# with one append via pyiceberg (which honors the property).
catalog = SqlCatalog("local", uri=f"sqlite:///{base}/catalog.db", warehouse=f"file://{base}")
catalog.create_namespace("ns")
rows = pa.table({"id": pa.array([1], pa.int64()), "val": pa.array(["a"], pa.string())})
table = catalog.create_table(
    "ns.t",
    schema=rows.schema,
    location=f"file://{table_location}",
    properties={"write.data.path": f"file://{data_path}"},
)
table.append(rows)
before = parquet_files(base)
print(f"write.data.path : file://{data_path}")
print(f"pyiceberg wrote : {[os.path.relpath(p, base) for p in sorted(before)]}")

# Append to the same table via Sail.
server = SparkConnectServer("127.0.0.1", 0)
server.start()
host, port = server.listening_address
spark = SparkSession.builder.remote(f"sc://{host}:{port}").getOrCreate()
spark.createDataFrame([(2, "b")], ["id", "val"]).write.format("iceberg").mode("append").save(
    f"file://{table_location}"
)
print(f"sail wrote      : {[os.path.relpath(p, base) for p in sorted(parquet_files(base) - before)]}")
spark.stop()
server.stop()

Output:

write.data.path : file:///tmp/sail-wdp-repro-qxjbqnxz/managed-data
pyiceberg wrote : ['managed-data/00000-0-8afdb823-3974-4004-8865-3ce5ede7456e.parquet']
sail wrote      : ['table/data/part-0e03af87-f928-4b6f-b554-c3f9ae945be4-00000000000000000000.parquet']

PyIceberg honors the property; Sail silently falls back to <table location>/data/.

Note that a write.data.path that is a child of the table location (e.g. file://<table location>/custom-data) is honored — the property is treated purely as a table-root-relative subdirectory, never as an arbitrary absolute location.

Where it happens

In crates/sail-iceberg/src/physical_plan/writer_exec.rs (identical on v0.6.5, v0.6.6, and main @ 97169ba):

• resolve_data_dir_from_property_value only returns Some when the property URL has the same scheme/host as the table URL and strip_prefix(table base) succeeds, yielding a relative remainder. An absolute URL outside the table base returns None.
• resolve_data_dir_from_properties then falls back to "data".
• The writer output is anchored at the table URL itself: writer_root = url_to_object_path(&table_url), so there is no way for a resolved data dir to escape the table location.

The same resolution is applied to the write.data.path / write.folder-storage.path write options, so there is no configuration workaround either.

Expected behavior

An absolute write.data.path (at least one on the same scheme/host/object store, as in the Unity Catalog case) should be honored: data files written under it and manifest entries carrying the correct absolute file_path — matching PyIceberg (≥ 0.9.0, apache/iceberg-python#1611) and the Java/Spark LocationProvider semantics. Rejecting a different host explicitly (instead of silently ignoring the property) would also be reasonable.

This would unlock writing to Databricks Unity Catalog managed Iceberg tables, which are writable by external engines only when the engine honors the UC-advertised write.data.path (Databricks docs).

Related work

I could not find an existing issue, PR, or discussion about this — closest related threads:

• feat: implement ALTER TABLE SET/UNSET TBLPROPERTIES for Iceberg #1822 (ALTER TABLE SET/UNSET TBLPROPERTIES) introduced the current write.data.path handling in the writer (sub-path-only resolution).
• feat: catalog managed Iceberg table #1929 (catalog managed Iceberg table) added the catalog-managed write path where this surfaces against Unity Catalog.
• fix: align Unity managed Delta staging and commit replay #2195 fixed the analogous Unity-managed location semantics on the Delta Lake side (staging and commit replay); this issue is the Iceberg counterpart.
• Epic [Epic] Full Iceberg Integration #172 (Full Iceberg Integration) has no checklist item covering write.data.path / LocationProvider semantics yet.

Environment

• pysail 0.6.6 (also reproduced on 0.6.5), installed from PyPI
• pyspark-client 4.1.2, Python 3.12, Linux x86_64
• pyiceberg 0.11.1 (control writer)
• UC evidence: Databricks Unity Catalog managed Iceberg table via IRC + credential vending; Sail commit rejected with 500 ServiceFailureException ErrorCode 2012; same parquet committed via pyiceberg succeeds

[linhr]

Thanks for reporting this! We'll take a look shortly.