feat: support S3 Table Buckets with S3TablesCatalog

[felixscherz]

Hi, this is in regards to #1404.

I created a first draft of an S3TablesCatalog that uses the S3 Table Bucket API for catalog operations.

How to run tests

Since moto does not support mocking the S3 Tables API yet (WIP: getmoto/moto#8470) we have to run tests against a live AWS account. To do that, create an S3 Tables Bucket in one of the supported regions and then set the table bucket ARN and AWS Region as environment variables

AWS_REGION=us-east-2 AWS_TEST_S3_BUCKET_ARN=... poetry run pytest tests/catalog/integration_test_s3tables.py

[felixscherz]

I was able to work around the issue above by using FsspecFileIO instead of the default PyarrowFileIO. Using FsspecFileIO the catalog is now able to create new tables.

[kevinjqliu]

Thanks for working on this @felixscherz Feel free to tag me when its ready for review :)

[felixscherz]

I think you can now review this PR if you have time @kevinjqliu :)
The biggest issue for now will be that testing is only possible against AWS itself since moto does not support the s3tables API yet. I created an issue on the moto side but have not had the time to implement it myself getmoto/moto#8422.

I currently run tests by setting the ARN env variable to that of an s3 table bucket I created within my personal AWS account:

https://github.com/felixscherz/iceberg-python/blob/feat/s3tables-catalog/tests/catalog/test_s3tables.py#L24-L31

[kevinjqliu]

Thanks for the PR, i added a few comments to clarify the catalog behaviors

I'm a little hesitant to merge this in given that we have to run tests against a production S3 endpoint. Maybe we can mock the endpoint?

[kevinjqliu]

I ran the tests locally ARN=arn:aws:s3tables:us-east-2:... poetry run pytest tests/catalog/test_s3tables.py
had to manually add s3tables.region to the catalog config

    properties = {"s3tables.table-bucket-arn": table_bucket_arn, "s3tables.region": "us-east-2", "py-io-impl": "pyiceberg.io.fsspec.FsspecFileIO"}

And these 3 testa failed, everything else is ✅

FAILED tests/catalog/test_s3tables.py::test_s3tables_api_raises_on_conflicting_version_tokens - botocore.exceptions.NoRegionError: You must specify a region.
FAILED tests/catalog/test_s3tables.py::test_s3tables_api_raises_on_preexisting_table - botocore.exceptions.NoRegionError: You must specify a region.
FAILED tests/catalog/test_s3tables.py::test_creating_catalog_validates_s3_table_bucket_exists - botocore.exceptions.NoRegionError: You must specify a region.

[felixscherz]

Thank you for the review!

I removed tests related to boto3 and set the AWS region explicitly for the test run.
I agree with you that we should not merge this as long as the only option for running tests is to run them against a live AWS account. I'm currently working on supporting s3tables with moto: getmoto/moto#8470.
We can hold off on this PR until moto can support s3tables so we can run tests against a mock endpoint?

[kevinjqliu]

Added a few more comments.

I was able to run the test locally

AWS_REGION=us-east-2 ARN=... poetry run pytest tests/catalog/test_s3tables.py

after making a few local changes

• poetry update boto3
• add aws_region fixture
• pass aws_region to catalog

Could you update the PR description so others can test this PR out?

[HonahX]

@felixscherz Thanks for the great contribution! Looking forward to adding this to PyIceberg! I left some comments. Please let me know what you think.

[HonahX]

I did not find the logic for cases when table not exist, which means create_table_transaction will not be supported in the current version.

iceberg-python/pyiceberg/catalog/__init__.py

Lines 754 to 765 in e41c428

	def create_table_transaction(
	self,
	identifier: Union[str, Identifier],
	schema: Union[Schema, "pa.Schema"],
	location: Optional[str] = None,
	partition_spec: PartitionSpec = UNPARTITIONED_PARTITION_SPEC,
	sort_order: SortOrder = UNSORTED_SORT_ORDER,
	properties: Properties = EMPTY_DICT,
	) -> CreateTableTransaction:
	return CreateTableTransaction(
	self._create_staged_table(identifier, schema, location, partition_spec, sort_order, properties)
	)

We do not have to support everything in the initial PR. But it will be good to override create_table_transaction as "Not Implemented" for the s3tables

[felixscherz]

I added exceptions for this case for now along with a test. I will have a look at how to implement this properly

[HonahX]

If anything goes wrong after this point, I think we should clean up the created s3 table by s3tables' delete_table endpoint.

[felixscherz]

I added a try/except to delete the s3 table in case something goes wrong with writing the initial metadata.

[kevinjqliu]

can you run poetry lock --no-update for CI?

[kevinjqliu]

@felixscherz could you rebase this against main?

i see that getmoto/moto/8470 is now merged, thanks for driving that!
waiting for getmoto/moto@4f565fb to make it to the next release (5.0.28)

[felixscherz]

I rebased onto the main. I prepared the unit tests using the new moto features and I will commit them once the new moto release is available:)

[felixscherz]

@kevinjqliu moto==5.0.28 just got released, I added unit tests and documentation, could you have a look when you have time?

[kevinjqliu]

LGTM! Thanks for adding both the unit test and integration test. And for driving downstream dependency to add support for S3Tables!! (getmoto/moto#8470)

I pushed a few changes to resolve merge conflicts. And I verified the integration test locally

AWS_REGION=us-east-2 AWS_TEST_S3_TABLE_BUCKET_ARN=arn:aws:s3tables:us-east-2:033327485438:bucket/s3-table poetry run pytest tests/catalog/integration_test_s3tables.py

[felixscherz]

@geruh would you have another look at this PR when you have time?

[kevinjqliu]

hey @felixscherz thanks for following up on this PR. I haven't forgotten about this! :) I'll do another review soon

[felixscherz]

I believe this PR is no longer relevant since AWS started support the REST Catalog: #1404 (comment) :)

[mmadsen]

@felixscherz does this mean that nothing special is needed to use PyIceberg against an S3 Table bucket now? Thank you for your work on this -- our team has been waiting to use table buckets specifically until we could migrate some pyiceberg-based services.

[felixscherz]

Yes! You use the Rest catalog and should be able to follow this guide: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-open-source.html.
I haven't tested it myself however:)