Support XML API (#331)

[tustvold]

Closes #331

It has been years since I last wrote any Golang so please let me know if I've made some silly mistakes. Most of the necessary functionality already existed, the only necessary change was to add support for the XML put API

[tustvold]

This was incorrect, the XML APIs should not return a body and certainly shouldn't return JSON

[fsouza]

🤔 Oh are signed uploads powered by the XML API? Interesting!

[tustvold]

This can't have ever worked as insertObject expects query parameters encoding the object name, etc...

[fsouza]

insertObject does not necessarily read the object name from query parameters:

fake-gcs-server/fakestorage/upload.go

Lines 85 to 95 in d7832ec

	default:
	// Support Signed URL Uploads
	if r.URL.Query().Get("X-Goog-Algorithm") != "" {
	switch r.Method {
	case http.MethodPost:
	return s.resumableUpload(bucketName, r)
	case http.MethodPut:
	return s.signedUpload(bucketName, r)
	}
	}
	return jsonResponse{errorMessage: "invalid uploadType", status: http.StatusBadRequest}

I need to take a deeper look at this, but curious if you're basing on comment on the docs? There are many undocumented things in the GCS API that we run into when someone tries to integrate with some different SDK, and I've ironically done a poor job in documenting those in fake-gcs-server 🙈

[tustvold]

Indeed, insertObject has a peculiar special case that uses X-Goog-Algorithm to detect a signed URL and then falls back to implementing something that looks like the XML API, because that is what signed URLs are. With this PR that can probably be removed 🤔

[fsouza]

Yeah, absolutely. insertObject definitely looks like a little monster doing too many things. Thank you very much! I'll go over the PR in details later today.

[raz-amir]

Hi, bumping this one - any plans to fix and merge it soon? Thanks

[fsouza]

@ramir-savvy someone needs to take a look at the test failures and fix them. I can do it, but not in the near future. Maybe in a couple of weeks.

[raz-amir]

Thank you @fsouza. I have opened #1215 which better suits my needs.

[tustvold]

Just wondering what the status of this is, the lack of support for the XML APIs is causing some upstream confusion - apache/arrow-rs#5263

[Slach]

=( ohh, if could be a great functionality which allow us test GCS over S3 in Altinity/clickhouse-backup#695

[nickpresta]

@tustvold Is this something still on your radar?

[tustvold]

We've been using this branch to run integration tests for the last year. I'd be happy to see this incorporated but that's out of my hands 😅

[fsouza]

Yeah I think the code overall looks good, but I'd need conflicts resolved and CI to be green before merging. I can give it a shot myself, but I don't expect to have any time to work on this in the next 2-3 weeks.

[Slach]

@fsouza if i merge conclicts and create new PR, will it merge?

[adriangb]

it would be great to get this functionality included!

[adriangb]

I tried using this with a mix of Rust's object_store (via deltalake) and google-cloud-storage in Python and can't get them to both work:

import os

os.environ['STORAGE_EMULATOR_HOST'] = 'http://localhost:4443'
os.environ['GOOGLE_SERVICE_ACCOUNT_KEY'] = '{"gcs_base_url": "http://localhost:4443", "disable_oauth": true, "client_email": "", "private_key": "", "private_key_id": ""}'

import pyarrow
from deltalake import DeltaTable
from google.cloud import storage

client = storage.Client()

bucket = client.bucket('test-bucket')  # type: ignore

print(list(bucket.list_blobs()))

bucket.blob('test.txt').upload_from_string(b'Hello, world!')

print(list(bucket.list_blobs()))

blob = bucket.blob('test.txt')

print(blob.download_as_string())

DeltaTable.create('gs://test-bucket', schema=pyarrow.schema([('id', pyarrow.int64())]), mode='overwrite')

Running docker run -it -p 4443:4443 -v ./gcs_data:/data tustvold/fake-gcs-server -scheme http -public-host localhost:4443 -log-level debug -filesystem-root=/data makes the deltalake / object_store part work but the google sdk part fails at blob.download_as_string() with 'Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206> and I see "GET /download/storage/v1/b/test-bucket/o/test.txt?alt=media HTTP/1.1" 404 10 in the logs.

Switching to running docker run -it -p 4443:4443 -v ./gcs_data:/data tustvold/fake-gcs-server -scheme http -public-host http://localhost:4443 -log-level debug -filesystem-root=/data (so adding http:// to -public-host) makes the google sdk part work but the object_store part fail with OSError: Generic GCS error: Error performing list request: Client error with status 404 Not Found: Not Found and I see "GET /test%2Dbucket?list-type=2&prefix=_delta_log%2F HTTP/1.1" 404 10 in the logs.

I'm guessing this has to do with virtual bucket name parsing?