Skip to content

Return Mock on GetBucketRequestPayment #8924

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 10, 2025

Conversation

shirady
Copy link
Contributor

@shirady shirady commented Apr 1, 2025

Describe the Problem

Currently, we fail the request of GetBucketRequestPayment with InternalError although in our current code we have implemented returning a mock. In the case of PutBucketRequestPayment instead of InternalError it would be NotImplemented.

Explain the Changes

  1. Rename the file and actions related to GetBucketRequestPayment and add a log printing so we will not fail the request, also add a test for the mock in containerized and NC.
  2. Rename the file and actions related to PutBucketRequestPayment in the same way.

Issues:

  1. Example from the logs - GetBucketRequestPayment (on the master branch):
Apr-9 15:47:03.052 [nsfs/73360] [ERROR] core.endpoint.s3.s3_rest:: Got a not supported S3 op get_bucket_requestPayment - doesn't suppose to happen
Apr-9 15:47:03.052 [nsfs/73360] [ERROR] core.endpoint.s3.s3_rest:: S3 ERROR <?xml version="1.0" encoding="UTF-8"?><Error><Code>InternalError</Code><Message>We encountered an internal error. Please try again.</Message><Resource>/bucket-01-04?requestPayment</Resource><RequestId>m99xbrlc-czo41u-1c4y</RequestId></Error> GET /bucket-01-04?requestPayment {"host":"localhost:6443","accept-encoding":"identity","user-agent":"aws-cli/2.17.11 md/awscrt#0.20.11 ua/2.0 os/macos#24.3.0 md/arch#arm64 lang/python#3.11.10 md/pyimpl#CPython cfg/retry-mode#standard md/installer#source md/prompt#off md/command#s3api.get-bucket-request-payment","x-amz-date":"20250409T124703Z","x-amz-content-sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","authorization":"AWS4-HMAC-SHA256 Credential=Dwertyuiopasdfg11001/20250409/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=8baf2c3ea9bee8440a5d8237775b83364f6f1fb82580ffa97c972b28a2098b1a"} Error: We encountered an internal error. Please try again.
    at _get_method_from_req (/Users/shiradymnik/SourceCode/noobaa-core/src/endpoint/s3/s3_rest.js:329:15)
    at authorize_request_policy (/Users/shiradymnik/SourceCode/noobaa-core/src/endpoint/s3/s3_rest.js:236:20)
    at async Promise.all (index 1)
    at async authorize_request (/Users/shiradymnik/SourceCode/noobaa-core/src/endpoint/s3/s3_rest.js:220:5)
    at async handle_request (/Users/shiradymnik/SourceCode/noobaa-core/src/endpoint/s3/s3_rest.js:140:5)
    at async Object.s3_rest [as handler] (/Users/shiradymnik/SourceCode/noobaa-core/src/endpoint/s3/s3_rest.js:69:9)
  1. Example from the logs - GetBucketRequestPayment (on the master branch):
Apr-9 15:50:50.972 [nsfs/73360] [ERROR] core.endpoint.s3.s3_rest:: Got a not supported S3 op put_bucket_requestPayment - doesn't suppose to happen
Apr-9 15:50:50.972 [nsfs/73360] [ERROR] core.endpoint.s3.s3_rest:: S3 ERROR <?xml version="1.0" encoding="UTF-8"?><Error><Code>InternalError</Code><Message>We encountered an internal error. Please try again.</Message><Resource>/bucket-01-04?requestPayment</Resource><RequestId>m99xgnh1-bzns9c-1d42</RequestId></Error> PUT /bucket-01-04?requestPayment {"host":"localhost:6443","accept-encoding":"identity","user-agent":"aws-cli/2.17.11 md/awscrt#0.20.11 ua/2.0 os/macos#24.3.0 md/arch#arm64 lang/python#3.11.10 md/pyimpl#CPython cfg/retry-mode#standard md/installer#source md/prompt#off md/command#s3api.put-bucket-request-payment","content-md5":"stiwwZodlx5hsnK0pEYn+Q==","x-amz-date":"20250409T125050Z","x-amz-content-sha256":"aad02439b908cbad0ded5b0d4a418f4b2b9e8a4515d6134e2b124bdeab82f33e","authorization":"AWS4-HMAC-SHA256 Credential=Dwertyuiopasdfg11001/20250409/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date, Signature=e6187adecafb5c01756ff18545ce4d83386a49a86633a9b166f0eb094084b205","content-length":"131"} Error: We encountered an internal error. Please try again.
    at _get_method_from_req (/Users/shiradymnik/SourceCode/noobaa-core/src/endpoint/s3/s3_rest.js:329:15)
    at authorize_request_policy (/Users/shiradymnik/SourceCode/noobaa-core/src/endpoint/s3/s3_rest.js:236:20)
    at async Promise.all (index 1)
    at async authorize_request (/Users/shiradymnik/SourceCode/noobaa-core/src/endpoint/s3/s3_rest.js:220:5)
    at async handle_request (/Users/shiradymnik/SourceCode/noobaa-core/src/endpoint/s3/s3_rest.js:140:5)
    at async Object.s3_rest [as handler] (/Users/shiradymnik/SourceCode/noobaa-core/src/endpoint/s3/s3_rest.js:69:9)

Testing Instructions:

Unit tests:

Please run:
(NC):
sudo NC_CORETEST=true node ./node_modules/mocha/bin/mocha ./src/test/unit_tests/test_nsfs_integration.js

(Containerized):
Use this guide for local run and run: ./node_modules/mocha/bin/mocha.js src/test/unit_tests/test_s3_ops.js NOOBAA_LOG_LEVEL=all

Manual Testing instruction (NC deployment):

  1. Create an account with the CLI: sudo node src/cmd/manage_nsfs account add --name <account-name> --new_buckets_path /Users/buckets/ --access_key <access-key> --secret_key <secret-key> --uid <uid> --gid <gid>
    Note: before creating the account need to give permission to the new_buckets_path: chmod 777 /Users/buckets/
  2. Start the NSFS server with: sudo node src/cmd/nsfs --debug 5
  3. Create the alias for S3 service:alias nc-user-1-s3=‘AWS_ACCESS_KEY_ID=<access-key> AWS_SECRET_ACCESS_KEY=<secret-key> aws --no-verify-ssl --endpoint-url https://localhost:6443’
  4. Check the connection to the endpoint and try to list the buckets (should be empty): nc-user-1-s3 s3 ls; echo $?
  5. Add bucket to the account using AWS CLI: nc-user-1-s3 s3 mb s3://bucket-01-04 (bucket-01-04 is the bucket name in this example)
  6. Run the request of get-bucket-request-payment (expect to see a mock reply): nc-user-1-s3 s3api get-bucket-request-payment --bucket bucket-01-04
  7. Run the request of put-bucket-request-payment (expect to see InternalError - but from the s3ops layer): nc-user-1-s3 s3api put-bucket-request-payment --bucket bucket-01-04 --request-payment-configuration '{"Payer":"Requester"}'
  • Doc added/updated
  • Tests added

@shirady shirady self-assigned this Apr 1, 2025
@shirady shirady force-pushed the api-mock-get-bucket-request-payment branch from a9e51e8 to aea6980 Compare April 9, 2025 12:40
@pull-request-size pull-request-size bot added size/M and removed size/S labels Apr 9, 2025
@shirady shirady force-pushed the api-mock-get-bucket-request-payment branch from aea6980 to 066ad7e Compare April 9, 2025 12:41
@shirady shirady marked this pull request as ready for review April 9, 2025 13:23
@shirady shirady requested a review from nadavMiz April 9, 2025 13:24
Copy link
Contributor

@nadavMiz nadavMiz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

1. Rename the file and actions related to GetBucketRequestPayment and add a log printing so we will not fail the request, also add a test for the mock in containerized and NC.
2. Rename the file and actions related to PutBucketRequestPayment in the same way.

Signed-off-by: shirady <[email protected]>
@shirady shirady force-pushed the api-mock-get-bucket-request-payment branch from 066ad7e to e0f0922 Compare April 10, 2025 04:38
@shirady shirady merged commit e776be7 into noobaa:master Apr 10, 2025
11 checks passed
@shirady shirady deleted the api-mock-get-bucket-request-payment branch April 10, 2025 05:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants