Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use /etc/hosts binding, some APIs are not available #1187

Closed
jiashiwen opened this issue Sep 6, 2024 · 8 comments
Closed

Use /etc/hosts binding, some APIs are not available #1187

jiashiwen opened this issue Sep 6, 2024 · 8 comments
Labels
bug This issue is a bug.

Comments

@jiashiwen
Copy link

Describe the bug

When using /etc/hosts for binding, some APIs report errors. List_buckets, list_objects_v2, and get_object are verified here.

Expected Behavior

get bucket list ok,get objectlist ok,get object ok

Current Behavior

got bucket ok ,but error got objects list and get object

got buckets: Ok(
ListBucketsOutput {
buckets: Some(
[
Bucket {
name: Some(
"ai-datasets",
),
creation_date: Some(
2024-02-05T02:28:04Z,
),
},
Bucket {
name: Some(
"cs-monitor",
),
creation_date: Some(
2023-02-07T10:12:00Z,
),
},
Bucket {
name: Some(
"dataroom",
),
creation_date: Some(
2024-05-16T10:26:42Z,
),
},
Bucket {
name: Some(
"hackthon2023",
),
creation_date: Some(
2023-08-01T01:43:57Z,
),
},
Bucket {
name: Some(
"jsw-bucket",
),
creation_date: Some(
2023-02-23T03:11:46Z,
),
},
Bucket {
name: Some(
"jsw-bucket-1",
),
creation_date: Some(
2023-09-20T09:43:15Z,
),
},
Bucket {
name: Some(
"pingdata",
),
creation_date: Some(
2021-08-27T03:35:08Z,
),
},
Bucket {
name: Some(
"robot-test",
),
creation_date: Some(
2022-04-12T06:41:43Z,
),
},
Bucket {
name: Some(
"robots",
),
creation_date: Some(
2021-11-09T03:03:23Z,
),
},
Bucket {
name: Some(
"tsp-picture",
),
creation_date: Some(
2022-01-10T03:17:10Z,
),
},
],
),
owner: Some(
Owner {
display_name: Some(
"jcloud_edUmprJ",
),
id: Some(
"577257366345",
),
},
),
continuation_token: None,
_extended_request_id: None,
_request_id: Some(
"917104E844BBC1FA",
),
},
)
objects: Err(
DispatchFailure(
DispatchFailure {
source: ConnectorError {
kind: Io,
source: hyper::Error(
Connect,
ConnectError(
"dns error",
Custom {
kind: Uncategorized,
error: "failed to lookup address information: Name or service not known",
},
),
),
connection: Unknown,
},
},
),
)
obj: Err(
DispatchFailure(
DispatchFailure {
source: ConnectorError {
kind: Io,
source: hyper::Error(
Connect,
ConnectError(
"dns error",
Custom {
kind: Uncategorized,
error: "failed to lookup address information: Name or service not known",
},
),
),
connection: Unknown,
},
},
),
)

Reproduction Steps

code

#[tokio::main]
async fn main() {
    let ak = "xxx";
    let sk = "xxx";
    let region = "cn-north-1";
    let endpoint = "http://xxx.cn-north-1.xxx.com";
    let bucket = "jsw-bucket";

    let config = SdkConfig::builder()
        .credentials_provider(SharedCredentialsProvider::new(Credentials::new(
            ak, sk, None, None, "Static",
        )))
        .endpoint_url(endpoint)
        .region(Region::new(region))
        .behavior_version(BehaviorVersion::v2024_03_28())
        .build();

    let s3_config_builder = aws_sdk_s3::config::Builder::from(&config);
    let client = aws_sdk_s3::Client::from_conf(s3_config_builder.build());

    let buckets = client.list_buckets().send().await;
    println!("got buckets: {:#?}", buckets);

    let list = client
        .list_objects_v2()
        .bucket("jsw-bucket")
        .max_keys(20)
        .send()
        .await;
    println!("objects: {:#?}", list);

    let obj = client
        .get_object()
        .bucket(bucket)
        .key("100k/!!41725505223551787799")
        .send()
        .await;
    println!("obj: {:#?}", obj);
}

/etc/hosts bind name

ip xxx.cn-north-1.xxx.com

Possible Solution

No response

Additional Information/Context

No response

Version

"B0E80D4FC4528926",
        ),
    },
)
root@a30:~/rustprojects/oss_pipe# cargo tree | grep aws-
├── aws-config v1.5.5 (/root/rustprojects/aws-sdk-rust/sdk/aws-config)
│   ├── aws-credential-types v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-credential-types)
│   │   ├── aws-smithy-async v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-async)
│   │   ├── aws-smithy-runtime-api v1.7.2 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-runtime-api)
│   │   │   ├── aws-smithy-async v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-async) (*)
│   │   │   ├── aws-smithy-types v1.2.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-types)
│   │   ├── aws-smithy-types v1.2.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   ├── aws-runtime v1.4.2 (/root/rustprojects/aws-sdk-rust/sdk/aws-runtime)
│   │   ├── aws-credential-types v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-credential-types) (*)
│   │   ├── aws-sigv4 v1.2.3 (/root/rustprojects/aws-sdk-rust/sdk/aws-sigv4)
│   │   │   ├── aws-credential-types v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-credential-types) (*)
│   │   │   ├── aws-smithy-eventstream v0.60.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-eventstream)
│   │   │   │   ├── aws-smithy-types v1.2.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   │   │   ├── aws-smithy-http v0.60.10 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-http)
│   │   │   │   ├── aws-smithy-eventstream v0.60.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-eventstrea
│   │   │   │   ├── aws-smithy-runtime-api v1.7.2 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-runtime-api
│   │   │   │   ├── aws-smithy-types v1.2.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   │   │   ├── aws-smithy-runtime-api v1.7.2 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-runtime-api) (*
│   │   │   ├── aws-smithy-types v1.2.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   │   ├── aws-smithy-async v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-async) (*)
│   │   ├── aws-smithy-eventstream v0.60.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-eventstream) (*)
│   │   ├── aws-smithy-http v0.60.10 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-http) (*)
│   │   ├── aws-smithy-runtime v1.7.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-runtime)
│   │   │   ├── aws-smithy-async v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-async) (*)
│   │   │   ├── aws-smithy-http v0.60.10 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-http) (*)
│   │   │   ├── aws-smithy-runtime-api v1.7.2 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-runtime-api) (*
│   │   │   ├── aws-smithy-types v1.2.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   │   ├── aws-smithy-runtime-api v1.7.2 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-runtime-api) (*)
│   │   ├── aws-smithy-types v1.2.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   │   ├── aws-types v1.3.3 (/root/rustprojects/aws-sdk-rust/sdk/aws-types)
│   │   │   ├── aws-credential-types v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-credential-types) (*)
│   │   │   ├── aws-smithy-async v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-async) (*)
│   │   │   ├── aws-smithy-runtime-api v1.7.2 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-runtime-api) (*
│   │   │   ├── aws-smithy-types v1.2.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   ├── aws-sdk-sso v1.40.0 (/root/rustprojects/aws-sdk-rust/sdk/sso)
│   │   ├── aws-credential-types v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-credential-types) (*)
│   │   ├── aws-runtime v1.4.2 (/root/rustprojects/aws-sdk-rust/sdk/aws-runtime) (*)
│   │   ├── aws-smithy-async v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-async) (*)
│   │   ├── aws-smithy-http v0.60.10 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-http) (*)
│   │   ├── aws-smithy-json v0.60.7 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-json)
│   │   │   └── aws-smithy-types v1.2.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   │   ├── aws-smithy-runtime v1.7.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-runtime) (*)
│   │   ├── aws-smithy-runtime-api v1.7.2 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-runtime-api) (*)
│   │   ├── aws-smithy-types v1.2.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   │   ├── aws-types v1.3.3 (/root/rustprojects/aws-sdk-rust/sdk/aws-types) (*)
│   ├── aws-sdk-ssooidc v1.41.0 (/root/rustprojects/aws-sdk-rust/sdk/ssooidc)
│   │   ├── aws-credential-types v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-credential-types) (*)
│   │   ├── aws-runtime v1.4.2 (/root/rustprojects/aws-sdk-rust/sdk/aws-runtime) (*)
│   │   ├── aws-smithy-async v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-async) (*)
│   │   ├── aws-smithy-http v0.60.10 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-http) (*)
│   │   ├── aws-smithy-json v0.60.7 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-json) (*)
│   │   ├── aws-smithy-runtime v1.7.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-runtime) (*)
│   │   ├── aws-smithy-runtime-api v1.7.2 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-runtime-api) (*)
│   │   ├── aws-smithy-types v1.2.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   │   ├── aws-types v1.3.3 (/root/rustprojects/aws-sdk-rust/sdk/aws-types) (*)
│   ├── aws-sdk-sts v1.40.0 (/root/rustprojects/aws-sdk-rust/sdk/sts)
│   │   ├── aws-credential-types v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-credential-types) (*)
│   │   ├── aws-runtime v1.4.2 (/root/rustprojects/aws-sdk-rust/sdk/aws-runtime) (*)
│   │   ├── aws-smithy-async v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-async) (*)
│   │   ├── aws-smithy-http v0.60.10 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-http) (*)
│   │   ├── aws-smithy-json v0.60.7 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-json) (*)
│   │   ├── aws-smithy-query v0.60.7 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-query)
│   │   │   ├── aws-smithy-types v1.2.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   │   ├── aws-smithy-runtime v1.7.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-runtime) (*)
│   │   ├── aws-smithy-runtime-api v1.7.2 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-runtime-api) (*)
│   │   ├── aws-smithy-types v1.2.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   │   ├── aws-smithy-xml v0.60.8 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-xml)
│   │   ├── aws-types v1.3.3 (/root/rustprojects/aws-sdk-rust/sdk/aws-types) (*)
│   ├── aws-smithy-async v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-async) (*)
│   ├── aws-smithy-http v0.60.10 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-http) (*)
│   ├── aws-smithy-json v0.60.7 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-json) (*)
│   ├── aws-smithy-runtime v1.7.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-runtime) (*)
│   ├── aws-smithy-runtime-api v1.7.2 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-runtime-api) (*)
│   ├── aws-smithy-types v1.2.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   ├── aws-types v1.3.3 (/root/rustprojects/aws-sdk-rust/sdk/aws-types) (*)
├── aws-credential-types v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-credential-types) (*)
├── aws-sdk-s3 v1.47.0 (/root/rustprojects/aws-sdk-rust/sdk/s3)
│   ├── aws-credential-types v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-credential-types) (*)
│   ├── aws-runtime v1.4.2 (/root/rustprojects/aws-sdk-rust/sdk/aws-runtime) (*)
│   ├── aws-sigv4 v1.2.3 (/root/rustprojects/aws-sdk-rust/sdk/aws-sigv4) (*)
│   ├── aws-smithy-async v1.2.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-async) (*)
│   ├── aws-smithy-checksums v0.60.12 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-checksums)
│   │   ├── aws-smithy-http v0.60.10 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-http) (*)
│   │   ├── aws-smithy-types v1.2.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   ├── aws-smithy-eventstream v0.60.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-eventstream) (*)
│   ├── aws-smithy-http v0.60.10 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-http) (*)
│   ├── aws-smithy-json v0.60.7 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-json) (*)
│   ├── aws-smithy-runtime v1.7.1 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-runtime) (*)
│   ├── aws-smithy-runtime-api v1.7.2 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-runtime-api) (*)
│   ├── aws-smithy-types v1.2.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   ├── aws-smithy-xml v0.60.8 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-xml) (*)
│   ├── aws-types v1.3.3 (/root/rustprojects/aws-sdk-rust/sdk/aws-types) (*)
├── aws-smithy-types v1.2.4 (/root/rustprojects/aws-sdk-rust/sdk/aws-smithy-types) (*)
├── aws-types v1.3.3 (/root/rustprojects/aws-sdk-rust/sdk/aws-types) (*)

Environment details (OS name and version, etc.)

ubuntu 22.04

Logs

No response

@jiashiwen jiashiwen added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Sep 6, 2024
@njaard
Copy link

njaard commented Sep 6, 2024

I'm using strois, which uses ureq, which uses rust's std DNS resolver, and this just started happening to me as well, but only if I do a bunch of requests simultaneously.

I don't have any /etc/hosts entries for AWS

@ysaito1001
Copy link
Collaborator

Thank you for reporting the issue. Did it start occurring after you upgraded aws-sdk-s3 (and others) to the versions you pasted above? And if so, what was the last working version of aws-sdk-s3?

@ysaito1001 ysaito1001 added response-requested Waiting on additional info and feedback. Will move to 'closing-soon' in 7 days. and removed needs-triage This issue or PR still needs to be triaged. labels Sep 6, 2024
@njaard
Copy link

njaard commented Sep 6, 2024

@ysaito1001 I'm not using aws-sdk-s3

Are you using cloudflare DNS (1.1.1.1)?

@ysaito1001
Copy link
Collaborator

@njaard, Hmm, if the same issue occurs with strois and a different HTTP client, the issue could be outside our SDK.

@ysaito1001
Copy link
Collaborator

@ysaito1001 I'm not using aws-sdk-s3

Are you using cloudflare DNS (1.1.1.1)?

No we are not, at least not that I know of.

@njaard
Copy link

njaard commented Sep 6, 2024

Yes, I think the similarity in our problems is a coincidence.

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to 'closing-soon' in 7 days. label Sep 6, 2024
@jiashiwen
Copy link
Author

The problem is caused by request style, change to path style to solve the problem

let s3_config_builder = aws_sdk_s3::config::Builder::from(&config).force_path_style(true);
let client = aws_sdk_s3::Client::from_conf(s3_config_builder.build());

thanks!

Copy link

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a bug.
Projects
None yet
Development

No branches or pull requests

3 participants