Add support for version 2.1 of the IMDS credentials provider #4109

ysaito1001 · 2025-04-30T18:51:27Z

DO NOT MERGE THIS PR UNTIL READY TO RELEASE

Description

This PR adds support for version 2.1 of the IMDS credentials provider. With this update, if the underlying EC2 instance's IMDS supports it, the provider can now retrieve credentials that include an account ID. This account ID can then be used by account-based endpoints.

To support this functionality, the credentials provider first attempts to access the extended API endpoint, which ends with -extended (e.g., http://169.254.169.254/latest/meta-data/iam/security-credentials-extended). If this endpoint returns a 404, the provider falls back to the legacy API endpoint—the same one used in the IMDS credentials provider v2.0.

This "try extended API, then fall back to legacy API" pattern is applied to both retrieving the IMDS instance profile name and fetching credentials. Importantly:

Once the provider determines that a specific API version (extended or legacy) is supported, it will stick to that version.
Only the extended API can return credentials that include an account ID.

The PR also made the following IMDS credentials providers options configurable:

Disable IMDS credentials fetching
IMDS instance profile name

Both options can be configured via environment variables or a shared config file, with environment variables taking precedence over the config file.

Note: Feature tracking for this functionality will be handled in a separate PR.

Testing

Updated connection recordings in aws-config/test-data/ to reflect changes in the execution flow within .retrieve_credentials (Note: This update is not intended to add new test coverage, but to align recordings with the updated flow.)
Added unit tests to verify the behavior of the extended API, including falling back to the legacy API when necessary.
Added unit tests to verify the configuration options for disabling IMDS credentials fetching and specifying the IMDS instance profile name.

Checklist

For changes to the AWS SDK, generated SDK code, or SDK runtime crates, I have created a changelog entry Markdown file in the .changelog directory, specifying "aws-sdk-rust" in the applies_to key.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

github-actions · 2025-04-30T19:12:20Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)
No codegen difference in the Client Test
No codegen difference in the Server Test
No codegen difference in the Server Test Python
No codegen difference in the Server Test Typescript

A new doc preview is ready to view.

aajtodd · 2025-04-30T20:25:08Z

...me/aws-config/test-data/default-credential-provider-chain/imds_assume_role/http-traffic.json

@@ -89,7 +89,7 @@
            "action": {
                "Request": {
                    "request": {
-                        "uri": "http://169.254.169.254/latest/meta-data/iam/security-credentials/",
+                        "uri": "http://169.254.169.254/latest/meta-data/iam/security-credentials-extended/",


So this test is updated to hit extended but the response isn't updated?

Ah, good point, not strictly necessary since extended API can still return credentials without account ID. Admittedly, this is what happens when we manually tweak connection recording files for a feature that hasn't been supported in the real world.

aajtodd · 2025-04-30T20:35:06Z

aws/rust-runtime/aws-config/src/imds/credentials.rs

+        {
+            Ok(credentials) => {
+                let state = &mut self.provider_state.write().unwrap();
+                state.resolved_profile = Some(profile.to_string());


isn't this set in resolve_profile_name() already?

aajtodd · 2025-04-30T20:36:27Z

aws/rust-runtime/aws-config/src/imds/credentials.rs

+                Ok(credentials)
+            }
+            Err(ImdsError::ErrorResponse(raw)) if raw.response().status().as_u16() == 404 => {
+                // A block is required to drop the `RwLockWriteGuard`,


I don't think you need this or the other similar comments like it, this would be self evident to anyone trying to remove the block.

aajtodd · 2025-04-30T20:41:41Z

aws/rust-runtime/aws-config/src/imds/credentials.rs

+    }
+
+    #[tokio::test]
+    async fn should_return_valid_credentials_when_profile_is_unstable() {


(comment unrelated to this particular code I chose it at random): The SEP defines a bunch of test cases are we covering all of them? Is there anyway to turn the JSON provided into test cases such that new test cases or changes to the SEP are easy? I don't care how we do it but some level of traceability to the SEP would be good here.

Also unrelated, but related to SEP JSON tests in general. We don't really seem to use them, when I have implemented a SEP with tests I've had translated them into Rust/Kotlin because that seemed to be the standard, but it would be nice if we could start using the JSON tests directly where possible.

I don't care if we use the JSON directly or not (though in this case the JSON seems workable) but at the very least translating them such that we can match test cases seems desirable. SEP test cases probably ought to be required to have an identifier 🤔

Yes, unit tests have covered all the test cases except for ones I deemed redundant from an equivalence testing point of view, which test scenarios where the extended API returns credentials without account ID; I only included one test case for that using profile-name my-profile-0005.

SEP test cases probably ought to be required to have an identifier

Agreed. The closest thing to the test case ID in the SEP is a profile name, e.g. my-profile-0001. If we used JSON directly, that would be the best traceability. We sort of have the test infrastructure as shown under aws-config/test-data. They work well with connection recording, but for the feature like this that hasn't been supported in the real world we cannot generate recording, so it's easier to write the tests in unit tests.

Is there anyway to turn the JSON provided into test cases such that new test cases or changes to the SEP are easy?

That's invent and simplify :) Requires some level of design for the to-be-built infrastructure to be useful not just in one SEP, but in many others in the future.

landonxjames

One question about the RwLock, but otherwise LGTM!

landonxjames · 2025-04-30T21:43:46Z

aws/rust-runtime/aws-config/src/imds/credentials.rs

+
+        match self.client.get(self.uri_base()).await {
+            Ok(profile) => {
+                let state = &mut self.provider_state.write().unwrap();


.read() and .write() on a RwLock can panic, should we use .try_read() and .try_write() instead or do you not expect it to be an issue in practice?

Good question, I don't expect it to be an issue in practice. read and write can panic upon unwrapping a LockResult whenever a writer panics while holding an exclusive lock. If we look at write critical sections in our code (where we call self.provider_state.write().), they either update the field of ProviderState or return a value from the method. Therefore, the lock will not be poisoned.

In terms of .read vs. .try_read (or .write vs. try_write), the difference is whether the caller blocks or not (note that the notion of a poisoned lock equally exists for .try_* variants). I think we can use blocking ones because

read critical sections and write critical sections are short (reading a field value or assigning a new value to the field), which is necessarily so since we are already in an async context and don't want to block for a long time anyway.

If try_* variants returns Err, we need to loop back and try acquiring the lock again, which I assume is better handled by blocking variants.

At least I switched to .expect with explanatory comment in bff4f26.

landonxjames · 2025-04-30T21:48:51Z

aws/rust-runtime/aws-config/src/imds/credentials.rs

+    }
+
+    #[tokio::test]
+    async fn should_return_valid_credentials_when_profile_is_unstable() {


Also unrelated, but related to SEP JSON tests in general. We don't really seem to use them, when I have implemented a SEP with tests I've had translated them into Rust/Kotlin because that seemed to be the standard, but it would be nice if we could start using the JSON tests directly where possible.

This commit addresses #4109 (comment)

github-actions · 2025-05-01T19:27:01Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)
No codegen difference in the Client Test
No codegen difference in the Server Test
No codegen difference in the Server Test Python
No codegen difference in the Server Test Typescript

A new doc preview is ready to view.

ysaito1001 added 2 commits April 30, 2025 12:45

Add support for IMDS v2.1

8fcfd16

Update connection recording tests

fd51389

aajtodd reviewed Apr 30, 2025

View reviewed changes

landonxjames approved these changes Apr 30, 2025

View reviewed changes

ysaito1001 added 2 commits May 1, 2025 13:24

Remove redundant comments that are self-explanatory

fcdc1b8

This commit addresses #4109 (comment)

Use .expect instead of unwrap when accessing LockResult

bff4f26

This commit addresses #4109 (comment)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for version 2.1 of the IMDS credentials provider #4109

Add support for version 2.1 of the IMDS credentials provider #4109

ysaito1001 commented Apr 30, 2025 •

edited

Loading

github-actions bot commented Apr 30, 2025

aajtodd Apr 30, 2025

ysaito1001 May 1, 2025

aajtodd Apr 30, 2025

aajtodd Apr 30, 2025

aajtodd Apr 30, 2025

landonxjames Apr 30, 2025

aajtodd May 1, 2025

ysaito1001 May 1, 2025 •

edited

Loading

landonxjames left a comment

landonxjames Apr 30, 2025

ysaito1001 May 1, 2025

landonxjames Apr 30, 2025

github-actions bot commented May 1, 2025

Add support for version 2.1 of the IMDS credentials provider #4109

Are you sure you want to change the base?

Add support for version 2.1 of the IMDS credentials provider #4109

Conversation

ysaito1001 commented Apr 30, 2025 • edited Loading

Description

Testing

Checklist

github-actions bot commented Apr 30, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ysaito1001 May 1, 2025 • edited Loading

Choose a reason for hiding this comment

landonxjames left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented May 1, 2025

ysaito1001 commented Apr 30, 2025 •

edited

Loading

ysaito1001 May 1, 2025 •

edited

Loading