Skip to content

[FLINK-37683][Connectors/AWS] Enable SSO credentials #197

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 9 commits into
base: main
Choose a base branch
from
Open

[FLINK-37683][Connectors/AWS] Enable SSO credentials #197

Changes from 6 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
2aad3fb
add sso and ssooidc dependencies to the base pom, similar to sts
soisyourface Apr 7, 2025
6a3fc51
Merge pull request #1 from soisyourface/fix_kinesis_sso_credentials
soisyourface Apr 9, 2025
65ac913
fix: Remove unsupported 'allowUndeclaredRTE' property from Checkstyle…
soisyourface Apr 16, 2025
60d4bfc
add a test... but i think an e2e test is really the only thing to tes…
soisyourface Apr 17, 2025
f19f84a
Merge branch 'apache:main' into main
soisyourface Apr 17, 2025
e516173
whoops, removed this to try and fix an error
soisyourface Apr 17, 2025
6d1ab91
Merge branch 'main' of github.com:soisyourface/flink-connector-aws
soisyourface Apr 17, 2025
c987345
grr, IDEA added this plugin...
soisyourface Apr 17, 2025
f9c8ed4
rename the test
soisyourface Apr 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions ...nector-aws-base/src/test/java/org/apache/flink/connector/aws/util/AWSGeneralUtilTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
import software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider;
import software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider;
import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
import software.amazon.awssdk.auth.credentials.ProfileProviderCredentialsContext;
import software.amazon.awssdk.auth.credentials.SystemPropertyCredentialsProvider;
import software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider;
import software.amazon.awssdk.core.exception.SdkClientException;
Expand All @@ -34,11 +35,16 @@
import software.amazon.awssdk.http.async.SdkAsyncHttpClient;
import software.amazon.awssdk.http.nio.netty.NettyNioAsyncHttpClient;
import software.amazon.awssdk.http.nio.netty.internal.NettyConfiguration;
import software.amazon.awssdk.profiles.Profile;
import software.amazon.awssdk.profiles.ProfileFile;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.sso.auth.SsoProfileCredentialsProviderFactory;
import software.amazon.awssdk.services.sts.auth.StsAssumeRoleCredentialsProvider;
import software.amazon.awssdk.utils.AttributeMap;
import software.amazon.awssdk.utils.ImmutableMap;
import software.amazon.awssdk.utils.StringInputStream;

import java.io.IOException;
import java.net.URI;
import java.nio.file.Paths;
import java.time.Duration;
Expand Down Expand Up @@ -952,4 +958,41 @@ void testNewInstanceOfDefaultCredentialsProviderCreatedForEachClient() {

return builder;
}

@Test
void testGetCredentialsProviderReturnsSsoProviderForSsoProfile() throws IOException {
Copy link
Contributor

@leekeiabstraction leekeiabstraction Apr 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test case name indicates that we're expecting SsoProvider to be returned but actual assertion is on UncheckedIOException being thrown. Can you elaborate?

Copy link
Author

@soisyourface soisyourface Apr 25, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

likely a leftover name from a previous iteration of writing the tests.
renamed.

the bigger question is: does this test even do anything?

String profileName = "my-sso-profile";
String ssoStartUrl = "https://my-dummy-sso-portal.awsapps.com/start";
String ssoRegion = "us-east-1";
String ssoAccountId = "123456789012";
String ssoRoleName = "MyTestRole";

String configFileContent =
"[profile " + profileName + "]\n"
+ "sso_start_url = " + ssoStartUrl + "\n"
+ "sso_region = " + ssoRegion + "\n"
+ "sso_account_id = " + ssoAccountId + "\n"
+ "sso_role_name = " + ssoRoleName + "\n"
+ "region = " + ssoRegion + "\n";

ProfileFile profileFile = ProfileFile.builder()
.content(new StringInputStream(configFileContent))
.type(ProfileFile.Type.CONFIGURATION)
.build();

Profile profile = profileFile.profile(profileName).orElseThrow();

ProfileProviderCredentialsContext context = ProfileProviderCredentialsContext.builder()
.profile(profile)
.profileFile(profileFile)
.build();

SsoProfileCredentialsProviderFactory factory = new SsoProfileCredentialsProviderFactory();

assertThatThrownBy(() -> factory.create(context))
.isInstanceOf(java.io.UncheckedIOException.class)
.hasRootCauseInstanceOf(java.nio.file.NoSuchFileException.class)
.hasMessageContaining(".aws/sso/cache/");
}
Comment on lines +991 to +995
Copy link
Contributor

@leekeiabstraction leekeiabstraction Apr 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you elaborate why we're asserting that an exception is thrown here? What would the full exception message look like?

Copy link
Author

@soisyourface soisyourface Apr 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here is the full exception message:

[ERROR] org.apache.flink.connector.aws.util.AWSGeneralUtilTest.testGetCredentialsProviderReturnsSsoProviderForSsoProfile  Time elapsed: 0.925 s  <<< ERROR!
java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /home/me/.aws/sso/cache/b6c9aa45a3f785e14b3cdb559fe03c92b9a2fb00.json
        at software.amazon.awssdk.services.sso.internal.SsoAccessTokenProvider.tokenFromFile(SsoAccessTokenProvider.java:58)
        at software.amazon.awssdk.services.sso.internal.SsoAccessTokenProvider.resolveToken(SsoAccessTokenProvider.java:51)
        at software.amazon.awssdk.services.sso.auth.SsoProfileCredentialsProviderFactory$SsoProfileCredentialsProvider.<init>(SsoProfileCredentialsProviderFactory.java:107)
        at software.amazon.awssdk.services.sso.auth.SsoProfileCredentialsProviderFactory$SsoProfileCredentialsProvider.<init>(SsoProfileCredentialsProviderFactory.java:88)
        at software.amazon.awssdk.services.sso.auth.SsoProfileCredentialsProviderFactory.create(SsoProfileCredentialsProviderFactory.java:68)
        at org.apache.flink.connector.aws.util.AWSGeneralUtilTest.testGetCredentialsProviderReturnsSsoProviderForSsoProfile(AWSGeneralUtilTest.java:991)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
        at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
        at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
        at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
        at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
        at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
        at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
        at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
        at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
        at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
        at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
        at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
        at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
        at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
        at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214)
        at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210)
        at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
        at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
        at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
        at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
        at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
        at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
        at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
        at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
        at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
        at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
        at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
        at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
        at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
        at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
        at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
        at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
        at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
        at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
        at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
        at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
        at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
        at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
        at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
        at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
        at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
        at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.lambda$execute$1(JUnitPlatformProvider.java:199)
        at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
        at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.execute(JUnitPlatformProvider.java:193)
        at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:154)
        at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:120)
        at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
        at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
        at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)
Caused by: java.nio.file.NoSuchFileException: /home/me/.aws/sso/cache/b6c9aa45a3f785e14b3cdb559fe03c92b9a2fb00.json
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:261)
        at java.base/java.nio.file.Files.newByteChannel(Files.java:380)
        at java.base/java.nio.file.Files.newByteChannel(Files.java:432)
        at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420)
        at java.base/java.nio.file.Files.newInputStream(Files.java:160)
        at software.amazon.awssdk.services.sso.internal.SsoAccessTokenProvider.tokenFromFile(SsoAccessTokenProvider.java:55)
        ... 75 more

in this case the exception is being thrown because there is no cache with the credentials in it. we can always just remove the check but this essentially means that if it makes it this far then the sso/ssooidc class loader stuff worked because the dependencies were found.

at least that was my initial thinking. but i really don't know if these tests are valuable, as mentioned in a previous comment.


}