NIFI-14956 - AWS Credentials Provider - Add support for STS AssumeRoleWithWebIdentity

[pvillard31]

Summary

NIFI-14956 - AWS Credentials Provider - Add support for STS AssumeRoleWithWebIdentity

Given the EOL of the SDK v1, I only implemented a version for the SDK v2

Steps to setup an AWS only test environment for testing this new feature.
(note the commands are using --profile perso, change base on your needs)

Setup variables

export REGION=eu-central-1
export POOL_NAME=nifi-oidc-demo
export USERNAME=test
export PASSWORD='NewStrongPassword#123'
export DOMAIN_PREFIX=nifi-demo-pv
export CONF_CLIENT_NAME=NiFi
export CALLBACK_URL=http://localhost:53682/callback
export LOGOUT_URL=http://localhost:53682/logout

Create Cognito User Pool

aws cognito-idp create-user-pool --region $REGION --pool-name "$POOL_NAME" --policies 'PasswordPolicy={MinimumLength=12,RequireUppercase=true,RequireLowercase=true,RequireNumbers=true,RequireSymbols=true}' --auto-verified-attributes email --profile perso


USER_POOL_ID=$(aws cognito-idp list-user-pools --region "$REGION" --max-results 60 --profile perso | jq -r --arg name "$POOL_NAME" '.UserPools[] | select(.Name==$name) | .Id')

Create Hosted UI Domain

aws cognito-idp create-user-pool-domain --region $REGION --user-pool-id $USER_POOL_ID --domain $DOMAIN_PREFIX --profile perso

Create Confidential App Client (with Auth Code Flow)

aws cognito-idp create-user-pool-client \
    --region $REGION \
    --user-pool-id $USER_POOL_ID \
    --client-name "$CONF_CLIENT_NAME" \
    --generate-secret \
    --allowed-o-auth-flows-user-pool-client \
    --allowed-o-auth-flows code \
    --supported-identity-providers COGNITO \
    --allowed-o-auth-scopes openid email profile \
    --callback-urls "$CALLBACK_URL" \
    --logout-urls "$LOGOUT_URL" \
    --explicit-auth-flows ALLOW_USER_PASSWORD_AUTH ALLOW_USER_SRP_AUTH ALLOW_REFRESH_TOKEN_AUTH \
    --profile perso

From the output, capture the below:

export CONF_CLIENT_ID=...
export CONF_CLIENT_SECRET=...

And:

export TOKEN_ENDPOINT=https://$DOMAIN_PREFIX.auth.$REGION.amazoncognito.com/oauth2/token
export ISSUER=https://cognito-idp.$REGION.amazonaws.com/$USER_POOL_ID

Create and confirm a user

aws cognito-idp admin-create-user --region $REGION --user-pool-id $USER_POOL_ID --username $USERNAME --temporary-password "$PASSWORD" --profile perso

aws cognito-idp admin-set-user-password --region $REGION --user-pool-id $USER_POOL_ID --username $USERNAME --password "$PASSWORD" --permanent --profile perso

Get a Refresh Token for the user

SECRET_HASH=$(printf '%s' "$USERNAME""$CONF_CLIENT_ID" | openssl dgst -sha256 -hmac "$CONF_CLIENT_SECRET" -binary | base64)

REFRESH_TOKEN=$(aws cognito-idp initiate-auth \
        --region $REGION \
        --client-id $CONF_CLIENT_ID \
        --auth-flow USER_PASSWORD_AUTH \
        --auth-parameters USERNAME=$USERNAME,PASSWORD="$PASSWORD",SECRET_HASH="$SECRET_HASH" \
        --profile perso \
        | jq -r .AuthenticationResult.RefreshToken)

Go to AWS IAM and Create Role

• Create Role
• Select Web Identity
• Select your Cognito pool
• Select the audience with the client ID of your app
• Next
• Select a policy, for example AmazonSQSFullAccess
• Next
• Give a name
• Create Role

Capture the Role ARN for the role you just created. Example:

export ROLE_ARN=arn:aws:iam::802551555016:role/nifi-test

Configure NiFi

• StandardOauth2AccessTokenProvider

  • Authorization Server URL: $TOKEN_ENDPOINT
  • Client Authentication Strategy: Basic Authentication
  • Grant Type: Refresh Token
  • Refresh Token: $REFRESH_TOKEN
  • Client ID: $CONF_CLIENT_ID
  • Client Secret: $CONF_CLIENT_SECRET
  • Scope: (leave empty)

• AWS Credentials Provider (Web Identity)

  • OAuth2 Access Token Provider: select provider above
  • Assume Role ARN: $ROLE_ARN
  • Assume Role Session Name: nifi-oidc-test
  • Assume Role STS Region: $REGION

• Use an AWS SDK v2 processor (e.g., PutSQS) with this credentials provider and test

Important notes

• STS requires aud; using refresh_token (or code) yields an id_token with aud that matches the app client ID.
• Only v2 implementation so cannot be used with S3 processors at the moment as they are still using SDK v1.

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

• Apache NiFi Jira issue created

Pull Request Tracking

• Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
• Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

• Pull Request based on current revision of the main branch
• Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

• Build completed using ./mvnw clean install -P contrib-check
  • JDK 21

Licensing

• New dependencies are compatible with the Apache License 2.0 according to the License Policy
• New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

• Documentation formatting appears as expected in rendered files

[exceptionfactory]

Thanks for proposing the additional feature for AWS authentication @pvillard31.

On initial review of the AssumeRoleWithWebIdentity along with the instructions provided, there appears to be a mismatch between the intended use case for this strategy and how it would be used. As described in the documentation, and as the name implies, the identity comes from an authenticated web or mobile application user. The setup steps describe provisioning a token, including a refresh token, with a process that normally involves authentication to a web application. Since NiFi does not make use of an authenticated user's credentials directly to drive authentication to other services, the WebIdentity strategy does not seem to align. Additionally, using a user's credentials for service-based authentication is not a good strategy in general.

If there are other use cases, they are worth considering, but I'm not initially inclined to move forward in this particular direction.

[pvillard31]

Thanks for the feedback @exceptionfactory. As I noted in the PR description, what you're pointing out is a pure limitation of using Cognito but I wanted to provide an example that is easy to reproduce for a reviewer willing to try the change in AWS. I would not expect users to actually use Cognito in this context.

• STS WebIdentity is not inherently end‑user only. AWS uses it for machine identities too (e.g., EKS IRSA). What STS requires is a valid OIDC JWT with an issuer it trusts and an aud that matches a registered client ID.
• Cognito client‑credentials returns an access token without aud, and Cognito does not let you override standard claims for access tokens. It also does not issue ID tokens for client‑credentials. So Cognito + client‑credentials cannot satisfy STS trust policy checks wrt aud.

I also tested this change with NiFi running on GCP:

• NiFi runs on GCP
• I created a dummy OAuth Token Provider to get a Google OIDC ID token using the metadata server of my GCE instance
• Created a Google OIDC provider in AWS and specify an audience such as sts.amazonaws.com
• Then create a role using WebIdentity referencing that identity provider and also specifying the audience (as well as the service account attached to my GCE instance)

As you can see this is way more involved for a reviewer and I did have to build a custom NAR for the token provider implementation in GCP to get the token.

For reference:
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-idp_oidc.html
https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html
https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html

The last link gives an example where STS AssumeRoleWithWebIdentity is used with a service account in the context of EKS with the below role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::$account_id:oidc-provider/$oidc_provider"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "$oidc_provider:aud": "sts.amazonaws.com",
          "$oidc_provider:sub": "system:serviceaccount:$namespace:$service_account"
        }
      }
    }
  ]
}

Let me know what you think.

[exceptionfactory]

Thanks for the helpful and substantive background on WebIdentity @pvillard31. The use case for platforms other than web application is clear now, and I see why the Cognito example was useful for testing purposes, if not applicable for general use cases.

I provided some feedback on the implementation details. Most of it seems clear, although I think some adjustment of property descriptors would help make it clear when this particular type of credentials provider is being configured.

[exceptionfactory]

@pvillard31 Following the merge of changes to remove AWS SDK 1, this pull request can be updated to remove SDK 1 references, then it looks close to completion.

[pvillard31]

Thanks @exceptionfactory, rebased against main, hopefully I didn't mess it up :)

[exceptionfactory]

Thanks @pvillard31, the latest version looks good! +1 merging