Replies: 1 comment
-
@rankol I would suggest two ways Option 1 Quick: airflow/airflow/providers/amazon/aws/operators/emr.py Lines 111 to 113 in 648e224 Option 2: |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I've attempted to create this as a feature request but when I submit the Issue I get re-directed to a webpage that says
This site can't be reached
so I'm just posting this here for now.The EmrAddStepsOperator does not have an input for Execution role ARNs.
airflow/airflow/providers/amazon/aws/operators/emr.py
Line 34 in bf14d14
The Boto3 EMR client does have an option for this input.
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/emr.html#EMR.Client.add_job_flow_steps
EMR version 6.7 recently released which allows for Execution role ARNs to be used with EMR job steps. This seems to be done at the CLI with
--execution-role-arn = $RUNTIME_ROLE_ARN
https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-steps-runtime-roles.html
This would allow for EMR jobs to use an execution role for IAM permissions instead of using the EC2 instance profile attached to the cluster for permissions. Other EMR operators like the container operator or serverless operator allow for execution role ARNs to be passed in.
Beta Was this translation helpful? Give feedback.
All reactions