Skip to content

Commit 4ab33c0

Browse files
[xy] Include ec2_key_name in project's metadata.yaml template (mage-ai#869)
* [xy] Add ec2_key_name to project metadata.yaml template. * [xy] Update README.
1 parent 23afebe commit 4ab33c0

File tree

2 files changed

+3
-1
lines changed

2 files changed

+3
-1
lines changed

docs/kernels/README.md

-1
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,6 @@ Instructions for running PySpark kernel
1717
* Specify PySpark kernel related [metadata](#metadata) in project's metadata.yaml file
1818
* Launch a remote AWS EMR Spark cluster. Install mage_ai library in bootstrap actions. Make sure the EMR cluster is publicly accessible.
1919
* You can use the `create_emr.py` script under [scripts/spark](https://github.com/mage-ai/mage-ai/tree/master/scripts/spark) folder to launch a new EMR cluster. Example: `python3 create_cluster.py [project_path]`. Please make sure your AWS crendentials are provided in `~/.aws/credentials` file or environment variables (`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`) when executing the script.
20-
* After the EMR cluster is launched, please grant S3 read and write permissions of the bucket you want to access in your pipeline and the `remote_variabls_dir` bucket to the EMR cluster's IAM roles (`EMR_DefaultRole` and `EMR_EC2_DefaultRole`).
2120
* Connect to the remote spark cluster with command `ssh -i [path_to_key_pair] -L 0.0.0.0:9999:localhost:8998 [master_ec2_public_dns_name]`
2221
* `path_to_key_pair` is the path to the `ec2_key_pair_name` configured in `metadata.yaml` file
2322
* Find the `master_ec2_public_dns_name` in your newly created EMR cluster page under attribute `Master public DNS`

mage_ai/data_preparation/templates/repo/metadata.yaml

+3
Original file line numberDiff line numberDiff line change
@@ -6,3 +6,6 @@ emr_config:
66
slave_security_group: 'sg-yyyyyyyyyyyy'
77
master_instance_type: 'r5.4xlarge'
88
slave_instance_type: 'r5.4xlarge'
9+
# ec2_key_name must be configured during cluster launch to enable SSH access.
10+
# You can create a key pair in page https://console.aws.amazon.com/ec2#KeyPairs and download the key file.
11+
ec2_key_name: '[ec2_key_pair_name]'

0 commit comments

Comments
 (0)