Skip to content

Commit 44028b0

Browse files
dy46dy46
authored
[dy] Staging/production GitHub Actions docs (mage-ai#2609)
* [dy] Add docs * [dy] Add docs for staging/production github actions * [dy] Separate to staging and production task
1 parent 634cf76 commit 44028b0

File tree

3 files changed

+232
-0
lines changed

3 files changed

+232
-0
lines changed

‎docs/mint.json

+1
Original file line numberDiff line numberDiff line change
@@ -323,6 +323,7 @@
323323
"pages": [
324324
"production/ci-cd/local-cloud/repository-setup",
325325
"production/ci-cd/local-cloud/github-actions",
326+
"production/ci-cd/staging-production/github-actions",
326327
"production/ci-cd/local-cloud/gitlab-ci-cd",
327328
"production/ci-cd/local-cloud/buildkite"
328329
]

‎docs/production/ci-cd/staging-production/github-actions.mdx

+102
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
---
2+
title: "GitHub Actions"
3+
description: "Development (local), staging (cloud), and production (cloud) using GitHub Actions"
4+
sidebarTitle: "Deploy with staging environment"
5+
---
6+
7+
## Mage project setup
8+
9+
Follow the [Mage project setup instructions](/production/ci-cd/local-cloud/repository-setup).
10+
11+
---
12+
13+
## GitHub Actions setup
14+
15+
1. Create a new repository on GitHub.
16+
2. Open your repository on GitHub, then click the tab labeled **Settings**.
17+
3. Click the section labeled **Secrets and variables** on the left hand side to expand it.
18+
4. Create separate staging and production GitHub environments in the **Environments** section.
19+
1. You can also choose to require approval before running jobs in your production environment.
20+
More information [here](https://docs.github.com/en/actions/deployment/targeting-different-environments/using-environments-for-deployment#required-reviewers).
21+
2. You may need to change the `environment` variable in the jobs section of the Github Actions below
22+
based on the name of your GitHub environments.
23+
5. Click the link labeled **Actions**.
24+
6. Click the button labeled **New repository secret** in the top right corner.
25+
7. Follow the instructions below for your specific cloud provider:
26+
27+
---
28+
29+
## AWS
30+
31+
1. If you haven’t already, create a new AWS ECR repository.
32+
2. You’ll need AWS credentials with the following policy permissions:
33+
```json
34+
{
35+
"Version": "2012-10-17",
36+
"Statement": [
37+
{
38+
"Effect": "Allow",
39+
"Action": [
40+
"ecr:BatchCheckLayerAvailability",
41+
"ecr:CompleteLayerUpload",
42+
"ecr:GetAuthorizationToken",
43+
"ecr:InitiateLayerUpload",
44+
"ecr:PutImage",
45+
"ecr:UploadLayerPart",
46+
"ecs:DeregisterTaskDefinition",
47+
"ecs:DescribeClusters",
48+
"ecs:DescribeServices",
49+
"ecs:DescribeTaskDefinition",
50+
"ecs:RegisterTaskDefinition",
51+
"ecs:UpdateService",
52+
"iam:PassRole"
53+
],
54+
"Resource": "*"
55+
}
56+
]
57+
}
58+
```
59+
3. In the field labeled **Name**, enter the value `AWS_ACCESS_KEY_ID`.
60+
4. In the field labeled **Secret**, enter your AWS Access Key ID.
61+
5. Click the button labeled **Add secret** to save.
62+
6. Add a 2nd secret by clicking the button labeled **New repository secret** in
63+
the top right corner.
64+
7. In the field labeled **Name**, enter the value `AWS_SECRET_ACCESS_KEY`.
65+
8. In the field labeled **Secret**, enter your AWS Secret Access Key.
66+
9. Click the button labeled **Add secret** to save.
67+
10. Click on the tab labeled **Actions**.
68+
11. On the left side, click the button labeled **New workflow**.
69+
12. Find the link labeled **`set up a workflow yourself`** and click it.
70+
13. Copy the contents from the GitHub Action YAML file for AWS at
71+
[templates/github_actions/build_and_deploy_to_aws_ecs_staging_production.yml](https://github.com/mage-ai/mage-ai/blob/master/templates/github_actions/build_and_deploy_to_aws_ecs_staging_production.yml),
72+
and paste it into the textarea.
73+
14. Change the following values under the key labeled `env`:
74+
75+
```yaml
76+
env:
77+
AWS_REGION: ...
78+
CONTAINER_NAME: ...
79+
ECR_REPOSITORY: ...
80+
ECS_CLUSTER: ...
81+
ECS_STAGING_SERVICE: ...
82+
ECS_PRODUCTION_SERVICE: ...
83+
ECS_STAGING_TASK_DEFINITION: ...
84+
ECS_PRODUCTION_TASK_DEFINITION: ...
85+
```
86+
87+
| Key | Description | Sample value |
88+
| --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ |
89+
| `AWS_REGION` | Region of your AWS ECS cluster. | `us-west-2` |
90+
| `CONTAINER_NAME` | Set this to the name of the container in the containerDefinitions section of your task definition. | `mage-data-production-container` |
91+
| `ECR_REPOSITORY` | The name of the AWS ECR repository you created to store your Docker images. | `mage-data` |
92+
| `ECS_CLUSTER` | The name of your AWS ECS cluster. | `mage-production-cluster` |
93+
| `ECS_STAGING_SERVICE` | The name of your AWS ECS staging service. | `mage-production-cluster` |
94+
| `ECS_PRODUCTION_SERVICE` | The name of your AWS ECS production service. | `mage-production-ecs-service` |
95+
| `ECS_STAGING_TASK_DEFINITION` | Go to your AWS ECS task definition for the staging service. Click on the **JSON** tab on the task definition detail page. Copy the JSON string content and save it to a file in your root folder containing your Mage project. Use the path to that file as the value in this field. | `some_path/ecs-task-definition.json` |
96+
| `ECS_PRODUCTION_TASK_DEFINITION` | Go to your AWS ECS task definition for the production service. Click on the **JSON** tab on the task definition detail page. Copy the JSON string content and save it to a file in your root folder containing your Mage project. Use the path to that file as the value in this field. | `some_path/ecs-task-definition.json` |
97+
98+
1. Click the button labeled **Start commit** in the top right corner.
99+
1. Click the button labeled **Commit new file**.
100+
1. Every time you merge a pull request into the master branch, this GitHub
101+
Action will run, building a Docker image using your GitHub code, then
102+
updating AWS ECS to use the new image with the updated code.

‎templates/github_actions/build_and_deploy_to_aws_ecs_staging_production.yml

+129
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
name: Deploy to Amazon ECS staging and production
2+
3+
on:
4+
push:
5+
branches:
6+
- master
7+
8+
env:
9+
AWS_REGION: MY_AWS_REGION # set this to your preferred AWS region, e.g. us-west-1
10+
ECR_REPOSITORY: MY_ECR_REPOSITORY # set this to your Amazon ECR repository name
11+
ECS_STAGING_SERVICE: MY_STAGING_ECS_SERVICE # set this to your Amazon ECS staging service name
12+
ECS_PRODUCTION_SERVICE: MY_PROD_ECS_SERVICE # set this to your Amazon ECS production service name
13+
ECS_CLUSTER: MY_ECS_CLUSTER # set this to your Amazon ECS cluster name
14+
ECS_STAGING_TASK_DEFINITION: MY_ECS_TASK_DEFINITION # set this to the path to your Amazon ECS staging task definition
15+
# file, e.g. .aws/task-definition.json
16+
ECS_PRODUCTION_TASK_DEFINITION: MY_ECS_TASK_DEFINITION # set this to the path to your Amazon ECS production task definition
17+
# file, e.g. .aws/task-definition.json
18+
CONTAINER_NAME: MY_CONTAINER_NAME # set this to the name of the container in the
19+
# containerDefinitions section of your task definition
20+
21+
jobs:
22+
push-to-ecr:
23+
name: Push image to AWS ECR
24+
runs-on: ubuntu-latest
25+
26+
outputs:
27+
image: ${{ steps.build-image.outputs.image }}
28+
29+
steps:
30+
- name: Checkout
31+
uses: actions/checkout@v3
32+
33+
- name: Configure AWS credentials
34+
uses: aws-actions/configure-aws-credentials@13d241b293754004c80624b5567555c4a39ffbe3
35+
with:
36+
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
37+
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
38+
aws-region: ${{ env.AWS_REGION }}
39+
mask-aws-account-id: 'no'
40+
41+
- name: Login to Amazon ECR
42+
id: login-ecr
43+
uses: aws-actions/amazon-ecr-login@aaf69d68aa3fb14c1d5a6be9ac61fe15b48453a2
44+
45+
- name: Build, tag, and push image to Amazon ECR
46+
id: build-image
47+
env:
48+
ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
49+
IMAGE_TAG: ${{ github.sha }}
50+
run: |
51+
# Build a docker container and
52+
# push it to ECR so that it can
53+
# be deployed to ECS.
54+
docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .
55+
docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
56+
echo "::set-output name=image::$ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG"
57+
58+
deploy-staging:
59+
name: Deploy staging
60+
runs-on: ubuntu-latest
61+
environment: staging
62+
needs: push-to-ecr
63+
64+
outputs:
65+
image: ${{ steps.build-image.outputs.image }}
66+
67+
steps:
68+
- name: Configure AWS credentials
69+
uses: aws-actions/configure-aws-credentials@13d241b293754004c80624b5567555c4a39ffbe3
70+
with:
71+
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
72+
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
73+
aws-region: ${{ env.AWS_REGION }}
74+
75+
- name: Download task definition
76+
run: |
77+
aws ecs describe-task-definition --task-definition ${{ env.ECS_STAGING_TASK_DEFINITION }} \
78+
--query taskDefinition > task-definition.json
79+
80+
- name: Fill in the new image ID in the Amazon ECS task definition
81+
id: task-def
82+
uses: aws-actions/amazon-ecs-render-task-definition@v1
83+
with:
84+
task-definition: task-definition.json
85+
container-name: ${{ env.CONTAINER_NAME }}
86+
image: ${{ needs.push-to-ecr.outputs.image }}
87+
88+
- name: Deploy Amazon ECS staging task definition
89+
uses: aws-actions/amazon-ecs-deploy-task-definition@v1
90+
with:
91+
task-definition: ${{ steps.task-def.outputs.task-definition }}
92+
service: ${{ env.ECS_STAGING_SERVICE }}
93+
cluster: ${{ env.ECS_CLUSTER }}
94+
wait-for-service-stability: true
95+
96+
deploy-production:
97+
name: Deploy production
98+
runs-on: ubuntu-latest
99+
environment: production
100+
needs: [push-to-ecr, deploy-staging]
101+
102+
steps:
103+
- name: Configure AWS credentials
104+
uses: aws-actions/configure-aws-credentials@13d241b293754004c80624b5567555c4a39ffbe3
105+
with:
106+
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
107+
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
108+
aws-region: ${{ env.AWS_REGION }}
109+
110+
- name: Download task definition
111+
run: |
112+
aws ecs describe-task-definition --task-definition ${{ env.ECS_PRODUCTION_TASK_DEFINITION }} \
113+
--query taskDefinition > task-definition.json
114+
115+
- name: Fill in the new image ID in the Amazon ECS task definition
116+
id: task-def
117+
uses: aws-actions/amazon-ecs-render-task-definition@v1
118+
with:
119+
task-definition: task-definition.json
120+
container-name: ${{ env.CONTAINER_NAME }}
121+
image: ${{ needs.push-to-ecr.outputs.image }}
122+
123+
- name: Deploy Amazon ECS production task definition
124+
uses: aws-actions/amazon-ecs-deploy-task-definition@v1
125+
with:
126+
task-definition: ${{ steps.task-def.outputs.task-definition }}
127+
service: ${{ env.ECS_PRODUCTION_SERVICE }}
128+
cluster: ${{ env.ECS_CLUSTER }}
129+
wait-for-service-stability: true

0 commit comments

Comments
 (0)