feat: blue green 무중단 배포 (#746)

* feat: 무중단 배포 롤링 3대 * fix: health check 30초 딜레이 추가 * fix: 종료된 컨테이너도 이름 변경하도록 수정 * fix: 종료되어야 하는 컨테이너가 남아 있는 경우에 삭제하는 명령어 추가 * fix: 잘못된 개행 삭제 * feat: blue green 무중단 배포 스크립트 * feat: blue green 무중단 배포 스크립트 * feat: blue green 무중단 배포 스크립트 * feat: blue green 무중단 배포 스크립트 * feat: blue green 무중단 배포 스크립트 * feat: blue green 무중단 배포 스크립트 * feat: blue green 무중단 배포 스크립트 * feat: blue green 무중단 배포 스크립트 * feat: blue green 무중단 배포 스크립트 * feat: blue green 무중단 배포 스크립트 * feat: blue green 무중단 배포 스크립트 * feat: blue green 무중단 배포 스크립트 * feat: blue green 무중단 배포 스크립트 * feat: blue green 무중단 배포 스크립트 * feat: blue green 무중단 배포 스크립트 * feat: blue green 무중단 배포 스크립트 * feat: blue green 무중단 배포 스크립트 * feat: blue green 무중단 배포 스크립트 * feat: blue green 무중단 배포 스크립트 * feat: blue green 무중단 배포 스크립트 * feat: blue green 무중단 배포 스크립트 * feat: blue green 무중단 배포 스크립트 * feat: blue green 무중단 배포 스크립트 * feat: blue green 무중단 배포 스크립트 * feat: blue green 무중단 배포 스크립트 * feat: blue green 무중단 배포 스크립트 * feat: blue green 무중단 배포 스크립트 * feat: blue green 무중단 배포 스크립트 * feat: blue green 무중단 배포 스크립트 * feat: blue green 무중단 배포 스크립트 * feat: blue green 무중단 배포 스크립트 * feat: blue green 무중단 배포 스크립트 * feat: blue green 무중단 배포 스크립트 * feat: blue green 무중단 배포 스크립트 * feat: blue green 무중단 배포 스크립트 * feat: nginx 파일 용량 변경 * feat: nginx 파일 용량 변경 * feat: nginx renuseport * feat: nginx renuseport * feat: nginx renuseport * feat: nginx renuseport * feat: 테스트 브랜치 제거 * feat: prometheus 9100 포트 활성화 * test: 테스트 브랜치 추가 * test: 9100 포트 포워딩 * test: 9100 포트 포워딩 * test: 9100 포트 포워딩 * test: 9100 포트 포워딩 * feat: branch 트리거 삭제 --------- Co-authored-by: 3juhwan <[email protected]>
woowacourse-teams · Oct 23, 2024 · a989389 · a989389
1 parent f1ddcbd
commit a989389
Show file tree

Hide file tree

Showing 3 changed files with 195 additions and 10 deletions.
diff --git a/.github/workflows/backend-prod.yml b/.github/workflows/backend-prod.yml
@@ -53,26 +53,202 @@ jobs:
       - name: Build and push
         run: |
           docker buildx build --platform linux/arm64 -t \
-          	${{ secrets.DOCKER_USERNAME }}/${{ secrets.DOCKER_IMAGE_BE_PROD }} --push .
+              ${{ secrets.DOCKER_USERNAME }}/${{ secrets.DOCKER_IMAGE_BE_PROD }} --push .
 
   deploy:
     needs: build
     strategy:
       matrix:
-        runner: [ prod-1, prod-2 ]
+        runner: [ prod-1 , prod-2 ]
     runs-on: [ self-hosted, '${{ matrix.runner }}' ]
     steps:
-      - name: Docker remove
+      - name: Create Docker network if not exists
         run: |
-          CONTAINER_IDS=$(sudo docker ps -qa)
-          if [ -n "$CONTAINER_IDS" ]; then
-            sudo docker rm -f $CONTAINER_IDS
+          NETWORK_NAME="haengdong_network"
+          if ! sudo docker network ls --format '{{.Name}}' | grep -w "$NETWORK_NAME" > /dev/null 2>&1; then
+            echo "Creating Docker network $NETWORK_NAME..."
+            sudo docker network create $NETWORK_NAME
           else
-            echo "No running containers found."
+            echo "Docker network $NETWORK_NAME already exists."
           fi
 
       - name: Docker Image pull
         run: sudo docker pull ${{ secrets.DOCKER_USERNAME }}/${{ secrets.DOCKER_IMAGE_BE_PROD }}
 
-      - name: Docker run
-        run: sudo docker run -d -p 80:8080 -e SPRING_PROFILES_ACTIVE=prod -v log-volume:/app/logs --name haengdong-backend ${{ secrets.DOCKER_USERNAME }}/${{ secrets.DOCKER_IMAGE_BE_PROD }}
+      - name: Check running container and determine the next port
+        id: check-port
+        run: |
+          echo "Checking if any container is running on ports 8080 or 8081..."
+          CURRENT_PORT=$(sudo docker ps --format "{{.Names}}" | grep "haengdong-backend" | grep -o '8080\|8081' || echo "")
+          echo "CURRENT_PORT on port $CURRENT_PORT."
+          # CURRENT_PORT가 빈 문자열이거나 8081인 경우
+          if [ "$CURRENT_PORT" == "8081" ]; then
+            echo "IF CURRENT_PORT : null or 8081"
+            NEXT_PORT=8080
+          else
+            echo "IF CURRENT_PORT : 8080"
+            NEXT_PORT=8081
+          fi
+
+          echo "Next container will run on port $NEXT_PORT."
+          echo "NEXT_PORT=$NEXT_PORT" >> $GITHUB_ENV
+          echo "CURRENT_PORT=$CURRENT_PORT" >> $GITHUB_ENV
+
+      - name: Run new container on the alternate port
+        run: |
+          NETWORK_NAME="haengdong_network"
+          echo "Running new container on port $NEXT_PORT..."
+          sudo docker run -d \
+            --name haengdong-backend-$NEXT_PORT \
+            --network $NETWORK_NAME \
+            -p $NEXT_PORT:8080 \
+            -e SPRING_PROFILES_ACTIVE=prod \
+            -v log-volume:/app/logs \
+            ${{ secrets.DOCKER_USERNAME }}/${{ secrets.DOCKER_IMAGE_BE_PROD }}
+
+      - name: Health check the new container
+        run: |
+          echo "Performing health check for the new container on port $NEXT_PORT..."
+          MAX_ATTEMPTS=30
+          SLEEP_INTERVAL=2
+          ATTEMPT=1
+      
+          while [ $ATTEMPT -le $MAX_ATTEMPTS ]; do
+            HEALTH_STATUS=$(curl -s http://localhost:$NEXT_PORT/actuator/health | sed -n 's/.*"status":"\([^"]*\)".*/\1/p')
+
+            if [ "$HEALTH_STATUS" = "UP" ]; then
+              echo "Health check passed on attempt $ATTEMPT."
+              break
+            else
+              echo "Attempt $ATTEMPT: Health check status: $HEALTH_STATUS. Retrying in $SLEEP_INTERVAL seconds..."
+              ATTEMPT=$((ATTEMPT+1))
+              sleep $SLEEP_INTERVAL
+            fi
+
+            if [ $ATTEMPT -gt $MAX_ATTEMPTS ]; then
+              echo "Health check failed after $MAX_ATTEMPTS attempts. Rolling back..."
+              sudo docker rm -f haengdong-backend-$NEXT_PORT
+              exit 1
+            fi
+          done
+
+      - name: Update or create Nginx container to point to new container port
+        run: |
+          NGINX_CONTAINER_NAME="nginx-proxy"
+          NETWORK_NAME="haengdong_network"
+          NEXT_PORT=$NEXT_PORT
+
+          echo "Next port is $NEXT_PORT"
+
+          echo "Checking if Nginx container exists..."
+          if sudo docker ps -a --filter "name=$NGINX_CONTAINER_NAME" --format "{{.Names}}" | grep -w $NGINX_CONTAINER_NAME; then
+
+            # Nginx 컨테이너가 실행 중인지 확인
+            if sudo docker ps --filter "name=$NGINX_CONTAINER_NAME" --format "{{.Names}}" | grep -w $NGINX_CONTAINER_NAME; then
+              echo "Nginx container is running."
+            else
+              echo "Nginx container exists but is not running. Starting Nginx container..."
+              sudo docker start $NGINX_CONTAINER_NAME
+              echo "Nginx container started."
+            fi
+
+            # nginx-conf 디렉토리 생성
+            mkdir -p ./nginx-conf
+
+            # default.conf 파일 생성 또는 업데이트
+            echo "Creating or updating default.conf..."
+            echo "
+            server {
+                listen 80 reuseport;
+
+                location / {
+                    proxy_pass http://haengdong-backend-$NEXT_PORT:8080;
+                    proxy_set_header Host \$host;
+                    proxy_set_header X-Real-IP \$remote_addr;
+                    proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
+                    proxy_set_header X-Forwarded-Proto \$scheme;
+                    client_max_body_size 300M;
+                }
+            }
+            server {
+                listen 9100;
+            
+                location /actuator {
+                proxy_pass http://haengdong-backend-$NEXT_PORT:8080;
+                proxy_set_header Host \$host;
+                proxy_set_header X-Real-IP \$remote_addr;
+                proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
+                proxy_set_header X-Forwarded-Proto \$scheme;
+              }
+            }" > ./nginx-conf/default.conf
+
+            # Nginx 컨테이너 내부에서 설정 리로드
+            echo "Reloading Nginx configuration..."
+            sudo docker cp ./nginx-conf/default.conf $NGINX_CONTAINER_NAME:/etc/nginx/conf.d/default.conf
+            sudo docker exec $NGINX_CONTAINER_NAME nginx -s reload
+            echo "Nginx configuration reloaded."
+
+          else
+            echo "Nginx container not found. Creating a new Nginx container..."
+
+            # nginx-conf 디렉토리 생성
+            mkdir -p ./nginx-conf
+
+            # default.conf 파일 생성
+            echo "Creating default.conf..."
+            echo "
+            server {
+                listen 80 reuseport;
+
+                location / {
+                    proxy_pass http://haengdong-backend-$NEXT_PORT:8080;
+                    proxy_set_header Host \$host;
+                    proxy_set_header X-Real-IP \$remote_addr;
+                    proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
+                    proxy_set_header X-Forwarded-Proto \$scheme;
+                    client_max_body_size 300M;
+                }
+            }
+            server {
+                listen 9100;
+          
+                location /actuator {
+                proxy_pass http://haengdong-backend-$NEXT_PORT:8080;
+                proxy_set_header Host \$host;
+                proxy_set_header X-Real-IP \$remote_addr;
+                proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
+                proxy_set_header X-Forwarded-Proto \$scheme;
+              }
+            }" > ./nginx-conf/default.conf
+
+            # 새로운 Nginx 컨테이너 실행
+            sudo docker run -d \
+              --name $NGINX_CONTAINER_NAME \
+              --network $NETWORK_NAME \
+              -p 80:80 \
+              -p 9100:9100 \
+              -v $(pwd)/nginx-conf:/etc/nginx/conf.d \
+              nginx
+
+            echo "New Nginx container created and running."
+          fi
+
+      - name: Stop and remove the old container
+        run: |
+          CURRENT_PORT=$CURRENT_PORT
+          if [ -n "$CURRENT_PORT" ]; then
+            echo "Stopping on port $CURRENT_PORT..."
+            sudo docker ps --filter "publish=$CURRENT_PORT" --format "{{.ID}}" | xargs sudo docker stop
+            echo "Sleeping on port $CURRENT_PORT..."
+            sleep 15
+            echo "Removing on port $CURRENT_PORT..."
+            sudo docker ps -a --format "{{.Names}}" | grep "haengdong-backend-$CURRENT_PORT" | xargs sudo docker rm
+          else
+            echo "No container to stop and remove."
+          fi
+
+      - name: Remove unused Docker images
+        run: |
+          echo "Removing unused Docker images..."
+          sudo docker image prune -af --filter "until=24h"
+          echo "Unused Docker images older than 24 hours have been removed."
diff --git a/server/src/main/resources/application.yml b/server/src/main/resources/application.yml
@@ -51,6 +51,12 @@ management:
     web:
       exposure:
         include: health, metrics, logfile, prometheus
+    health:
+      show-details: always
+
+  endpoint:
+    prometheus:
+      enabled: true
 
 server:
   servlet:
@@ -59,6 +65,9 @@ server:
       enabled: true
       force: true
 
+  tomcat:
+    mbeanregistry:
+      enabled: true
 ---
 
 spring:

diff --git a/server/src/main/resources/config b/server/src/main/resources/config