Skip to content

[SOU-489] 블루그린 배포 적용에 따른 시스템 모니터링 수정 #221

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bum0w0 merged 1 commit into from
Mar 12, 2026
Merged

[SOU-489] 블루그린 배포 적용에 따른 시스템 모니터링 수정 #221

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 30 additions & 22 deletions deploy/shared/system-monitor.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,15 @@
#!/bin/bash
set -e

ENV=${1:-dev}

WORK_DIR="/home/ubuntu/souzip"
if [ "$ENV" = "prod" ]; then
WORK_DIR="/home/souzip-prod/souzip"
else
WORK_DIR="/home/souzip-dev/souzip"
fi
DEPLOY_DIR="$WORK_DIR/deploy/$ENV"
HEALTH_CHECK_URL="http://localhost:8080/actuator/health"
STATUS_FILE="$DEPLOY_DIR/.system-status"

DISK_THRESHOLD=80
MEMORY_THRESHOLD=90
MAX_HEALTH_FAILURES=3
Expand All @@ -30,19 +33,35 @@ else
CONTAINER_WARNING_SENT=false
fi

if [ "$ENV" = "dev" ]; then
HEALTH_CHECK_URL="http://localhost:8080/actuator/health"
CONTAINER_NAME="souzip-api"
else
CONTAINER_NAME="souzip-api-${CONTAINER_COLOR:-blue}"
case "$CONTAINER_COLOR" in
Comment on lines +40 to +41
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge 활성 색상 기반으로 모니터 대상을 계산하세요

prod 분기에서 CONTAINER_COLOR를 기준으로 포트/컨테이너를 고르지만 이 스크립트는 .env에서 DISCORD_WEBHOOK_URL만 읽고 CONTAINER_COLOR를 로드하지 않아 unset 시 항상 blue(8081)로 고정됩니다. deploy/prod/scripts/deploy.sh는 blue/green을 번갈아 배포한 뒤 이전 색상을 down 하므로 green이 active인 배포 사이클에는 서비스가 정상이어도 헬스체크/컨테이너 중지 알림이 오탐될 수 있습니다; active 색상을 upstream에서 계산하거나 CONTAINER_COLOR를 확실히 주입하는 방식이 필요합니다.

Useful? React with 👍 / 👎.

blue)
HEALTH_CHECK_URL="http://localhost:8081/actuator/health"
;;
green)
HEALTH_CHECK_URL="http://localhost:8082/actuator/health"
;;
*)
HEALTH_CHECK_URL="http://localhost:8081/actuator/health"
;;
esac
fi

if curl -f -s --max-time 5 $HEALTH_CHECK_URL > /dev/null 2>&1; then
if [ "$HEALTH_STATUS" = "down" ]; then
if [ ! -z "$HEALTH_DOWN_SINCE" ]; then
DOWN_TIMESTAMP=$(date -d "$HEALTH_DOWN_SINCE" +%s)
UP_TIMESTAMP=$(date +%s)
DOWN_DURATION=$(( UP_TIMESTAMP - DOWN_TIMESTAMP ))
DOWNTIME_MIN=$(( DOWN_DURATION / 60 ))

if [ ! -z "$DISCORD_WEBHOOK_URL" ]; then
notify_server_up "${DOWNTIME_MIN}분"
notify_server_up "${DOWNTIME_MIN}분" "$ENV"
fi
fi

HEALTH_STATUS="up"
HEALTH_FAILURE_COUNT=0
HEALTH_DOWN_SINCE=""
Expand All @@ -51,24 +70,21 @@ if curl -f -s --max-time 5 $HEALTH_CHECK_URL > /dev/null 2>&1; then
fi
else
HEALTH_FAILURE_COUNT=$((HEALTH_FAILURE_COUNT + 1))

if [ $HEALTH_FAILURE_COUNT -ge $MAX_HEALTH_FAILURES ] && [ "$HEALTH_STATUS" = "up" ]; then
HEALTH_STATUS="down"
HEALTH_DOWN_SINCE=$(date '+%Y-%m-%d %H:%M:%S')

if [ ! -z "$DISCORD_WEBHOOK_URL" ]; then
notify_server_down
notify_server_down "$ENV"
fi
fi
fi

DISK_USAGE=$(df -h / | awk 'NR==2 {print $3 " / " $2}')
DISK_PERCENT=$(df / | awk 'NR==2 {print int($5)}')

if [ $DISK_PERCENT -ge $DISK_THRESHOLD ]; then
if [ "$DISK_WARNING_SENT" = "false" ]; then
if [ ! -z "$DISCORD_WEBHOOK_URL" ]; then
notify_disk_warning "$DISK_USAGE" "$DISK_PERCENT"
notify_disk_warning "$DISK_USAGE" "$DISK_PERCENT" "$ENV"
fi
DISK_WARNING_SENT=true
fi
Expand All @@ -78,30 +94,22 @@ fi

MEMORY_USAGE=$(free -h | awk 'NR==2 {print $3 " / " $2}')
MEMORY_PERCENT=$(free | awk 'NR==2 {print int($3/$2 * 100)}')

if [ $MEMORY_PERCENT -ge $MEMORY_THRESHOLD ]; then
if [ "$MEMORY_WARNING_SENT" = "false" ]; then
if [ ! -z "$DISCORD_WEBHOOK_URL" ]; then
notify_memory_warning "$MEMORY_USAGE" "$MEMORY_PERCENT"
notify_memory_warning "$MEMORY_USAGE" "$MEMORY_PERCENT" "$ENV"
fi
MEMORY_WARNING_SENT=true
fi
else
MEMORY_WARNING_SENT=false
fi

if [ "$ENV" = "dev" ]; then
CONTAINER_NAME="souzip-api"
else
CONTAINER_NAME="souzip-api-prod"
fi

CONTAINER_RUNNING=$(docker ps --filter "name=$CONTAINER_NAME" --filter "status=running" -q)

if [ -z "$CONTAINER_RUNNING" ]; then
if [ "$CONTAINER_WARNING_SENT" = "false" ]; then
if [ ! -z "$DISCORD_WEBHOOK_URL" ]; then
notify_container_stopped
notify_container_stopped "$ENV"
fi
CONTAINER_WARNING_SENT=true
fi
Expand All @@ -116,4 +124,4 @@ HEALTH_DOWN_SINCE="$HEALTH_DOWN_SINCE"
DISK_WARNING_SENT=$DISK_WARNING_SENT
MEMORY_WARNING_SENT=$MEMORY_WARNING_SENT
CONTAINER_WARNING_SENT=$CONTAINER_WARNING_SENT
EOF
EOF
Loading