Skip to content

🐰 Add RabbitMQ Integration for Chat Log Processing Triggers#2

Open
ChanYoungHan wants to merge 7 commits intoAlexiFeng:mainfrom
ChanYoungHan:feature/enable_mq
Open

🐰 Add RabbitMQ Integration for Chat Log Processing Triggers#2
ChanYoungHan wants to merge 7 commits intoAlexiFeng:mainfrom
ChanYoungHan:feature/enable_mq

Conversation

@ChanYoungHan
Copy link

🐰 Add RabbitMQ Integration for Chat Log Processing Triggers

📋 Development Background

The existing MCP Chat Logger only supported simple file saving, but we have extended it to use AMQP (RabbitMQ) as a trigger for processing chat logs.

Key reasons for introducing AMQP:

  • 🔒 Prevent trigger loss: Publish messages to RabbitMQ simultaneously with file saving to ensure that subsequent processing triggers are not lost
  • 🔄 Asynchronous processing: Perform chat log processing tasks reliably in the background
  • 📈 Scalability: Provide architecture where multiple consumers can process the same message
  • 🛠️ Reliability: Prevent message loss through RabbitMQ's persistence and delivery confirmation

✨ Key Changes

🏗️ Architecture Improvements

  • New module structure: Added utils/rabbitmq_publisher.py to separate AMQP logic
  • Development environment tools: Centralized development Docker environment in dev-tools/ directory
  • Automatic mode detection: Automatic RabbitMQ/file-only mode switching based on environment variable configuration

🔧 Core Feature Additions

  • RabbitMQ Integration:
    • Exchange: llmLogger (direct type)
    • Queue: llm_logger
    • Routing Key: llm_logger
  • Dual storage mechanism: File saving + RabbitMQ message publishing
  • Environment variable-based automatic configuration: RABBITMQ_HOST, RABBITMQ_PORT, RABBITMQ_USERNAME, RABBITMQ_PASSWORD

🌍 Internationalization and Usability Improvements

  • Multi-language README: Support for Chinese, Korean, and English
  • Development/Production environment separation: CloudAMQP (production) vs Docker (development)
  • Simplified execution: Removed complex CLI options, mode determination through environment variables only

📊 Change Statistics

12 files changed, 1328 insertions(+), 277 deletions(-)

🆕 New Files

  • utils/rabbitmq_publisher.py - RabbitMQ message publishing module
  • dev-tools/docker-compose.yml - Development RabbitMQ environment
  • dev-tools/.env.example - Environment variable template
  • dev-tools/test_rabbitmq.py - Development connection testing
  • rabbitmq_init/init.sh - RabbitMQ initialization script

🔄 Major Modifications

  • chat_logger.py - Added AMQP integration and automatic mode detection logic
  • README.md, README_ko.md, README_en.md - Complete overhaul reflecting new architecture
  • pyproject.toml - Added pika, python-dotenv dependencies

❌ Removed Elements

  • README_zh.md - Duplicate file cleanup (README.md is in Chinese)
  • Makefile - Replaced with environment variable-based automatic configuration
  • Unnecessary MCP tools (test_rabbitmq_connection, get_rabbitmq_config)

🚀 Usage Examples

RabbitMQ Mode (with environment variables)

# After setting .env file
uv run chat_logger.py
# Output: 🔧 MCP Chat Logger starting with RabbitMQ enabled

File-only Mode (without environment variables)

# Run without environment variables
uv run chat_logger.py  
# Output: 🔧 MCP Chat Logger starting in file-only mode (RabbitMQ not configured)

🔗 Related Issues and Background

This PR aims to build a reliable trigger system for subsequent processing of chat log data. Since file saving alone poses a risk of processing trigger loss, we introduced a message-based trigger system through RabbitMQ to ensure data processing stability and scalability.


This Pull Request is a significant update that advances the existing functionality into a scalable architecture while maintaining backward compatibility. 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant