Luciq Distributed Chat System

A high-performance, distributed chat system built with a microservices architecture. The system handles message creation through an event-driven pipeline using Go for high-throughput message ingestion, Rails for business logic, Redis for queuing, and Elasticsearch for full-text search.

Architecture

┌─────────┐     ┌──────────┐     ┌─────────┐     ┌────────────┐     ┌───────────┐
│  Client │────▶│  Nginx   │────▶│   Go    │────▶│   Redis    │────▶│  Sidekiq  │
└─────────┘     │ Gateway  │     │ Service │     │   Queue    │     │  Worker   │
                └──────────┘     └─────────┘     └────────────┘     └─────┬─────┘
                     │                                                      │
                     │                                                      ▼
                     │           ┌─────────────────────────────────────────────┐
                     └──────────▶│           Rails API Service                 │
                                 │  - REST API endpoints                       │
                                 │  - Business logic & validations             │
                                 │  - Database operations                      │
                                 └──────────┬──────────────────────────────────┘
                                            │
                        ┌───────────────────┼───────────────────┐
                        ▼                   ▼                   ▼
                   ┌─────────┐      ┌──────────────┐    ┌──────────────┐
                   │  MySQL  │      │    Redis     │    │Elasticsearch │
                   │Database │      │ (Sequences)  │    │   (Search)   │
                   └─────────┘      └──────────────┘    └──────────────┘

Components

• Nginx: API gateway with rate limiting (10 req/s, burst 20) and request routing
• Go Service: High-performance message ingestion service that queues messages to Redis
• Rails API: RESTful API for managing applications, chats, and messages
• Sidekiq Worker: Background job processor that consumes messages from Redis queue
• MySQL: Primary database for persistent storage
• Redis: Message queue and distributed sequence generation
• Elasticsearch: Full-text search engine for message search functionality

Data Model

Applications

• Fields: name, token (UUID), chats_count
• Purpose: Top-level entity representing a client application
• Token: Auto-generated UUID for API authentication

Chats

• Fields: application_id, number, messages_count
• Purpose: Conversation threads within an application
• Numbering: Sequential per application (1, 2, 3...)

Messages

• Fields: chat_id, number, body
• Purpose: Individual messages within a chat
• Numbering: Sequential per chat using Redis atomic counters
• Search: Indexed in Elasticsearch for full-text search

🚀 Getting Started

Prerequisites

• Docker & Docker Compose
• Git

Installation & Running

1. Clone the repository

   git clone https://github.com/aroo530/luciq-distributed-chat-system.git
   cd luciq-distributed-chat-system

2. Start all services

   docker-compose up --build

   This will start:

   • MySQL (port 3306)
   • Redis (port 6379)
   • Elasticsearch (port 9200)
   • Rails API (port 3000)
   • Sidekiq Worker
   • Go Service (port 8080)
   • Nginx Gateway (port 80)

3. Verify services are running

   docker-compose ps

   All services should show as "Up" or "healthy"

4. Create the database (if not auto-created)

   docker-compose exec rails bundle exec rails db:create db:migrate

Stopping Services

docker-compose down

To remove volumes (data will be lost):

docker-compose down -v

📡 API Endpoints

Applications

Tip: You can import the Luciq Chat System API.postman_collection.json file into Postman to easily test all API endpoints.

Method	Endpoint	Description
POST	/applications	Create a new application
GET	/applications	List all applications
GET	/applications/:token	Get application details
PATCH	/applications/:token	Update application name

Chats

Method	Endpoint	Description
POST	/applications/:token/chats	Create a new chat
GET	/applications/:token/chats	List all chats for an application
GET	/applications/:token/chats/:number	Get chat details

Messages

Method	Endpoint	Description
POST	/messages	Create a message (via Go service)
GET	/applications/:token/chats/:number/messages	List messages in a chat
GET	/applications/:token/chats/:number/messages/search?q=query	Search messages

Example Usage

1. Create an application

   curl -X POST http://localhost/applications \
     -H "Content-Type: application/json" \
     -d '{"name": "MyApp"}'

   Response:

   {
     "token": "token",
     "name": "MyApp",
     "chats_count": 0
   }

2. Create a chat

   curl -X POST http://localhost/applications/token/chats \
     -H "Content-Type: application/json"

3. Send a message (via Go service)

   curl -X POST http://localhost/messages \
     -H "Content-Type: application/json" \
     -d '{
       "application_token": "token",
       "chat_number": 1,
       "body": "Hello, World!"
     }'

4. Search messages

   curl "http://localhost/applications/token/chats/1/messages/search?q=Hello"

Testing

Running Tests

The project uses RSpec for testing:

Run all tests:

docker-compose exec rails bundle exec rspec

Run specific test file:

docker-compose exec rails bundle exec rspec spec/requests/messages_spec.rb

Run tests with coverage report:

docker-compose exec rails bundle exec rspec
# Coverage report will be generated in coverage/index.html

Test Infrastructure

• Factory Bot: Test data generation
• Faker: Random test data
• Database Cleaner: Test isolation
• SimpleCov: Code coverage reporting
• RSpec: Testing framework

🔧 Configuration

Environment Variables

Rails Service

• RAILS_ENV: Environment (development/test/production)
• DATABASE_URL: MySQL connection string
• REDIS_URL: Redis connection for Sidekiq
• ELASTICSEARCH_URL: Elasticsearch endpoint

Go Service

• REDIS_URL: Redis connection for message queue
• RAILS_API_URL: Rails API endpoint
• LOG_LEVEL: Logging level (info/debug/error)

Database Configuration

MySQL credentials (development):

• Host: localhost:3306
• Database: luciq_development
• User: luciq
• Password: password
• Root Password: root

📝 Message Flow

1. Client sends POST request to /messages via Nginx
2. Nginx routes to Go service (rate-limited)
3. Go Service validates the request and enqueues the job to Redis
4. Sidekiq Worker (MessageConsumerWorker) picks up the job
5. Worker generates message number using Redis INCR
6. Worker saves message to MySQL database
7. Worker triggers ChatCounterWorker to update counts
8. Message is indexed in Elasticsearch for search

🔍 Monitoring & Logging

Nginx Logs

• Format: JSON with correlation IDs
• Location: /var/log/nginx/access.log
• Fields: timestamp, remote_addr, request_id, status, upstream

Application Logs

• Rails: Structured logging with Lograge
• Go: Structured logging with context
• Sidekiq: Job execution logs

Viewing Logs

# Nginx logs
docker-compose logs -f nginx

# Rails logs
docker-compose logs -f rails

# Go service logs
docker-compose logs -f go_service

# Sidekiq worker logs
docker-compose logs -f worker

Development

Project Structure

.
├── docker-compose.yml          # Service orchestration
├── nginx/
│   └── conf.d/
│       └── luciq.conf         # Nginx routing & rate limiting
├── go/
│   ├── main.go                # Go service entry point
│   ├── handlers/              # HTTP handlers
│   ├── queue/                 # Redis queue integration
│   └── logging/               # Structured logging
├── rails/
│   ├── app/
│   │   ├── controllers/       # API controllers
│   │   ├── models/            # ActiveRecord models
│   │   └── jobs/              # Sidekiq workers
│   ├── config/                # Rails configuration
│   ├── db/
│   │   ├── migrate/           # Database migrations
│   │   └── schema.rb          # Database schema
│   └── spec/                  # RSpec tests
└── data/                      # Persistent data (gitignored)

Adding Migrations

docker-compose exec rails bundle exec rails generate migration MigrationName
docker-compose exec rails bundle exec rails db:migrate

Accessing Rails Console

docker-compose exec rails bundle exec rails console

Accessing Redis CLI

docker-compose exec redis redis-cli

Accessing MySQL

docker-compose exec mysql mysql -u luciq -ppassword luciq_development

Technology Stack

Component	Technology	Version
API Framework	Ruby on Rails	7.1.3
Go Framework	Chi Router	v5
Database	MySQL	8.0
Cache/Queue	Redis	7-alpine
Search	Elasticsearch	8.14.1
Background Jobs	Sidekiq	Latest
Web Server	Nginx	Latest
Testing	RSpec	Latest