A self-service platform that provisions, configures, monitors, and destroys Apache Hadoop + Apache Spark clusters using a fully automated, end-to-end infrastructure pipeline.
┌─────────────────────────────────────────────────────────────────────┐
│ User Browser │
└─────────────────────────┬───────────────────────────────────────────┘
│ HTTP
┌─────────────────────────▼───────────────────────────────────────────┐
│ React + Vite + TypeScript (Port 5173) │
│ Dashboard · Create Cluster · Cluster Detail + Logs │
└─────────────────────────┬───────────────────────────────────────────┘
│ REST API
┌─────────────────────────▼───────────────────────────────────────────┐
│ FastAPI Backend (Port 8000) │
│ JWT Auth · SQLAlchemy ORM · Pydantic Schemas │
└──────┬──────────────────┬──────────────────────────────────────────┘
│ PostgreSQL │ Enqueue Task
┌──────▼──────┐ ┌───────▼──────────────────────────────────────────┐
│ PostgreSQL │ │ Redis Queue │
│ (Port 5432)│ └───────┬──────────────────────────────────────────┘
└─────────────┘ │ Consume
┌───────▼──────────────────────────────────────────┐
│ Celery Workers │
│ create_cluster · destroy_cluster · get_logs │
└───────┬──────────────────────────────────────────┘
│
┌───────────┼───────────┐
│ │ │
┌─────────▼──┐ ┌──────▼────┐ ┌───▼────────┐
│ Terraform │ │ Ansible │ │ Docker │
│ (IaC) │ │ (Config) │ │ Cluster │
│ network │ │ java │ │ Hadoop │
│ nodes │ │ hadoop │ │ Spark │
│ storage │ │ spark │ │ YARN │
└────────────┘ └───────────┘ └────────────┘
| Layer | Technology |
|---|---|
| Frontend | React 18, Vite, TypeScript |
| Backend | FastAPI, SQLAlchemy, Alembic |
| Database | PostgreSQL 15 |
| Queue | Redis 7 + Celery 5 |
| IaC | Terraform (Docker provider) |
| Config Mgmt | Ansible |
| Containers | Docker Desktop + Docker Compose |
| Monitoring | Prometheus + Grafana |
| Auth | JWT + bcrypt |
- Windows 11 with WSL2 enabled
- Docker Desktop (latest) with WSL2 backend
- Git for cloning
git clone https://github.com/loueylahwel/Automated-Multi-Node-Distributed-Cluster-Orchestrator
cd cluster-orchestrator.\setup.ps1This will:
- Build all Docker images
- Start PostgreSQL, Redis
- Start FastAPI backend + Celery workers
- Seed the database (admin user)
- Start the React frontend
- Start Prometheus + Grafana
Navigate to http://localhost:5173
Login with: admin / admin123
chmod +x setup.sh
./setup.shOr step by step:
# Build
docker-compose build
# Start infrastructure
docker-compose up -d postgres redis
# Wait for DB, then start services
sleep 8
docker-compose up -d backend celery-worker celery-flower frontend prometheus grafana
# Seed default users
docker-compose exec backend python /app/../scripts/seed.py| Service | URL | Credentials |
|---|---|---|
| Frontend UI | http://localhost:5173 | admin / admin123 |
| API (FastAPI) | http://localhost:8000 | — |
| API Docs (Swagger) | http://localhost:8000/docs | — |
| Celery Flower | http://localhost:5555 | — |
| Grafana | http://localhost:3001 | admin / admin |
| Prometheus | http://localhost:9090 | — |
| HDFS UI (when cluster running) | http://localhost:9870 | — |
| YARN UI (when cluster running) | http://localhost:8088 | — |
| Spark UI (when cluster running) | http://localhost:8080 | — |
POST /api/auth/register Register new user
POST /api/auth/token Login (returns JWT)
GET /api/auth/me Get current user
POST /api/clusters/ Create cluster (async)
GET /api/clusters/ List all clusters
GET /api/clusters/{id} Get cluster details
DELETE /api/clusters/{id} Destroy cluster (async)
GET /api/clusters/{id}/logs Get provisioning logs
# Login
TOKEN=$(curl -s -X POST http://localhost:8000/api/auth/token \
-d "username=admin&password=admin123" \
| jq -r .access_token)
# Create cluster
curl -X POST http://localhost:8000/api/clusters/ \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "my-cluster",
"master_nodes": 1,
"worker_nodes": 3,
"hadoop_version": "3.3.6",
"spark_version": "3.5.0",
"cpu_per_node": 2,
"memory_per_node": "4g"
}'cluster-orchestrator/
├── frontend/ # React + Vite + TypeScript UI
│ ├── src/
│ │ ├── api/client.ts # Axios API client
│ │ ├── components/ # Layout, StatusBadge
│ │ ├── hooks/useAuth.tsx # Auth context
│ │ ├── pages/ # Dashboard, CreateCluster, ClusterDetail
│ │ └── types/index.ts # TypeScript interfaces
│ ├── Dockerfile
│ └── package.json
│
├── backend/ # FastAPI application
│ ├── app/
│ │ ├── api/ # clusters.py, auth.py routers
│ │ ├── core/ # config, database, security
│ │ ├── models/ # SQLAlchemy ORM models
│ │ ├── schemas/ # Pydantic request/response schemas
│ │ ├── tasks/ # Celery tasks (cluster_tasks.py)
│ │ └── main.py # FastAPI app entry point
│ ├── alembic/ # DB migrations
│ ├── requirements.txt
│ └── Dockerfile
│
├── terraform/ # Infrastructure as Code
│ ├── main.tf # Root module
│ └── modules/
│ ├── network/ # Docker network
│ ├── master_node/ # Hadoop NameNode containers
│ ├── worker_node/ # Hadoop DataNode containers
│ ├── security_group/ # Firewall rules
│ └── storage/ # Persistent volumes
│
├── ansible/ # Configuration management
│ ├── inventory.ini # Node inventory
│ ├── site.yml # Main playbook
│ └── roles/
│ ├── install_java/ # OpenJDK 11
│ ├── install_hadoop/ # Apache Hadoop
│ ├── install_spark/ # Apache Spark
│ ├── configure_cluster/ # HDFS, YARN config
│ └── start_services/ # Start daemons
│
├── docker/
│ └── hadoop/ # Custom Hadoop Docker image
│ ├── Dockerfile.master
│ ├── conf/ # core-site.xml, hdfs-site.xml, etc.
│ └── entrypoint.sh
│
├── monitoring/
│ ├── prometheus/
│ │ └── prometheus.yml # Scrape configuration
│ └── grafana/
│ └── dashboards/ # Dashboard JSON + provisioning
│
├── scripts/
│ └── seed.py # DB seed (admin user)
│
├── docs/ # Additional documentation
├── docker-compose.yml # Full stack orchestration
├── setup.ps1 # Windows PowerShell setup
├── setup.sh # Linux/WSL2 setup
└── README.md
When you click Deploy Cluster:
1. POST /api/clusters/
└── FastAPI creates DB record (status: pending)
└── Enqueues Celery task → Redis
└── Returns immediately (non-blocking)
2. Celery Worker picks up task
└── Updates status: provisioning
└── Generates Terraform .tfvars
└── Runs: terraform init
└── Runs: terraform apply (creates Docker containers)
└── Updates infrastructure_output in DB
3. Updates status: configuring
└── Runs Ansible playbooks:
install_java.yml
install_hadoop.yml
install_spark.yml
configure_cluster.yml
start_services.yml
4. Updates status: running
└── Logs all steps to cluster_logs table
5. Frontend polls every 3s
└── Displays live status + logs
cd backend
pip install -r requirements.txt
uvicorn app.main:app --reloadcd frontend
npm install
npm run devcd backend
celery -A app.tasks.celery_app worker --loglevel=info# Inside backend container
alembic revision --autogenerate -m "description"
alembic upgrade head# Windows
.\setup.ps1 -Down
# Linux/WSL2
./setup.sh down
# Or directly
docker-compose down -v- Designed and implemented a full-stack distributed cluster orchestration platform using FastAPI, React/TypeScript, Celery, and Redis, enabling self-service provisioning of Apache Hadoop and Spark clusters
- Built a decoupled async pipeline (FastAPI → Redis → Celery → Terraform → Ansible) that provisions multi-node clusters end-to-end with real-time status updates and structured log streaming
- Automated infrastructure lifecycle using Terraform modular IaC (network, nodes, storage, security groups) and Ansible configuration management (Java, Hadoop, Spark installation and service startup)
- Implemented JWT-based RBAC (admin/user roles) with bcrypt password hashing using FastAPI security, SQLAlchemy ORM, and Alembic migrations on PostgreSQL
- Containerized the full stack with Docker Compose including backend, Celery workers, Redis, PostgreSQL, and a Prometheus + Grafana monitoring stack, deployable locally on Windows 11 with WSL2
— Cluster list with status indicators
— Cluster creation form with live summary
MIT — Free and open source.