🚀 SMARTDRIVE - File Storage Service

Intelligent File Upload, Storage, and Management Service

📋 Overview

The File Storage Service is a core component of the SMARTDRIVE platform, responsible for handling file uploads, storage management, and providing secure access to files. It integrates with AWS S3 for scalable cloud storage and DynamoDB for metadata management.

🏗️ Architecture

┌─────────────────┐    ┌──────────────┐    ┌─────────────┐
│   Client App    │───▶│ File Storage │───▶│   AWS S3    │
│                 │    │   Service    │    │   Storage   │
└─────────────────┘    └──────────────┘    └─────────────┘
                              │
                              ▼
                       ┌─────────────┐
                       │ DynamoDB    │
                       │ Metadata    │
                       └─────────────┘

✨ Features

🔐 Secure File Upload

• Multi-part upload support for large files
• File validation (type, size, content)
• Virus scanning integration
• Encryption at rest and in transit

📁 File Management

• Metadata storage in DynamoDB
• File versioning support
• Access control and permissions
• File sharing capabilities

🔍 Search Integration

• Real-time indexing for search
• Metadata extraction and tagging
• AI processing integration

📊 Monitoring & Analytics

• Upload progress tracking
• Performance metrics
• Error monitoring
• Usage analytics

🛠️ Technology Stack

• Framework: Spring Boot 3.2
• Language: Java 17
• Storage: AWS S3
• Database: AWS DynamoDB
• Message Queue: AWS SQS
• Monitoring: OpenTelemetry, Micrometer
• Documentation: OpenAPI 3.0 (Swagger)

🚀 Quick Start

Prerequisites

• Java 17+
• Docker & Docker Compose
• AWS Account with S3, DynamoDB, and SQS access

Environment Variables

# AWS Configuration
AWS_ACCESS_KEY_ID=your_access_key
AWS_SECRET_ACCESS_KEY=your_secret_key
AWS_REGION=us-east-1

# AWS Services
S3_BUCKET_NAME=smartdrive-uploads
DYNAMODB_TABLE_NAME=smartdrive-metadata
SQS_QUEUE_URL=https://sqs.us-east-1.amazonaws.com/your-queue

# Service Configuration
SERVER_PORT=8081

Local Development

# Clone the repository
git clone <repository-url>
cd file-storage-service

# Build the project
./gradlew build

# Run with Docker Compose
docker compose up -d

# Or run locally
./gradlew bootRun

Docker Deployment

# Build Docker image
docker build -t smartdrive-file-storage .

# Run container
docker run -p 8081:8081 \
  -e AWS_ACCESS_KEY_ID=your_key \
  -e AWS_SECRET_ACCESS_KEY=your_secret \
  -e AWS_REGION=us-east-1 \
  smartdrive-file-storage

📡 API Endpoints

File Upload

POST /api/v1/files/upload
Content-Type: multipart/form-data

# Response
{
  "contentId": "uuid",
  "fileName": "document.pdf",
  "fileSize": 1024000,
  "uploadStatus": "COMPLETED",
  "s3Key": "uuid-document.pdf"
}

File Download

GET /api/v1/files/{contentId}/download

# Returns file content with appropriate headers

File Metadata

GET /api/v1/files/{contentId}/metadata

# Response
{
  "contentId": "uuid",
  "fileName": "document.pdf",
  "contentType": "application/pdf",
  "fileSize": 1024000,
  "uploadedAt": "2024-01-01T12:00:00Z",
  "processingStatus": "COMPLETED"
}

Upload Progress

GET /api/v1/files/{contentId}/progress

# Server-Sent Events stream
data: {"progress": 75, "status": "UPLOADING"}

🔧 Configuration

Application Properties

server:
  port: 8081

spring:
  application:
    name: file-storage-service

app:
  s3:
    bucket-name: ${S3_BUCKET_NAME:smartdrive-uploads}
  dynamodb:
    table-name: ${DYNAMODB_TABLE_NAME:smartdrive-metadata}
  sqs:
    queue-url: ${SQS_QUEUE_URL}
  upload:
    max-file-size: 100MB
    allowed-types:
      - image/*
      - application/pdf
      - text/*

AWS Configuration

@Configuration
public class AwsConfig {
    @Bean
    public S3Client s3Client() {
        return S3Client.builder()
            .region(Region.of(awsRegion))
            .credentialsProvider(DefaultCredentialsProvider.create())
            .build();
    }
}

📊 Monitoring

Health Checks

GET /actuator/health
GET /actuator/health/s3
GET /actuator/health/dynamodb
GET /actuator/health/sqs

Metrics

• Upload Success Rate: smartdrive.upload.success.rate
• File Size Distribution: smartdrive.file.size.distribution
• Processing Time: smartdrive.processing.time
• Error Rate: smartdrive.error.rate

Logging

logging:
  level:
    com.smartdrive.filestorageservice: INFO
    software.amazon.awssdk: INFO
  pattern:
    console: "%d{HH:mm:ss.SSS} [%X{traceId:-}] %-5level %logger{20} - %msg%n"

🔒 Security

Authentication & Authorization

• JWT Token Validation
• Role-based Access Control
• File-level Permissions
• Audit Logging

Data Protection

• Encryption at Rest (AWS S3 SSE)
• Encryption in Transit (TLS 1.3)
• Secure File Handling
• Virus Scanning

🧪 Testing

Unit Tests

./gradlew test

Integration Tests

./gradlew integrationTest

Load Testing

# Using Apache Bench
ab -n 1000 -c 10 -p upload.json -T multipart/form-data http://localhost:8081/api/v1/files/upload

🚀 Deployment

Docker Compose

version: '3.8'
services:
  file-storage-service:
    build: .
    ports:
      - "8081:8081"
    environment:
      - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
      - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
      - AWS_REGION=${AWS_REGION}
    depends_on:
      - elasticsearch
      - redis

Kubernetes

apiVersion: apps/v1
kind: Deployment
metadata:
  name: file-storage-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: file-storage-service
  template:
    metadata:
      labels:
        app: file-storage-service
    spec:
      containers:
      - name: file-storage-service
        image: smartdrive/file-storage:latest
        ports:
        - containerPort: 8081
        env:
        - name: AWS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              name: aws-credentials
              key: access-key-id

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

🆘 Support

• Documentation: SMARTDRIVE Docs
• Issues: GitHub Issues
• Discussions: GitHub Discussions

---

Built with ❤️ by the SMARTDRIVE Team
🌐 Website • 📚 Docs • �� GitHub