Adding Restic for data backups

Restic/README.md

@@ -0,0 +1,149 @@
+# Data Warehouse Restic-Docker Backup System
+
+## This project sets up a simple and secure backup system using Restic on Docker
+
+## 🚀 Features
+
+- **Automated Backups**: Back up multiple Docker volumes using Restic.
+- **Snapshot Management**: Easily manage and restore snapshots.
+- **Customizable**: Modify the backup script to suit your needs.
+
+---
+
+## 📋 Prerequisites
+
+1. **Docker**: Ensure Docker is installed on your system. [Install Docker](https://docs.docker.com/get-docker/).
+2. **Docker Compose**: Ensure Docker Compose is installed. [Install Docker Compose](https://docs.docker.com/compose/install/).
+
+---
+
+## 🛠️ Setup Instructions
+
+### 1. Clone the Repository
+
+```bash
+cd Restic
+```
+
+### 2. Create Required Folders
+
+Before proceeding, create the `backup` and `restore` folders inside the Restic directory. These folders will be used to store backup data and restored files, respectively.
+
+Run the following commands:
+
+```bash
+mkdir backup restore
+```
+
+### 3. Restic Monitors and Backs Up the Following Volumes
+
+Restic is configured to back up data from several external Docker volumes on Redback VM. Before running the backup system, make sure these volumes exist.
+
+```bash
+data-lakehouse_minio-data
+data-lakehouse_minio-config
+fileuploadservice_dremio-data
+dp-postgres-data
+dp-es-data
+dp-logstash-data
+```
+
+Run the following commands to create them:
+
+```bash
+docker volume create data-lakehouse_minio-data
+docker volume create data-lakehouse_minio-config
+docker volume create fileuploadservice_dremio-data
+docker volume create dp-postgres-data
+docker volume create dp-es-data
+docker volume create dp-logstash-data
+```
+
+### 4. Configure Restic Password
+
+Create a `restic-password.txt` file in the project directory and add your Restic repository password:
+
+```plaintext
+your-secure-password
+```
+
+### 5. Make Scripts Executable
+
+Ensure the backup script is executable:
+
+```bash
+chmod +x scripts/backup.sh
+```
+
+---
+
+## ▶️ Running the Backup System
+
+### Start the Restic Container
+
+Run the following command to start the Restic container:
+
+```bash
+docker-compose up -d
+```
+
+### Verify the Logs
+
+Check the logs of the `restic` container to ensure backups are running:
+
+```bash
+docker logs -f restic-backup
+```
+
+---
+
+## 🔄 Managing Snapshots
+
+### List Available Snapshots
+
+To list all available snapshots, run:
+
+```bash
+docker exec -it restic-backup sh
+```
+
+### Restore a Snapshot
+
+To restore a specific snapshot, use the following command:
+
+```bash
+restic restore `snapshot-id` --target /restore
+```
+
+Replace `<snapshot-id>` with the ID of the snapshot you want to restore.
+
+### Access Restored Files
+
+The restored files will be available in the `./restore` directory on your host machine.
+
+---
+
+## 🛑 Stopping the Backup System
+
+To stop the Restic container, run:
+
+```bash
+docker-compose down
+```
+
+---
+
+## 📝 Notes
+
+- Ensure all required volumes are created before starting the container.
+- Create the `backup` and `restore` folders before running the system.
+- Modify the `backup.sh` script to customize the backup process.
+- Use `docker-compose logs` to troubleshoot any issues.
+
+---
+
+## 📖 Resources
+
+- [Restic Documentation](https://restic.readthedocs.io/)
+- [Docker Documentation](https://docs.docker.com/)
+- [Docker Compose Documentation](https://docs.docker.com/compose/)

Restic/docker-compose.yml

@@ -0,0 +1,43 @@
+version: '3.8'
+services:
+  restic:
+    image: restic/restic
+    container_name: restic-backup
+    #depends_on:
+      #minioserver:
+        #condition: service_started
+    environment:
+      RESTIC_REPOSITORY: "/backup/restic-repo"
+      RESTIC_PASSWORD_FILE: "/restic-password.txt/restic-password.txt"
+    volumes:
+      - data-lakehouse_minio-data:/data-lakehouse_minio-data
+      - data-lakehouse_minio-config:/data-lakehouse_minio-config
+      - fileuploadservice_dremio-data:/fileuploadservice_dremio-data
+      - dp-postgres-data:/dp-postgres-data
+      - dp-es-data:/dp-es-data
+      - dp-logstash-data:/dp-logstash-data
+      - ./backup:/backup
+      - ./restic-password.txt:/restic-password.txt
+      - ./restore:/restore
+      - ./scripts/backup.sh:/scripts/backup.sh
+    entrypoint: ["/bin/sh", "-c", "/scripts/backup.sh"]
+    networks:
+      - coredwinfrastructure_dw_network
+
+volumes:
+  data-lakehouse_minio-data:
+    external: true
+  data-lakehouse_minio-config:
+    external: true
+  fileuploadservice_dremio-data:
+    external: true
+  dp-postgres-data:
+    external: true
+  dp-es-data:
+    external: true
+  dp-logstash-data:
+    external: true
+
+networks:
+  coredwinfrastructure_dw_network:
+    external: true

Restic/restic-password.txt/restic-password.txt

@@ -0,0 +1 @@
+your-secure-password

Restic/scripts/backup.sh

@@ -0,0 +1,21 @@
+#!/bin/sh
+
+echo "Initializing Restic repository if not exists..."
+if ! restic snapshots > /dev/null 2>&1; then
+  restic init
+fi
+
+while true; do
+  echo "Starting Restic backup..."
+  restic backup\
+  /data-lakehouse_minio-data\
+  /data-lakehouse_minio-config \
+  /fileuploadservice_dremio-data \
+  /dp-postgres-data \
+  /dp-es-data \
+  /dp-logstash-data
+  echo "Backup completed. Sleeping for 4 minutes..."
+  #find /localbackup -type f -delete
+  echo "Sleeping for 4 minutes..."
+  sleep 240
+done