diff --git a/Restic/README.md b/Restic/README.md new file mode 100644 index 0000000..efad9e3 --- /dev/null +++ b/Restic/README.md @@ -0,0 +1,149 @@ +# Data Warehouse Restic-Docker Backup System + +## This project sets up a simple and secure backup system using Restic on Docker + +## 🚀 Features + +- **Automated Backups**: Back up multiple Docker volumes using Restic. +- **Snapshot Management**: Easily manage and restore snapshots. +- **Customizable**: Modify the backup script to suit your needs. + +--- + +## 📋 Prerequisites + +1. **Docker**: Ensure Docker is installed on your system. [Install Docker](https://docs.docker.com/get-docker/). +2. **Docker Compose**: Ensure Docker Compose is installed. [Install Docker Compose](https://docs.docker.com/compose/install/). + +--- + +## 🛠️ Setup Instructions + +### 1. Clone the Repository + +```bash +cd Restic +``` + +### 2. Create Required Folders + +Before proceeding, create the `backup` and `restore` folders inside the Restic directory. These folders will be used to store backup data and restored files, respectively. + +Run the following commands: + +```bash +mkdir backup restore +``` + +### 3. Restic Monitors and Backs Up the Following Volumes + +Restic is configured to back up data from several external Docker volumes on Redback VM. Before running the backup system, make sure these volumes exist. + +```bash +data-lakehouse_minio-data +data-lakehouse_minio-config +fileuploadservice_dremio-data +dp-postgres-data +dp-es-data +dp-logstash-data +``` + +Run the following commands to create them: + +```bash +docker volume create data-lakehouse_minio-data +docker volume create data-lakehouse_minio-config +docker volume create fileuploadservice_dremio-data +docker volume create dp-postgres-data +docker volume create dp-es-data +docker volume create dp-logstash-data +``` + +### 4. Configure Restic Password + +Create a `restic-password.txt` file in the project directory and add your Restic repository password: + +```plaintext +your-secure-password +``` + +### 5. Make Scripts Executable + +Ensure the backup script is executable: + +```bash +chmod +x scripts/backup.sh +``` + +--- + +## ▶️ Running the Backup System + +### Start the Restic Container + +Run the following command to start the Restic container: + +```bash +docker-compose up -d +``` + +### Verify the Logs + +Check the logs of the `restic` container to ensure backups are running: + +```bash +docker logs -f restic-backup +``` + +--- + +## 🔄 Managing Snapshots + +### List Available Snapshots + +To list all available snapshots, run: + +```bash +docker exec -it restic-backup sh +``` + +### Restore a Snapshot + +To restore a specific snapshot, use the following command: + +```bash +restic restore `snapshot-id` --target /restore +``` + +Replace `` with the ID of the snapshot you want to restore. + +### Access Restored Files + +The restored files will be available in the `./restore` directory on your host machine. + +--- + +## 🛑 Stopping the Backup System + +To stop the Restic container, run: + +```bash +docker-compose down +``` + +--- + +## 📝 Notes + +- Ensure all required volumes are created before starting the container. +- Create the `backup` and `restore` folders before running the system. +- Modify the `backup.sh` script to customize the backup process. +- Use `docker-compose logs` to troubleshoot any issues. + +--- + +## 📖 Resources + +- [Restic Documentation](https://restic.readthedocs.io/) +- [Docker Documentation](https://docs.docker.com/) +- [Docker Compose Documentation](https://docs.docker.com/compose/) diff --git a/Restic/docker-compose.yml b/Restic/docker-compose.yml new file mode 100644 index 0000000..3b79d7a --- /dev/null +++ b/Restic/docker-compose.yml @@ -0,0 +1,43 @@ +version: '3.8' +services: + restic: + image: restic/restic + container_name: restic-backup + #depends_on: + #minioserver: + #condition: service_started + environment: + RESTIC_REPOSITORY: "/backup/restic-repo" + RESTIC_PASSWORD_FILE: "/restic-password.txt/restic-password.txt" + volumes: + - data-lakehouse_minio-data:/data-lakehouse_minio-data + - data-lakehouse_minio-config:/data-lakehouse_minio-config + - fileuploadservice_dremio-data:/fileuploadservice_dremio-data + - dp-postgres-data:/dp-postgres-data + - dp-es-data:/dp-es-data + - dp-logstash-data:/dp-logstash-data + - ./backup:/backup + - ./restic-password.txt:/restic-password.txt + - ./restore:/restore + - ./scripts/backup.sh:/scripts/backup.sh + entrypoint: ["/bin/sh", "-c", "/scripts/backup.sh"] + networks: + - coredwinfrastructure_dw_network + +volumes: + data-lakehouse_minio-data: + external: true + data-lakehouse_minio-config: + external: true + fileuploadservice_dremio-data: + external: true + dp-postgres-data: + external: true + dp-es-data: + external: true + dp-logstash-data: + external: true + +networks: + coredwinfrastructure_dw_network: + external: true diff --git a/Restic/restic-password.txt/restic-password.txt b/Restic/restic-password.txt/restic-password.txt new file mode 100644 index 0000000..912829b --- /dev/null +++ b/Restic/restic-password.txt/restic-password.txt @@ -0,0 +1 @@ +your-secure-password \ No newline at end of file diff --git a/Restic/scripts/backup.sh b/Restic/scripts/backup.sh new file mode 100644 index 0000000..1bf6079 --- /dev/null +++ b/Restic/scripts/backup.sh @@ -0,0 +1,21 @@ +#!/bin/sh + +echo "Initializing Restic repository if not exists..." +if ! restic snapshots > /dev/null 2>&1; then + restic init +fi + +while true; do + echo "Starting Restic backup..." + restic backup\ + /data-lakehouse_minio-data\ + /data-lakehouse_minio-config \ + /fileuploadservice_dremio-data \ + /dp-postgres-data \ + /dp-es-data \ + /dp-logstash-data + echo "Backup completed. Sleeping for 4 minutes..." + #find /localbackup -type f -delete + echo "Sleeping for 4 minutes..." + sleep 240 +done