- Introduction
- A Homelab
- Hardware
- Software
- Provisioning
- Monitoring and Alerting
- Maintenance
- Backup Strategy
- Documentation
- Still to do
Welcome to my Homelab definition-in-code! The repository consists largely of Ansible code which is used to provision and maintain the services which make up the functionality within my Homelab.
Please feel free to reuse any of the code or provide feedback and suggestions.
A Homelab is many things to many people, but I use it primarily to
-
self host capabilities that I want to run locally, without exposing data outside my home
-
develop skills and knowledge around cloud native technologies
-
exploring different technologies and how they can be used to enhance my home
-
enjoy building something I use everyday
Some good sources of information about what a Homelab is include:
In addition, sources of information about setting up a Homelab include:
-
MORE TO COME!!
The major hardware compoenents which makes up the Homelab are shown in the following diagram.
A Ubiquity Cloud Gateway (UCG) Ultra is used to create and manage the primary network aspects of the Homelab.
The UCG runs UniFi OS as the operating system which hosts the Unifi Network Application.
Some good sources of information for setting up a Unifi network include:
- Full Unifi Config - Setup from Start to Finish
- COMPLETE UniFi Network Setup Guide (Detailed for Beginners)
In addition, a Raspberry Pi 3 Model B+ single board computer, running Rasbian Bullseye, is used to host an instance of AdGuard Home, to provide a local DNS service.
All hosts within the Homelab are configured automatically by the Gateway, to use the DNS capability within AdGuard Home to resolve hostnames internal to the Homelab.
I wrote an article about using AdGuard Home as a DNS service which is available on Medium.
A Beelink SEi12 i5-1235U Intel 12 Gen Mini PC was selected as the Homelab compute server.
The system uses an i5-1235U processor, which with 10 cores, provides a good amount of parallel processing in a relatviely efficient, compact, unit.
Server Specifications
| Component | Details |
|---|---|
| CPU | Intel Core i5-1235U 2P-8E-12H 3.3-4.4GHz / 15-55 W TDP / 10 nm (Intel 7) |
| GPU | Intel Xe / 80 EU / 1200 MHz |
| RAM | 64GB DDR4 3200MHz max which is fullyu utilised by 2x Crucial 32GB SODIMM modules |
| Storage | 500GB M.2 2280 NVMe plus 2TB SATA 3 2.5″ SSD |
| Network | 1x Gigabit Ethernet (Realtek) |
| Ports | 1x USB 3.1 Type-C (data) / 2x USB 3.0 / 2x USB 2.0 / 2x HDMI 2.1 / Audio Jack / BIOS Reset |
A ZimaBlade 7700 NAS kit, which uses a quad-core version of the ZimaBlade single-board x86 computer, is used as the storage server.
Server Specifications
| Component | Details |
|---|---|
| CPU | Intel® Celeron with 2.4 GHz Turbo Speed |
| Intel® AES New Instructions | |
| Intel® Virtualization Technology (VT-x) | |
| Intel® Virtualization Technology for Directed I/O (VT-d) | |
| Memory | 16 GB DDR3L RAM |
| Storage | Integrated 32 GB eMMC |
| Network | 1 x 10/100/1000 Mbps Gigabit Ethernet |
| PCIe | 1 x PCIe 2.0, four lanes |
| SATA | 2 x SATA 3.0 |
| Power | 45 W USB Type-C power adapter |
| Thermal | 6 W TDP with passive cooling |
For storage, two Seagate Barracuda Green 2TB SATA hard drives are used.
The compute server is running Proxmox Virtual Environment 8 hypervisor as the OS. Proxmox supports both Virtual Machines and LXC Linux Containers to be used to host the services and applications.
Proxmox 8 was installed on top of Debian 12 as I was unable to boot the Compute Server directly into the Proxmox installer.
Some good sources of information for setting up Proxmox are:
ADD DETAILS!!
The storage service is running TrueNAS Scale Community Edition as the storage solution.
TrueNAS Scale ElectricEel-24.10.2 was installed directly on to the ZimBlade.
The initial configuration of TrueNAS was based on details described in 6 Crucial Settings to Enable on TrueNAS SCALE.
The majority of applications and services running within the Homelab are deployed as Docker containers, running within LXC Linux containers. The exceptions are Nextcloud and Home Assistant, which are running in virtual machines, and Wazuh is installed directly on an LXC Linux Container.
The Linux containers run Debian 12 Bookworm as the host OS.
The following resource details are used for LXC Linux Containers:
Low resource services, those which are largely stateless:
| Resource | Value |
|---|---|
| Memory | 1.00 GiB (1024 MiB) |
| Swap | 1.00 GiB (1024 MiB) |
| Cores | 1 |
| Disk size | 16GB |
Medium resource services, those which host datastores of some type:
| Resource | Value |
|---|---|
| Memory | 2.00 GiB (1024 MiB) |
| Swap | 2.00 GiB (1024 MiB) |
| Cores | 2 |
| Disk size | 24GB |
Each application is hosted in a single LXC Linux Container, using an architecture as shown in the following diagram.
The current iteration of the Homelab hosts the following applications:
Nextcloud - Open source content collaboration platform.
Home Assistant - Open source home automation that puts local control and privacy first.
Authentik - Identity manager.
Dozzle - Realtime log viewer for docker containers.
EMQX - MQTT platform.
Firefly III - A free and open source personal finance manager.
Frigate - An open source NVR.
Grafana - An open source tool to create dashboards. - REPLACED BY DOZZLE
Grocy - An open source web-based self-hosted groceries & household management solution.
Homebox - An open source inventory and organization system.
Homepage - A modern, fully static, fast, secure fully proxied, highly customizable application dashboard.
Hortusfox - A free and open-sourced self-hosted plant manager system.
InfluxDB - A time-series database.
- 📓 Journal - A simple self-hosted journaling app.
Portainer - A universal container management platform.
Semaphore - User friendly web interface for executing Ansible playbooks, Terraform, OpenTofu code and Bash scripts.
Uptime Kuma - An easy-to-use self-hosted monitoring tool.
Vaultwarden - An alternative server implementation of the Bitwarden Client API.
Wazuh - A free and open-source platform for threat prevention, detection, and response, capable of protecting workloads across on-premises, virtualized, containerized, and cloud-based environments.
WUD (What's up Docker) - A tool to keep Docker containers up-to-date.
Underpinning the applications are a number of support services:
- Docker Socket Proxy - A security-enhanced proxy for the Docker socket.
- Docker Volume Backup - Companion container to backup Docker volumes.
- Node Exporter- Prometheus exporter for hardware and OS metrics. - REPLACED BY DOZZLE
Prometheus - An open-source systems monitoring and alerting toolkit. - REPLACED BY DOZZLE
Traefik - An open source application proxy.
Ansible is used to define and automate the configuration of the Linux containers, including the installation of Docker, and deployment of the containerised applications. The Ansible files make up the majority of this repository, which is structured to work with Semaphore UI.
The adoption of Ansible allows the configuration to be defined in code, and therefore version controlled in a source code repository.
All Ansible code is linted using Ansible Lint, using the configuration in .ansible-lint.yaml.
The following naming convention is to be adopted within the repository:
-
Playbooks: Use names like site.yml, webapp.yml, database.yml.
-
Roles: Use lowercase letters and hyphens to separate words, e.g. web-server or database-backup.
-
Task file names: Use lowercase letters and underscores, and end with .yml instead of .yaml.
-
Task names: Start with a verb to indicate the action, e.g. "start_webserver".
-
Variable names: Use lowercase letters and underscores to separate words.
TO DO: update code to align with naming convention
Semaphore UI is used to manage and run the various Ansible artifacts, including the inventories, and playbooks. All initial provisioning of the services within the Homelab is run from Semaphore. The history of all playbook runs is maintained within Semaphore UI in order to be able to monitor the the outcome of each run.
Monitoring is a key aspect of the Homelab I wanted to work on.
A number of different tools are utilised to support monitoring.
-
Portainer, which provides a frontend to Docker to:
-
check the configuration and state of deployed containers.
-
cleanup artifacts (images, volumes) which are no longer being utilised.
-
check logs of individual containers.
-
-
Dozzle, which provides a frontend to Docker hosts to:
-
visualise the state of multiple hosts.
-
check logs from multiple containers running on a host.
-
-
Uptime Kuma, which provides a frontend to:
- configure and monitor probes into different resource types, include host systems, Docker containers, and MQTT clients.
In terms of an overall monitoring and alerting system, Home Assistant is utilised, supported by the following integration, which provide sensors to ingest data from various services and enable controlling of services:
Using Home Assistant has a number of advantages in my case:
-
Provides an ability to manually monitor and control systems.
-
Supports automating controls and alerts for systems.
-
Provides a secure remote access mechanism via its cloud offering.
Current 'prototype' implementations of dashboards implemented within Home Assistant are shown in the following images.
The automation capability of Home Assistant is utilised to trigger notifications or take automated actions to resolve issues.
For example:
-
Notifications generated when hosts become unavailable from the network.
-
Notifications generated when host resources (CPU, Disk and memory) exceed thresholds.
-
Automating the restarting of resources which appear in an unstable state.
-
Automating the shutdown and restart of systems to reduce power consumption.
The intent is to automate as much of the maintenance of the Homelab as possible. Although tools such as Watchtower automate the updating of a running system, my intention is to have the Ansible code as the source-of-truth, that the running system needs then to reflect, in the style of GitOps.
The repository uses Mend Renovate, via the Renovate Github App, to assist with maintaining versions defined within the source code.
At present Renovate is successfully automatically creating pull requests for updates to Ansible dependencies, however, I had hoped it would also update Docker image versions, but have not seen this to occur.
The following are sources used to setup and configure Renovate:
- Meet Renovate - Your Update Automation Bot for Kubernetes and More!
- Renovate bot cheat sheet – the 11 most useful customizations
A major benefit for the adoption of Semaphore UI to manage Ansible is the capability to be able to schedule the running of Ansible playbooks.
For example, an automation is configured to run daily updates of the host systems, using the hosts-update.yaml playbook.
Pre-commit is utilised to automate the running of tools to ensure code quality and security is maintained before any code changes are committed to the source code. The tools run are configured within the .pre-commit-config.yaml file.
All secrets, such as passwords, tokens and personal identifiers, are maintained within the code base using ansible-vault functionality to ensure they are encrypted and not exposed in plain text. In addition, tools are run by pre-commit, to ensure the encrypted files are in an encrypted state before committing to source code, and all files are scanned for passwords and tokens.
A critical aspect of the Homelab is having a robust capability to ensure all data is being regularly backed-up. A number of different approaches are used to support the different services running in the Homelab.
All configuration for the Homelab is maintained in this version controlled repository so is not separately backed up.
The majority of the services are configured at runtime using either environment variables or labels. Writing configuration, or .env, files to the file system is kept to a minimum.
All containers which persist data use Docker Volumes as data stores, rather than bind mounting directly to the file system.
In order to backup the volumes, the service docker-volume-backup is utilised. The approach offers a lightweight containerised solution which can backup locally, to shared volumes, or cloud.
The configuration for docker-volume-backup is managed via the templated .env file, which is derived from docker-volume-backup Configuration reference.
A local copy of the data is retained, as well as a copy pushed to the Storage server, via ssh. The data is encrypted via GPG.
Pruning of the backups also is enabled, to ensure only 7 days of backups are retained. Backups are initiated @daily which
occurs at midnight.
For services which store data outside of a dedicated database, the associated data volume is mounted into the docker-volume-backup container.
An example of how this is achieved, taken from the deploy-homebox playbook, is shown below.
- name: Deploy a containerised instance of Docker Volume Backup
ansible.builtin.include_tasks:
file: tasks/deploy-docker-volume-backup.yaml
vars:
docker_volume_backup_volume_to_backup:
- "{{ homebox_volume_name }}:/backup/homebox-backup:ro"
docker_volume_backup_backup_label: "{{ homebox_service_name }}"A label is added to the associated container, to ensure the container is stopped before the data volume is backed up. An example of this, taken from deploy-homebox task, is shown below.
- name: Create Homebox container labels
ansible.builtin.set_fact:
homebox_container_labels: "{{ homebox_container_labels | default({}) | combine({item.key: item.value}) }}"
with_items:
...
# Docker Volume Backup labels
- { "key": "docker-volume-backup.stop-during-backup", "value": "true" }
...For services which store data in a dedicated database container, the database contents are dumped to a file, which is then backed up.
For MariaDB databases, the backup is achieved using mariadb-dump, an example of which, taken from
deploy-hortusfox task, is shown below.
- name: Create Hortusfox DB container labels
ansible.builtin.set_fact:
hortusfox_db_container_labels: "{{ hortusfox_db_container_labels | default({}) | combine({item.key: item.value}) }}"
with_items:
# Docker Volume Backup labels
- {
"key": "docker-volume-backup.archive-pre",
"value": "/bin/sh -c 'mariadb-dump --single-transaction --user={{ hortusfox.db_user }} -p{{ hortusfox.db_password }} --all-databases > /tmp/dumps/dump.sql'"
}In addition, a dedicated volume is created to store the backup file, an example of which, taken from deploy-hortusfox task, is shown below.
- name: Create Hortusfox backup volume # noqa: syntax-check[unknown-module]
community.docker.docker_volume:
name: "{{ hortusfox_db_backup_volume_name }}"
state: presentWithin the associated deployment of the database service, the volume is mounted into the container, an example of which, taken from hortusfox/docker-compose.yml, is shown below.
db:
container_name: hortusfox_db
image: mariadb:lts
environment:
MYSQL_ROOT_PASSWORD: {{ hortusfox.db_root_password | default('dummy') }}
MYSQL_DATABASE: {{ hortusfox.db_database | default('hortusfox') }}
MYSQL_USER: {{ hortusfox.db_user | default('hortusfox') }}
MYSQL_PASSWORD: {{ hortusfox.db_password | default('dummy') }}
hostname: db
labels: {{ (hortusfox_db_container_labels | default({})) | default(omit) }}
networks:
- hortusfox
restart: always
volumes:
- db_data:/var/lib/mysql
- {{ hortusfox_db_backup_volume_name | default('hortusfox_db_backup') }}:/tmp/dumpsIn the case where docker-compose is used to deploy the containers, the volume needs to assigned as external, an example of which,
taken from hortusfox/docker-compose.yml,
is shown below.
volumes:
db_data:
{{ hortusfox_db_backup_volume_name | default('hortusfox_db_backup') }}:
external: true
app_images:
app_logs:
app_backup:
app_themes:
app_migrate:As volume must also be mounted to the docker-volume-backup container, an example of which, taken from deploy-hortusfox.yaml playbook, is shown below.
- name: Deploy a containerised instance of Docker Volume Backup
ansible.builtin.include_tasks:
file: tasks/deploy-docker-volume-backup.yaml
vars:
docker_volume_backup_volume_to_backup:
- "{{ hortusfox_db_backup_volume_name }}:/backup/hortusfox_db-backup:ro"
docker_volume_backup_backup_label: "{{ hortusfox_service_name }}-db"TO DO: I could not get the database backups to work using docker-socket-proxy and had to bind to the docker socket directly.
For off-site backup, a Cloud Sync Task is configured within the Storage server to push the backup files created by
docker-volume-backup to a cloud storage provider. The task is scheduled to be run daily at 1am.
Nextcloud AOI (All in One) uses BorgBackup to manage backups.
An issue with the way the back up location is configured is that it cannot easily be changed once it is initially set.
A way to manage this is described in the Github project page using the following to be able to have the
Reset backup location button show in the AIO Interface.
root@nextcloud:~# docker exec nextcloud-aio-mastercontainer rm /mnt/docker-aio-config/data/borg.configAll documentation is written in Markdown, using the Markdown Guide as a reference, and linted, using markdownlint-cli2. The configuration for markdownlint is defined in .markdownlint.yaml.
- Explore using an Proxmox network for Proxmox nodes, rather than having them on main network.