Use new vault method

README.md

@@ -1,38 +1,32 @@
 # Montagu DB backup
 
-As the database has grown (from a few 10s of MB up to GB) the dump-and-restore
-approach we are using is becoming painful.  Downloading from S3 takes quite a
-while (especially verification with duplicati) and the restore (`pg_restore`)
-takes a long time to rebuild the indices.
+These docs remain curent for the next little bit until we give up on the db for most data, after which we will probably explore dump-and-restore again.  Doing much with these backups is a nuisance and quite slow in practice.
 
-## Design
+This doc dates from the setup of annex2 after moving production2.  All archaeological content has been removed, but git history is available.
 
-See [`design.md`](design.md) for a description of the system with less archaelogical content.
+## Overall design
 
-## Deployment
+Our primary database sits on `production2.montagu.dide.ic.ac.uk` on port 5432 (behind the VPN).  It uses streaming replication to stream to a [barman](https://www.pgbarman.org/) instance on `annex2.montagu.dide.ic.ac.uk`.
 
-Barman is running on `annex.montagu.dide.ic.ac.uk` as container `montagu-barman`
+Two concepts are important:
 
-```
-git clone https://github.com/vimc/montagu-db-backup
-cd montagu-db-backup
-pip3 install -r requirements.txt
-```
+1. a **base backup** is a snapshot of the postgres database at some point in time
+2. the **WAL (Write Ahead Log)** which can be replayed from a base backup to the present to restore the current database state
+
+## Deployment
 
-Then install the command with
+Barman is running on `annex2.montagu.dide.ic.ac.uk` as container `montagu-barman`
 
 ```
-sudo ./install
+git clone --recursive https://github.com/vimc/montagu-db-backup
+cd montagu-db-backup
+pip3 install --user -r requirements.txt
 ```
 
-which sets up a `barman-montagu` command that can be used from anywhere on the
-computer that refers to the configuration in *this* directory (you can also use
-`./barman-montagu` and avoid installing).
-
 Then you can interact with the barman container the `barman-montagu` command:
 
 ```
-$ barman-montagu --help
+$ ./barman-montagu --help
 Set up and use barman (Postgres streaming backup) for montagu
 
 Usage:
@@ -64,6 +58,11 @@ barman-montagu setup --pull-image --slot barman production.montagu.dide.ic.ac.uk
 ./start-metrics.sh       # Exposes Prometheus metrics on port 5000
 ```
 
+```
+barman-montagu setup --pull-image --slot barman production2.montagu.dide.ic.ac.uk
+./start-metrics.sh       # Exposes Prometheus metrics on port 5000
+```
+
 Or, for local testing you would want:
 
 ```
@@ -107,9 +106,11 @@ instance_name. And if they are tracking different databases they will also
 differ by database.
 
 ### Tests
+
 To test the metrics Flask app, run
+
 ```
-cd ./backup/metrics
+cd ./cached-metrics
 sudo -H  pip3 install -r ./bin/requirements.txt
 sudo -H  pip3 install -r ./bin/requirements-dev.txt
 pytest
@@ -297,3 +298,8 @@ barman-montagu setup --pull-image localhost
 ```
 
 (you may want to specify `--image-tag` too to run the branch you're working on).
+
+
+
+```
+```

barman-montagu

@@ -177,7 +177,7 @@ def vault_client():
         vault_token = input().strip()
     vault = hvac.Client(url=vault_url)
     print("Authenticating vault with GitHub")
-    vault.auth_github(vault_token)
+    vault.auth.github.login(vault_token)
     return vault
 
 

buildkite/pipeline.yml

@@ -1,3 +1,3 @@
 steps:
   - label: ":whale: Build"
-    command: docker-build
+    command: docker/docker-build

docker/Dockerfile

@@ -0,0 +1,55 @@
+FROM ubuntu:24.04
+
+# We need python for our scripts as well as for barman.  It's put
+# first because this also downloads wget which we need to get the gpg
+# key from the postgres deb repo
+#
+# For the "EXTERNALLY-MANAGED" bit, see:
+# https://elliottback.medium.com/python-this-environment-is-externally-managed-error-and-docker-6062aac20a6e
+RUN apt-get update && \
+        apt-get install -y --no-install-recommends \
+        build-essential \
+        curl \
+        ca-certificates \
+        git \
+        gnupg2 \
+        python3-dev \
+        python3-pip \
+        python3-setuptools \
+        python3-wheel && \
+        rm /usr/lib/python*/EXTERNALLY-MANAGED && \
+        pip3 install \
+        docopt \
+        yacron
+
+# Setting TZ here is necessary to stop apt interactively prompting for
+# configuration options, followed by the environment variable
+# DEBIAN_FRONTEND=noninteractive before the install itself.
+ENV TZ=Europe/London
+RUN install -d /usr/share/postgresql-common/pgdg && \
+        curl -o /usr/share/postgresql-common/pgdg/apt.postgresql.org.asc --fail https://www.postgresql.org/media/keys/ACCC4CF8.asc && \
+        sh -c 'echo "deb [signed-by=/usr/share/postgresql-common/pgdg/apt.postgresql.org.asc] https://apt.postgresql.org/pub/repos/apt noble-pgdg main" > /etc/apt/sources.list.d/pgdg.list' && \
+        apt-get update && \
+        DEBIAN_FRONTEND=noninteractive apt-get install -y \
+                barman \
+                postgresql-client-10
+
+# This is a peculiar way of "installing" python packages that I do not
+# have the energy to refactor right now.
+ENV METRICS_UTILS_REF=4b2ef9b
+RUN git clone https://github.com/vimc/metrics-utils /tmp/metrics_utils && \
+        git -C /tmp/metrics_utils reset --hard $METRICS_UTILS_REF && \
+        rm -rf /tmp/metrics_utils/.git && \
+        mv /tmp/metrics_utils /usr/local/lib/python3.12/dist-packages
+
+VOLUME /var/lib/barman
+VOLUME /var/log/barman
+VOLUME /recover
+VOLUME /nightly
+VOLUME /metrics
+
+COPY etc /etc
+COPY bin /usr/local/bin
+COPY schedule.yml /schedule.yml
+
+ENTRYPOINT ["barman-entrypoint"]

docker-build → docker/build

@@ -28,7 +28,7 @@ APP_DOCKER_BRANCH_TAG=$REGISTRY/$NAME:$GIT_BRANCH
 docker build --pull \
        --tag $APP_DOCKER_COMMIT_TAG \
        --tag $APP_DOCKER_BRANCH_TAG \
-       .
+       docker
 
-docker push $APP_DOCKER_BRANCH_TAG
-docker push $APP_DOCKER_COMMIT_TAG
+# docker push $APP_DOCKER_BRANCH_TAG
+# docker push $APP_DOCKER_COMMIT_TAG