docker: Switch to Debian and build all master packages

p12tic · p12tic · commit 24df4707dfce · 2021-05-20T15:59:45.000+03:00
Previously we were installing buildbot-www from whatever is available on
Pypi. This leads to wrong versions being taken for docker containers
built from tags. E.g. if 2.10.5 is built, one expects buildbot-www to
correspond to 2.10.5 code, not what's latest on Pypi. Fortunately we can
just reuse the existing release build code to build www packages for the
container and solve the problem this way.

Current Dockerfile uses Alpine, which is great for reducing the
container footprint, but for master this is less important because by
design there will be few instances running. Additionally, most of the
users of Buildbot (myself included) are much more likely to have more
experience with Debian. Given that I need to make significant changes to
the Dockerfile anyway, I'm switching it to Debian.
diff --git a/.dockerignore b/.dockerignore
@@ -0,0 +1,16 @@
+**/*.pyc
+**/dist
+**/*.egg-info
+**/build
+**/node_modules
+
+# Note: not ignoring .git so that build scripts can figure out which version of Buildbot we're
+# building.
+
+.dockerignore
+.venv*
+Dockerfile.master
+master/_build
+master/docs/manual/mydashboard.html
+master/docs/manual/mydashboard.py
+_trial_temp
diff --git a/Dockerfile.master b/Dockerfile.master
@@ -0,0 +1,107 @@
+# buildbot/buildbot-master
+
+# please follow docker best practices
+# https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/
+
+# Use a multi-stage build:
+# https://docs.docker.com/develop/develop-images/multistage-build/
+
+# Provides a base Debian (10) image with latest buildbot mater installed
+# the master image is not optimized for size, but rather uses Debian for wider package availability
+
+# Provide an intermediate Docker image named "buildbot-build".
+# This intermediate image builds binary wheels
+# which get installed in the final image.
+# This allows us to avoid installing build tools like gcc in the final image.
+
+FROM        debian:10 AS buildbot-build
+MAINTAINER  Buildbot maintainers
+
+# Last build date - this can be updated whenever there are security updates so
+# that everything is rebuilt
+ENV         security_updates_as_of 2021-04-28
+
+RUN \
+    apt-get update && \
+    apt-get -y upgrade && \
+    apt-get -y install -q \
+        curl \
+        git \
+        libcairo-gobject2 \
+        libcairo2-dev \
+        libgirepository1.0-dev \
+        libglib2.0-dev \
+        libffi-dev \
+        libpq-dev \
+        libssl-dev \
+        pkg-config \
+        python3 \
+        python3-dev \
+        python3-pip \
+        yarnpkg \
+        tar \
+        tzdata \
+        virtualenv \
+        && \
+    rm -rf /var/lib/apt/lists/*
+
+COPY . /usr/src/buildbot
+
+RUN cd /usr/src/buildbot && make tarballs
+RUN virtualenv --python=python3 /buildbot_venv && \
+    /buildbot_venv/bin/pip3 install -r /usr/src/buildbot/requirements-master-docker-extras.txt && \
+    env CRYPTOGRAPHY_DONT_BUILD_RUST=1 /buildbot_venv/bin/pip3 install /usr/src/buildbot/dist/*.whl
+
+RUN mkdir -p /wheels && \
+    /buildbot_venv/bin/pip3 list --format freeze | grep -v '^buildbot' | grep -v '^pkg-resources' > /wheels/wheels.txt && \
+    cd /wheels && \
+    /buildbot_venv/bin/pip3 wheel -r wheels.txt && \
+    rm /wheels/wheels.txt && \
+    cp /usr/src/buildbot/dist/*.whl /wheels
+
+#==============================================================================================
+# Build the final image here.  Use build artifacts from the buildbot-build
+# container.
+
+# Note that the UI and worker packages are the latest version published on pypi
+# This is to avoid pulling node inside this container
+
+FROM        debian:10-slim
+MAINTAINER  Buildbot maintainers
+
+# Last build date - this can be updated whenever there are security updates so
+# that everything is rebuilt
+ENV         security_updates_as_of 2021-04-28
+
+COPY . /usr/src/buildbot
+
+RUN \
+    apt-get update && \
+    apt-get -y upgrade && \
+    apt-get -y install -q \
+        curl \
+        dumb-init \
+        git \
+        libpq5 \
+        libcairo2 \
+        openssh-client \
+        python3 \
+        python3-pip \
+        tar \
+        tzdata \
+        virtualenv \
+        && \
+    rm -rf /var/lib/apt/lists
+
+# Build wheels in other container using the Dockerfile.build
+# and copy them into this container.
+# We do this to avoid having to pull gcc for building native extensions.
+COPY --from=buildbot-build /wheels /wheels
+
+# install pip dependencies
+RUN pip3 install --upgrade pip setuptools && \
+    cd /wheels && pip3 install $(ls -1 | grep -v 'buildbot-worker') && \
+    rm -r /root/.cache /wheels
+
+WORKDIR /buildbot
+CMD ["dumb-init", "/usr/src/buildbot/docker/start_buildbot.sh"]
diff --git a/master/.dockerignore b/master/.dockerignore
diff --git a/master/Dockerfile b/master/Dockerfile
diff --git a/master/buildbot/newsfragments/master-docker-cleanups.feature b/master/buildbot/newsfragments/master-docker-cleanups.feature
@@ -0,0 +1 @@
+The docker container for the master has been switched to Debian.
diff --git a/master/buildbot/newsfragments/master-docker-full-build.bugfix b/master/buildbot/newsfragments/master-docker-full-build.bugfix
@@ -0,0 +1,2 @@
+The docker container for the master now fully builds the www packages.
+Previously they were downloaded from pypi which resulted in downloading whatever version was newest at the time (:issue:`4998`).
diff --git a/requirements-master-docker-extras.txt b/requirements-master-docker-extras.txt
@@ -1,3 +1,4 @@
 requests==2.25.1
 psycopg2==2.8.6
 txrequests==0.9.6
+pycairo==1.20.0

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1 @@`
	`1`	`+The docker container for the master has been switched to Debian.`
Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,2 @@`
	`1`	`+The docker container for the master now fully builds the www packages.`
	`2`	+Previously they were downloaded from pypi which resulted in downloading whatever version was newest at the time (:issue:`4998`).