Skip to content

Release Process

Jump to bottom
Otto van der Schaaf edited this page Oct 25, 2023 · 13 revisions

Overview

The release process has two components:

  1. Things that can be done anywhere (cutting the branch, writing release notes, building binaries)
  2. Things that have to be done at Google (signing the binaries, deploying them to dl.google.com)

Most of the process is in the first category, and this doc only covers that category.

We have two kinds of releases:

  1. Feature releases: cut a new branch off of master, which contains all work up to this point.
  2. Point release: patch a few small changes onto an existing branch.
  3. Stable release: re-release an existing beta release as stable.

For example:

  • Release 1.9.32.1 was a feature release
  • All later 1.9.x.x releases (1.9.31.2 through 1.9.32.14) were point releases.
  • Release 1.9.32.3 was released as beta on 2015-01-05 and re-released as stable on 2015-01-14.

A feature release is generally a lot more work to get out, because we've made larger changes and it's more likely that something has broken that needs to be fixed before releasing. A point release, on the other hand, should be as small as possible to minimize the risk of breaking one thing while fixing another. Often this means we'll fix a bug two ways if the real fix involves a refactor: release a minimal fix for a point release, and put the larger refactor on master for the future.

Point releases are further broken down into "bugfix" and "security" releases. A bugfix release does exactly that: fixes bugs. We won't generally put one out unless there are serious bugs, but if there's a bug serious enough to be putting out a release then we'll generally backport fixes for less serious bugs to release along with it. Security releases are ones that people need to apply to avoid their site being exploited. Because we want people to apply these as quickly as possible, other fixes should go out in a separate bugfix release.

We release both beta and stable versions of mps and nps. Version x.x.x.x-beta and x.x.x.x-stable always have the same code; beta vs stable is just metadata to let sites know what they're likely to want to run. New feature releases start as beta, often have a few rounds of bugfix releases, have a few months of time for issues to crop up, and then once we have a version we think is solid we release the most recent point release as stable. For example, 1.9.32.1 was released as beta on 2014-09-16, then we made a security update to it on 2014-10-27 as 1.9.32.2 and a bug fix release on 2015-01-05 as 1.9.32.3, and then re-released 1.9.32.3 as stable on 2015-01-14.

Branching approach

For each feature release series we maintain a branch. The name (number) of the branch is the third component of the release version. For example, the 1.12.x.x series of releases is based on branch 34, and the 1.10.x.x and 1.11.x.x series were based on branch 33. When working on a release you work with the branch, and then when you have something ready to release you'll tag it.

Feature releases: cutting a branch

cd .../mod_pagespeed
git checkout master
git checkout -b NN
git push --set-upstream origin NN

cd .../ngx_pagespeed
git checkout master
git checkout -b NN
git push --set-upstream origin NN

If the current most recent branch is 34 (1.12.x.x) then you'd make a new branch and call it 35. Any changes you need to make to get this release ready should be committed to this branch.

Update VERSION

Edit net/instaweb/public/VERSION to match the expected release version. For example, change:

MAJOR=1
MINOR=13
BUILD=0
PATCH=0

to:

MAJOR=1
MINOR=13
BUILD=35
PATCH=1

Then commit:

git commit -a -m "Update VERSION for release 1.13.35.1"

Update lastchange.sh

When building from development branches mod_pagespeed reports its version as x.x.x.x-REVISION, where REVISION is the number of commits we've made on the master branch. For a release branch, however, we want it to report x.x.x.x-0:

git cherry-pick 2a4b2047

Update version on master

Builds from master always report with a minor version number one more than the current one, a branch number of 0, and a patch number of 0. For example, currently our most recent release is 1.12.34.2 so master builds report as 1.13.0.0. When you cut a feature release, update net/instaweb/public/VERSION on master to increment MINOR and zero BUILD and PATCH.

Point releases: use the existing branch

If you're making a point release, say 1.13.35.7, then first check out the branch for that release (35). Now you can cherry-pick commits onto it. For example:

cd .../mod_pagespeed
git checkout 35
git cherry-pick COMMIT-HASH
git cherry-pick OTHER-COMMIT-HASH
git push

Then update net/instaweb/public/VERSION to increment the patch number.

Stable releases: no code changes

When re-releasing a beta release as stable we don't make any code changes, we just run the commands below with stable instead of beta.

Release Notes

Start a draft of the release notes early: they take a surprisingly long time to get right.

The goal is to list everything that has changed between releases that users might care about. For a patch release this is simple: list the bugs fixed and the commits applied. For a feature release it's a bit more work to get a picture of what changed:

  • Look over bugs that have been marked fixed since the last release (both for nps and mps)
  • Look over pull requests that have been closed since the last release (both for nps and mps)
  • Don't look at the full diff between versions, because that's way too big, but look at diffs for files that often correspond to changes we should be noting:
    • apache/mod_instaweb.cc
    • system/system_rewrite_options.cc
    • rewriter/rewrite_filter_names.gperf
    • rewriter/rewrite_options.cc
    • everything under doc/
    • ngx_rewrite_options.cc in ngx_pagespeed
  • Also look over the open issues for mps and nps to find ones that are fixed but are still marked as open

Make edits to doc/release_notes.html, and send out a pull request for review.

Building binaries

We normally build the following files for release on dl.google.com:

  • mod_pagespeed: 32bit rpm
  • mod_pagespeed: 32bit deb
  • mod_pagespeed: 64bit rpm
  • mod_pagespeed: 64bit deb
  • mod_pagespeed: source tarball
  • psol: 32 bit headers+binary tarball
  • psol: 64 bit headers+binary tarball

We build these on GCE instances

Building mod_pagespeed rpms, debs, and PSOL tarballs

Note: if you're internal to google you already have the cloud sdk and a gcloud project; see the internal release supplement doc.

Install the cloud sdk for talking to GCE (https://cloud.google.com/sdk/) and create a gcloud project to do the builds under.

PATH=/path/to/cloud-sdk
CLOUDSDK_COMPUTE_ZONE=us-central1-a
BRANCH=36  # can't use a tag because CentOS git is too old to understand tags in "git clone -b BRANCH"
gcloud config set project [gcloud project name]
gcloud auth login
git checkout $BRANCH
# For beta releases, omit --stable
install/build_on_vm.sh --build_branch=$BRANCH --centos -- --stable
install/build_on_vm.sh --build_branch=$BRANCH --centos -- --32bit --stable
install/build_on_vm.sh --build_branch=$BRANCH -- --skip_psol --stable
install/build_on_vm.sh --build_branch=$BRANCH -- --32bit --skip_psol --stable

build_on_vm.sh will spinup a vm, clone $BRANCH, build and test it and then (if successful) drop the final objects in ~/release/$RELEASE.

Building the source tarball

This is what the OpenSUSE maintainer builds from, and potentially other distributions might use this as well.

First check out the branch you're working on:

git checkout NN

Then run:

devel/build_release_tarball.sh <beta|stable>

This will install deps, build a tarball, check that it compiles, and run tests on it. It puts the tarball in ~/release/VERSION/.

[note: this is written assuming https://github.com/we-amp/mod_pagespeed/pull/1474 lands without interface changes]

Verify status

At this point, your ~/release/VERSION should contain:

  • OS packages:
    • mod-pagespeed-[beta|stable]-VERSION-0.i386.rpm
    • mod-pagespeed-[beta|stable]-VERSION-0.x86_64.rpm
    • mod-pagespeed-[beta|stable]_VERSION-r0_amd64.deb
    • mod-pagespeed-[beta|stable]_VERSION-r0_i386.deb
  • PSOL binary tarballs: * *
  • Source tarball:
    • mod-pagespeed-[beta|stable]-VERSION-r0.tar.bz2

Load tests

Point releases should only contain changes small enough that they shouldn't be performance relevant, and so can generally skip load tests. If you're doing a feature release, though, you definitely need to run load tests.

We have two kinds of load tests: one that tries to run PageSpeed in full, like in real usage, and another test that runs a series of microbenchmarks. We call these the "load test" and the "siege tests" though really they're all load tests.

The goal of load testing is to make sure our performance doesn't go down over time as we add features and things get more complex. Because machines vary over time, to get an apples-to-apples comparison we run load tests for the previous release and the current release.

First we need to get copies of the two .so files to compare: the currently released beta version and the version you're preparing to release. These are in the .deb files, so we need to get those first:

  • For the most recent beta, download https://dl-ssl.google.com/dl/linux/direct/mod-pagespeed-beta_current_amd64.deb
  • For your current release: this should be ~/release/VERSION/mod-pagespeed-beta_VERSION-r0_amd64.deb

Now we can use devel/extract_so_from_deb.sh to pull the apache 2.2 and 2.4 .so files out:

$ ls
mod-pagespeed-beta_VERSION-r0_amd64.deb
mod-pagespeed-beta_current_amd64.deb

$ .../net/instaweb/scripts/extract_so_from_deb.sh mod-pagespeed-beta_VERSION-r0_amd64.deb
The .so files are:
  /tmp/tmp.AbrjbobkzA/mod_pagespeed.so
  /tmp/tmp.AbrjbobkzA/mod_pagespeed_ap24.so

$ cp /tmp/tmp.AbrjbobkzA/mod_pagespeed.so VERSION-mod_pagespeed.so
$ cp /tmp/tmp.AbrjbobkzA/mod_pagespeed_ap24.so VERSION-mod_pagespeed_ap24.so

... repeat for the other deb ...

$ ls .*.so
VERSION-mod_pagespeed_ap24.so
VERSION-mod_pagespeed.so
CURRENT-mod_pagespeed_ap24.so
CURRENT-mod_pagespeed.so

Traditional load tests

The traditional load test needs a corpus. There's one that Google uses internally that can't be open sourced for IP reasons, but as of 2017-01-11 Maks is writing an open-source corpus generator: https://github.com/we-amp/mod_pagespeed/pull/1473

For now I'll assume you have a corpus in /path/to/corpus

This load test selects pages and resources from the corpus and loads them in a random order. Best not to do anything else with your machine at the same time, to get more reproducible results. To start it run:

devel/mps_load_test.sh -custom_so /path/to/so_to_test /path/to/corpus

At the end it prints out the queries-per-second, a histogram of response codes, and greps the Apache error_log for "exit signal". The presence of such text should be investigated. Absence of such makes us happy.

To get comparisons across versions you'll need to make a pair of config files that should give similar behavior. For example, if we added a new feature that's optional and slow, we should make sure that we're not load testing with that feature on.

Run the load test with both the new version and the old version, and send the QPS numbers to [email protected]. If there's a major regression, figure out what it is, bisecting if necessary. With 1.10 we saw a qps decrease that, on bisecting, turned out to be us removing an extra layer of caching from our slurp, which was a load-test specific change. (And then by the time we cut the final version we had merged some last-minute changes with large QPS improvements.)

Multiple Apache Versions

Please load-test using multiple Apache version & MPMs. These can all be built using install/build_development_apache.sh, which builds a new ~/apache2 with the version you specify. For example:

install/build_development_apache.sh 2.2 prefork
[run load tests for both new and old versions]
install/build_development_apache.sh 2.2 worker
[run ...]
install/build_development_apache.sh 2.4 event

What if there are crashes?

We need to resolve crashes before releasing. Rare shutdown crashes might be ok in some cases, since the way we run arbitrary duration background optimization tasks isn't a good fit for graceful restarts, but if you run into anything like this make sure everyone else on pagespeed-dev@ is ok with you continuing.

What if I don't get the expected QPS?

You really need to stop and consider that the release might be broken performance-wise. It is pretty important that we don't ruin performance in the pursuit of features!

Siege tests

To run the siege load tests, just do devel/siege/siege_all.sh

[TODO: doc how to run this with custom .sos.]

Tagging the release

If the tests results look good, tag the mod_pagespeed release: For 1.13.35.1 I ran:

git tag -a "v1.13.35.1" -m "Release 1.13.35.1"
git push origin v1.13.35.1

ngx_pagespeed release

Follow https://github.com/we-amp/ngx_pagespeed/wiki/creating-a-release but don't tag latest-beta until we're ready to go live.

Documentation changes

Update versions

When we release all our recent work moves from UNRELEASED to a specific version. So cd into the html/ directory and run:

RELEASE=a.b.c.d
sed -i -e "s/\<UNRELEASED\>/$RELEASE/" $(grep -rwl UNRELEASED .)

and send it out for review.

Examples tarball

We need to tar up the examples directory to copy up to modpagespeed.com:

cd mod_pagespeed/install
tar -czf mod_pagespeed_examples.tar.gz mod_pagespeed_example/

PSOL documentation

First, install doxygen:

sudo apt-get install doxygen

Build the documentation from an open-source client synced to the same revision as the released code (for branch 34, you will need to copy the devel/ directory from master):

cd devel
make doxygen

This creates psol_doc.tar.gz in the git root. Copy that to modpagespeed.com and install it:

scp psol_doc.tar.gz [email protected]:
ssh [email protected]
cd /var/www/html
mkdir psol-$RELEASE
cd psol-$RELEASE
tar xzf ~/psol_doc.tar.gz
chmod -R a+rX .
cd ..
ln -Tsf psol-$RELEASE/html psol

Verify that your new version is visible on the internet, by browsing to http://modpagespeed.com/psol/. Check the version at the top.

Make sure everything is ready

Before asking google to make things live, make sure:

  • the release notes are approved and ready to be deployed
  • the documentation changes are ready to go in
  • the announcement emails are drafted
  • the mps examples tarball is ready to deploy

Google internal steps

Now people at google need to follow the google-internal release docs to:

  • sign binaries
  • put binaries on dl.google.com
  • stage the linux repos
    • they won't deploy the changes until it's time to announce

Time to announce:

  • tell google to deploy the linux repo changes
  • update modpagespeed.com and ngxpagespeed.com to the new version
  • submit the release notes and documentation pull requests, and deploy them to modpagespeed.com
  • deploy the examples tarball to modpagespeed.com and ngxpagespeed.com following our doc
  • send out the announcement emails to mod-pagespeed-announce and ngx-pagespeed-announce, ccing mod-pagespeed-discuss and ngx-pagespeed-discuss
  • retag latest-[beta|stable] for both mod_pagespeed and ngx_pagespeed
  • update https://github.com/we-amp/mod_pagespeed/issues/968 with a new source tarball link
Clone this wiki locally