Skip to content

Commit 7199c87

Browse files
Geoff Andersongwenshap
Geoff Anderson
authored and
gwenshap
committed
MINOR: Added scripts to automate Vagrant setup for system tests
Updated testing README accordingly. Author: Geoff Anderson <[email protected]> Reviewers: Ewen Cheslack-Postava, Gwen Shapira Closes apache#201 from granders/minor-vagrant-package-script
1 parent d88b5bd commit 7199c87

File tree

4 files changed

+181
-33
lines changed

4 files changed

+181
-33
lines changed

Diff for: ‎tests/README.md

+11-33
Original file line numberDiff line numberDiff line change
@@ -14,51 +14,29 @@ https://cwiki.apache.org/confluence/display/KAFKA/tutorial+-+set+up+and+run+Kafk
1414

1515
* Install Virtual Box from [https://www.virtualbox.org/](https://www.virtualbox.org/) (run `$ vboxmanage --version` to check if it's installed).
1616
* Install Vagrant >= 1.6.4 from [http://www.vagrantup.com/](http://www.vagrantup.com/) (run `vagrant --version` to check if it's installed).
17-
* Install Vagrant Plugins:
17+
* Install system test dependiences, including ducktape, a command-line tool and library for testing distributed systems.
1818

19-
# Required
20-
# Note that vagrant-hostmanager v1.6.0 and up breaks our Vagrant scripts
21-
$ vagrant plugin install vagrant-hostmanager --plugin-version 1.5.0
22-
$ vagrant plugin install vagrant-cachier
19+
$ python tests/setup.py develop
2320

24-
* Build a specific branch of Kafka
25-
26-
$ cd kafka
27-
$ git checkout $BRANCH
28-
$ gradle
29-
$ ./gradlew jar
30-
31-
* Setup a testing cluster with Vagrant. Configure your Vagrant setup by creating the file
32-
`Vagrantfile.local` in the directory of your Kafka checkout. For testing purposes,
33-
`num_brokers` and `num_kafka` should be 0, and `num_workers` should be set high enough
34-
to run all of you tests. An example resides in kafka/vagrant/system-test-Vagrantfile.local
35-
36-
# Example Vagrantfile.local for use on local machine
37-
# Vagrantfile.local should reside in the base Kafka directory
38-
num_zookeepers = 0
39-
num_kafka = 0
40-
num_workers = 9
21+
* Run the bootstrap script to set up Vagrant for testing
22+
23+
$ tests/bootstrap-test-env.sh
4124

42-
* Bring up the cluster (note that the initial provisioning process can be slow since it involves
43-
installing dependencies and updates on every vm.):
25+
* Bring up the test cluster
4426

4527
$ vagrant up
4628

47-
* Install ducktape:
29+
* Build the desired branch of Kafka
4830

49-
$ pip install ducktape
50-
31+
$ git checkout $BRANCH
32+
$ gradle # (only if necessary)
33+
$ ./gradlew jar
34+
5135
* Run the system tests using ducktape:
5236

5337
$ cd tests
5438
$ ducktape kafkatest/tests
5539

56-
* If you make changes to your Kafka checkout, you'll need to rebuild and resync to your Vagrant cluster:
57-
58-
$ cd kafka
59-
$ ./gradlew jar
60-
$ vagrant rsync # Re-syncs build output to cluster
61-
6240
EC2 Quickstart
6341
--------------
6442
This quickstart will help you run the Kafka system tests on EC2. In this setup, all logic is run

Diff for: ‎tests/bootstrap-test-env.sh

+94
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
#!/usr/bin/env bash
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
# This script automates the process of setting up a local machine for running Kafka system tests
18+
19+
# Helper function which prints version numbers so they can be compared lexically or numerically
20+
function version { echo "$@" | awk -F. '{ printf("%03d%03d%03d%03d\n", $1,$2,$3,$4); }'; }
21+
22+
base_dir=`dirname $0`/..
23+
cd $base_dir
24+
25+
echo "Checking Virtual Box installation..."
26+
bad_vb=false
27+
if [ -z `vboxmanage --version` ]; then
28+
echo "It appears that Virtual Box is not installed. Please install and try again (see https://www.virtualbox.org/ for details)"
29+
bad_vb=true
30+
else
31+
echo "Virtual Box looks good."
32+
fi
33+
34+
echo "Checking Vagrant installation..."
35+
vagrant_version=`vagrant --version | egrep -o "\d+\.\d+\.\d+"`
36+
bad_vagrant=false
37+
if [ "$(version $vagrant_version)" -lt "$(version 1.6.4)" ]; then
38+
echo "Found Vagrant version $vagrant_version. Please upgrade to 1.6.4 or higher (see http://www.vagrantup.com for details)"
39+
bad_vagrant=true
40+
else
41+
echo "Vagrant installation looks good."
42+
fi
43+
44+
if [ "x$bad_vagrant" == "xtrue" -o "x$bad_vb" == "xtrue" ]; then
45+
exit 1
46+
fi
47+
48+
echo "Checking for necessary Vagrant plugins..."
49+
install_hostmanager=false
50+
hostmanager_version=`vagrant plugin list | grep vagrant-hostmanager | egrep -o "\d+\.\d+\.\d+"`
51+
if [ -z "$hostmanager_version" ]; then
52+
install_hostmanager=true
53+
elif [ "$hostmanager_version" != "1.5.0" ]; then
54+
echo "You have the wrong version of vagrant plugin vagrant-hostmanager. Uninstalling..."
55+
vagrant plugin uninstall vagrant-hostmanager
56+
install_hostmanager=true
57+
fi
58+
if [ "x$install_hostmanager" == "xtrue" ]; then
59+
vagrant plugin install vagrant-hostmanager --plugin-version 1.5.0
60+
fi
61+
62+
echo "Creating and packaging a reusable base box for Vagrant..."
63+
vagrant/package-base-box.sh
64+
65+
# Set up Vagrantfile.local if necessary
66+
if [ ! -e Vagrantfile.local ]; then
67+
echo "Creating Vagrantfile.local..."
68+
cp vagrant/system-test-Vagrantfile.local Vagrantfile.local
69+
else
70+
echo "Found an existing Vagrantfile.local. Keeping without overwriting..."
71+
fi
72+
73+
# Sanity check contents of Vagrantfile.local
74+
echo "Checking Vagrantfile.local..."
75+
vagrantfile_ok=true
76+
num_brokers=`egrep -o "num_brokers\s*=\s*\d+" Vagrantfile.local | cut -d '=' -f 2 | xargs`
77+
num_zookeepers=`egrep -o "num_zookeepers\s*=\s*\d+" Vagrantfile.local | cut -d '=' -f 2 | xargs`
78+
num_workers=`egrep -o "num_workers\s*=\s*\d+" Vagrantfile.local | cut -d '=' -f 2 | xargs`
79+
if [ "x$num_brokers" == "x" -o "$num_brokers" != 0 ]; then
80+
echo "Vagrantfile.local: bad num_brokers. Update to: num_brokers = 0"
81+
vagrantfile_ok=false
82+
fi
83+
if [ "x$num_zookeepers" == "x" -o "$num_zookeepers" != 0 ]; then
84+
echo "Vagrantfile.local: bad num_zookeepers. Update to: num_zookeepers = 0"
85+
vagrantfile_ok=false
86+
fi
87+
if [ "x$num_workers" == "x" -o "$num_workers" == 0 ]; then
88+
echo "Vagrantfile.local: bad num_workers (size of test cluster). Set num_workers high enough to run your tests."
89+
vagrantfile_ok=false
90+
fi
91+
92+
if [ "$vagrantfile_ok" == "true" ]; then
93+
echo "Vagrantfile.local looks good."
94+
fi

Diff for: ‎vagrant/package-base-box.sh

+75
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
#!/usr/bin/env bash
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
# This script automates the process of creating and packaging
18+
# a new vagrant base_box. For use locally (not aws).
19+
20+
base_dir=`dirname $0`/..
21+
cd $base_dir
22+
23+
backup_vagrantfile=backup_Vagrantfile.local
24+
local_vagrantfile=Vagrantfile.local
25+
26+
# Restore original Vagrantfile.local, if it exists
27+
function revert_vagrantfile {
28+
rm -f $local_vagrantfile
29+
if [ -e $backup_vagrantfile ]; then
30+
mv $backup_vagrantfile $local_vagrantfile
31+
fi
32+
}
33+
34+
function clean_up {
35+
echo "Cleaning up..."
36+
vagrant destroy -f
37+
rm -f package.box
38+
revert_vagrantfile
39+
}
40+
41+
# Name of the new base box
42+
base_box="kafkatest-worker"
43+
44+
# vagrant VM name
45+
worker_name="worker1"
46+
47+
echo "Destroying vagrant machines..."
48+
vagrant destroy -f
49+
50+
echo "Removing $base_box from vagrant..."
51+
vagrant box remove $base_box
52+
53+
echo "Bringing up a single vagrant machine from scratch..."
54+
if [ -e $local_vagrantfile ]; then
55+
mv $local_vagrantfile $backup_vagrantfile
56+
fi
57+
echo "num_workers = 1" > $local_vagrantfile
58+
echo "num_brokers = 0" >> $local_vagrantfile
59+
echo "num_zookeepers = 0" >> $local_vagrantfile
60+
vagrant up
61+
up_status=$?
62+
if [ $up_status != 0 ]; then
63+
echo "Failed to bring up a template vm, please try running again."
64+
clean_up
65+
exit $up_status
66+
fi
67+
68+
echo "Packaging $worker_name..."
69+
vagrant package $worker_name
70+
71+
echo "Adding new base box $base_box to vagrant..."
72+
vagrant box add $base_box package.box
73+
74+
clean_up
75+

Diff for: ‎vagrant/system-test-Vagrantfile.local

+1
Original file line numberDiff line numberDiff line change
@@ -20,3 +20,4 @@
2020
num_zookeepers = 0
2121
num_brokers = 0
2222
num_workers = 9
23+
base_box = "kafkatest-worker"

0 commit comments

Comments
 (0)