Skip to content
This repository was archived by the owner on Apr 27, 2022. It is now read-only.

Latest commit

 

History

History
134 lines (100 loc) · 5.3 KB

testing.md

File metadata and controls

134 lines (100 loc) · 5.3 KB

Testing

Overview

Testing for HIL is done using pytest. All tests are placed in the "tests" subdirectory, which is subdivided into:

  • unit - for basic unit tests. These are safe to run without a full HIL enviornment (i.e. you don't need libvirt etc).
  • deployment - for tests that need to run against an actual setup, with libvirtd and at least one real switch.

Developers should run at least the unit and pycodestyle tests before making a commit. Ideally, the deployment tests should also be run, though we're less strict about this. Developers should also introduce tests for any new functions/methods they write, or any new or fixed functionality. Ideally, bug fixes should include a test (whether unit or functional) that reproduces the problem scenario so that we can ensure the bug doesn't return!

Configuration

Install HIL and its test dependencies in the virtual environment, done via: pip install -e .[tests] in the root directory. This will pull a recent version of pytest and you'll be ready to test the code. Using pip's -e option installs the hil in editable mode, which has the advantage that one need not reinstall every time a file is changed!

Most of the tests use a common set of default configuration options, as seen in testsuite.cfg.default. If you wish to override these parameters, you may copy it to testsuite.cfg and edit. In the future, this will allow developers to do things like test against different DBMSes (for the moment some changes are needed to the test suite to actually support this). For now, it's mostly interesting when running the deployment tests (see below).

Running

To run all tests, from the HIL root directory run:

py.test tests/

To run just a subset of them, specify a particular file or directory:

py.test tests/unit
py.test tests/unit/api.py

To run just a pycodestyle test:

pycodestyle *.py tests/ hil/

As stated above, running at least tests/unit is mandatory before each commit.

Code Coverage

The pytest-cov plugin (included in requirements.txt) allows us to generate reports indicating what code is/is not executed by our tests. setup.cfg configures pytest to output coverage information by default, so you mostly don't need to worry about this from a usage perspective -- just make sure you pay attention to the output.

More information is available on the projects PyPI page.

Test structure

Tests are kept in the tests directory, which is further organized into a few subdirectories:

  • unit
  • integration
  • deployment

For each file in the hil code, there should be a file with the same name in the unit directory. Within those files, classes (class names must begin with "Test") can be used to organize tests into functional areas. Function names must also begin with "test". See tests/unit/api.py for examples.

There also may be a few files loose in the tests directory that do not clearly fit into one of the above categories.

Integration tests

The tests/integration directory contains tests that require substantial setup of external software, or special configuration. These tend to be expensive in terms of time. These are run automatically by our travis-ci configuration.

Deployment tests

The deployment tests (tests/deployment) are a set of unit tests which are most useful when executed in an environment with real hardware and a libvirtd instance available. Some of these are executed by our travis-ci configuration using mock drivers, but (obviously) not against real hardware. They should be run before merging any patch relating to specific hardware support, or interacting with headnodes. To run the deployment tests, you must do the following:

  • Write a testsuite.cfg reflecting your environment. Copy examples/testsuite.cfg-deployment and edit. In particular, you will need to load the extensions for your switch drivers and the corresponding network allocator (see drivers.md), and specify extension-specific options.
  • Write a site-layout.json describing the layout of your environment. The file examples/site-layout.json provides an example. Here is a full description of the file format:

site-layout.json must contain a single json object, with two fields: "switches" and "nodes".

"switches" must be a list of json objects, each of which describes a switch in your environment, and must have the same fields as required in the body of the switch_register API call (see rest_api.md), plus a "switch" field, which supplies the name of the switch (normally specified in the URL).

"nodes" must be a list of json objects, each of which defines a node, and has three fields:

  • "name", a string which specifies the name of the node.
  • "nics", a list of objects each describing a nic on the node, with the fields (all of them strings):
    • "name", the name of the nic
    • "mac", the mac address of the nic
    • "switch", the name of the switch that the nic is connected to
    • "port", the name/label of the port on the switch that the nic is connected to
  • "obm", An object with the same set of fields as required by the obm field in the node_register API call.

The tests currently require at least four nodes to be specified in site-layout.json, each of which must have at least one nic connected to the switch.