Skip to content

Testing

Jump to bottom Edit New page
Ben Gur edited this page Nov 16, 2023 · 19 revisions

Introduction

Pykern provides a framework built on top of pytest to eliminate much of the boilerplate in testing.

The example below is a basic pykern test.

All tests are located in the tests directory in a repository. tests is structured identically to the repository; for example, if you have a pkcli test you would put the test file in tests/pkcli directory.

A test file ends _test.py. For example, the test for pykern.pkyaml is tests/pkyaml_test.py.

Let's write a test for echo() in the pykern.pkcli.pkexample module. echo() concatenates a suffix string and an optional prefix string. The first case test_echo() asserts that "hello world" is returned when prefix "hello " and suffix "world" are passed:

"""Test pykern.pkcli.pkexample

A collection of example command line tests:

    test_echo(): tests the concatenation of a prefix and suffix

:copyright: Copyright (c) 2022 RadiaSoft LLC.  All Rights Reserved.
:license: http://www.apache.org/licenses/LICENSE-2.0.html
"""
import pytest

def test_echo():
    from pykern import pkunit
    from pykern.pkcli import pkexample

    pkunit.pkeq('hello world', pkexample.echo(prefix='hello ', suffix='world'))

We can then run this test with:

$ pykern test tests/pkcli/pkexample_test.py
pkexample_test.py pass
passed=1

We return to this example in the sections below.

Test file

docstring

The module starts with a docstring. Use it to describe the general purpose of the module and the specific cases it contains.

import

In contrast to our usual style, pytest is only one global import. This is necessary to prevent other modules from initializing before the environment for the test is set up. In our example it doesn't matter, but more sophisticated tests[TODO(???): provide a link to such case] will break.

Case functions

The case functions must begin with test_. To exclude a case, change the name to, say, xtest_, and pytest will omit it from the cases to run.

A case function begins with some imports, typically including the module under test. These imports are unqualified to prevent modification of the global name space. [TODO(robnagler): need to link to DesignHints discussion which has yet to be written.] For convenient comparison functions, also import pykern.pkunit (unqualified).

The rest of the case function calls functions in the module under test and checks conditions. Our example has only one call and only one condition, but a case function may contain any number of conditions.

pykern.pkunit

pkunit supplies many functions. The most common is pkeq which compares an expected value on the left to the actual value on the right. pkeq also accepts the keyword arguments expect and actual; our example would then be

pkunit.pkeq(expect='hello world', actual=pkexample.echo(prefix='hello ', suffix='world'))

However, the simpler syntax is more convenient when multiplied over many conditions.

When expect and actual are equal, pkeq returns without a message. This is typical for condition functions: the point is to know when the actual value is unexpected. For example, let's change the suffix world to venus:

    pkunit.pkeq('hello world', pkexample.echo(prefix='hello ', suffix='venus'))

and run the test:

$ pykern test tests/pkcli/pkexample_test.py
pkexample_test.py FAIL pkexample_test.log
================================ test session starts =================================
platform linux -- Python 3.7.2, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /home/vagrant/.pyenv/versions/3.7.2/envs/py3/bin/python
cachedir: .pytest_cache
rootdir: /home/vagrant/src/radiasoft/pykern
plugins: anyio-3.3.0
collecting ... collected 1 item

pkexample_test.py::test_echo FAILED

====================================== FAILURES ======================================
_____________________________________ test_echo ______________________________________
Traceback (most recent call last):
  File "/home/vagrant/src/radiasoft/pykern/tests/pkcli/pkexample_test.py", line 12, in test_echo
    pkunit.pkeq('hello world', pkexample.echo(prefix='hello ', suffix='venus'))
  File "/home/vagrant/src/radiasoft/pykern/pykern/pkunit.py", line 292, in pkeq
    pkfail('expect={} != actual={}', expect, actual)
  File "/home/vagrant/src/radiasoft/pykern/pykern/pkunit.py", line 305, in pkfail
    raise PKFail('{} {}'.format(call, msg))
pykern.pkunit.PKFail: pkexample_test.py:12:test_echo expect=hello world != actual=hello venus
================================= 1 failed in 0.21s ==================================
error: FAILED=1 passed=0

Most of the output -- which can be voluminous -- is uninteresting. However, it is clear that the test failed, and why:

pykern.pkunit.PKFail: pkexample_test.py:12:test_echo expect=hello world != actual=hello venus

pkunit outputs all failures it detects the same way. Search for pkunit.PKFail to find the error message.

Run specific cases

To run a specific test case:

pykern test <path to python test file> case=<pattern to match the case name>

To run all test files under . or ./tests:

pykern test

Test file names must end with _test.py in order for pykern test to find them. Test output goes into _test.log

Note also that if you are using Bivio's emacs macros[TODO(???) add a link to these if possible], you can run the test with \C-c \C-m

PKunit module

The pykern.pkunit module is the bread and butter of the pykern testing framework. It contains various useful operations for unit tests.

Basics

Some basic pkunit operations include

pkfail

fmt_string = '{} failed'
pkunit.pkfail(fmt_string, 'this function')

Raises PKFail exception, with *args or **kwargs passed to fmt.format(*args, **kwargs) as message. PKFail is a subclass of AssertionError. All the operations below call pkfail() on failure.

pkeq

pkeq(expect, actual)

Passes if expect == actual; fails otherwise

pkexcept

with pkunit.pkexcept(<exception class>, <fmt_str>):
    <code that should trigger exception>

If the argument is an exception class, passes when the code raises that exception; fails otherwise. If the argument is a regex, passes when the code raises any exception whose message matches that regex; fails otherwise

For example, passing an exception:

with pkunit.pkexcept(ZeroDivisionError):
    a = 0
    b = 1
    c = b / a

(note that code under with pkunit.pkexcept() will not raise an exception of class <exception class>)

Passing a regex:

with pkunit.pkexcept(r'^missing'):
    assert 0, 'missing parameter in call'

pkok

pkunit.pkok(expression)

Passes if expression is truthy; fails otherwise

pkre

pkunit.pkre(regex, actual)

Passes if actual matches regex via re.search(); fails otherwise

This example from sirepo/tests/uri_router_test.py compares a uri value with a regex to make sure the uri has the correct format

pkunit.pkre('http://[^/]+/en$', uri)

data_dir

It's useful to keep files for a particular test in a single directory. data_dir() computes the data directory based on the test name. The test data directory is always <test>_data, where <test> is the name of the test's python module with the _test removed. For example, if the test file is setup_test.py then the directory will be setup_data.

For example, if the file setup_test.py contains

def test_example():
   f = data_dir('test_file.txt')
   text = pkio.read_text(f)
   actual = foo()
   pkeq(text, actual)

f would point to setup_data/test_file.txt.

We just have to know what the name of the file in data_dir is and we can easily get the path to it

work_dir

Also useful is a temporary directory for the output of test functions.

work_dir() returns ephemeral work directory, creating it if necessary, using the same rules as data_dir(). To continue the previous example, work_dir() in the file setup_test.py would create and reference setup_work/

empty_work_dir

If the directory defined in work_dir() exists, this removes it and its contents, recreates it, and returns the path

The following example from pykern/tests/pkio_test.py makes use of empty_work_dir() to test reading and writing text.

Note that res in res = pkio.write_text(str(expect_res), write_content) is the path to the file written by pkio.write_text, and that expect_res is the path to the work directory.

def test_write_text():
    """Also tests read_text"""
    from pykern import pkunit
    from pykern import pkio

    d = pkunit.empty_work_dir()
    expect_res = d.join('anything')
    expect_content = 'something\u2167'
    write_content = bytes(expect_content, 'utf-8')
    res = pkio.write_text(str(expect_res), write_content)
    assert expect_res == res, \
        'Verify result is file path as py.path.Local'
    with open(str(expect_res), 'rb') as f:
        assert write_content == f.read(), \
            'When write_text is called, it should write "something"'
    assert expect_content == pkio.read_text(str(expect_res)), \
        'When read_text, it should read "something"'

save_chdir_work

This calls empty_work_dir() and navigates to it with chdir() by calling pkio.save_chdir() on empty_work_dir().

with pkunit.save_chdir_work():
    <do stuff in work dir>

save_chdir_work() takes the default parameter is_pkunit_prefix (False by default). This flag determines if the newly created work_dir() will be the root of file I/O.

case_dirs

Sets up work_dir() by iterating the /*.in subdirectory in data_dir(). Every /<case-name>.in is copied recursively to /<case-name> in work_dir(). This function then yields that directory. If you want to only use cases from some specific /<case-name>.in subdir, and not all /*.in subdirs, you can pass a group_prefix default parameter value ('' by default) to case_dirs(). This will perform the regular operations but only on /<case-name>.in.

All files in data_dir() in the sub-directory <case-name>.out are expect files. Each of these expect files can be compared to the corresponding work_dir() actual file using file_eq.

To generate expect files for testing, touch the file in the appropriate .out directory. The test log will show you the differences and output a command to correct the data. This is easier and more reliable than generating the file yourself.

Returns: py.path.local: case directory created in work_dir (also PWD)

for d in pkunit.case_dirs():
    i = d.join(<fname>).read()
    ...

In the above example, d is the case directory copied to work_dir(). This means that in data_dir() there were subdirs named *.in. The subdirs copied to work_dir() can now be manipulated as needed for the test.

Example using group_prefix:

from pykern.pkcli import fmt
def test_group_prefix_example():
    for d in pkunit.case_dirs('fmt_dir'):
        fmt.run(d)

This example has group_prefix='fmt_dir'. It will iterate over data_dir/fmt_dir.in, copy those files to work_dir/fmt_dir to be used as input, run fmt.run() on the case_dir in question, and then file_eq() on the output and expected output copied from <case-name>.out. All other /*.in subdirs will be skipped over in this test

file_eq

file_eq is a flexible file comparison function.

The handle for file_eq is:

def file_eq(expect_path, *args, **kwargs):

It takes a path to a file in the work directory and either a positional argument specifying content for comparison, a keyword argument actual specifying the same content, or a keyword argument actual_path (string literal) specifying the path to a file for comparison. The three options are mutually exclusive. If file_eq() fails, it will output a diff to help pinpoint the discrepancies.

If expect_path and actual_path both exist, they will be compared as plain text. If actual_path does not exist, it will be created from actual. expect_path can optionally be passed to file_eq() as a keyword param.

When comparing a .json file to content specified by actual, file_eq() uses pkjson to load and parse the file to make the comparison, and to write the contents of actual to a file in work_dir().

file_eq() can compare compare .txt, .csv., jinja, or .json files, and, if the pandas package is installed, .xlsx. For file_eq() and case_dirs(), if the name of the expect (out) file is foo.csv, then the first sheet (sheet 0) in the corresponding work_dir xlsx will be converted to foo.csv before comparison. If the expect (out) file has a #, e.g. foo#3.csv, then the fourth sheet will be extracted from the actual xlsx to foo#3.csv in the work_dir.

When comparing .jinja, file_eq() renders it with pykern.pkjina.render_file using a jinja context supplied by the j2_ctx keyword argument

file_eq Examples:

Basic usage:

file_eq('expected_results.txt', actual_path='actual_results.txt')

The example below shows how to compare .xlsx actual with .csv expect:

file_eq(
    expect_path=’example.csv’,
    actual_path='example.xlsx'
)

Note:

If file_eq() fails and the diff includes a No newline at end of file message for both expect and actual, it can be ignored. However, there may be other differences, so carefully check the result of the test.

Add a custom footer
Add a custom sidebar
Clone this wiki locally