Feature: DragonTest Unit Testing Framework

[Rain336]

Feature: DragonTest Unit Testing Framework

Microdragon needs unit testing, but the default libtest framework cannot run without libstd, so a new testing framework needs to be written using the custom_test_frameworks nightly feature. This new test framework should offer most of the features of libtest and maybe some extensions specific to testing kernels.

Writing Tests

Writing tests should be as similar to libtest as possible. To do this we need to overwrite a few builtin macros normally used by libtest with our own proc-macros.

• #[test] and #[bench]
#[test] and #[bench] are used to write tests and benchmarks respectively.
  Our macros, similar to their builtin counterparts, will generate a static item with a #[test_case] attribute macro. This item will be of type TestSpec, our version of TestDescAndFn from libtest, but more simplified, since we don't need to support dynamic tests.

pub enum TestFn {
    Test(fn() -> bool),
    Benchmark(fn(&mut Bencher) -> bool),
}

pub enum ShouldPanic {
    No,
    Yes,
    YesWithMessage(&'static str),
}

pub struct TestSpec {
    pub func: TestFn,
    pub name: &'static str,
    pub ignore: bool,
    pub ignore_message: Option<&'static str>,
    pub source_file: &'static str,
    pub should_panic: ShouldPanic,
}

Here a few examples of how functions would be translated:

#[test]
fn sample_test() {}

#[bench]
fn sample_bench() {}

fn sample_test() {}

#[test_case]
static SAMPLE_TEST_SPEC: TestSpec = TestSpec {
    func: TestFn::Test(|| dragontest::test_result(sample_test())),
    name: concat!(module_path!(), "::", stringify!(sample_test)),
    ignore: false,
    ignore_message: None,
    source_file: file!(),
    should_panic: ShouldPanic::No
};

fn sample_bench() {}

#[test_case]
static SAMPLE_BENCH_SPEC: TestSpec = TestSpec {
    func: TestFn::Benchmark(|b| dragontest::test_result(sample_bench(b))),
    name: concat!(module_path!(), "::", stringify!(sample_bench)),
    ignore: false,
    ignore_message: None,
    source_file: file!(),
    should_panic: ShouldPanic::No
};

• #[shuld_panic]
#[shuld_panic] is an additional attribute macro that can be used in combination with #[test] and #[bench]. It cannot stand alone, so no proc-macro function is written, instead the #[test] and #[bench] macros check for it. It controls the should_panic field in the TestSpec item. Here a few examples of how functions would be translated:

#[test]
#[shuld_panic]
fn sample_test() {}

#[test]
#[shuld_panic(expected = "panic message contains this")]
fn sample_test_message() {}

fn sample_test() {}

#[test_case]
static SAMPLE_TEST_SPEC: TestSpec = TestSpec {
    func: TestFn::Test(|| dragontest::test_result(sample_test())),
    name: concat!(module_path!(), "::", stringify!(sample_test)),
    ignore: false,
    ignore_message: None,
    source_file: file!(),
    should_panic: ShouldPanic::Yes
};

fn sample_test_message() {}

#[test_case]
static SAMPLE_TEST_MESSAGE_SPEC: TestSpec = TestSpec {
    func: TestFn::Test(|| dragontest::test_result(sample_test_message())),
    name: concat!(module_path!(), "::", stringify!(sample_test_message)),
    ignore: false,
    ignore_message: None,
    source_file: file!(),
    should_panic: ShouldPanic::YesWithMessage("panic message contains this")
};

• #[ignore]
#[ignore], similar to #[shuld_panic] is an additional attribute macro that can be used in combination with #[test] and #[bench]. It also has no proc-macro function and instead the #[test] and #[bench] macros check for it. It controls the ignore and ignore_message fields in the TestSpec item. Here a few examples of how functions would be translated:

#[test]
#[ignore]
fn sample_test() {}

#[test]
#[ignore = "This test is ignored"]
fn sample_test_message() {}

fn sample_test() {}

#[test_case]
static SAMPLE_TEST_SPEC: TestSpec = TestSpec {
    func: TestFn::Test(|| dragontest::test_result(sample_test())),
    name: concat!(module_path!(), "::", stringify!(sample_test)),
    ignore: true,
    ignore_message: None,
    source_file: file!(),
    should_panic: ShouldPanic::No
};

fn sample_test_message() {}

#[test_case]
static SAMPLE_TEST_MESSAGE_SPEC: TestSpec = TestSpec {
    func: TestFn::Test(|| dragontest::test_result(sample_test_message())),
    name: concat!(module_path!(), "::", stringify!(sample_test_message)),
    ignore: true,
    ignore_message: Some("This test is ignored"),
    source_file: file!(),
    should_panic: ShouldPanic::No
};

• Optionally, assert!, assert_eq!, assert_ne! and assert_match!
  The default implementation for assert!, assert_eq!, assert_ne! and assert_match! can be used, but a more verbose and pretty version similar to pretty_assertions should be considered.

Running Tests

The plan is to integrate with the cargo test command, which requires us to write a runner that can package and iso and start qemu for running the tests. This would require either a complete reimplementation of the Justfile in rust or modifying the Justfile to be able to act as our test runner. I would go the route of rewriting the Justfile in rust and actually replacing it completely with it. Then we could also define a runner for normal builds and make cargo run work.

Additionally, cargo test accepts different options that are processed by libtest, most of these have to be implemented by us too to provide adequate support. The following are not supported:

• --force-run-in-process
  • Tests are always run in-process, out-of-process is not supported.
• --logfile <PATH>
  • The test framework is run inside a VM, so it cannot access the filesystem.
  • This could be implemented by the test runner capturing the serial port output and writing it to a file.
• --nocapture
  • All tests print directly to the console
  • Custom print! and println! macros could be provided
• --test-threads
  • Tests are not run in parallel
  • This could be a future addition, but out of scope for now
• -Z
  • Unstable options are always enabled. They are either implemented or not.

For the command line options to be considered, they also need to be passed into the VM, this can be done by passing the options as bootloader kernel arguments. Most bootloaders support this and it can be easily prepared by the test runner's packing step.

Integrating the Test Framework

The test framework supplies a main entry point that can be passed to the #![test_runner] macro. Then the re-exported test main function just has to be called to start the test framework.

Handling panics requires another function from the test framework to process the panic and continue testing.

Open Questions

• Should the processor be reset to a known good state, before each test?
  • Set all control registers to default values from the test framework?
• This feature seems to get pretty big, should it maybe be split out as an external project?
  • A lot of other rust kernels could make good use of it too
  • This is slowly turning into a kernel of it's own

[Rain336]

Considering we are running in a VM, getting a known good timer is actually easy, since the testing framework only needs to support one timer that is guaranteed to be provided by QEMU. For x86_64 this will most likely be the TimeStamp Counter (TSC) which can be easily used to implement a monotonic clock, similar to Instant. This is basically all we need for time tracking.