Skip to content

Latest commit

 

History

History
186 lines (154 loc) · 8.53 KB

ModelUnitTests.md

File metadata and controls

186 lines (154 loc) · 8.53 KB

Model Unit Tests

Oxidized includes automated unit tests for its models, which require very little effort to use. There are three different default unit tests for models:

You only need to provide test files under /spec/model/data, and the tests will be run automatically with rake test. See CONTRIBUTING.md for explanations on how to set up a development environment. In the following examples, we use Bundler and prefix all commands with bundle exec.

In addition, you can write custom unit tests for your model

Device Simulation

YAML Simulation File

You need a YAML simulation file for your device. See the link for instructions on how to produce it.

The YAML simulation files are stored under /spec/model/data/, with the naming convention <model>:<description>:simulation.yaml, where <model> is the lowercase name of the Oxidized model and <description> is the name of the test case. <description> is generally formatted as <hardware>_<software> or <hardware>_<software>_<information>.

Using a correct name for the file is important to ensure it is included in automatic model unit tests.

Expected Output

You need a second file that contains the expected output, which has the same name as the YAML simulation file but ends with :output.txt instead of :simulation.yaml.

You can automatically produce an output file based on the current model for all YAML simulation files missing an :output.txt:

bundle exec ruby spec/model/atoms_generate.rb

In the following example, spec/model/data/aoscx:R8N85A-C6000-48G-CL4_PL.10.08.1010:output.txt (the second file in the list) was missing:

oxidized$ bundle exec ruby spec/model/atoms_generate.rb
Run options: --seed 57811

# Running:

Generating output file for aoscx:R0X25A-6410_FL.10.10.1100:simulation... SKIP, output already exists
Generating output file for aoscx:R8N85A-C6000-48G-CL4_PL.10.08.1010:simulation... OK
Generating output file for arubainstant:IAP515_8.10.0.6_VWLC:simulation... SKIP, output already exists
Generating output file for asa:5512_9.12-4-67_single-context:simulation... SKIP, output already exists
Generating output file for garderos:R7709_003_006_068:simulation... SKIP, output already exists
Generating output file for ios:C8200L_16.12.1:simulation... FAIL, no simulation file
Generating output file for ios:C9200L-24P-4G_17.09.04a:simulation... SKIP, output already exists
Generating output file for ios:C9800-L-F-K9_17.06.05:simulation... SKIP, output already exists
Generating output file for ios:asr920_16.8.1b:simulation... SKIP, output already exists
Generating output file for junos:srx300_22.4:simulation... SKIP, output already exists
Generating output file for opnsense:nano_23.7:simulation... SKIP, output already exists
Generating output file for pfsense:CE_2.7.2:simulation... SKIP, output already exists
Generating output file for routeros:CHR_7.10.1:simulation... SKIP, output already exists
Generating output file for routeros:CHR_7.16:simulation... SKIP, output already exists
Generating output file for routeros:L009UiGS_7.15.2:simulation... SKIP, output already exists
.

Finished in 0.904792s, 1.1052 runs/s, 0.0000 assertions/s.

1 runs, 0 assertions, 0 failures, 0 errors, 0 skips
Coverage report generated for RSpec to /home/xxx/oxidized/coverage/coverage.xml. 651 / 1122 LOC (58.02%) covered
Coverage report generated for RSpec to /home/xxx/oxidized/coverage.
Line Coverage: 58.02% (651 / 1122)

Running the Tests

You can modify the :output.txt file to match your expectations and modify the model accordingly. Run bundle exec rake to run the tests.

Here is an example when the output of the VTP command is missing in the expected output:

oxidized$ bundle exec rake test
/usr/bin/ruby3.1 -I"lib:spec" /home/xxx/oxidized/vendor/bundle/ruby/3.1.0/gems/rake-13.2.1/lib/rake/rake_test_loader.rb "spec/cli_spec.rb" "spec/config_spec.rb" "spec/hook/githubrepo_spec.rb" "spec/input/ssh_spec.rb" "spec/manager_spec.rb" "spec/model/apc_aos_spec.rb" "spec/model/model_atoms_spec.rb" "spec/model/model_helper_spec.rb" "spec/node_spec.rb" "spec/nodes_spec.rb" "spec/output/file_spec.rb" "spec/output/git_spec.rb" "spec/refinements_spec.rb" "spec/source/csv_spec.rb" "spec/source/http_spec.rb" "spec/source/jsonfile_spec.rb" "spec/source/sql_spec.rb"
Run options: --seed 31447

# Running:

...............................................................SS..................F.............................SS..

Finished in 7.963602s, 14.6918 runs/s, 48.7217 assertions/s.

  1) Failure:
ATOMS tests#test_0006_ios:C9200L-24P-4G_17.09.04a:output has expected output [spec/model/model_atoms_spec.rb:12]:
--- expected
+++ actual
@@ -9,6 +9,21 @@
 ! CPU: ARM64
 ! Memory: nvram 2048K
 !
+! VTP: VTP Version capable             : 1 to 3
+! VTP: VTP version running             : 1
+! VTP: VTP Domain Name                 : Oxidized
+! VTP: VTP Pruning Mode                : Disabled (Operationally Disabled)
+! VTP: VTP Traps Generation            : Disabled
+! VTP: Device ID                       : 40f0.7800.0000
+! VTP: Feature VLAN:
+! VTP: --------------
+! VTP: VTP Operating Mode                : Transparent
+! VTP: Maximum VLANs supported locally   : 1005
+! VTP: Number of existing VLANs          : 10
+! VTP: Configuration Revision            : 0
+! VTP: MD5 digest                        : 0x35 0x00 0x00 0x00 0x00 0x00 0x7F 0xB4
+! VTP:                                     0x07 0x00 0x00 0x00 0x00 0x00 0x09 0x6D
+!
 ! NAME: \"c92xxL Stack\", DESCR: \"c92xxL Stack\"
 ! PID: C9200L-24P-4G     , VID: V01  , SN: JAE24FFFFFF
 !


117 runs, 388 assertions, 1 failures, 0 errors, 4 skips

You have skipped tests. Run with --verbose for details.
Coverage report generated for RSpec to /home/xxx/oxidized/coverage/coverage.xml. 2167 / 3131 LOC (69.21%) covered
Coverage report generated for RSpec to /home/xxx/oxidized/coverage.
Line Coverage: 69.21% (2167 / 3131)
rake aborted!
Command failed with status (1): [ruby -I"lib:spec" /home/xxx/oxidized/vendor/bundle/ruby/3.1.0/gems/rake-13.2.1/lib/rake/rake_test_loader.rb "spec/cli_spec.rb" "spec/config_spec.rb" "spec/hook/githubrepo_spec.rb" "spec/input/ssh_spec.rb" "spec/manager_spec.rb" "spec/model/apc_aos_spec.rb" "spec/model/model_atoms_spec.rb" "spec/model/model_helper_spec.rb" "spec/node_spec.rb" "spec/nodes_spec.rb" "spec/output/file_spec.rb" "spec/output/git_spec.rb" "spec/refinements_spec.rb" "spec/source/csv_spec.rb" "spec/source/http_spec.rb" "spec/source/jsonfile_spec.rb" "spec/source/sql_spec.rb" ]
/home/xxx/oxidized/vendor/bundle/ruby/3.1.0/gems/rake-13.2.1/exe/rake:27:in `<top (required)>'
Tasks: TOP => test
(See full trace by running task with --trace)

If you want to be sure that your model has been tested, run bundle exec rake test TESTOPTS="--verbose" and search for your models unter ATOMS tests

Device Prompt

You can specify device prompts to test in a YAML file named spec/model/data/<model>:generic:prompt.yaml.

The YAML file has three sections containing a list of prompts to test:

  • pass: these prompts will pass the prompt regexp.
  • pass_with_expect: these prompts will pass the prompt regexp after having been cleaned by the expect commands.
  • fail: these prompts will fail the prompt regexp.

Here is an example:

pass:
  - "LAB-R1234_Garderos# "
pass_with_expect:
  - "\e[4m\rLAB-R1234_Garderos#\e[m "
fail:
  - "\e[4m\rLAB-R1234_Garderos#\e[m "
  - "#LAB#"

Secrets

You can test if the model effectively removes secrets from your YAML simulation file with a YAML file named like the YAML simulation, but with the suffix :secret.yaml.

The YAML file has two sections containing a list of strings to test:

  • fail: the test will fail if the output contains these strings.
  • pass: the test will pass only if the output contains these strings.
fail:
  - 'AAAAAAAAAABBBBBBBBBBCCCCCCCCCC'
pass:
  - 'snmp-server host 10.10.42.12 version 2c <secret removed> inform'
  - 'hash-mgmt-user oxidized password hash <secret removed>'
  - 'hash-mgmt-user rocks password hash <secret removed> usertype read-only'

Custom tests

When you write custom tests for your models, please do not use the filenames mentioned above, as it will interfere with the standard tests. If you need to store a custom simulation file, use model:description:custom_simulation.yaml.

The cumulus test is an example of a custom test.