=Files=
* topology.svg - topology diagram
* topology.xml - ietf-network model based network spec. Edit to change network addresses, authorization credentials etc.
* configure.ac - autoconf build script
* Makefile.am - automake build script (contains the list of all testcases)

=Running=
 apt-get update
 apt-get -y upgrade
 
 apt-get -y install software-properties-common
 add-apt-repository -s "deb http://yuma123.org/repos/apt/debian stretch main"
 wget -O - http://yuma123.org/repos/yuma123.gpg.key | apt-key add -
 apt-get update

 apt-get -y install testsuite-h100-ecoc-demo
 cd /opt/testsuite-h100-ecoc-demo
 autoreconf -i -f
 ./configure

Make sure your topology.xml has correct server/port/username/password configuration:

 make check

This should complete with OKs (some scripts report fail if there is external traffic generator incrementing counters)

To start the continuous monitor (updates the files referenced by monitor/index.html for graphical representation of the traffic and periodic reports):

 ./test-monitor

Here is example of the output of the monitor:

[[File:testsuite-h100-ecoc-demo-topology.svg]]
 <nowiki>
Report time: 20:25:19

1. Period:
+------------+---------------------+---------------------+--------------+
|    node    |       start         |         stop        |     period   |
+------------+---------------------+---------------------+--------------+
| middle     | 2018-09-22T18:24:20 | 2018-09-22T18:25:00 |       40.000 |
| local      | 2018-09-22T18:24:20 | 2018-09-22T18:25:00 |       40.000 |
| remote     | 2018-09-22T18:24:20 | 2018-09-22T18:25:00 |       40.000 |
+------------+---------------------+---------------------+--------------+
2. Data transfer rate:
+------------+--------------+-------+
|  if name   |       MB/s   |   %   |
+------------+--------------+-------+
| middle.xe0 |       3412   |   29  |
| remote.xe0 |       6773   |   57  |
+------------+--------------+-------+

3. Timing:
+-----------+--------------+-------------+--------------+---------------+
|  if name  |  delay-min   |  delay-max  |      PDV     |     samples   |
+-----------+--------------+-------------+--------------+---------------+
| local.ge1 |      7353 ns |      7456 ns|       103 ns |    228883040  |
| local.ge2 |      7347 ns |      7449 ns|       102 ns |    229662467  |
| local.ge3 |      7398 ns |      7494 ns|        96 ns |    230612730  |
| local.ge4 |         0 ns |         0 ns|         0 ns |            0  |
+-----------+--------------+-------------+--------------+---------------+
</nowiki>

After the monitor is started one would need to enable traffic and change its parameters from another terminal. Here the traffic generators and analysers are started and stopped:

 ./test-setup-traffic-analyzers ; ./test-setup-traffic-generators-1
 # traffic pattern 1
 ./test-clear-traffic-generators
 # no traffic
 ./test-setup-traffic-analyzers ; ./test-setup-traffic-generators-2
 # traffic pattern 2
 ...

==Device side==
There are 3 scripts in ./setup (run-netconfd-local.sh, run-netconfd-middle.sh, run-netconfd-remote.sh) that can start the netconfd instances for each node with the correct parameters. Use those to either connect to the VCU-110 cards directly or to connect to another netconfd providing register access. In case there is external host that runs netconfd make sure the parameters are identical e.g. all expected modules are loaded.

=Reports=
* test-suite.log - overview
* test-*.log - individual test reports

=Test Network Setup=
2x H100 1.6 prototype VCU-110 systems,  1x H100 1.0 prototype and an optional traffic analyzer.

Further in the document the elements will be referred to as:
* local - the left H100 1.6
* remote - the right H100 1.6
* middle - the H100 1.0 in the middle
* analyzer (optional)

Details about the actual networking addresses and authorization are specified in the topology.xml file and if there is a change in the setup this file needs to be updated before running.

=OpenDaylight=
The 3 devices can be mounted as NETCONF devices in OpenDaylight southbound interface. The testsuite includes the ./test-odl-install testcase that downloads and installs OpenDaylight (Beryllium-SR4) and configures it using yangcli to mount the 3 nodes.
Once this testcase completes one can use the ./test-odl-clear-traffic-generators to stop all traffic in identical way as the non-odl ./test-clear-traffic-generators-[1,2]. This testcase is added as proof of concept that the mounted devices configuration is correctly responding to interaction with OpenDaylight Northbound RESTCONF interface.

The following links can be used to access OpenDaylight web interface (user:admin password:admin):

* http://localhost:8080/index.html - DLUX Web GUI
* http://localhost:8080/restconf/config/network-topology:network-topology/topology/topology-netconf/node/middle/yang-ext:mount/ - configuration of 'middle'

For more information check the OpenDaylight project documentation.

=References=
# Transpacket AS, "VCU-110 Prototype r1.6"
# RFC6241 Enns, R., Ed., Bjorklund, M., Ed., Schoenwaelder, J., Ed., and A. Bierman, Ed., "Network Configuration Protocol (NETCONF)", RFC 6241, DOI 10.17487/RFC6241, June 2011, <http://www.rfc-editor.org/info/rfc6241>.

=Test specification=
Test cases included in the suite:
# validate-topology-config
# netconf-node-access
# setup-idle
# setup-fusion-streams-yangcli
# setup-network-bridge-yangcli
# get-net-[1,2-1,3,2-2]
# setup-traffic-analyzers
# setup-loopbacks
# setup-traffic-generators
# diff-net
# set-net-2-1

Every test case is implemented as executable named test-<testcase-name> and can be started individually e.g ./test-setup-idle

The test cases produce output with formal NETCONF transactions data (state and configuration) so that every single action taken by the test case can be traced back and reported results according to the "Reported results" section described in each test case. All the information is printed to the standard output (no other files containing test report data are created). The return status of the executable is indication of the requirements for the specified validations and acceptance criteria were met. Following the common Unix convention 0=success, not(0)=error-code.

A detailed specification for each test case part of the suite follows with this information structure:

 + Objective
 + Procedure
 + DUT Configuration(s)
 + Steps
 +--- []
    +--- Validates ...
    +--- Returns ...
 + Reported results
 + Acceptance criteria

==Testcase: validate-topology-config==
===Objective===
Validate the topology.xml has correct XML format and no YANG syntactic or semantic errors.
===Procedure===
Call netconfd in --validate-config-only mode.

==Testcase: netconf-node-access==
===Objective===
Validate all nodes part of the test setup network are accessible over NETCONF.
===Procedure===
Establish connections to all nodes specified in the topology.xml file using the common testapi API.
===DUT configuration===
N.A
====Steps====
# Connect to all nodes specified in the topology.xml with testapi.network_connect(network)
## Validate the operation succeeds.

==Testcase: setup-idle==
===Objective===
Load idle configurations to local, remote and middle and verify idle devices behave as expected.
===Procedure===
After committing idle configurations validate not counters are running.
===DUT configuration===
Idle configuration e.g. delete /fusion-streams and other top level containers.

====Steps====
# Delete /fusion-streams on local and remote and commit.
## No counters are being incremented.

==Testcase: setup-*==
===Objective===
Each of the setup-* testcases carry out a incremental step in the sequence of creating a complete benchmark test.

==Testcase: get-net-[0-12]==
===Objective===
Using the get-net commannd line tool create snapshot of the network for each transaction.

==Testcase: set-net-*==
===Objective===
Using the set-net commannd line tool restore configuration snapshot of the network of a previously captured with get-net configuration.

==Testcase: diff-net==
===Objective===
Compare and print all counters of 2 consequtive network snapshot copies.