Skip to content

TESTBenches.md

Latest commit

 

History

History
419 lines (354 loc) · 23.4 KB

TESTBenches.md

File metadata and controls

419 lines (354 loc) · 23.4 KB
Raw

Table of Contents

[[TOC]]

Machines

Machine Lockable Resource Function Connected devices
obelix Obelix eNB (n40, n78), nrUE 172.21.19.13, X300 (192.168.60.2)
porcepix Porcepix Executor, EPC, 5GC --
up2 UP2 COTS UE Quectel RM520N
nepes Nepes gNB (n78), EPC/5GC B200mini (30C51EB)
ofqot Ofqot gNB (n78) B200mini (30C51D4)
idefix Idefix COTS UE Quectel RM500Q
caracal Caracal gNB/phytest, OAI UE N300 (192.168.10.2), AMD T2 card
amariue Amarisoft-UE nrUE Amarisoft UE simulator
nano Nano_EPC/Nano_5G_COTS_UE Executor, EPC, adb 2x COTS (adb)
hutch Hutch eNB (B7) B200mini (30C5239)
starsky Starsky eNB (B40) b200mini (30A3E3C)
carabe Carabe UE (B7UE) B200mini (30AE8C9)
avra Avra gNB (n78) AW2S Jaguar (192.168.80.239)
cacofonix Cacofonix gNB (n78, FHI7.2) --
matix Matix gNB (n77) N310
gracehopper1-oai Gracehopper1 gNB (n78, PNF/Nvidia CUBB + VNF) Foxconn RU, Nvidia Aerial integration
gracehopper3-oai Gracehopper3 build --
acamas Acamas 4G and 5G rfsimulator --
groot Groot gNB (n77) B210 (30AD30F)
rocket Rocket gNB (n77) B210 (31F8010)
raspix Raspix COTS UE Quectel RM520N
jetson1-oai Jetson1-oai nrUE USRP B210

Note: The available resources, and their current usage, is indicated here:

Testbenches

5G OTA Testbench

Purpose: Over-the-air 4G/5G (NSA/SA) tests, performance tests

5G OTA Testbench

LaTeX/TikZ version if you want to modify to reflect your setup

5G NSA/Faraday Cage Testbench

Purpose: Faraday cage 5G tests, functional tests

5G NSA/Faraday Cage Testbench

PDF version | LaTeX/TikZ version if you want to modify to reflect your setup

5G AW2S Testbench

Purpose: AW2S tests with Amarisoft UE simulator

5G AW2S Testbench

PDF version | LaTeX/TikZ version if you want to modify to reflect your setup

5G UE OTA Testbench

Purpose: Over-the-air 5G tests with OAI UE

OAI UE Testbench

PDF version | LaTeX/TikZ version if you want to modify to reflect your setup

4G Testbench(es)

Purpose: 4G/LTE testbenches

4G Faraday Cage Testbench

PDF version | LaTeX/TikZ version if you want to modify to reflect your setup

Pipelines

RAN-GitHub-Container-Parent

Purpose: automatically triggered tests on pull request creation or push https://github.com/duranta-project/openairinterface5g/labels/documentation https://github.com/duranta-project/openairinterface5g/labels/BUILD-ONLY https://github.com/duranta-project/openairinterface5g/labels/4G-LTE https://github.com/duranta-project/openairinterface5g/labels/5G-NR https://github.com/duranta-project/openairinterface5g/labels/nrUE

This pipeline has basically two main stages, as follows. For the image build, please also refer to the dedicated documentation for information on how the images are built.

Image Build pipelines

Image Test pipelines

How to reproduce CI results

The CI builds docker images at the beginning of every test run. To see the exact command line steps, please refer to the docker/Dockerfile.build.* files. Note that the console log of each pipeline run also lists the used docker files.

The CI uses these images for most of the pipelines. It uses docker-compose to orchestrate the images. To identify the docker-compose file, follow these steps:

  1. Each CI test run HTML lists the XML file used for a particular test run. Open the corresponding XML file under ci-scripts/xml_files/.
  2. The XML file has a "test case" that refers to deployment of the image; it will reference a directory containing a YAML file (the docker-compose file) using option yaml_path, which will be under ci-scripts/yaml_files/. Go to this directory and open the docker-compose file.
  3. The docker-compose file can be used to run images locally, or you can infer the used configuration file and possible additional options that are to be passed to the executable to run from source.

For instance, to see how the CI runs the multi-UE 5G RFsim test case, the above steps look like this:

  1. The first tab in the 5G RFsim test mentions xml_files/container_5g_rfsim.xml, so open ci-scripts/xml_files/container_5g_rfsim.xml.
  2. This XML file has a DeployGenObject test case, referencing the directory yaml_files/5g_rfsimulator. The corresponding docker-compose file path is ci-scripts/yaml_files/5g_rfsimulator/docker-compose.yaml.
  3. To know how to run the gNB, realize that there is a section oai-gnb. It mounts the configuration ci-scripts/conf_files/gnb.sa.band78.106prb.rfsim.conf (note that the path is relative to the directory in which the docker-compose file is located). Further, an environment variable USE_ADDITIONAL_OPTIONS is declared, referencing the relevant options -E --rfsim (you can ignore logging options). You would therefore run the gNB from source like this: 
    sudo ./cmake_targets/ran_build/build/nr-softmodem -O ci-scripts/conf_files/gnb.sa.band78.106prb.rfsim.conf -E --rfsim
    To run this on your local machine, assuming you have a 5GC installed, you might need to change IP information in the config to match your core.

If you wish, you can rebuild CI images locally following these steps and then use the docker-compose file directly.

Some tests are run from source (e.g. ci-scripts/xml_files/gnb_phytest_usrp_run.xml), which directly give the options they are run with.

How to debug CI failures

It is possible to debug CI failures using the generated core dump and the image used for the run. A script is provided (see developer instructions below) that, provided the core dump file, container image, and the source tree, executes gdb inside the container; using the core dump information, a developer can investigate the cause of failure.

Developer instructions

The CI team will send you a docker image and a core dump file, and the commit as of which the pipeline failed. Let's assume the coredump is stored at /tmp/coredump.tar.xz, and the image is in /tmp/oai-nr-ue.tar.gz. First, you should check out the corresponding branch (or directly the commit), let's say in ~/oai-branch-fail. Now, unpack the core dump, load the image into docker, and use the script docker/debug_core_image.sh to open gdb, as follows:

cd /tmp
tar -xJf /tmp/coredump.tar.xz
docker load < /tmp/oai-nr-ue.tar.gz
~/oai-branch-fail/docker/debug_core_image.sh <image> /tmp/coredump ~/oai-branch-fail

where you replace <image> with the image loaded in docker load. The script will start the container and open gdb; you should see information about where the failure (e.g., segmentation fault) happened. If you just see ??, the core dump and container image don't match. Be also on the lookout for the corresponding message from gdb:

warning: core file may not match specified executable file.

Once you quit gdb, the container image will be removed automatically.

CI team instructions

The entrypoint scripts of all containers print the core pattern that is used on the running machine. Search for core_pattern at the start of the container logs to retrieve the possible location. Possible locations might be:

See below for instructions on how to retrieve the core dump. Further, download the image and store it to a file using docker save. Make sure to pick the right image (Ubuntu or RHEL)!

Core dump in a file

This is not recommended, as files could pile up and fill the system disk completely! Prefer another method further down.

If the core pattern is a path: it should at least include the time in the pattern name (suggested pattern: /tmp/core.%e.%p.%t) to correlate the time the segfault occurred with the CI logs. If you identified the core dump, copy the core dump from that machine; if identification is difficult, consider rerunning the pipeline.

Core dump via systemd

Use the first command to list all core dumps. Scroll down to the core dump of interest (it lists the executables in the last column; use the time to correlate the segfault and the CI run). Take the PID of the executable (first column after the time). Dump the core dump to a location of your choice.

sudo coredumpctl list
sudo coredumpctl dump <PID> > /tmp/coredump

Core dump via abrt (automatic bug reporting tool)

TBD: use the documentation page for the moment.

Core dump via apport

I did not find an easy way to use apport. Anyway, the systemd approach works fine. So remove apport, install systemd-coredump, and verify it is the new coredump handler:

sudo systemctl stop apport
sudo systemctl mask --now apport
sudo apt install systemd-coredump
# Verify this changed the core pattern to a pipe to systemd-coredump
sysctl kernel.core_pattern