Skip to content

CableJenkinsMaintainerGuide

Jump to bottom
Ned Haughton edited this page Nov 2, 2016 · 1 revision

This document describes the CABLE automated testing infrastructure, its current features, and processes required to add new testing procedures.

System description

The CABLE automated testing infrastructure (currently hosted on NCI) run on Jenkins, a general-purpose testing and automation server (currently using version 2.0). Currently, testing includes an automated trunk testing suite, which runs automatically after any SVN update, and another automated testing suite that can be run on-demand for a given branch. Both testing suites are staged, and will stop after the first failure. The included tests are described in more detail below. The system is a work-in-progress and can relatively easily be modified to add new tests, or update existing tests.

Testing suites

Each of the testing suites is a Jenkins pipeline, which is a special job that runs other jobs. Each individual job is generally a set of rules that includes things like copying artefacts (files) from previous jobs, generally running a shell script, recording script results, and then archiving artefacts for later use. Each job's details can be viewed and edited by clicking the "Configure" link on the job. Each build of a job includes a log of the script output, which can be viewed by clicking the build, and then clicking "Console Output". Files created during the most recent build of a job can be accessed at /home/561/nnh561/jenkins/workspace/CABLE/[path/to/jenkins/job] on Raijin at NCI.

Trunk testing

The main trunk testing suite is trunk_testing in the CABLE folder. It is a Jenkins Pipeline that is triggered by the CABLE SVN checkout job, which checks every hour for updates to the trunk SVN repository, and updates the checkout and triggers the pipeline when changes are found.

The trunk_testing suite contains a number of stages:

  1. checkout: Update the checkout of CABLE-AUX, if required.
  2. build: Compile multiple versions of CABLE: Offline, MPI, and libcable (all with ifort), plus offline with gfortran.
  3. build_debug: Compile debug offline versions of CABLE (ifort, gfortran), plus debug versions with all checks and warnings enabled (ifort and gfortran).
  4. run: Run offline versions of CABLE: offline-ifort, MPI-ifort, offline-gfortran
  5. analyse: Compare the output of offline simulations produced by the offline-ifort and offline-gfortran runs, and by the offline-ifort and MPI-ifort runs.
  6. ACCESS: compiles two copies of libcable from the same code-base, compiles ACCESS 1.4 twice (scripts modified from versions generated by the UMUI), and runs ACCESS twice. Runs are compared for bitwise similarity

As the pipeline runs, stages become green if they succeed, or red if they fail. Details of each individual job can be accessed by clicking on the stage, clicking "logs", clicking the relevant job, then clicking the relevant build (the second link). Details a

Jobs within a stage are generally run in parallel, and stages are run in sequence. In general, if a job fails, the stage fails, and subsequent stages will not run.

Data (SVN files, compiled binaries, simulation output) are copied to each job from the relevant prerequisite jobs.

Individual jobs can be viewed in the trunk folder, and the run pipeline can be viewed via the configure option of the trunk_testing job.

Branch testing

The main branch testing suite, branch_testing. It is a Jenkins pipeline that is triggered manually, when anyone with permissions (anyone in the NCI cable user group) presses the build with parameters link, and enters a legitimate branch name. The pipeline layout is similar to the trunk_testing layout.

branch_testing stages include:

  1. checkout: Checks out the branch from SVN
  2. compile: Compiles offline with ifort
  3. run: Run branch version of CABLE offline.
  4. analyse: Compares branch offline run with latest trunk offline run output.
  5. ACCESS_trunk_similarity: compiles, runs and compares ACCESS runs from branch and trunk
  6. ACCESS_self_similarity: compiles two versions of the branch version of cable/ACCESS and runs and compares them (as per step 6 of trunk_testing)

Adding/modifying features

Adding jobs to Jenkins is a fairly straight-forward procedure: Simply go to the CABLE folder or one of it's subfolders, and press "New Item", select "Freestyle Project", and fill in the blanks. However, there are a number of things that must be added to a job for it to run, in particular it must be restricted to run on raijin && nnh561 for it to work. In most cases it is easier to copy a job from an existing job - instead of selecting "Freestyle Project", select "Copy from", and enter the path to a similar job to copy from, including the path if it is in a different folder (../ can be used to go up a directory, relatively, or enter the path as /CABLE/folder/job for an absolute path - Jenkins will let you know if it can't find the job). Then simply edit the job details as necessary - edit the build script in particular, but also check all the other details, including artefacts to copy, etc.

Once a new job is in-place, it can be added to one of the pipelines by configuring the pipeline, and adding the job to the relevant path in the Pipeline script section. The format is fairly straightforward for basic jobs, and can just be copied from another job in the script, changing the name and path. If more complexity is required, click "Pipeline Syntax", and use the snippet generator to add parameters as required.

Clone this wiki locally