cql-tests-runner

Test Runner for the CQL Tests repository. This node application allows you to run the tests in the CQL Tests repository against a server of your choice using the $cql operation. The runner in its current state uses only this operation, and there is no expectation of any other FHIR server capability made by this runner. Additional capabilities may be required in the future as we expand the runner to support full Library/$evaluate as well. None of the tests in the repository have any expectation of being able to access data (i.e. the tests have no retrieve expressions).

The application runs all the tests in the repository and outputs the results as a JSON file in the results directory. If the output directory does not exist, it will be created.

Results output from running these tests can be posted to the CQL Tests Results repository.

Setting up the Environment

This application requires Node v24 and makes use of the Axios framework for HTTP request/response processing. Node Download

Install application dependencies using

npm install

CQL Tests Submodule

The cql-tests folder has been added as a submodule. After pulling, you'll find a cql-tests folder inside cql-tests-runner. However, when you peek inside that folder, depending on your Git version, you might see nothing. Newer versions of Git will handle this automatically, but older versions may require you to explicitly instruct Git to download the contents of cql-tests.

git submodule update --init --recursive

Configuration Settings

Configuration settings are set in a JSON configuration file. The file conf/localhost.json provides a sample configuration.

{
    "FhirServer": {
      "BaseUrl": "https://fhirServerBaseUrl",
      "CqlOperation": "$cql"
    },
    "Build": {
      "CqlFileVersion": "1.0.000",
      "CqlOutputPath": "./cql"
    },
    "Tests": {
      "ResultsPath": "./results",
      "SkipList": []
    },
    "Debug": {
      "QuickTest": true
    }
}

Create your own configuration file and reference it when running the commands. You can use conf/localhost.json as a template for a new configuration with your own settings.

Running the tests

The CLI now requires a configuration file path as an argument. Run the tests with the following commands:

Running from Source Code

To run the application directly from source:

# Install dependencies
npm install

# Initialize the cql-tests submodule
git submodule update --init --recursive

# Run commands directly from TypeScript source
npx tsx src/bin/cql-tests.ts run-tests conf/localhost.json ./results # Run CQL tests
npx tsx src/bin/cql-tests.ts server                               # Run in server API mode
npx tsx src/bin/cql-tests.ts help                               # Hetailed command help

npx tsx src/bin/cql-tests.ts build-cql conf/localhost.json ./cql    # Unused legacy tool

Running from Pre-Built OCI/Docker Image

The application is available as the pre-built image tag hlseven/quality-cql-tests-runner:latest.

Using the Docker Image

By default, the image runs the CLI. When you bind in any local directories (such as configuration and results directories) you may use it as you would any other command line utility.

# Run CQL tests with a configuration file
docker run --rm -v $(pwd)/conf:/app/conf -v $(pwd)/results:/app/results \
  hlseven/quality-cql-tests-runner:latest run-tests conf/localhost.json ./results

# Build CQL libraries (Unused)
docker run --rm -v $(pwd)/conf:/app/conf -v $(pwd)/cql:/app/cql \
  hlseven/quality-cql-tests-runner:latest build-cql conf/localhost.json ./cql

# Start in REST server mode listening on port 3000.
docker run --rm -p 3000:3000 -v $(pwd)/conf:/app/conf \
  hlseven/quality-cql-tests-runner:latest server

# Using host networking to test against a server running on the host machine
docker run --rm --network host -v $(pwd)/conf:/app/conf -v $(pwd)/results:/app/results \
  hlseven/quality-cql-tests-runner:latest run-tests conf/localhost.json ./results

Building the Docker Image

# Build the Docker image locally
docker build -t cql-tests-runner .

# Build multi-platform image for distribution
docker buildx build --platform linux/arm64,linux/amd64 -t hlseven/quality-cql-tests-runner:latest .

# Run with built image
docker run --rm -v $(pwd)/conf:/app/conf -v $(pwd)/results:/app/results hlseven/quality-cql-tests-runner:latest run-tests conf/localhost.json ./results

# Using host networking with built image
docker run --rm --network host -v $(pwd)/conf:/app/conf -v $(pwd)/results:/app/results hlseven/quality-cql-tests-runner:latest run-tests conf/localhost.json ./results

Environment Variable Overrides

You can still override specific settings using environment variables:

export SERVER_BASE_URL=http://fhirServerBaseEndpoint
export CQL_OPERATION=$cql

Development Environment

If using vscode for development, below are some examples for running the tests with configuration files:

{
  "type": "node",
  "request": "launch",
  "name": "Launch Build Command",
  "skipFiles": ["<node_internals>/**"],
  "program": "${workspaceFolder}/src/bin/cql-tests.ts",
  "args": ["build-cql", "conf/localhost.json"],
  "runtimeArgs": ["--import", "tsx"]
},
{
  "type": "node",
  "request": "launch",
  "name": "Launch Run Command",
  "skipFiles": ["<node_internals>/**"],
  "program": "${workspaceFolder}/src/bin/cql-tests.ts",
  "args": ["run-tests", "conf/localhost.json"],
  "runtimeArgs": ["--import", "tsx"],
  "env": {
    "SERVER_BASE_URL": "http://localhost:3000"
  }
}

Server Command

The server command starts an HTTP server that provides a REST API for running CQL tests. This is mainly intended to be used by CQL Tests UI

Starting the Server

# Using tsx (development mode)
npx tsx src/bin/cql-tests.ts server

# Using Docker
docker run --rm -p 3000:3000 -v $(pwd)/conf:/app/conf \
  hlseven/quality-cql-tests-runner:latest server

# Using Docker with host networking
docker run --rm --network host -v $(pwd)/conf:/app/conf \
  hlseven/quality-cql-tests-runner:latest server

Using the Server API

The server provides the following endpoints:

• GET / - Server information and available endpoints
• POST / - Run CQL tests with configuration in request body (synchronous)
• POST /jobs - Create a new job to run CQL tests asynchronously
• GET /jobs/:id - Get job status and results by job ID
• GET /health - Health check endpoint

Asynchronous Job Processing

For long-running test suites, the server supports asynchronous job processing:

# Create a job (returns immediately with job ID)
curl -X POST http://localhost:3000/jobs \
  -H "Content-Type: application/json" \
  -d @conf/localhost.json

# Poll job status and results
curl http://localhost:3000/jobs/{job-id}

Jobs support progress tracking and can be polled for status updates. The original synchronous endpoint (POST /) remains available for quick tests.

Example Usage

# Start the server
npx tsx src/bin/cql-tests server --port 3000

# In another terminal, run tests via synchronous execution API
curl -X POST http://localhost:3000/ \
  -H "Content-Type: application/json" \
  -d @conf/localhost.json \
  -o results.json

# Check server health
curl http://localhost:3000/health

The server accepts a configuration object in the request body and returns the test results as JSON.

Unit Testing

Unit testing is implemented with Vitest. This is for only testing of the cql-test-runner logic, and not for testing FHIR operations.

Test cases are stored in the test/ folder.

Executing Unit Tests

Unit tests can be run from the command line using the following

$ npm run unit-tests