Testing Strategy

Testing Strategy

SecondLayer employs a multi-layered testing strategy covering frontend unit tests, backend unit and integration tests, end-to-end (E2E) browser tests, load and quality testing, and CI-driven automated verification. Tests run both locally during development and in CI pipelines on merge to main.

1. Test Frameworks by Service

Service	Framework	Config File	Test Command
lexwebapp (frontend)	Vitest + React Testing Library	lexwebapp/vitest.config.ts	npm run test / npx vitest run
mcp_backend	Jest (ts-jest)	mcp_backend/jest.config.js	npm test / npx jest --no-cache
mcp_rada	Jest (ts-jest)	mcp_rada/jest.config.js	npm test
mcp_openreyestr	Jest (ts-jest)	mcp_openreyestr/jest.config.js	npm test
packages/shared	Jest (ts-jest)	packages/shared/jest.config.cjs	npm test
E2E	Playwright	tests/playwright.config.ts	npx playwright test
Load/Quality	Custom TypeScript (tsx)	scripts/testing/load-test-quality.ts	npx tsx scripts/testing/load-test-quality.ts

2. Frontend Testing (Vitest)

The frontend uses Vitest with the happy-dom environment (not jsdom) and React Testing Library for component interaction.

2.1 Configuration

Key settings from lexwebapp/vitest.config.ts:

• Environment: happy-dom (faster than jsdom)
• Pool: forks with maxForks: 1 (single-process to avoid memory issues)
• Test timeout: 15,000ms
• Coverage provider: v8 (text, json, html reporters)
• Setup file: src/__tests__/setup.ts (mocks fetch, env vars, ReadableStream)
• Path alias: @ maps to ./src

2.2 Test Categories

Frontend tests are organized by concern:

Category	Location	Examples
Services	src/services/api/__tests__/	SSEClient, MCPService
Hooks	src/hooks/__tests__/	useMCPTool, useVideoSignaling
Stores	src/stores/__tests__/	chatStore, uiStore, localeStore, undoStore, videoCallStore
Components	src/components/*/__tests__/	ConsultationChatTab, SupportWidget, VideoCallControls
Pages	src/pages/__tests__/	AdminCostsPage, ConsultationsPage, LegalCodesLibraryPage
Crypto	src/services/crypto/__tests__/	E2EE integration, crypto primitives
Router	src/router/__tests__/	Route definitions

2.3 Running Frontend Tests

cd lexwebapp
npm run test              # Interactive watch mode
npx vitest run            # Single run with exit
npx vitest run --reporter=verbose  # Verbose output
npm run test:coverage     # With coverage report

3. Backend Testing (Jest)

All backend services use Jest with ts-jest for TypeScript transformation. Tests run in a Node.js environment.

3.1 Configuration (shared across backend services)

Key settings:

• Preset: ts-jest
• Test timeout: 120,000ms (tests may call external APIs)
• Max workers: 1 (sequential to avoid resource contention)
• Verbose: true, silent: false (real-time output)
• Setup file: jest.setup.js (loads dotenv, provides test JWT_SECRET)

3.2 Backend Test Categories (mcp_backend)

Category	Directory	Scope
Middleware	src/middleware/__tests__/	Auth, JWT, rate limiting, balance checks
Controllers	src/controllers/__tests__/	Diia auth, password auth, Diia signing
Services	src/services/__tests__/	Billing, consultation, embedding, legislation, Monobank, uploads
Routes	src/routes/__tests__/	Payment, consultation, document management, video calls
API/Tools	src/api/__tests__/	Tool smoke tests, integration, EDRSR, OSINT, vault
SSE	src/api/__tests__/sse/	SSE authentication, streaming, protocol compliance
Adapters	src/adapters/__tests__/	RADA legislation adapter, ZO adapter
Factories	src/factories/__tests__/	Service composition, dependency injection
Utils	src/utils/__tests__/	HTML parser, LLM retry, Redis client, sanitization

3.3 Shared Package Tests

The @secondlayer/shared package has its own test suite covering:

• LLM client managers (OpenAI, Anthropic, Bedrock)
• Model selector (budget-aware routing)
• Base database and HTTP server abstractions
• Cost tracker
• SSE handler

cd packages/shared && npm test

3.4 Running Backend Tests

# Full suite
cd mcp_backend && npm test

# Specific directories (as CI runs them)
cd mcp_backend && npx jest --no-cache --forceExit \
  src/controllers/__tests__/ \
  src/middleware/__tests__/ \
  src/adapters/__tests__/ \
  src/services/__tests__/

# Single file
cd mcp_backend && npx jest --no-cache src/services/__tests__/billing-service-downgrade.test.ts --forceExit

4. End-to-End Testing (Playwright)

E2E tests validate full user flows through the browser against a running environment.

4.1 Configuration

From tests/playwright.config.ts:

• Browser: Chromium (Desktop Chrome)
• Base URL: https://dev.legal.org.ua
• Workers: 1 (sequential)
• Timeout: 30,000ms per test
• Retries: 0
• Artifacts: screenshots on failure, video on failure, trace on first retry

4.2 E2E Spec Coverage

Spec File	Scope
test-billing-tariffs.spec.ts	Subscription and payment flows
test-google-auth.spec.ts	Google OAuth login
test-diia-auth.spec.ts	Diia digital identity login
test-dev-auth.spec.ts	Developer/password login
test-chat-comprehensive.spec.ts	Full chat interaction flow
test-chat-tool-chain.spec.ts	Multi-tool orchestration
test-datasource-coverage.spec.ts	All datasource integrations
test-legislation-library.spec.ts	Legislation browsing and search
test-monobank-payments.spec.ts	Payment processing
test-e2ee-documents.spec.ts	End-to-end encrypted documents
test-profile-real.spec.ts	User profile management
test-all-environments.spec.ts	Cross-environment validation
test-kmu-news-viewer.spec.ts	KMU news data source
test-chat-auth-login.spec.ts	Chat with authentication

4.3 Running E2E Tests

cd tests
npx playwright test                          # Run all
npx playwright test e2e/test-chat-comprehensive.spec.ts  # Single spec
npx playwright test --headed                 # With browser visible

5. Load and Quality Testing

Introduced in PR #1708-#1710, the load test validates the chat pipeline under concurrent load across all 45 user-facing MCP tools.

5.1 Architecture

The load test (scripts/testing/load-test-quality.ts):

1. Creates 10 test users with $100 balance each
2. Runs a preflight check (health endpoint + auth verification)
3. Executes two phases:
   • Simple phase: 45 queries (one per tool), batches of 10, 3s pause between batches
   • Complex phase: 45 queries (one per tool), batches of 5, 8s pause between batches
4. Grades answers using Bedrock Claude Haiku for quality scoring
5. Stores all results in load_test_results PostgreSQL table
6. Cleans up test users (results are preserved)

5.2 Metrics Tracked

Metric	Description
Response time (ms)	Total time from request to final SSE event
First byte (ms)	Time to first SSE data byte
Tool routing accuracy	Whether the expected tool was triggered (exact and contains match)
Cost (USD)	Per-request LLM API cost
Quality score	LLM-graded: relevant, partially_relevant, not_relevant, hallucinated
Status	success, error, timeout, no_answer

5.3 Running Load Tests

# Full run
npx tsx scripts/testing/load-test-quality.ts

# View report from last run
npx tsx scripts/testing/load-test-quality.ts --report

# View specific run
npx tsx scripts/testing/load-test-quality.ts --report loadtest-2026-05-15_14-30-00

# Environment overrides
LOAD_TEST_BASE_URL=https://legal.org.ua npx tsx scripts/testing/load-test-quality.ts

5.4 Quality Scoring

Quality is assessed by Bedrock Claude Haiku (eu.anthropic.claude-haiku-4-5-20251001-v1:0) using a structured grading prompt:

• relevant: Directly answers the query with correct legal information
• partially_relevant: Related but incomplete or tangential
• not_relevant: Does not address the query
• hallucinated: Contains fabricated case numbers, fake citations, or invented legal norms

6. CI/CD Test Integration

Tests are integrated into two GitHub Actions pipelines running on a self-hosted runner.

6.1 Local CI Pipeline (ci-local-deploy.yml)

Triggered on push to main:

detect-changes --> build-and-test --> local-deploy-and-test

build-and-test (runs on local-build VM):

1. Builds packages/shared
2. If backend changed: overlays proprietary source, builds, runs unit tests
3. If frontend changed: installs deps, runs Vitest (60s timeout wrapper), builds production bundle
4. Backend unit tests run specific directories only:
   • src/controllers/__tests__/
   • src/middleware/__tests__/
   • src/adapters/__tests__/
   • src/services/__tests__/

local-deploy-and-test:

1. Builds Docker images for changed services
2. Runs database migrations
3. Starts containers, waits for health checks
4. Verifies production containers were not affected

6.2 Production Deploy Pipeline (deploy-prod.yml)

Triggered after successful local CI or manually:

detect-changes --> pre-deploy-tests --> deploy-preview --> promote-to-prod --> create-release

pre-deploy-tests:

• Same as build-and-test but targeting production build
• Backend: build + Jest unit tests (same directories)
• Frontend: install + Vitest + production build (VITE_API_URL=https://legal.org.ua)

deploy-preview:

• Blue-green deployment to inactive color
• Health checks on new containers before switching

promote-to-prod:

• Switches nginx upstreams to new color
• Stops old containers
• Final health check via SSH

6.3 Self-Healing (Currently Disabled)

Both pipelines have Claude Code self-heal steps that:

• Analyze build/test failures from CI logs
• Create fix branches with [ci-autofix] commits
• Open PRs automatically
• Guard against infinite loops (skip if autofix found in recent commits)

These are currently disabled (if: false) as they were not producing useful fixes.

6.4 Change Detection

Tests only run for changed services. The detect-changes action compares HEAD against the previous deploy tag to determine which services need building and testing.

7. Testing Scripts

Additional shell scripts in scripts/testing/ for manual and specialized testing:

Script	Purpose
run-all-tests.sh	Run all test suites
run-e2e-tests.sh	Run Playwright E2E
test-gateway.sh	Validate unified gateway routing
test-chat-tool-coverage.sh	Verify all tools are reachable via chat
test-batch-processing.sh	Test batch API endpoint
test-document-service.sh	Document parsing validation
run-document-service-test.sh	Document service integration
test-all-mcp-logging.sh	MCP tool logging verification
test-case-chain-improved.sh	Case chain retrieval
test-new-tool.sh	Validate a newly added tool

8. Test Environment Requirements

8.1 Environment Variables

Variable	Used By	Purpose
JWT_SECRET	Backend tests	Authentication in tests (auto-set in jest.setup.js)
POSTGRES_*	Load test	Database connection for test user management
AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY	Load test quality grading	Bedrock access for LLM-based grading
LOAD_TEST_BASE_URL	Load test	Target environment URL
VITE_API_URL	Frontend build	API endpoint for production builds

8.2 Infrastructure Requirements

• Unit tests (frontend/backend): No infrastructure needed (everything is mocked)
• Integration tests (src/api/__tests__/): Require running database, Redis, and backend services
• E2E tests: Require full deployed environment accessible via HTTPS
• Load tests: Require running backend with database access for user provisioning