Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions .claude/guidelines/changelog_protocol.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# CHANGELOG Update Protocol

## Scope

Every PR that changes deployer-visible behaviour requires a corresponding entry in [`CHANGELOG.md`](../../CHANGELOG.md), recorded in the same PR. Behaviour changes include:

- Adding, modifying, or removing a `terraform.tfvars` variable
- Modifying the configuration of an existing component (container env var, nginx directive, image version, etc.)
- Adding or removing a component (container, module, service)
- Security fixes affecting the deployed system

Pure refactors with identical observable behaviour, internal-only test changes, and comment-only changes do not require an entry.

## Rules

1. **Same-PR.** The entry lands in the same PR as the change. [`documentation/upgrade_steps.md`](../../documentation/upgrade_steps.md) step 5 routes upgraders to `CHANGELOG.md` for `terraform.tfvars` migration guidance, and reviewers rely on `Notable Changes` to understand release content — a missing entry silently breaks both audiences.

2. **Pick the right `Notable Changes` subsection** under the in-progress release:
- **General** — component additions/removals, config changes, version bumps, behaviour changes (default for most entries)
- **Security** — security-relevant fixes
- **Documentation** — documentation additions/changes (including `TEMPLATE_terraform.tfvars` updates)
- **Testing** — test surface changes

3. **`terraform.tfvars` variable changes need an additional entry.** Beyond the `Notable Changes` bullet (typically `Documentation` for variable additions), a variable add/modify/delete also requires a row under `### Configuration File Changes → #### terraform.tfvars`. Row format is defined in [`variable_default_values.md`](variable_default_values.md#changelog-entry-format).

## Bullet format

```
- <Concise change description>. [`#<issue>`](https://github.com/seqeralabs/cx-field-tools-installer/issues/<issue>)
```

Link a tracking issue when one exists.
5 changes: 5 additions & 0 deletions .claude/guidelines/python_standards.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Python Standards

Default to the user-level global Python guidelines (`lang:python/python.md`). Project-level standards have not yet been codified.

TODO: codify project-level Python conventions specific to this repository — e.g., conventions around the custom tfvars parser ([`scripts/installer/utils/extractors.py`](../../scripts/installer/utils/extractors.py)), the project logger ([`scripts/installer/utils/logger.py`](../../scripts/installer/utils/logger.py)), validation scripts, and test conventions that diverge from the global standard.
7 changes: 7 additions & 0 deletions .claude/guidelines/security_considerations.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Security Considerations

- Sensitive values are stored in AWS SSM Parameter Store only — never in `terraform.tfvars` or repo-tracked files.
- Private-subnet access uses AWS Instance Connect Endpoint with `ProxyCommand`.
- Template files use the `.tpl` extension to avoid accidental secret exposure via filename pattern matching.
- Custom certificates are supported via [`assets/src/customcerts/`](../../assets/src/customcerts/).
- Run `make verify-full` to invoke `tfsec` security scanning before applying.
8 changes: 8 additions & 0 deletions .claude/guidelines/terraform_conventions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# Terraform Conventions

## Conventions

- **No default values** in [`variables.tf`](../../variables.tf) — every value must be explicitly set in `terraform.tfvars`. Rules and rationale in [`variable_default_values.md`](variable_default_values.md).
- Use `null_resource` with `local-exec` provisioners instead of `local_file` resources. Rationale: `local_file` writes during plan-evaluation, which conflicts with the project's "regenerate everything on apply" model.
- State is stored locally by default at `DONTDELETE/terraform.tfstate`.
- Templates live under [`assets/src/`](../../assets/src/) with `.tpl` extension; rendered output goes to `assets/target/` (not source-controlled).
54 changes: 54 additions & 0 deletions .claude/guidelines/testing_commands.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# Testing Commands

## Quick reference

| Command | Purpose |
| ------- | ------- |
| `make run_tests` | Run all tests under [`tests/`](../../tests/). |
| `make run_tests_no_containers` | Run all tests except those marked `@pytest.mark.testcontainer`. |
| `make generate_test_data` | Copy `TEMPLATE_terraform.tfvars` into `tests/datafiles/` and seed core data. |
| `make generate_json_plan` | Produce `tfplan.json` from a fresh `terraform plan` for plan-based tests. |
| `make purge_cache` | Clear cached plans and templatefiles under `tests/`. |
| `make test_cleanse` | Remove generated `tests/datafiles/` artefacts. |

## Running specific test subsets

Pytest markers are defined in [`tests/pytest.ini`](../../tests/pytest.ini) — that file is authoritative; do not duplicate the marker list elsewhere.

```bash
# Plan-based tests only (no containers, no AWS)
pytest -m "local" -v

# Database tests
pytest -m "db" -v

# Redis tests
pytest -m "redis" -v

# Single test file or directory
pytest tests/unit/test_module_connection_strings/ -v -s -x
```

## Structured logging

Tests emit structured logs to `tests/logs/pytest_structured.log`. Parser tooling lives at [`tests/utils/log_parser.py`](../../tests/utils/log_parser.py):

```bash
# Summarise the most recent run
python tests/utils/log_parser.py summarize

# Extract failures
python tests/utils/log_parser.py extract-failures

# Format recent entries for LLM consumption
python tests/utils/log_parser.py llm-format --recent 100

# Validate log format
python tests/utils/log_parser.py validate

# Export as JSON
python tests/utils/log_parser.py export-json

# Tail in real-time during a run
tail -f tests/logs/pytest_structured.log
```
28 changes: 28 additions & 0 deletions .claude/guidelines/testing_strategy.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Testing Strategy

## Approach

The project uses a hybrid testing approach:

1. **Plan-based tests** — Mock resources using `terraform plan` output. Plan outputs are cached in `tests/.plan_cache` to speed up n+1 test cycles when underlying values are unchanged.
2. **Unit tests** — Test individual modules in isolation.
3. **Integration tests** — Full deployment and validation cycle.
4. **Local value tests** — Validate computed values and data sources.

All tests are designed to run without requiring actual AWS resources for basic validation.

## Layout

| Directory | Purpose |
| --------- | ------- |
| [`tests/unit/`](../../tests/unit/) | Unit tests for modules. |
| [`tests/integration/`](../../tests/integration/) | Integration tests for end-to-end workflows. |
| [`tests/datafiles/`](../../tests/datafiles/) | Generated test data. |
| [`tests/logs/`](../../tests/logs/) | Structured pytest logs for LLM analysis. |
| [`tests/utils/`](../../tests/utils/) | Test utilities including log parsing and formatting. |

Pytest markers are defined in [`tests/pytest.ini`](../../tests/pytest.ini) — that file is authoritative.

## Running tests

See [`testing_commands.md`](testing_commands.md).
94 changes: 9 additions & 85 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,19 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
## Path Prefixes
References in the table below use these prefixes:

- `guid:` → `~/.claude/guidelines/`
- `proj:` → `<repo>/.claude/guidelines/`

## Important Reminders - LOAD BEFORE EVERY TASK
- Identify task type(s) and load applicable resources per the table below.

| Task Type | Required |
|-----------|----------|
| ALWAYS DO FOR ALL TASKS | `guid:variable_default_values.md` |
| ALWAYS DO FOR ALL TASKS | `proj:variable_default_values.md` |
| Any PR that changes deployer-visible behaviour (variables, component config, components added/removed) | `proj:changelog_protocol.md` |
| Writing or editing Terraform (`*.tf`, `terraform.tfvars`) | `proj:terraform_conventions.md` |
| Writing or editing Python | `proj:python_standards.md` |
| Running, writing, or analysing tests | `proj:testing_strategy.md`, `proj:testing_commands.md` |
| Working on security-sensitive code (secrets, IAM, SSM, certificates) | `proj:security_considerations.md` |

## Project Overview

Expand Down Expand Up @@ -65,58 +70,7 @@ make verify
make verify-full
```

### Testing Commands
```bash
# Run all tests
./tests/run_tests.sh

# Run unit tests only
pytest tests/unit/test_module_connection_strings/ -v -s -x

# Run plan-based tests only
make test_plan_only

# Generate test data
make generate_test_data

# Generate JSON plan for testing
make generate_json_plan
```

### Python Development
```bash
# Run Python validation script
python3 scripts/installer/validation/check_configuration.py

# Format Python files (required after any Python modifications)
black <filename>.py

# Run pytest with specific markers
pytest -m "local" -v
pytest -m "db" -v
pytest -m "redis" -v
```

### Testing with Structured Logging
```bash
# View structured test results
python tests/utils/log_parser.py summarize

# Extract failed tests for LLM analysis
python tests/utils/log_parser.py extract-failures

# View pytest logs for debugging
tail -f tests/logs/pytest_structured.log

# Parse logs for LLM analysis
python tests/utils/log_parser.py llm-format --recent 100

# Validate log format
python tests/utils/log_parser.py validate

# Export logs as JSON for programmatic analysis
python tests/utils/log_parser.py export-json
```
For test commands and structured-logging tooling, see [`proj:testing_commands.md`](.claude/guidelines/testing_commands.md).

## Project Structure

Expand All @@ -143,7 +97,7 @@ Sequential numbered files defining infrastructure resources in dependency order.
- `tests/datafiles/` - Test data generation
- `tests/logs/` - Structured pytest logs for LLM analysis
- `tests/utils/` - Test utilities including log parsing and formatting
- Pytest markers: `local`, `db`, `db_new`, `db_existing`, `redis`, `urls`, `urls_insecure`, `log_enabled`
- Pytest markers are defined in [`tests/pytest.ini`](tests/pytest.ini).

## Configuration Files

Expand All @@ -157,25 +111,6 @@ Sequential numbered files defining infrastructure resources in dependency order.
- Generated files placed in `assets/target/`
- Templates support variable substitution via Terraform `templatefile()` function

## Development Notes

### Python Code Standards
- **Must run `black` formatter after any Python file modifications**
- Uses custom tfvars parser (`installer/utils/extractors.py`)
- Logging configured via `installer/utils/logger.py`
- No `__pycache__` directories created (disabled in pytest.ini)

### Terraform Conventions
- **No default values** in `variables.tf` - all values must be explicitly defined in `terraform.tfvars`
- Uses `null_resource` with `local-exec` provisioners instead of `local_file` resources
- State stored locally by default (`DONTDELETE/terraform.tfstate`)

### Security Considerations
- Sensitive values stored in AWS SSM Parameter Store only
- Uses AWS Instance Connect Endpoint for private subnet access
- Templates contain `.tpl` extension to avoid accidental secret exposure
- Custom certificates supported via `assets/src/customcerts/`

## Troubleshooting

### Common Issues
Expand All @@ -187,14 +122,3 @@ Sequential numbered files defining infrastructure resources in dependency order.
- Configuration validation via `check_configuration.py` before Terraform execution
- Destroy validation via `check_destroy.py` before teardown
- Security scanning available via `tfsec` (use `make verify-full`)

## Testing Strategy

The project uses a hybrid testing approach:
1. **Plan-based tests** - Mock resources using `terraform plan` output
1. Plan outputs are cached in `tests/.plan_cache` to speed up n+1 test cycles when underlying values are unchanged.
2. **Unit tests** - Test individual modules in isolation
3. **Integration tests** - Full deployment and validation cycle
4. **Local value tests** - Validate computed values and data sources

All tests designed to run without requiring actual AWS resources for basic validation.
Loading