Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
308 changes: 306 additions & 2 deletions .github/copilot-instructions.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,19 @@
# Repository Instructions for GitHub Copilot

## Project Overview

**node-json-db** is a lightweight, file-based JSON database for Node.js/TypeScript with zero production dependencies. Key features:

- **DataPath navigation**: XPath-like syntax (`/users/0/name`, `/items[]`, `/data[-1]`) to navigate nested JSON data
- **Async-first API**: All database operations return Promises (async/await)
Comment on lines +7 to +8
Copy link

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The DataPath examples use /users/0/name, but the project’s documented/featured array syntax uses bracket notation (e.g. /users[0]/name) to support [] append and [-1] semantics consistently (see README array examples). Consider updating this example to bracket form to avoid implying /0 is the recommended array access pattern.

Copilot uses AI. Check for mistakes.
- **Optional AES-256-GCM encryption**: Via `config.setEncryption(key32bytes)`, stored as `.enc.json`
- **Concurrency safety**: Built-in reader-writer locks prevent race conditions
- **Pluggable adapters**: `IAdapter<T>` interface for custom storage backends
- **Automatic persistence**: `saveOnPush: true` (default) writes to disk on every `push()`/`delete()`
- **Generic type-safe retrieval**: `getObject<T>(path)` and `getObjectDefault<T>(path, default)`

---

## Package Manager

This project uses **pnpm** as the package manager, NOT npm.
Expand Down Expand Up @@ -38,6 +52,296 @@ pnpm install
### Common Commands

```bash
pnpm run build # Build the project
pnpm test # Run tests with coverage
pnpm run build # Build the project (TypeScript → dist/)
pnpm test # Run all tests with coverage (jest --coverage)
pnpm run build:doc # Generate TypeDoc API documentation (→ docs/)
```

---

## Architecture & Directory Structure

```
node-json-db/
├── src/ # TypeScript source (compiled to dist/)
│ ├── JsonDB.ts # Main exported class — all public API methods
│ ├── adapter/
│ │ ├── IAdapter.ts # Interface: readAsync() / writeAsync()
│ │ ├── data/
│ │ │ └── JsonAdapter.ts # JSON serialize/deserialize + date revival
│ │ └── file/
│ │ ├── FileAdapter.ts # Raw filesystem I/O
│ │ └── CipheredFileAdapter.ts # AES-256-GCM encryption layer
│ ├── lib/
│ │ ├── JsonDBConfig.ts # Config & ConfigWithAdapter classes
│ │ ├── Errors.ts # DatabaseError / DataError (extend NestedError)
│ │ ├── ArrayInfo.ts # Array path parsing (e.g. items[0], items[])
│ │ ├── DBParentData.ts # Parent node resolution for writes/deletes
│ │ └── Utils.ts # merge(), removeTrailingChar(), KeyValue type
│ └── lock/
│ ├── Lock.ts # readLockAsync() / writeLockAsync() helpers
│ ├── ReadWriteLock.ts # High-perf reader-writer lock with pooling
│ └── Error.ts # TimeoutError class
├── test/ # Jest test suite (ts-jest)
│ ├── 01-utils.test.ts # ArrayInfo regex + safe-regex ReDoS tests
│ ├── 02-jsondb.test.ts # Core CRUD, merge, types, errors
│ ├── 03-existing-db.test.ts # File persistence and reload
│ ├── 04-array-utils.test.ts # Array indexing, append, nested arrays
│ ├── 05-errors-test.ts # Error types and DataPath error codes
│ ├── 06-concurrency.test.ts # Lock timeouts, concurrent access
Comment on lines +89 to +91
Copy link

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the directory tree, test/05-errors-test.ts is described as covering error types / DataPath error codes, but the file is currently empty. Consider updating the annotation to point to the actual error coverage (e.g. test/02-jsondb.test.ts) or removing this entry to avoid sending agents to a dead file.

Copilot uses AI. Check for mistakes.
│ ├── 07-cyphered.test.ts # Encryption key validation, encrypt/decrypt
│ ├── ArrayInfo.test.ts # Unit tests for ArrayInfo class
│ ├── DBParentData.test.ts # Unit tests for DBParentData class
│ ├── JsonDB.test.ts # Unit tests for JsonDB class
│ ├── adapter/
│ │ └── adapters.test.ts # Unit tests for adapters
│ └── lock/ # Lock unit tests
├── dist/ # Compiled JS output (git-ignored, built by tsc)
├── jest.config.js # Jest configuration (preset: ts-jest)
├── tsconfig.json # TypeScript config (strict, commonjs, esnext)
└── package.json # Scripts, devDependencies, commitlint config
```

**Adapter chain**: `JsonDB → JsonAdapter → (CipheredFileAdapter | FileAdapter)`

---

## TypeScript Conventions

- **Strict mode** is enabled (`"strict": true` in `tsconfig.json`) — no implicit `any`
- **Target**: `esnext` compiled to `commonjs` for Node.js compatibility
- **Naming**:
- Classes/Interfaces: `PascalCase` (e.g., `JsonDB`, `ArrayInfo`, `IAdapter`)
- Private class members: prefixed with `_` (e.g., `_filename`)
- Methods/variables: `camelCase`
- **Visibility**: Mark implementation details `private`, keep the public API minimal
- **Generics**: Use generic type parameters for type-safe retrieval (e.g., `getObject<T>()`)
- **Imports**: Named imports from relative paths; no barrel `index.ts` files
- **JSDoc**: Required for all public API methods and configuration options

---

## Public API Reference

### `JsonDB` class (`src/JsonDB.ts`)

All methods are async and protected by reader-writer locks internally.

Comment on lines +128 to +129
Copy link

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The statement “All methods are async and protected by reader-writer locks internally” isn’t accurate for the current JsonDB API: resetData() is synchronous, and load(), save(), reload(), and fromPath() are not wrapped in readLockAsync/writeLockAsync. Please adjust this wording to reflect which methods are actually lock-protected/async to avoid misleading API consumers and agents.

Copilot uses AI. Check for mistakes.
| Method | Signature | Description |
|--------|-----------|-------------|
| `push` | `push(path, data, override=true)` | Write data; `override=false` deep-merges |
| `getData` | `getData(path): Promise<any>` | Read data; throws `DataError` if not found |
| `getObject<T>` | `getObject<T>(path): Promise<T>` | Type-safe read |
| `getObjectDefault<T>` | `getObjectDefault<T>(path, default?)` | Read with fallback value |
| `exists` | `exists(path): Promise<boolean>` | Check if path exists |
| `delete` | `delete(path): Promise<void>` | Remove node at path |
| `count` | `count(arrayPath): Promise<number>` | Array length |
| `getIndex` | `getIndex(path, value, prop='id')` | Find array index by property value |
| `getIndexValue` | `getIndexValue(path, value)` | Find array index by value |
| `find<T>` | `find<T>(rootPath, callback)` | First match in array/object |
| `filter<T>` | `filter<T>(rootPath, callback)` | All matches in array/object |
| `fromPath` | `fromPath(routePath, prop='id')` | Convert route-style path to DataPath |
| `load` | `load(): Promise<void>` | Manually load from file (auto-called) |
| `save` | `save(force?): Promise<void>` | Manually save to file |
| `reload` | `reload(): Promise<void>` | Reload from disk, discarding in-memory state |
| `resetData` | `resetData(data): void` | ⚠️ Replaces all in-memory data directly |

### `Config` class (`src/lib/JsonDBConfig.ts`)

```typescript
new Config(
filename: string, // File path (auto-appends .json if no extension)
saveOnPush: boolean, // Default: true — save after every push/delete
humanReadable: boolean, // Default: false — pretty-print JSON
separator: string, // Default: '/' — DataPath separator
syncOnSave: boolean, // Default: false — fsync after writes
parseDates: boolean // Default: true — revive ISO date strings
)

config.setEncryption(key: CipherKey)
// key must be exactly 32 bytes (Buffer, string, or symmetric KeyObject)
// Changes filename from .json → .enc.json (idempotent)
```

### `ConfigWithAdapter` class

```typescript
new ConfigWithAdapter(
adapter: IAdapter<any>, // Custom adapter implementation
saveOnPush: boolean, // Default: true
separator: string // Default: '/'
)
```

---

## DataPath Syntax

DataPaths use `/` as the default separator (configurable):

| Path | Meaning |
|------|---------|
| `/` | Root object |
| `/users` | `users` property of root |
| `/users/0/name` | `name` of first user |
| `/items[0]` | First element of `items` array |
Comment on lines +185 to +187
Copy link

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This table entry uses /users/0/name, which is inconsistent with the library’s standard array indexing syntax (/users[0]/name) used throughout the README and tests. Using bracket notation here will better match how ArrayInfo parses arrays (and avoids confusion with object property names that happen to be numeric strings).

Copilot uses AI. Check for mistakes.
| `/items[]` | Append new element to `items` array |
| `/items[-1]` | Last element of `items` array |
| `/matrix[0][1]` | Nested array access |
| `/items[]/id` | Append new object and set its `id` property |

---

## Error Handling

All errors extend `NestedError` which includes `message`, `id` (numeric code), and optional `inner` (wrapped error).

```typescript
import { DatabaseError, DataError } from 'node-json-db'

// DataError — problems with path or data (ids 3–13, 100, 200)
// id 3: Can't merge another type with Array
// id 4: Can't merge Array with Object
// id 5: Path not found (used by getObjectDefault to return default)
// id 10: Can't find array index
Comment on lines +202 to +206
Copy link

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This section says it documents all DataError ids 3–13, 100, 200, but it currently omits id 6 (The Data Path can't be empty). Also, ids 7–9 aren’t used in the codebase, so the “3–13” range wording is misleading—consider listing only the ids that actually exist.

Copilot uses AI. Check for mistakes.
// id 11: Target is not an array
// id 12: filter/find on non-array/non-object
// id 13: fromPath value not found
// id 100: Can't get/delete appended data
// id 200: Non-numeric array index

Comment on lines +208 to +212
Copy link

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DataError id mappings here don’t match the implementation in a few places: id 100 is specifically “Can’t get data when appending” (not delete), and id 10 is also used for “Can’t delete an appended data” in ArrayInfo.delete(). Please align these descriptions with the actual throw sites to keep the error-code reference reliable.

Copilot uses AI. Check for mistakes.
// DatabaseError — I/O or load/save issues (ids 1–2, 7)
// id 1: Can't load database
// id 2: Can't save database
// id 7: Database not loaded, can't write
```

---

## Encryption

Encrypted databases use AES-256-GCM via Node.js `crypto`:

```typescript
import { Config } from 'node-json-db'
import { randomBytes } from 'crypto'

const config = new Config('mydb', true)
const key = randomBytes(32) // Must be exactly 32 bytes
config.setEncryption(key) // filename: mydb.json → mydb.enc.json

// Encrypted file format (JSON):
// { "iv": "<hex>", "tag": "<hex>", "data": "<hex>" }
// A fresh IV and tag are generated on every write for security
```

- Accepts: `Buffer` (32 bytes), `string` (32 chars), or symmetric `KeyObject` (256-bit)
- Asymmetric keys are rejected with an `Error`
Comment on lines +238 to +239
Copy link

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The encryption key bullet says a string of “32 chars”, but the implementation validates 32 bytes via Buffer.byteLength(). A 32-character non-ASCII string may fail this check; consider rewording to “32 bytes (e.g., 32 ASCII characters)” to match actual behavior.

Copilot uses AI. Check for mistakes.
- The `.enc.json` extension prevents accidental unencrypted access

---

## Testing Conventions

- **Framework**: Jest 30 with `ts-jest` preset (no compilation step needed for tests)
- **Run tests**: `pnpm test` — runs all tests with coverage
- **Run single file**: `pnpm test -- test/02-jsondb.test.ts`
- **Temporary files**: Use `/tmp/<uuid>` paths (via `randomUUID()`) or `test/` directory with cleanup
- **Cleanup**: Always add `afterEach` hooks to remove test files created during tests
- **Test file naming**: Numbered files (`01-`, `02-`) are integration tests; unnumbered are unit tests
- **Coverage**: Reported to codecov via CI

### Test patterns

```typescript
// Cleanup pattern (used in integration tests)
afterEach(() => {
try { fs.rmSync(testFile + ".json") } catch (e) {}
})

// Temporary encrypted DB pattern (in encryption tests)
const getDbPath = () => `/tmp/${randomUUID()}`
afterEach(() => {
const files = ['/tmp/test.enc.json', '/tmp/test.json']
files.forEach(f => { if (existsSync(f)) unlinkSync(f) })
})
```

---

## Commit Message Conventions

This repository enforces [Conventional Commits](https://www.conventionalcommits.org/) via `commitlint`:

```
<type>: <description>

Types: feat, fix, docs, style, refactor, test, chore, perf, ci, build, revert
```

Examples:
- `feat: add support for custom separators`
- `fix: prevent race condition in concurrent writes`
- `test: add coverage for encryption key validation`
- `docs: update README with encryption examples`

The `body-max-line-length` rule is disabled (long PR descriptions allowed).

---

## CI/CD Pipeline

**Trigger branches**: `develop` (PRs & pushes), `master` (PRs, pushes, and releases)

### `nodejs.yml` — Build & Test
- Runs on Node.js `lts/*` and `current` (matrix)
- Steps: `pnpm install` → `pnpm test` → upload coverage to Codecov

### `release.yml` — Release (master only)
- `build` job: same as above
- `ci-cd-check`: dry-run `semantic-release`
- `deploy-pages`: builds TypeDoc → deploys to GitHub Pages
- `release`: runs `semantic-release` → publishes to npm + creates GitHub Release

### `codeql-analysis.yml` — Security Scanning
- Runs CodeQL analysis on pushes/PRs to master/develop

Comment on lines +306 to +308
Copy link

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CodeQL workflow description says it runs on PRs to both master and develop, but .github/workflows/codeql-analysis.yml is configured to run on pull requests targeting develop only. Please update this line to match the actual triggers so agents don’t assume CodeQL runs on every PR to master.

Copilot uses AI. Check for mistakes.
**Release automation**: `semantic-release` reads conventional commits to determine version bump and generates `CHANGELOG.md` automatically.

---

## Adapter Pattern

To implement a custom storage backend, implement `IAdapter<T>`:

```typescript
import { IAdapter } from 'node-json-db'

class MyAdapter implements IAdapter<string> {
async readAsync(): Promise<string | null> {
// Return raw string data or null if not found
}
async writeAsync(data: string): Promise<void> {
// Persist raw string data
}
}

// Use with ConfigWithAdapter:
import { JsonDB, ConfigWithAdapter } from 'node-json-db'
const db = new JsonDB(new ConfigWithAdapter(new JsonAdapter(new MyAdapter())))
```
Comment on lines +318 to +332
Copy link

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Adapter Pattern example uses new JsonAdapter(...) but the snippet doesn’t import JsonAdapter, so it won’t compile as written. Either add JsonAdapter to the import list or show an example where the custom adapter already reads/writes JSON and can be passed directly to ConfigWithAdapter.

Copilot uses AI. Check for mistakes.

The `JsonAdapter` wraps a `string`-based adapter and handles JSON serialization/deserialization (including optional date revival). The `FileAdapter` → `CipheredFileAdapter` extend chain is the built-in file-based implementation.

---

## Known Patterns & Gotchas

1. **`push()` with `override=false`** merges recursively for objects, concatenates for arrays, and skips `null` values
2. **`getObjectDefault<T>()`** catches only `DataError` with `id === 5` (path not found); other errors still propagate
Comment on lines +340 to +341
Copy link

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gotcha #1 says push() merge “skips null values”, but the merge() implementation in src/lib/Utils.ts does not skip nulls—source[prop] can overwrite with null, and only the special-case is when the destination is null. Please adjust this description to match the actual merge semantics (arrays concat; objects recurse; primitives including null overwrite).

Copilot uses AI. Check for mistakes.
3. **`getData("/")`** returns the entire root object
4. **Array index caching**: `ArrayInfo.processArray()` caches regex results in `regexCache` for performance
5. **Concurrency**: `readLockAsync`/`writeLockAsync` from `src/lock/Lock.ts` wrap all public read/write operations; custom adapter implementations do not need to add their own locking
6. **Date parsing**: Enabled by default (`parseDates: true`); ISO 8601 strings are automatically revived as `Date` objects
Comment on lines +344 to +345
Copy link

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gotcha #5 claims readLockAsync/writeLockAsync wrap all public read/write operations, but load(), save(), reload(), fromPath(), and resetData() are not lock-wrapped in src/JsonDB.ts. Consider narrowing this statement to the specific methods that are locked (getData/push/delete, and methods that call getData).

Copilot uses AI. Check for mistakes.
7. **File auto-created**: The JSON file is only created on the first `save()` (triggered by `push()`/`delete()` with `saveOnPush: true`); the file does NOT exist after a bare `new JsonDB(config)` call
8. **`fromPath()`**: Converts Express-style routes (e.g., `/users/123`) to DataPaths using `getIndex()` internally
Comment on lines +346 to +347
Copy link

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gotcha #7 says the JSON file is created only on the first save() (via push/delete), but the default file-based adapter path will also create the file during the first load()/getData() when it doesn’t exist (because JsonAdapter.readAsync() writes {} on empty/missing data). Please update this to reflect that the first read can create the file.

Copilot uses AI. Check for mistakes.