Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions .cursor/rules/tests-nvm.mdc
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
description: Run nvm use before tests to avoid SyntaxError (e.g. from "with" or other Node-version-dependent syntax)
alwaysApply: true
---

# Running tests

Before running any test commands in this project (e.g. `npm test`, `npm run mocha`, `npm run test:unit`), run:

```bash
nvm use
```

This ensures the Node version from `.nvmrc` is active. Skipping it can cause SyntaxError or other version-related failures.
76 changes: 63 additions & 13 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,57 @@ All notable changes to this project will be documented in this file. Dates are d

Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).

#### [v1.0.0](https://github.com/oceanprotocol/ocean-node/compare/v0.2.3...v1.0.0)
#### [v1.0.7](https://github.com/oceanprotocol/ocean-node/compare/v1.0.6...v1.0.7)

- fix query [`#1252`](https://github.com/oceanprotocol/ocean-node/pull/1252)

#### [v1.0.6](https://github.com/oceanprotocol/ocean-node/compare/v1.0.5...v1.0.6)

> 3 March 2026

- fix engine hash creation [`#1250`](https://github.com/oceanprotocol/ocean-node/pull/1250)
- Release 1.0.6 [`b7a7f65`](https://github.com/oceanprotocol/ocean-node/commit/b7a7f655573903595f4ac76a917385b3a14ba60b)

#### [v1.0.5](https://github.com/oceanprotocol/ocean-node/compare/v1.0.4...v1.0.5)

> 3 March 2026

- Locks catch error [`#1248`](https://github.com/oceanprotocol/ocean-node/pull/1248)
- Release 1.0.5 [`080a370`](https://github.com/oceanprotocol/ocean-node/commit/080a370a9f5f436279247b43a6a254cacbeeccf1)

#### [v1.0.4](https://github.com/oceanprotocol/ocean-node/compare/v1.0.3...v1.0.4)

> 3 March 2026

- fix job deletion [`#1247`](https://github.com/oceanprotocol/ocean-node/pull/1247)
- Release 1.0.4 [`152f8d1`](https://github.com/oceanprotocol/ocean-node/commit/152f8d108c86a9d0031f15a042c0851902a83971)

#### [v1.0.3](https://github.com/oceanprotocol/ocean-node/compare/v1.0.2...v1.0.3)

> 2 March 2026

- fix access lists validation format [`#1244`](https://github.com/oceanprotocol/ocean-node/pull/1244)
- Release 1.0.3 [`188710b`](https://github.com/oceanprotocol/ocean-node/commit/188710b51ba461476974eb80bd9c5e38d71a8d9d)

#### [v1.0.2](https://github.com/oceanprotocol/ocean-node/compare/v1.0.1...v1.0.2)

> 2 March 2026

- allow custom cpu & ram [`#1242`](https://github.com/oceanprotocol/ocean-node/pull/1242)
- Release 1.0.2 [`3ae416e`](https://github.com/oceanprotocol/ocean-node/commit/3ae416efd0c6ac8014b0ccac4bf1ee399780d186)

#### [v1.0.1](https://github.com/oceanprotocol/ocean-node/compare/v1.0.0...v1.0.1)

> 1 March 2026

- update dns bootstraps [`#1239`](https://github.com/oceanprotocol/ocean-node/pull/1239)
- Do not abort on ping failure [`#1238`](https://github.com/oceanprotocol/ocean-node/pull/1238)
- Bump minimatch from 3.1.2 to 3.1.5 [`#1237`](https://github.com/oceanprotocol/ocean-node/pull/1237)
- Release 1.0.1 [`c79f33b`](https://github.com/oceanprotocol/ocean-node/commit/c79f33bbc4765aa5fdf73ad52cc0245f942a0068)

### [v1.0.0](https://github.com/oceanprotocol/ocean-node/compare/v0.2.3...v1.0.0)

> 27 February 2026

- Release 1 0 [`#1235`](https://github.com/oceanprotocol/ocean-node/pull/1235)
- retry connection lost [`#1213`](https://github.com/oceanprotocol/ocean-node/pull/1213)
Expand Down Expand Up @@ -177,36 +227,36 @@ Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
- Fix validUntil value for free c2d. [`#899`](https://github.com/oceanprotocol/ocean-node/pull/899)
- Bump axios from 1.7.4 to 1.8.2 [`#892`](https://github.com/oceanprotocol/ocean-node/pull/892)
- Bump serialize-javascript and mocha [`#891`](https://github.com/oceanprotocol/ocean-node/pull/891)
- Release 1.0.0 [`bbdf97b`](https://github.com/oceanprotocol/ocean-node/commit/bbdf97b55eb62a985650f0f044c7e4e300169c2d)
- add integration tests for getJobs handler [`b4bdb40`](https://github.com/oceanprotocol/ocean-node/commit/b4bdb4058807c0315d6ed1c2982893ed5354aad8)
- remove retrial. [`1190c46`](https://github.com/oceanprotocol/ocean-node/commit/1190c46f7bb676083e12c9741a477a7166b9165f)
- Fix retrial in worker. [`8d5ed53`](https://github.com/oceanprotocol/ocean-node/commit/8d5ed53825b5b1026c0b2968ae4e4222a7401829)

#### [v0.2.3](https://github.com/oceanprotocol/ocean-node/compare/v0.2.1...v0.2.3)

> 24 March 2025

- Update node script - C2D [`#896`](https://github.com/oceanprotocol/ocean-node/pull/896)
- Update node script - C2D [`#896`](https://github.com/oceanprotocol/ocean-node/pull/896)
- fix docker-compose [`#895`](https://github.com/oceanprotocol/ocean-node/pull/895)
- re-indexing old DDOs [`#867`](https://github.com/oceanprotocol/ocean-node/pull/867)
- Upgrade tsx dep to v4.x [`#893`](https://github.com/oceanprotocol/ocean-node/pull/893)
- C2D Docker [`#705`](https://github.com/oceanprotocol/ocean-node/pull/705)
- Updating codeowners [`#887`](https://github.com/oceanprotocol/ocean-node/pull/887)
- fix issue with empty nft fields [`#886`](https://github.com/oceanprotocol/ocean-node/pull/886)
- fix issue with empty nft fields [`#886`](https://github.com/oceanprotocol/ocean-node/pull/886)
- add allowed admins access list [`#841`](https://github.com/oceanprotocol/ocean-node/pull/841)
- Update error message for invalid peer connection [`#874`](https://github.com/oceanprotocol/ocean-node/pull/874)
- add AUTHORIZED_DECRYPTERS_LIST [`#836`](https://github.com/oceanprotocol/ocean-node/pull/836)
- fix status code if policy server not available [`#869`](https://github.com/oceanprotocol/ocean-node/pull/869)
- Fix DDO: Stats and Prices for exchanges/dispensers [`#774`](https://github.com/oceanprotocol/ocean-node/pull/774)
- Fix DDO: Stats and Prices for exchanges/dispensers [`#774`](https://github.com/oceanprotocol/ocean-node/pull/774)
- move p2p getters as handlers [`#862`](https://github.com/oceanprotocol/ocean-node/pull/862)
- always check remote peerId [`#864`](https://github.com/oceanprotocol/ocean-node/pull/864)
- Test if dashboard changes are already committed [`#842`](https://github.com/oceanprotocol/ocean-node/pull/842)
- add AUTHORIZED_PUBLISHERS_*** env variables [`#826`](https://github.com/oceanprotocol/ocean-node/pull/826)
- add AUTHORIZED*PUBLISHERS*\*\*\* env variables [`#826`](https://github.com/oceanprotocol/ocean-node/pull/826)
- Issue 814 credentials types [`#823`](https://github.com/oceanprotocol/ocean-node/pull/823)
- remove echo command [`#839`](https://github.com/oceanprotocol/ocean-node/pull/839)
- Issue 808 - new accesslist credentials type [`#819`](https://github.com/oceanprotocol/ocean-node/pull/819)
- add ALLOWED_VALIDATORS_LIST [`#829`](https://github.com/oceanprotocol/ocean-node/pull/829)
- update build files and hash [`#821`](https://github.com/oceanprotocol/ocean-node/pull/821)
- add * as match all rule for address types [`#837`](https://github.com/oceanprotocol/ocean-node/pull/837)
- add \* as match all rule for address types [`#837`](https://github.com/oceanprotocol/ocean-node/pull/837)
- fix: return correct message on policy server call [`#834`](https://github.com/oceanprotocol/ocean-node/pull/834)
- add policyServerPassthrough routes [`#832`](https://github.com/oceanprotocol/ocean-node/pull/832)
- Bump fast-xml-parser from 4.3.6 to 4.5.0 in /dashboard [`#711`](https://github.com/oceanprotocol/ocean-node/pull/711)
Expand Down Expand Up @@ -242,7 +292,7 @@ Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
- fix system tests. Running old version of node? [`#733`](https://github.com/oceanprotocol/ocean-node/pull/733)
- rm console.logs [`#731`](https://github.com/oceanprotocol/ocean-node/pull/731)
- fix wrong block for log [`#727`](https://github.com/oceanprotocol/ocean-node/pull/727)
- nonce db sql lite [`#723`](https://github.com/oceanprotocol/ocean-node/pull/723)
- nonce db sql lite [`#723`](https://github.com/oceanprotocol/ocean-node/pull/723)
- Bump version axios 1.6.0 -> 1.7.4. [`#716`](https://github.com/oceanprotocol/ocean-node/pull/716)
- Bump version express 4.18.2 -> 4.21.0. [`#717`](https://github.com/oceanprotocol/ocean-node/pull/717)
- Feature/ add Elasticsearch database alternative for typesense [`#599`](https://github.com/oceanprotocol/ocean-node/pull/599)
Expand All @@ -251,7 +301,7 @@ Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
- Bump micromatch from 4.0.5 to 4.0.8 in /dashboard [`#649`](https://github.com/oceanprotocol/ocean-node/pull/649)
- Bump undici from 5.27.0 to 5.28.4 [`#610`](https://github.com/oceanprotocol/ocean-node/pull/610)
- testing changes [`#718`](https://github.com/oceanprotocol/ocean-node/pull/718)
- Policy Server [`#694`](https://github.com/oceanprotocol/ocean-node/pull/694)
- Policy Server [`#694`](https://github.com/oceanprotocol/ocean-node/pull/694)
- fix missing/invalid db_url log message, put warn at startup [`#654`](https://github.com/oceanprotocol/ocean-node/pull/654)
- move c2d engines under OceanNode class [`#702`](https://github.com/oceanprotocol/ocean-node/pull/702)
- improve error message, transfer fees and tweak node response [`#701`](https://github.com/oceanprotocol/ocean-node/pull/701)
Expand Down Expand Up @@ -381,7 +431,7 @@ Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
- Differentiate error messages indexer [`#570`](https://github.com/oceanprotocol/ocean-node/pull/570)
- Issue 565 optimize get status [`#566`](https://github.com/oceanprotocol/ocean-node/pull/566)
- fix get indexing queue [`#564`](https://github.com/oceanprotocol/ocean-node/pull/564)
- Changes on logging transports (.env var based locations) [`#553`](https://github.com/oceanprotocol/ocean-node/pull/553)
- Changes on logging transports (.env var based locations) [`#553`](https://github.com/oceanprotocol/ocean-node/pull/553)
- Check if ddo state is active before executing node's commands. [`#542`](https://github.com/oceanprotocol/ocean-node/pull/542)
- use static rpc provider [`#548`](https://github.com/oceanprotocol/ocean-node/pull/548)
- Fix downloading full content of the file. [`#559`](https://github.com/oceanprotocol/ocean-node/pull/559)
Expand Down Expand Up @@ -435,7 +485,7 @@ Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
- Remove chain id from get compute envs task. [`#460`](https://github.com/oceanprotocol/ocean-node/pull/460)
- Issue 397 warn env db logs [`#457`](https://github.com/oceanprotocol/ocean-node/pull/457)
- fix p2p peers [`#449`](https://github.com/oceanprotocol/ocean-node/pull/449)
- c2d v2 arhitecture [`#381`](https://github.com/oceanprotocol/ocean-node/pull/381)
- c2d v2 arhitecture [`#381`](https://github.com/oceanprotocol/ocean-node/pull/381)
- Fix: dashboard failing build if NODE_ENV is changed [`#450`](https://github.com/oceanprotocol/ocean-node/pull/450)
- Dashboard: get ocean peers polling [`#445`](https://github.com/oceanprotocol/ocean-node/pull/445)
- Replace hardcoded values in Dashboard [`#444`](https://github.com/oceanprotocol/ocean-node/pull/444)
Expand Down Expand Up @@ -514,15 +564,15 @@ Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).
- Provider fees compute [`#252`](https://github.com/oceanprotocol/ocean-node/pull/252)
- Updates to package.json bringing it in line with our other repositories [`#260`](https://github.com/oceanprotocol/ocean-node/pull/260)
- Issue 205 ddo handling [`#239`](https://github.com/oceanprotocol/ocean-node/pull/239)
- fix error on publish + no signer/no account / metadata events error [`#255`](https://github.com/oceanprotocol/ocean-node/pull/255)
- fix error on publish + no signer/no account / metadata events error [`#255`](https://github.com/oceanprotocol/ocean-node/pull/255)
- add config option for network interfaces, p2p and http [`#248`](https://github.com/oceanprotocol/ocean-node/pull/248)
- Feature/ Add handle decrypt method [`#221`](https://github.com/oceanprotocol/ocean-node/pull/221)
- Added checks for metadata events. [`#237`](https://github.com/oceanprotocol/ocean-node/pull/237)
- Create collections specific for ddo versions [`#225`](https://github.com/oceanprotocol/ocean-node/pull/225)
- Move commands from constants.ts to @types/commands.ts [`#244`](https://github.com/oceanprotocol/ocean-node/pull/244)
- Issue 227 get environments [`#238`](https://github.com/oceanprotocol/ocean-node/pull/238)
- fix unit test on commands.ts - pick mismatches in both directions [`#246`](https://github.com/oceanprotocol/ocean-node/pull/246)
- Expose validateDDO on http [`#234`](https://github.com/oceanprotocol/ocean-node/pull/234)
- Expose validateDDO on http [`#234`](https://github.com/oceanprotocol/ocean-node/pull/234)
- Missing param validation for directCommand DOWNLOAD [`#242`](https://github.com/oceanprotocol/ocean-node/pull/242)
- add c2d in ci [`#241`](https://github.com/oceanprotocol/ocean-node/pull/241)
- add C2C cluster env config [`#240`](https://github.com/oceanprotocol/ocean-node/pull/240)
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,7 @@ Your node is now running. To start additional nodes, repeat these steps in a new
- [API Endpoints](docs/API.md)
- [Environmental Variables](docs/env.md)
- [Database Guide](docs/database.md)
- [Storage Types](docs/Storage.md)
- [Testing Guide](docs/testing.md)
- [Network Configuration](docs/networking.md)
- [Logging & accessing logs](docs/networking.md)
Expand Down
173 changes: 173 additions & 0 deletions docs/Storage.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,173 @@
# Storage Types

Ocean Node supports four storage backends for assets (e.g. algorithm or data files). Each type is identified by a `type` field on the file object and has its own shape and validation rules.

## Supported types

| Type | `type` value | Description |
| ---------- | ------------- | ------------------------------------ |
| **URL** | `url` | File served via HTTP/HTTPS |
| **IPFS** | `ipfs` | File identified by IPFS CID |
| **Arweave**| `arweave` | File identified by Arweave transaction ID |
| **S3** | `s3` | File in S3-compatible storage (AWS, Ceph, MinIO, etc.) |

All file objects can optionally include encryption metadata: `encryptedBy` and `encryptMethod` (e.g. `AES`, `ECIES`).

---

## URL storage

Files are fetched from a given URL using HTTP GET or POST.

### File object shape

```json
{
"type": "url",
"url": "https://example.com/path/to/file.zip",
"method": "get",
"headers": {}
}
```

| Field | Required | Description |
| --------- | -------- | ------------------------------------------------ |
| `type` | Yes | Must be `"url"` |
| `url` | Yes | Full HTTP/HTTPS URL to the file |
| `method` | Yes | `"get"` or `"post"` |
| `headers` | No | Optional request headers (key-value object) |

### Validation

- `url` and `method` must be present.
- `method` must be `get` or `post` (case-insensitive).
- If the node config defines `unsafeURLs` (list of regex patterns), any URL matching a pattern is rejected.
- The URL must look like a real URL (`http://` or `https://`); path-like values are rejected.

### Node configuration

- Optional: `unsafeURLs` – array of regex strings; URLs matching any of them are considered unsafe and rejected.

---

## IPFS storage

Files are resolved via an IPFS gateway using a content identifier (CID).

### File object shape

```json
{
"type": "ipfs",
"hash": "QmXoypizjW3WknFiJnKLwHCnL72vedxjQkDDP1mXWo6uco"
}
```

| Field | Required | Description |
| ------ | -------- | ------------------------------ |
| `type` | Yes | Must be `"ipfs"` |
| `hash` | Yes | IPFS content identifier (CID) |

The node builds the download URL as: `{ipfsGateway}/ipfs/{hash}` (e.g. `https://ipfs.io/ipfs/QmXoy...`).

### Validation

- `hash` (CID) must be present.
- The value must not look like an HTTP(S) URL (use URL storage for gateway URLs).
- The value must not look like a file path.

### Node configuration

- **Required**: `ipfsGateway` – base URL of the IPFS HTTP gateway (e.g. `https://ipfs.io`).

---

## Arweave storage

Files are identified by an Arweave transaction ID and fetched via an Arweave gateway.

### File object shape

```json
{
"type": "arweave",
"transactionId": "abc123..."
}
```

| Field | Required | Description |
| --------------- | -------- | -------------------------- |
| `type` | Yes | Must be `"arweave"` |
| `transactionId` | Yes | Arweave transaction ID |

The node builds the download URL as: `{arweaveGateway}/{transactionId}`.

### Validation

- `transactionId` must be present.
- The value must not look like an HTTP(S) URL (use URL storage for direct URLs).
- The value must not look like a file path.

### Node configuration

- **Required**: `arweaveGateway` – base URL of the Arweave gateway (e.g. `https://arweave.net`).

---

## S3 storage

Files are stored in S3-compatible object storage. The node uses the AWS SDK and works with Amazon S3, Ceph, MinIO, DigitalOcean Spaces, and other S3-compatible services. Credentials and endpoint are provided on the file object; no node-level S3 config is required.

### File object shape

```json
{
"type": "s3",
"s3Access": {
"endpoint": "https://s3.amazonaws.com",
"region": "us-east-1",
"bucket": "my-bucket",
"objectKey": "path/to/file.zip",
"accessKeyId": "AKIAIOSFODNN7EXAMPLE",
"secretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
}
}
```

| Field | Required | Description |
| --------- | -------- | ----------- |
| `type` | Yes | Must be `"s3"` |
| `s3Access` | Yes | Object with endpoint, bucket, object key, and credentials (see below). |

**`s3Access` fields:**

| Field | Required | Description |
| ----------------- | -------- | ----------- |
| `endpoint` | Yes | S3 endpoint URL (e.g. `https://s3.amazonaws.com`, `https://nyc3.digitaloceanspaces.com`, or `https://my-ceph.example.com`) |
| `bucket` | Yes | Bucket name |
| `objectKey` | Yes | Object key (path within the bucket) |
| `accessKeyId` | Yes | Access key for the S3-compatible API |
| `secretAccessKey` | Yes | Secret key for the S3-compatible API |
| `region` | No | Region (e.g. `us-east-1`). Optional; defaults to `us-east-1` if omitted. Some backends (e.g. Ceph) may ignore it. |
| `forcePathStyle` | No | If `true`, use path-style addressing (e.g. `endpoint/bucket/key`). Required for some S3-compatible services (e.g. MinIO). Default `false` (virtual-host style, e.g. `bucket.endpoint/key`, standard for AWS S3). |

### Validation

- `s3Access` must be present.
- Within `s3Access`, `bucket`, `objectKey`, `endpoint`, `accessKeyId`, and `secretAccessKey` must be present and non-empty.
- `region` and `forcePathStyle` are optional; when provided they are used when creating the S3 client.

### Node configuration

- None. All S3 connection details (endpoint, credentials, bucket, key) come from the file object’s `s3Access`.

---

## Summary

- **URL**: flexible HTTP(S) endpoints; optional custom headers and `unsafeURLs` filtering.
- **IPFS**: CID-based; requires `ipfsGateway` in config.
- **Arweave**: transaction-ID-based; requires `arweaveGateway` in config.
- **S3**: S3-compatible object storage (AWS, Ceph, MinIO, etc.); credentials and endpoint in the file object; `region` optional (defaults to `us-east-1`).

The storage implementation lives under `src/components/storage/`. The node selects the backend from the file object’s `type` (case-insensitive) and validates the shape and config before fetching or streaming the file.
Loading
Loading