Skip to content

Commit 5e9ded4

Browse files
author
tcsenpai
committed
Merge branch 'stabilisation' into petri
# Conflicts: # src/libs/blockchain/chainBlocks.ts # src/libs/consensus/v2/routines/broadcastBlockHash.ts # testing/loadgen/src/features/consensus/petri_block_production.ts # testing/loadgen/src/features/consensus/petri_relay_flow.ts # testing/loadgen/src/features/consensus/petri_soak.ts # testing/loadgen/src/features/consensus/petri_tx_inclusion.ts
2 parents 8751305 + 530111b commit 5e9ded4

224 files changed

Lines changed: 1003 additions & 633 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.dockerignore‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,10 +5,10 @@
55
node_modules
66

77
# Exclude devnet runtime files
8-
devnet/identities
9-
devnet/.env
10-
devnet/demos_peerlist.json
11-
devnet/postgres-data
8+
testing/devnet/identities
9+
testing/devnet/.env
10+
testing/devnet/demos_peerlist.json
11+
testing/devnet/postgres-data
1212
ipfs
1313
ipfs_53550
1414
docker_data

‎.gitignore‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -254,9 +254,9 @@ certs/
254254
# Ops junk
255255
src/libs/network/routines/transactions/*.html
256256
# Devnet runtime files (generated, not tracked)
257-
devnet/identities/
258-
devnet/.env
259-
devnet/postgres-data/
257+
testing/devnet/identities/
258+
testing/devnet/.env
259+
testing/devnet/postgres-data/
260260
ipfs_53550/data_53550/ipfs
261261
.tlsnotary-key
262262
src/features/tlsnotary/SDK_INTEGRATION.md
@@ -286,7 +286,7 @@ documentation/demos_yp_v5.pdf
286286
/.taskmaster
287287
/internal-docs
288288
/.beads_bak
289-
/better_testing/runs
289+
/testing/runs
290290
/documentation/internal-docs
291291
/PR_DUMP.md
292292
/.beads

‎MISSING_SP_ENDPOINTS.md‎

Lines changed: 314 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,314 @@
1+
# Missing StorageProgram Query Endpoints
2+
3+
## Summary
4+
5+
The Demos node at `node2.demos.sh` currently supports StorageProgram **write** transactions
6+
(`CREATE_STORAGE_PROGRAM`, `WRITE_STORAGE`) but returns `501 Method not implemented` for all
7+
**read/query** RPC methods. These methods are invoked via the `nodeCall` RPC mechanism — a POST
8+
request to the node with `{ method: "nodeCall", params: [{ message: "<METHOD_NAME>", data: {...}, muid: "..." }] }`.
9+
10+
The SDK expects a response shape of `{ result: 200, response: <payload> }` on success.
11+
12+
A total of **9 RPC message handlers** are missing. Until they are implemented, the only workaround
13+
is replaying state from `getTransactionHistory` — which is slow, brittle, and not scalable.
14+
15+
**Test contract:** `stor-30556b12e724b0e7c7c45ef920df7ea822cad04c` — a deployed StorageProgram with JSON data (multi-user registry with alice/bob entries). Good address to probe against when implementing these endpoints.
16+
17+
---
18+
19+
## RPC Transport
20+
21+
All query methods below are sent as HTTP POST to the node URL with the following envelope:
22+
23+
```json
24+
{
25+
"method": "nodeCall",
26+
"params": [
27+
{
28+
"message": "<RPC_METHOD_NAME>",
29+
"data": { ... },
30+
"muid": "storage-<timestamp>"
31+
}
32+
]
33+
}
34+
```
35+
36+
Success response envelope:
37+
38+
```json
39+
{ "result": 200, "response": <payload> }
40+
```
41+
42+
---
43+
44+
## Missing Endpoints
45+
46+
### 1. `getStorageProgram`
47+
48+
- **RPC Method Name:** `getStorageProgram`
49+
- **SDK Method:** `StorageProgram.getByAddress(rpcUrl, storageAddress, identity?)`
50+
- **Request Parameters:**
51+
```json
52+
{
53+
"storageAddress": "stor-<40-char-hash>",
54+
"requesterAddress": "<optional demos1... address>"
55+
}
56+
```
57+
- **Expected Response:** `StorageProgramData`
58+
```typescript
59+
{
60+
storageAddress: string; // "stor-<hash>"
61+
owner: string; // "demos1..."
62+
programName: string;
63+
encoding: "json" | "binary";
64+
data?: Record<string, unknown> | string | null;
65+
metadata?: Record<string, unknown> | null;
66+
storageLocation: string; // "onchain" or other
67+
sizeBytes: number;
68+
createdAt: string; // ISO timestamp
69+
updatedAt: string; // ISO timestamp
70+
createdByTx?: string; // tx hash
71+
lastModifiedByTx?: string; // tx hash
72+
interactionTxs?: string[]; // all tx hashes that touched this program
73+
}
74+
```
75+
- **Description:** Fetches the full storage program record by its address. Respects ACL — if
76+
`requesterAddress` is provided it is used for access control checks; otherwise only public
77+
programs are readable. Returns `null` (or a 404) if not found.
78+
79+
---
80+
81+
### 2. `getStorageProgramAll`
82+
83+
- **RPC Method Name:** `getStorageProgramAll`
84+
- **SDK Method:** `StorageProgram.getAll(rpcUrl, storageAddress, identity?)`
85+
- **Request Parameters:**
86+
```json
87+
{
88+
"storageAddress": "stor-<40-char-hash>",
89+
"requesterAddress": "<optional demos1... address>"
90+
}
91+
```
92+
- **Expected Response:** `StorageProgramData` (same shape as `getStorageProgram` above)
93+
- **Description:** Alias/variant of `getStorageProgram` that is explicitly intended to return the
94+
full data payload (as opposed to a summary). In practice the SDK treats these as equivalent — the
95+
node may implement both with the same handler or differentiate them (e.g. `getStorageProgram`
96+
could omit `data` while `getStorageProgramAll` always includes it). Either approach is acceptable
97+
as long as both return the full `StorageProgramData` shape.
98+
99+
---
100+
101+
### 3. `getStorageProgramsByOwner`
102+
103+
- **RPC Method Name:** `getStorageProgramsByOwner`
104+
- **SDK Method:** `StorageProgram.getByOwner(rpcUrl, owner, identity?)`
105+
- **Request Parameters:**
106+
```json
107+
{
108+
"owner": "demos1...",
109+
"requesterAddress": "<optional demos1... address>"
110+
}
111+
```
112+
- **Expected Response:** `StorageProgramListItem[]`
113+
```typescript
114+
Array<{
115+
storageAddress: string; // "stor-<hash>"
116+
programName: string;
117+
encoding: "json" | "binary";
118+
sizeBytes: number;
119+
storageLocation: string;
120+
createdAt: string; // ISO timestamp
121+
updatedAt: string; // ISO timestamp
122+
}>
123+
```
124+
- **Description:** Returns a list of all storage programs owned by the given address. The list
125+
items intentionally omit the full `data` payload — callers follow up with `getStorageProgram` or
126+
`getStorageProgramAll` for individual program data. Should return an empty array (not an error)
127+
when the owner has no programs.
128+
129+
---
130+
131+
### 4. `searchStoragePrograms`
132+
133+
- **RPC Method Name:** `searchStoragePrograms`
134+
- **SDK Method:** `StorageProgram.searchByName(rpcUrl, nameQuery, options?)`
135+
- **Request Parameters:**
136+
```json
137+
{
138+
"query": "partialOrFullName",
139+
"options": {
140+
"limit": 10, // optional, number
141+
"offset": 0, // optional, number (for pagination)
142+
"exactMatch": false // optional, boolean
143+
},
144+
"requesterAddress": "<optional demos1... address>"
145+
}
146+
```
147+
- **Expected Response:** `StorageProgramListItem[]` (same shape as `getStorageProgramsByOwner`)
148+
- **Description:** Full-text / partial name search across all storage programs. When `exactMatch`
149+
is `false` (the default), the `query` string should match any program whose `programName`
150+
contains it as a substring (case-insensitive preferred). When `exactMatch` is `true`, only exact
151+
name matches are returned. Supports pagination via `limit` and `offset`. Should return an empty
152+
array (not an error) when there are no matches.
153+
154+
---
155+
156+
### 5. `getStorageProgramFields`
157+
158+
- **RPC Method Name:** `getStorageProgramFields`
159+
- **SDK Method:** `StorageProgram.getFields(rpcUrl, storageAddress, identity?)`
160+
- **Request Parameters:**
161+
```json
162+
{
163+
"storageAddress": "stor-<40-char-hash>",
164+
"requesterAddress": "<optional demos1... address>"
165+
}
166+
```
167+
- **Expected Response:** `StorageProgramFieldsResponse`
168+
```typescript
169+
{
170+
fields: string[]; // top-level key names in the JSON data object
171+
count: number; // fields.length
172+
}
173+
```
174+
- **Description:** Returns the list of top-level field (key) names present in a JSON-encoded
175+
storage program. Only applicable to programs with `encoding: "json"`. Should return `null` (or
176+
an appropriate error) for binary-encoded programs.
177+
178+
---
179+
180+
### 6. `getStorageProgramValue`
181+
182+
- **RPC Method Name:** `getStorageProgramValue`
183+
- **SDK Method:** `StorageProgram.getValue(rpcUrl, storageAddress, field, identity?)`
184+
- **Request Parameters:**
185+
```json
186+
{
187+
"storageAddress": "stor-<40-char-hash>",
188+
"field": "fieldName",
189+
"requesterAddress": "<optional demos1... address>"
190+
}
191+
```
192+
- **Expected Response:** `StorageProgramValueResponse`
193+
```typescript
194+
{
195+
field: string; // echoes the requested field name
196+
value: unknown; // the field's current value
197+
type: "string" | "number" | "boolean" | "array" | "object" | "null" | "undefined";
198+
}
199+
```
200+
- **Description:** Returns the current value of a single named field from a JSON-encoded storage
201+
program, along with its JavaScript type. Only applicable to `encoding: "json"` programs.
202+
203+
---
204+
205+
### 7. `getStorageProgramItem`
206+
207+
- **RPC Method Name:** `getStorageProgramItem`
208+
- **SDK Method:** `StorageProgram.getItem(rpcUrl, storageAddress, field, index, identity?)`
209+
- **Request Parameters:**
210+
```json
211+
{
212+
"storageAddress": "stor-<40-char-hash>",
213+
"field": "arrayFieldName",
214+
"index": 0,
215+
"requesterAddress": "<optional demos1... address>"
216+
}
217+
```
218+
Note: `index` supports negative values (`-1` = last element).
219+
- **Expected Response:** `StorageProgramItemResponse`
220+
```typescript
221+
{
222+
field: string; // echoes the requested field name
223+
index: number; // the resolved (non-negative) index actually accessed
224+
value: unknown; // the item at that index
225+
arrayLength: number; // total length of the array
226+
}
227+
```
228+
- **Description:** Returns a single element from an array-typed field within a JSON-encoded storage
229+
program. Supports negative indexing (e.g. `-1` returns the last element). Should return `null`
230+
(or an error) if the field is not an array or if the index is out of bounds.
231+
232+
---
233+
234+
### 8. `hasStorageProgramField`
235+
236+
- **RPC Method Name:** `hasStorageProgramField`
237+
- **SDK Method:** `StorageProgram.hasField(rpcUrl, storageAddress, field, identity?)`
238+
- **Request Parameters:**
239+
```json
240+
{
241+
"storageAddress": "stor-<40-char-hash>",
242+
"field": "fieldName",
243+
"requesterAddress": "<optional demos1... address>"
244+
}
245+
```
246+
- **Expected Response:** `StorageProgramHasFieldResponse`
247+
```typescript
248+
{
249+
field: string; // echoes the requested field name
250+
exists: boolean;
251+
}
252+
```
253+
- **Description:** Checks whether a named field exists in a JSON-encoded storage program without
254+
returning its value. Useful for cheap existence checks before performing a full `getValue` call.
255+
Only applicable to `encoding: "json"` programs.
256+
257+
---
258+
259+
### 9. `getStorageProgramFieldType`
260+
261+
- **RPC Method Name:** `getStorageProgramFieldType`
262+
- **SDK Method:** `StorageProgram.getFieldType(rpcUrl, storageAddress, field, identity?)`
263+
- **Request Parameters:**
264+
```json
265+
{
266+
"storageAddress": "stor-<40-char-hash>",
267+
"field": "fieldName",
268+
"requesterAddress": "<optional demos1... address>"
269+
}
270+
```
271+
- **Expected Response:** `StorageProgramFieldTypeResponse`
272+
```typescript
273+
{
274+
field: string;
275+
type: "string" | "number" | "boolean" | "array" | "object" | "null" | "undefined";
276+
}
277+
```
278+
- **Description:** Returns only the type of a named field without fetching its value. Primarily
279+
used to guard array operations — callers check `type === "array"` before calling
280+
`getStorageProgramItem`. Only applicable to `encoding: "json"` programs.
281+
282+
---
283+
284+
## Priority Ranking
285+
286+
Implement in this order to unblock SDK consumers as quickly as possible:
287+
288+
| Priority | RPC Method | Reason |
289+
|----------|------------------------------|------------------------------------------------------------------------------------------|
290+
| 1 | `getStorageProgram` | Core read operation — every read flow starts here. Blocks all other usage. |
291+
| 2 | `getStorageProgramAll` | Used by `StorageProgram.getAll()`; many consumers call this instead of `getByAddress`. |
292+
| 3 | `getStorageProgramsByOwner` | Required to enumerate programs owned by an address (dashboard / management UIs). |
293+
| 4 | `getStorageProgramFields` | Needed to discover the schema of a JSON program before reading individual fields. |
294+
| 5 | `getStorageProgramValue` | Needed to read a single field efficiently without fetching the full program. |
295+
| 6 | `searchStoragePrograms` | Needed for name-based discovery; lower urgency than direct-address reads. |
296+
| 7 | `hasStorageProgramField` | Convenience existence check; can be emulated by calling `getStorageProgramValue`. |
297+
| 8 | `getStorageProgramFieldType` | Type guard for array operations; can be emulated via `getStorageProgramValue`. |
298+
| 9 | `getStorageProgramItem` | Array-element access; can be emulated via `getStorageProgramAll` + client-side indexing. |
299+
300+
---
301+
302+
## Current Workaround
303+
304+
Until these endpoints are implemented, state can be reconstructed by calling
305+
`getTransactionHistory` for the relevant address and replaying all `CREATE_STORAGE_PROGRAM`,
306+
`WRITE_STORAGE`, `SET_FIELD`, `SET_ITEM`, `APPEND_ITEM`, `DELETE_FIELD`, and `DELETE_ITEM`
307+
transactions in order. This approach is correct but has significant drawbacks:
308+
309+
- Requires fetching and processing an unbounded number of transactions.
310+
- Does not scale as transaction history grows.
311+
- Cannot efficiently answer field-level or item-level queries.
312+
- Provides no ACL enforcement at the node level.
313+
314+
This workaround is not suitable for production use and should be treated as a temporary measure only.

‎PROJECT_INDEX.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -99,7 +99,7 @@
9999
{ "file": "tsconfig.json", "purpose": "TypeScript compiler settings" },
100100
{ "file": "ormconfig.json", "purpose": "TypeORM/database config" },
101101
{ "file": "knip.json", "purpose": "unused code/dependency analysis" },
102-
{ "file": "devnet/docker-compose.yml", "purpose": "local multi-node network" },
102+
{ "file": "testing/devnet/docker-compose.yml", "purpose": "local multi-node network" },
103103
{ "file": "tlsnotary/docker-compose.yml", "purpose": "TLSNotary stack" },
104104
{ "file": "monitoring/prometheus/prometheus.yml", "purpose": "metrics scraping" },
105105
{ "file": "monitoring/grafana/provisioning/dashboards/dashboard.yml", "purpose": "Grafana dashboard provisioning" }
@@ -109,7 +109,7 @@
109109
{ "file": "INSTALL.md", "topic": "installation guide" },
110110
{ "file": "CONTRIBUTING.md", "topic": "contribution workflow" },
111111
{ "file": "monitoring/README.md", "topic": "observability stack" },
112-
{ "file": "devnet/README.md", "topic": "local devnet" },
112+
{ "file": "testing/devnet/README.md", "topic": "local devnet" },
113113
{ "file": "OMNIPROTOCOL_SETUP.md", "topic": "omni protocol setup" },
114114
{ "file": "L2PS_TESTING.md", "topic": "L2PS testing" }
115115
],

‎PROJECT_INDEX.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ Generated: 2026-02-18T13:55:45Z
88
- `tests/` - OmniProtocol and fixture-driven tests
99
- `scripts/` - Operational scripts (ZK setup, load tests, wallet/proof helpers)
1010
- `docs/`, `documentation/`, `monitoring/` - User and operator docs
11-
- `devnet/` - Local 4-node Docker dev network
11+
- `testing/devnet/` - Local 4-node Docker dev network
1212
- `tlsnotary/`, `postgres/`, `monitoring/` - Infra/service compose stacks
1313

1414
## 🚀 Entry Points
@@ -64,7 +64,7 @@ Generated: 2026-02-18T13:55:45Z
6464
- `knip.json` - unused export/dependency analysis
6565
- `monitoring/prometheus/prometheus.yml` - Prometheus scrape config
6666
- `monitoring/grafana/provisioning/**/*.yml` - Grafana dashboard/datasource provisioning
67-
- `devnet/docker-compose.yml` - local multi-node devnet orchestration
67+
- `testing/devnet/docker-compose.yml` - local multi-node devnet orchestration
6868
- `tlsnotary/docker-compose.yml` - TLSNotary service stack
6969

7070
## 📚 Documentation
@@ -74,7 +74,7 @@ Generated: 2026-02-18T13:55:45Z
7474
- `CONTRIBUTING.md` - contribution workflow
7575
- `L2PS_TESTING.md` - L2PS test guidance
7676
- `monitoring/README.md` - metrics/Grafana setup
77-
- `devnet/README.md` - local devnet lifecycle
77+
- `testing/devnet/README.md` - local devnet lifecycle
7878
- `OMNIPROTOCOL_SETUP.md` - OmniProtocol setup
7979

8080
## 🧪 Test Coverage

0 commit comments

Comments
 (0)