feat: add numeric collation for balance sorting and support sort parameter in RPC calls by TheCrazyGM · Pull Request #129 · hive-engine/hivesmartcontracts

TheCrazyGM · 2025-09-21T13:18:33Z

Add ability to sort by balance / stake / etc.

API Call

curl -X POST http://localhost:5000/contracts -H "Content-Type: application/json" -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "find",
    "params": {
      "contract": "tokens",
      "table": "balances",
      "query": {"symbol": "PIZZA"},
      "sort": {"field": "balance", "order": "desc"},
      "limit": 10
    }
  }'

Output as expected:

{
    "jsonrpc": "2.0",
    "id": 1,
    "result": [
        {
            "_id": 241391,
            "account": "null",
            "symbol": "PIZZA",
            "balance": "1232816.82",
            "stake": "3.15",
            "pendingUnstake": "0",
            "delegationsIn": "0",
            "delegationsOut": "0",
            "pendingUndelegations": "0"
        },
        {
            "_id": 1739899,
            "account": "moon.deposit",
            "symbol": "PIZZA",
            "balance": "115243.02",
            "stake": "0",
            "pendingUnstake": "0",
            "delegationsIn": "0",
            "delegationsOut": "0",
            "pendingUndelegations": "0"
        },
        {
            "_id": 525742,
            "account": "tipcc",
            "symbol": "PIZZA",
            "balance": "50073.47",
            "stake": "0",
            "pendingUnstake": "0",
            "delegationsIn": "0",
            "delegationsOut": "0",
            "pendingUndelegations": "0"
        },
        {
            "_id": 208477,
            "account": "ecoinstant",
            "symbol": "PIZZA",
            "balance": "18308.18",
            "stake": "102013.11",
            "pendingUnstake": "0.00",
            "delegationsIn": "0",
            "delegationsOut": "0",
            "pendingUndelegations": "0"
        },
        {
            "_id": 186743,
            "account": "huzzah",
            "symbol": "PIZZA",
            "balance": "17552.33",
            "stake": "18659.28",
            "pendingUnstake": "0",
            "delegationsIn": "0",
            "delegationsOut": "0",
            "pendingUndelegations": "0"
        },
        {
            "_id": 1775991,
            "account": "moon.deposit2",
            "symbol": "PIZZA",
            "balance": "16042.86",
            "stake": "0",
            "pendingUnstake": "0",
            "delegationsIn": "0",
            "delegationsOut": "0",
            "pendingUndelegations": "0"
        },
        {
            "_id": 186723,
            "account": "thebeardflex",
            "symbol": "PIZZA",
            "balance": "15090.89",
            "stake": "59966.62",
            "pendingUnstake": "0",
            "delegationsIn": "0",
            "delegationsOut": "0",
            "pendingUndelegations": "0"
        },
        {
            "_id": 208474,
            "account": "a1-shroom-spores",
            "symbol": "PIZZA",
            "balance": "10497.16",
            "stake": "23.33",
            "pendingUnstake": "0.00",
            "delegationsIn": "0",
            "delegationsOut": "0",
            "pendingUndelegations": "0"
        },
        {
            "_id": 1301276,
            "account": "muterra-cards",
            "symbol": "PIZZA",
            "balance": "10080.00",
            "stake": "0",
            "pendingUnstake": "0",
            "delegationsIn": "0",
            "delegationsOut": "0",
            "pendingUndelegations": "0"
        },
        {
            "_id": 634678,
            "account": "wrestorgonline",
            "symbol": "PIZZA",
            "balance": "9981.34",
            "stake": "53807.70",
            "pendingUnstake": "0.00",
            "delegationsIn": "0",
            "delegationsOut": "0",
            "pendingUndelegations": "0"
        }
    ]
}

No index needed, using:

db.collection.find(query)
  .sort({ balance: -1, _id: 1 })                          // Still sorts by string, but...
  .collation({ locale: "en_US", numericOrdering: true })  // Treats as numbers

…meter in RPC calls

…strings correctly

TheCrazyGM · 2025-09-21T13:32:44Z

libs/Database.js

            }
+
+            // Add numeric collation for balance/stake sorting (both are numeric strings)
+            const collation = ind.some(idx => idx.index === 'balance' || idx.index === 'stake')


I believe these are probably the only fields that would be popular, but any of the others could be added as needed.

I think that if you don't have an index with collation, this likely will stream everything and sort it, right? I suppose either way, if it is problematic, an index can be added directly to the table as a setup.

Yes, without index it does pull the whole thing into memory to sort. I had included a script to add the indexes, but i decided against it, was trying to avoid touching the database directly.

yeah, makes sense. But don't worry, index setup is not something that impacts hashes, and can make things better (or worse if you get too carried away with indexes i suppose, would impact writes i think)

der1sebi · 2025-09-21T17:52:36Z

That could be quite helpful. Good job Michael.

eonwarped · 2025-09-25T14:24:29Z

libs/Database.js

            }
+
+            // Add numeric collation for balance/stake sorting (both are numeric strings)
+            const collation = ind.some(idx => idx.index === 'balance' || idx.index === 'stake')


I think that if you don't have an index with collation, this likely will stream everything and sort it, right? I suppose either way, if it is problematic, an index can be added directly to the table as a setup.

eonwarped · 2025-09-25T14:25:10Z

libs/Database.js

-              // and should be rare in production. Otherwise, contract code is asking for an index that does
-              // not exist.
-              log.info(`Index ${JSON.stringify(ind)} not available for ${finalTableName}`); // eslint-disable-line no-console
+              // Build sort array even when indexes don't exist


we should be careful with this, because without an index on large collections it can be problematic to allow

I have it running right now, can you think of a problematic query and I'll test. But I haven't noticed any impact, e.g the pizza token has 18K+ records and no delay.

You probably want a table that has a lot more entries, like nft.

TheCrazyGM · 2025-09-25T16:13:55Z

Test

time curl -s -X POST http://localhost:5000/contracts -H "Content-Type: application/json" --data '{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "find",
  "params": {
    "contract": "nft",
    "table": "STARinstances",
    "query": {"ownedBy":"u"},
    "sort": {"field": "account", "order": "desc"},
    "limit": 10
  }
}'

Response

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": [
    {
      "_id": 1128384,
      "account": "wilson112000",
      "ownedBy": "u",
      "lockedTokens": {},
      "properties": {
        "type": "EnergyBoost1",
        "class": "energyboost",
        "stats": "0,0,0,0"
      },
      "previousAccount": "ivespino",
      "previousOwnedBy": "u"
    },
    {
      "_id": 2099507,
      "account": "santiagolecuona",
      "ownedBy": "u",
      "lockedTokens": {},
      "properties": {
        "type": "EnergyBoost1",
        "class": "energyboost",
        "stats": "0,0,0,0"
      },
      "previousAccount": "ivespino",
      "previousOwnedBy": "u"
    },
    {
      "_id": 2381250,
      "account": "ruviksan",
      "ownedBy": "u",
      "lockedTokens": {},
      "properties": {
        "type": "R221 PT 100 Bass Head",
        "class": "instrument",
        "stats": "0,50,0,0"
      },
      "previousAccount": "ivespino",
      "previousOwnedBy": "u"
    },
    {
      "_id": 2743030,
      "account": "n1t0",
      "ownedBy": "u",
      "lockedTokens": {},
      "properties": {
        "type": "R237 Ivespino",
        "class": "people",
        "stats": "125,3,75,2"
      },
      "previousAccount": "ivespino",
      "previousOwnedBy": "u"
    },
    {
      "_id": 2463713,
      "account": "marcos1977",
      "ownedBy": "u",
      "lockedTokens": {},
      "properties": {
        "type": "R223 Mint Kit",
        "class": "instrument",
        "stats": "0,50,0,0"
      },
      "previousAccount": "ivespino",
      "previousOwnedBy": "u"
    },
    {
      "_id": 981739,
      "account": "luke-wtp",
      "ownedBy": "u",
      "lockedTokens": {},
      "properties": {
        "type": "EnergyBoost1",
        "class": "energyboost",
        "stats": "0,0,0,0"
      },
      "previousAccount": "stewball",
      "previousOwnedBy": "u"
    },
    {
      "_id": 986482,
      "account": "luke-wtp",
      "ownedBy": "u",
      "lockedTokens": {},
      "properties": {
        "type": "66 Stewie",
        "class": "people",
        "stats": "5,0,10,0"
      },
      "previousAccount": "stewball",
      "previousOwnedBy": "u"
    },
    {
      "_id": 3336889,
      "account": "koko556",
      "soulBound": false,
      "ownedBy": "u",
      "lockedTokens": {},
      "properties": {
        "type": "S44 Old Blud",
        "class": "people",
        "stats": "100,1,100,1"
      },
      "previousAccount": "robkingdom",
      "previousOwnedBy": "u"
    },
    {
      "_id": 178322,
      "account": "juancarlosqp",
      "ownedBy": "u",
      "lockedTokens": {},
      "properties": {
        "type": "i6 Mid Range Acoustic",
        "class": "instrument",
        "stats": "0,10,0,0"
      },
      "previousAccount": "ivespino",
      "previousOwnedBy": "u"
    },
    {
      "_id": 1585024,
      "account": "juancarlosqp",
      "ownedBy": "u",
      "lockedTokens": {},
      "properties": {
        "type": "R189 Roman",
        "class": "people",
        "stats": "125,3,75,2"
      },
      "previousAccount": "ivespino",
      "previousOwnedBy": "u"
    }
  ]
}

Results

curl -s -X POST http://localhost:5000/contracts -H --data 0.01s user 0.01s system 0% cpu 27.538 total

It took almost 30 seconds and put my mongo instance at 100% CPU. (one core)

Suggestions

Put limits on what tables you can query with sort params? Or add an index to balance and stake and put the check back allowing only those two to be sorted?

…erver

TheCrazyGM · 2025-10-04T11:03:39Z

I think it's worth noting that the same api call on the nft table, without sort:

time curl -s -X POST http://localhost:5000/contracts -H "Content-Type: application/json" --data '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "find",
    "params": {
        "contract": "nft",
        "table": "STARinstances",
        "query": {"ownedBy":"u"},
        "limit": 10
    }
}'

came back with a very similar response time, so I don't think it added much more overhead than I thought.

curl -s -X POST http://localhost:5000/contracts -H --data 0.02s user 0.00s system 0% cpu 28.162 total

But, I've also added a catch in there to only allow sort on fields that are a) indexed or b) specifically balance or stake in the tokens contract.

I don't know if this is the direction you want to take it.

eonwarped · 2025-10-06T14:41:52Z

I think it's worth noting that the same api call on the nft table, without sort:
time curl -s -X POST http://localhost:5000/contracts -H "Content-Type: application/json" --data '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "find",
    "params": {
        "contract": "nft",
        "table": "STARinstances",
        "query": {"ownedBy":"u"},
        "limit": 10
    }
}'
came back with a very similar response time, so I don't think it added much more overhead than I thought.

curl -s -X POST http://localhost:5000/contracts -H --data 0.02s user 0.00s system 0% cpu 28.162 total

But, I've also added a catch in there to only allow sort on fields that are a) indexed or b) specifically balance or stake in the tokens contract.

I don't know if this is the direction you want to take it.

I think very likely what we need is just a way to enable the feature and allow node operators to choose.

Also, the reason why the query is slow is probably the index is ['account', 'ownedBy'] and likely you'd want just 'ownedBy' directly as an index. To improve the queries alongside this change, we should set up the indexes on the fly, but again, leave that to node operator.

TheCrazyGM added 2 commits September 21, 2025 09:03

feat: add numeric collation for balance sorting and support sort para…

4c0c32a

…meter in RPC calls

fix: add numeric collation for stake field sorting to handle numeric …

f25cd6c

…strings correctly

TheCrazyGM commented Sep 21, 2025

View reviewed changes

eonwarped reviewed Sep 25, 2025

View reviewed changes

feat: validate sort fields against allowed indexed fields in JsonRPCS…

93ed924

…erver

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add numeric collation for balance sorting and support sort parameter in RPC calls#129

feat: add numeric collation for balance sorting and support sort parameter in RPC calls#129
TheCrazyGM wants to merge 3 commits intohive-engine:qafrom
TheCrazyGM:balance_sort

TheCrazyGM commented Sep 21, 2025

Uh oh!

TheCrazyGM Sep 21, 2025

Uh oh!

eonwarped Sep 25, 2025

Uh oh!

TheCrazyGM Sep 25, 2025

Uh oh!

eonwarped Sep 25, 2025

Uh oh!

der1sebi commented Sep 21, 2025

Uh oh!

eonwarped Sep 25, 2025

Uh oh!

eonwarped Sep 25, 2025

Uh oh!

TheCrazyGM Sep 25, 2025

Uh oh!

eonwarped Sep 25, 2025

Uh oh!

TheCrazyGM commented Sep 25, 2025 •

edited

Loading

Uh oh!

TheCrazyGM commented Oct 4, 2025

Uh oh!

eonwarped commented Oct 6, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

TheCrazyGM commented Sep 21, 2025

Add ability to sort by balance / stake / etc.

API Call

Output as expected:

No index needed, using:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

der1sebi commented Sep 21, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TheCrazyGM commented Sep 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test

Response

Results

Suggestions

Uh oh!

TheCrazyGM commented Oct 4, 2025

Uh oh!

eonwarped commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

TheCrazyGM commented Sep 25, 2025 •

edited

Loading

eonwarped commented Oct 6, 2025 •

edited

Loading