Skip to content
Open
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,12 @@ looping (though with transactions there is less risk of data corruption). Anothe
that you may need to clear node processes after a stop if the process does not terminate on its own.
Otherwise it will interfere with logging.

#### Database Sorting (No Index Required)

Balance sorting now uses MongoDB's built-in numeric collation (`numericOrdering: true`), which treats balance strings as numbers without requiring database indexes.

**No additional setup needed** - sorting works out of the box with proper numeric ordering.

Also, with isolated-vm on node 20 and later, you will need to pass in --no-node-snapshot to node:

E.g.
Expand Down
16 changes: 12 additions & 4 deletions libs/Database.js
Original file line number Diff line number Diff line change
Expand Up @@ -684,20 +684,28 @@ class Database {
}
});
} else {
// This can happen when creating a table and using find with index all in the same transaction
// and should be rare in production. Otherwise, contract code is asking for an index that does
// not exist.
log.info(`Index ${JSON.stringify(ind)} not available for ${finalTableName}`); // eslint-disable-line no-console
// Build sort array even when indexes don't exist
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should be careful with this, because without an index on large collections it can be problematic to allow

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have it running right now, can you think of a problematic query and I'll test. But I haven't noticed any impact, e.g the pizza token has 18K+ records and no delay.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You probably want a table that has a lot more entries, like nft.

ind.forEach((el) => {
sort.push([el.index === '$loki' ? '_id' : el.index, el.descending === true ? 'desc' : 'asc']);
});
log.info(`Index ${JSON.stringify(ind)} not available for ${finalTableName}, using collation for numeric sorting`); // eslint-disable-line no-console
}
if (sort.findIndex(el => el[0] === '_id') < 0) {
sort.push(['_id', 'asc']);
}

// Add numeric collation for balance/stake sorting (both are numeric strings)
const collation = ind.some(idx => idx.index === 'balance' || idx.index === 'stake')
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe these are probably the only fields that would be popular, but any of the others could be added as needed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that if you don't have an index with collation, this likely will stream everything and sort it, right? I suppose either way, if it is problematic, an index can be added directly to the table as a setup.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, without index it does pull the whole thing into memory to sort. I had included a script to add the indexes, but i decided against it, was trying to avoid touching the database directly.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, makes sense. But don't worry, index setup is not something that impacts hashes, and can make things better (or worse if you get too carried away with indexes i suppose, would impact writes i think)

? { locale: "en_US", numericOrdering: true }
: undefined;

result = await tableData.find(EJSON.deserialize(query), {
limit: lim,
skip: off,
sort,
session: this.session,
projection: prj,
...(collation && { collation }),
}).toArray();

result = EJSON.serialize(result);
Expand Down
Loading