Add pLimit when loading file explorer in refresh placeholders#1379
Conversation
|
| export async function loadInMemoryPaths({ ctx }: { ctx: SyncContext }) { | ||
| const files: FileExplorerFiles = new Map(); | ||
| const folders: FileExplorerFolders = new Map(); | ||
| const limit = pLimit(20); |
There was a problem hiding this comment.
I have seen that you already defined a concurrency limit of 20 on refresh-items-placeholders do you see a benefit in centralizing this value?
There was a problem hiding this comment.
We could, but technically the values can change individually. I've set 20 because it was the recommendation of claude. I've tried changing both but in general terms 20 was the one that was working the best. However, it doesn't mean that is has to be the same for both. Each step has it's own concurrency, maybe in the future one should be 20 and the other 15. Basically they don't change because of the same reason that's why the should be 2 different values (at least with the knowledge I have).
2 participants
What
After trying a lot of things to reduce RAM usage (explained in this ticket https://inxt.atlassian.net/browse/PB-6240), we have been able to reduce the RAM usage by around 290MB for a test case of 1000 folders x 100 files (100k files). Previously it was ~835MB, now it's ~545MB.
We have done that by limiting the number of concurrent promises that can run when refreshing placeholders using
pLimitand also by only storing in the Map the required properties.I've also implemented a fix to not block the main thread when
createOrUpdateBatchfor files and folders.