Approaches for incremental data loaders

[ltostes]

Hello Observable Framework community!

I'm building a Framework dashboard page for a CS2 team I have with friends, and I would like to get your help on approaches for implementing incremental data loaders. My main goal in this scenario is minimizing the number of external API requests, and I'm thinking data loaders with an incremental approach would be most suitable since I can leave behind requests about data we already have (older matches details). But I'm not sure how to implement it and I haven't found anything related to that in the docs/discussions.

I don't have concrete solutions, but draft ideas on how to do it: My first thought would be to leave the data as a static file and update it every time before deploy (as a kind of separate build process for this data specifically). A second approach I thought would be to have the data loader effectively providing part of the data (the profile stats), but then indirectly changing another static file with the matches information (similar to the previous one, but would't require running scripts outside the build process). Although I have the feeling neither of these are good uses of Framework nor file versioning.

What are your thoughts or approaches on how to build incremental data loaders in Framework?

It might be an overkill for my specific scenario, but it does sound like something useful to learn for future endeavor so I deemed worth wile to ask around here.

I immensely appreciate the help!!
Also I'm a huge fan of nearly everything you build. Thank you!! :)

Appendix: some context about my case and personal thoughts

The basic data retrieval process

The data comes from the Leetify's app public API.
The process is as follows:

• A static list of player IDs (around 10 team members) leads to 1 API request per ID, each of which will return player information/stats as well as a list of all the matches this player has played (around 500).
• Based on this matches list, I collect all the unique match IDs across all players make a new API request for each to retrieve detailed information about each match (around 600 unique matches).

Here's a simple sketch highlighting it.

Current thoughts and options

1. Retrieve at runtime: Having it this way has the advantage of data always being 100% fresh, but it sounds troublesome to make 600+ requests each time the page is loaded.
2. Weekly/Daily data loader for all data: We don't get a lot of extra value to have data fresher than daily/weekly, so it feels like an awesome win for Framework's data loaders approach. Although, at build time, we would still make a decent amount of API requests (610+ and ever increasing) every time we run the data loader. (Not sure for how long Leetify would let this fly by, but maybe I'm just paranoid)
3. Incremental Weekly/Daily data loader: A nicer approach would be to only collect the data that has effectively changed since the last run (updated profile stats and new matches details), leaving behind the ~580 historial matches details requests since we already have this information.

Implementation approaches

Option 1 seems simple enough (I built a draft in a notebook, although I have to study more about CORS to have it running smoothly inside Framework). Option 2 seems straightforward with zip archives or a DuckDB file. Option 3 is about the discussion above.

[Fil]

Here's an incremental data loader, which uses Framework's own cache if present and appends a line to it.

❯ cat src/incremental.csv.js
import { readFileSync } from "node:fs";

try {
  const c = readFileSync("src/.observablehq/cache/incremental.csv", "utf-8");
  process.stdout.write(c);
} catch (err) {
  process.stdout.write("a\n");
}
process.stdout.write(`${Math.random()}\n`);

Of course you an do something more interesting when you read it—here I'm just piping it straight back to framework with the additional line.

To run it again and generate a newer version, I only need to run touch src/incremental.csv.js.

(Note that I'm using the new source root location src—you might need to replace that with docs.)

[ltostes]

Thanks a lot, Fil! That's exactly what I was looking for :)

I'll implement it on my side based on yours and share here if any interesting extra comes out.