Skip to content

feat: add cache to terrakit data fetch#48

Open
Beldine-Moturi wants to merge 10 commits intomainfrom
cache-inference-images
Open

feat: add cache to terrakit data fetch#48
Beldine-Moturi wants to merge 10 commits intomainfrom
cache-inference-images

Conversation

@Beldine-Moturi
Copy link
Copy Markdown
Contributor

@Beldine-Moturi Beldine-Moturi commented Apr 21, 2026

Summary

Related Issue (optional)

How to test this PR?

Screenshots / Logs (optional)

Checklist

  • This PR targets the main branch
  • I have added or updated relevant docs.
  • I have not included any secrets or credentials.
  • Linting and formatting checks pass.

@Beldine-Moturi Beldine-Moturi force-pushed the cache-inference-images branch from 7c172b7 to 9feba28 Compare April 22, 2026 09:40
@EugeneGene EugeneGene self-requested a review April 27, 2026 08:03
EugeneGene
EugeneGene previously approved these changes Apr 27, 2026
Copy link
Copy Markdown
Contributor

@fMurugi fMurugi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Can we have atomic writes.

In the case we have N pods writing or reading tp avoid partial files read:
Time →
[ create file ][ write... ][ write... ][ write... ]

another pod reads here → 💥 corrupted/partial file

 **Sol:**
     Time →
  [ write tmp file completely ]
  [ atomically rename ]

Other pods:

  • before rename → file doesn't exist
  • after rename → file is 100% complete
  1. The indexing should be done for each day ,so when we fetch data for 18-20 .we cache data for each day ,so as for the next request we have if dates expand or reduce, we can just fetch for the additional dates not all of the other dates.eg 18-21 will fetch only the missing date.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants