R2D2 scrubber

[Dooruk]

R2D2 has a lifetime for indexed items with different purposes:

https://github.com/JCSDA-internal/r2d2?tab=readme-ov-file#delete_experiment_by_name

JCSDA runs a weekly cronjob to scrub items past their lifetime. This is not urgent but we should also start looking into this.

@mer-a-o pointed me to this:
https://www.nccs.nasa.gov/nccs-users/instructional/using-discover/miscellaneous/using-cron

Might need @mathomp4 or @jardizzo insight here.

[Dooruk]

I used this before GPT was cool:

https://crontab.guru/

[ftgoktas]

Where should this cron job be set up on Discover?

R2D2Index is not available in the r2d2-client package (it's server-side only), the client side uses the deregister() method, which makes requests to the server's delete_experiment_by_name() endpoint and handles both metadata and physical file deletion.

I've created a draft scrubber script here, it uses r2d2.deregister() to delete experiments that have exceeded their lifetime. It filters experiments by specified users, checks each experiment's age against its lifetime (debug=14 days, science=180 days, publication=5 years, release=indefinite), and calls deregister() for expired experiments.

Also added a dry-run mode (default) and it can be run with --scrub to actually delete (for safer testing).

[mer-a-o]

@ftgoktas, r2d2.deregister calculates the age and based on the lifetime decides to delete the experiment or not. If you want to delete and experiment eariler you need to set ignore_lifetime=True. You can use the scrubber here as a reference: https://github.com/JCSDA-internal/r2d2-client/blob/develop/scrubber/scrub_experiments.py

[ftgoktas]

Looks like scrub_experiments.py script searches for and deletes experiments with lifetime='debug'. Do we intentionally only want to scrub debug experiments, or should we also be cleaning up science and publication experiments once they exceed their retention?

[Dooruk]

I'm thinking it should be a weekly running cron job under the gmao_ci account. I don't remember if you have access to that account @ftgoktas? I think we can just replicate and adopt the existing JCSDA script, including the wrappers but I leave that decision to the SI-team and @ashiklom.

Have we set any r2d2 admins yet for our database?
https://github.com/JCSDA-internal/r2d2-client/blob/505d01925988912d5c4b0ce8a2cfe72730b520c9/scrubber/scrub_experiments.py#L15-L21

For testing now debug is sufficient. I foresee we would have to define our own lifetime definitions at some point and who can utilize those etc.