Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Table History Policy in Snapshot Expiration Job #274

Merged

Conversation

Will-Lo
Copy link
Collaborator

@Will-Lo Will-Lo commented Dec 24, 2024

Summary

[Issue] Briefly discuss the summary of the changes made in this
pull request in 2-3 lines.

Following up from #262 and #259
this PR adds support for snapshot expiration table maintenance job to use the history policies defined.

Most notably snapshot expiration will follow the settings of maxAge, granularity, and versions as follows:

  1. If maxAge is provided, it will expire snapshots older than maxAge in granularity timeunit.
  2. If versions is provided, it will retain the last versions snapshots regardless of their age.
  3. If both are provided, it will prioritize maxAge; only retain up to versions number of snapshots younger than the maxAge. This is done by pruning the snapshots older than maxAge, and then running a second expiration to keeping N versions after that.

Note: If versions are defined and there are less than N versions in the history, then there were not enough commits (within that timespan if defined). Snapshot expiration will always keep at least 1 version.

The default behavior of snapshot expiration job will remain the same, keep snapshots within the last 3 days.

Changes

  • Client-facing API Changes
  • Internal API Changes
  • Bug Fixes
  • New Features
  • Performance Improvements
  • Code Style
  • Refactoring
  • Documentation
  • Tests

For all the boxes checked, please include additional details of the changes made in this pull request.

Testing Done

  • Manually Tested on local docker setup. Please include commands ran, and their output.
  • Added new tests for the changes made.
  • Updated existing tests to reflect the changes made.
  • No tests added or updated. Please explain why. If unsure, please feel free to ask for help.
  • Some other form of testing like staging or soak time in production. Please explain.

For all the boxes checked, include a detailed description of the testing done for the changes made in this pull request.

Additional Information

  • Breaking Changes
  • Deprecations
  • Large PR broken into smaller PRs, and PR plan linked in the description.

For all the boxes checked, include additional details of the changes made in this pull request.

@Will-Lo Will-Lo force-pushed the modify-snapshot-expiration-job-configurations branch from 5da42cc to 1f11e11 Compare January 2, 2025 17:23
Copy link
Collaborator

@jiang95-dev jiang95-dev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One small change request.

Copy link
Collaborator

@autumnust autumnust left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for adding the test cases for both configuration provided

Copy link
Member

@abhisheknath2011 abhisheknath2011 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Will-Lo.

Copy link
Collaborator

@jiang95-dev jiang95-dev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Will-Lo Will-Lo merged commit 3e8d387 into linkedin:main Jan 6, 2025
1 check passed
@Will-Lo Will-Lo deleted the modify-snapshot-expiration-job-configurations branch January 6, 2025 22:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants