-
Notifications
You must be signed in to change notification settings - Fork 52
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support Table History Policy in Snapshot Expiration Job #274
Support Table History Policy in Snapshot Expiration Job #274
Conversation
5da42cc
to
1f11e11
Compare
apps/spark/src/main/java/com/linkedin/openhouse/jobs/spark/Operations.java
Show resolved
Hide resolved
apps/spark/src/test/java/com/linkedin/openhouse/jobs/spark/OperationsTest.java
Outdated
Show resolved
Hide resolved
…d number of snapshots
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One small change request.
apps/spark/src/main/java/com/linkedin/openhouse/jobs/spark/SnapshotsExpirationSparkApp.java
Outdated
Show resolved
Hide resolved
apps/spark/src/main/java/com/linkedin/openhouse/jobs/spark/Operations.java
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks for adding the test cases for both configuration provided
apps/spark/src/main/java/com/linkedin/openhouse/jobs/spark/SnapshotsExpirationSparkApp.java
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @Will-Lo.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Summary
[Issue] Briefly discuss the summary of the changes made in this
pull request in 2-3 lines.
Following up from #262 and #259
this PR adds support for snapshot expiration table maintenance job to use the history policies defined.
Most notably snapshot expiration will follow the settings of
maxAge
,granularity
, andversions
as follows:Note: If versions are defined and there are less than N versions in the history, then there were not enough commits (within that timespan if defined). Snapshot expiration will always keep at least 1 version.
The default behavior of snapshot expiration job will remain the same, keep snapshots within the last 3 days.
Changes
For all the boxes checked, please include additional details of the changes made in this pull request.
Testing Done
For all the boxes checked, include a detailed description of the testing done for the changes made in this pull request.
Additional Information
For all the boxes checked, include additional details of the changes made in this pull request.