Initial draft for timestamping DLO generation and compaction. #182
Conversation
|
As discussed, the timestamp for the DLO strategy is added as part of the strategy in the table properties. For the compaction app, I've added a separate property ('write.data-layout.compaction') and used the same mechanism to alter the table properties. Let me know what you think. |
| @@ -58,6 +65,23 @@ protected void runInner(Operations ops) { | |||
| fileGroupRewriteResult.rewrittenDataFilesCount(), | |||
| fileGroupRewriteResult.rewrittenBytesCount()); | |||
| } | |||
| // Add compaction timestamp to table properties. | |||
| CompactionEvent event = | |||
There was a problem hiding this comment.
@teamurko can you confirm if this will be run after all partitions are compacted in Table or after each partition is compacted in a Table?
There was a problem hiding this comment.
it will run after compaction is done on the whole table
There was a problem hiding this comment.
Thank you for the quick PR, sorry for delayed response @anjagruenheid
Let's wait until we iterate on the e2e design, I hope that we can land on something quickly to clarity how we support these timestamps
The idea for these timestamp is to tell generator when to regenerate new strategy and to tell scheduler whether the strategy has not yet been applied, based on this idea I thought that the apply timestamp should be kept in the same entity as the create timestamp, but I'm not sure how it would work when we have multiple strategies per table, e.g. partition (range) scope
.../datalayout/src/main/java/com/linkedin/openhouse/datalayout/strategy/DataLayoutStrategy.java
Show resolved
Hide resolved
| @@ -58,6 +65,23 @@ protected void runInner(Operations ops) { | |||
| fileGroupRewriteResult.rewrittenDataFilesCount(), | |||
| fileGroupRewriteResult.rewrittenBytesCount()); | |||
| } | |||
| // Add compaction timestamp to table properties. | |||
| CompactionEvent event = | |||
There was a problem hiding this comment.
it will run after compaction is done on the whole table
| Gson gson = new GsonBuilder().create(); | ||
| Type type = new TypeToken<CompactionEvent>() {}.getType(); | ||
| ops.spark() | ||
| .sql( | ||
| String.format( | ||
| "ALTER TABLE %s SET TBLPROPERTIES ('%s' = '%s')", | ||
| ops.getTable(fqtn), | ||
| DATA_LAYOUT_COMPACTION_PROPERTY_KEY, | ||
| StringEscapeUtils.escapeJava(gson.toJson(event, type)))); |
There was a problem hiding this comment.
There was a problem hiding this comment.
Hmm, but it's not related to the strategy, right? It's a compaction event. Should I push this functionality into a separate util?
Summary
Issue] Briefly discuss the summary of the changes made in this
pull request in 2-3 lines.
Changes
For all the boxes checked, please include additional details of the changes made in this pull request.
Testing Done
For all the boxes checked, include a detailed description of the testing done for the changes made in this pull request.
Additional Information
For all the boxes checked, include additional details of the changes made in this pull request.