Add stats-q-error to stage level metrics#560
Open
gabotechs wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
stats_q_errormetric at the stage level that computes an error rate between the estimated stats VS what actually happened, taken from https://www.vldb.org/pvldb/vol2/vldb09-657.pdf.I find the current approach to be a bit messy...
It requires a lot of code for wiring this up. There's something special about this metric: is a metric attached to the
Stagerather than a specific node, and it's collected after runtime finished.Until now, all the metrics we stamp at the
Stagelevel are known during planning (either static or dynamic). However, this time, we need to wait for the full execution to finish.I'm very much open to other alternatives, as what I have here, is just too much code for just showing a metric.
@jayshrivastava any ideas on how to make this cleaner?