Open
Conversation
Adding a logging & alert system when data latency is greater than 16 hours (12 hours expected).
action.yaml, cribbed from alerting .yaml
Quickstart on job
query the api and check if most recent data is more than a certain number of hours stale compared to now. If it is, send an alert with a statement about the time difference.
12 hour latency expectation
getting rid of MAAP specific stuff
piping results of latency_logging.py to $GITHUB_OUTPUT for display in workflow
trying to capture output from stdout
adding action id and output
Adding alert parameter
add alert boolean
first terrible draft of workflow
Runs hourly and will alert if the data latency is longer than the # of hours as defined in latency_logging.py
Contributor
Author
|
This is partially in response to #244 -- but notably this will only "log" by issuing lots of alerts when we have data older than 12 hours. |
Contributor
Author
|
@zebbecker could you glance at this and tell me if you think it will break our existing alerts system? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
It's been really hard to stay on top of our data latency issues. This is in part because the data the users see can be delayed for a variety of reasons, but we only have alerts for run failures. This is a tack on to the existing alerting system that will let us know if the data the users are seeing is going stale faster than we want it to.