feat(metrics): add counter for retry create stream reader #20258
Conversation
| let init_stream_reader_retry_count = register_guarded_int_counter_vec_with_registry!( | ||
| "source_init_stream_reader_retry_count", | ||
| "Total number of retries to initialize stream reader", | ||
| &["actor_id", "source_name",], | ||
| registry | ||
| ) | ||
| .unwrap(); |
There was a problem hiding this comment.
should we add this to grafana dashboard?
There was a problem hiding this comment.
I think it overlaps with source_status_is_up (from source enumerator), so I don't add it to grafana.
But here we focus more on the retry on the compute node, planning add it to cloud alert.
There was a problem hiding this comment.
Oh, I don't know we have source_status_is_up!
Then what about adding it to the same panel (Source Upstream Status), so that we can both know whether the source is up on meta and each CN. 🤔
| let source_ctx_inner = source_ctx_clone.clone(); | ||
| source_ctx_inner.metrics.init_stream_reader_retry_count.with_guarded_label_values(&[ | ||
| &source_ctx_inner.actor_id.to_string(), | ||
| &source_ctx_inner.source_name, | ||
| ]).inc(); |
There was a problem hiding this comment.
Could we simply move this into source_reader.build_stream? And perhaps rename to stream_reader_build_count.
There was a problem hiding this comment.
Actually, I feel the figure is a little hard to understand. Why did it go up and down 🤔
There was a problem hiding this comment.
I am struggling with it too, suspect somewhere calls a rate.
There was a problem hiding this comment.
I believe the reason is RW is reporting the metric from multiple nodes and the count value from each node varies, leading to the fluctuation.
But it is irrelevant to the alert part, we always alert when the value is non-zero.
|
This pull request has been modified. If you want me to regenerate unit test for any of the files related, please find the file in "Files Changed" tab and add a comment |
|
let's wait for #20295, to minimize the changes. |
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
follow up #19805
manually test
Checklist
Documentation
Release note