Missing some specific metrics from statsd.

[sshkim]

Hi guys.

I'm using official helm chart with KubernetesExecutor.

I'm testing metrics but some metric is not showing.

I can see that most merics. But I can't see things like below list.

• operator_failures_{operator_name}
• operator_successes_{operator_name}
• dagrun.duration.success.{dag_id}
• dagrun.duration.failed.{dag_id}
• dag.{dag_id}.{task_id}.duration
• dagrun.schedule_delay.{dag_id}

[qzyu999]

I actually am seeing no airflow-related metrics. This is similar to here: #11514. I'm not sure how related this is to KubernetesExecutor, since this seems to show 3 cases of people getting varying degrees of metrics. This is an example of my output:

# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 4.7361e-05
go_gc_duration_seconds{quantile="0.25"} 6.6503e-05
go_gc_duration_seconds{quantile="0.5"} 8.2487e-05
go_gc_duration_seconds{quantile="0.75"} 0.00010118
go_gc_duration_seconds{quantile="1"} 0.000177828
go_gc_duration_seconds_sum 0.001412977
go_gc_duration_seconds_count 16
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 15
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.18.6"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 2.19952e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 3.9215376e+07
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.450108e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 61768
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 4.697416e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 2.19952e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 4.58752e+06
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 3.11296e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 9363
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 2.490368e+06
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 7.70048e+06
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.690243749729096e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 71131
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 2400
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 15600
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 47872
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 65280
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 4.194304e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 670292
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 688128
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 688128
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 1.5287304e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 8
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0.43
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 12
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.3500416e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.69024347541e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 7.3132032e+08
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 63
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0
# HELP statsd_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which statsd_exporter was built.
# TYPE statsd_exporter_build_info gauge
statsd_exporter_build_info{branch="HEAD",goversion="go1.18.6",revision="aecad1a2faf31d4a6c27323a29ca8c7a23d88f6b",version="0.22.8"} 1
# HELP statsd_exporter_event_queue_flushed_total Number of times events were flushed to exporter
# TYPE statsd_exporter_event_queue_flushed_total counter
statsd_exporter_event_queue_flushed_total 1379
# HELP statsd_exporter_events_unmapped_total The total number of StatsD events no mapping was found for.
# TYPE statsd_exporter_events_unmapped_total counter
statsd_exporter_events_unmapped_total 0
# HELP statsd_exporter_lines_total The total number of StatsD lines received.
# TYPE statsd_exporter_lines_total counter
statsd_exporter_lines_total 0
# HELP statsd_exporter_loaded_mappings The current number of configured metric mappings.
# TYPE statsd_exporter_loaded_mappings gauge
statsd_exporter_loaded_mappings 13
# HELP statsd_exporter_samples_total The total number of StatsD samples received.
# TYPE statsd_exporter_samples_total counter
statsd_exporter_samples_total 0
# HELP statsd_exporter_tag_errors_total The number of errors parsing DogStatsD tags.
# TYPE statsd_exporter_tag_errors_total counter
statsd_exporter_tag_errors_total 0
# HELP statsd_exporter_tags_total The total number of DogStatsD tags processed.
# TYPE statsd_exporter_tags_total counter
statsd_exporter_tags_total 0
# HELP statsd_exporter_tcp_connection_errors_total The number of errors encountered reading from TCP.
# TYPE statsd_exporter_tcp_connection_errors_total counter
statsd_exporter_tcp_connection_errors_total 0
# HELP statsd_exporter_tcp_connections_total The total number of TCP connections handled.
# TYPE statsd_exporter_tcp_connections_total counter
statsd_exporter_tcp_connections_total 0
# HELP statsd_exporter_tcp_too_long_lines_total The number of lines discarded due to being too long.
# TYPE statsd_exporter_tcp_too_long_lines_total counter
statsd_exporter_tcp_too_long_lines_total 0
# HELP statsd_exporter_udp_packets_total The total number of StatsD packets received over UDP.
# TYPE statsd_exporter_udp_packets_total counter
statsd_exporter_udp_packets_total 0
# HELP statsd_exporter_unixgram_packets_total The total number of StatsD packets received over Unixgram.
# TYPE statsd_exporter_unixgram_packets_total counter
statsd_exporter_unixgram_packets_total 0
# HELP statsd_metric_mapper_cache_gets_total The count of total metric cache gets.
# TYPE statsd_metric_mapper_cache_gets_total counter
statsd_metric_mapper_cache_gets_total 0
# HELP statsd_metric_mapper_cache_hits_total The count of total metric cache hits.
# TYPE statsd_metric_mapper_cache_hits_total counter
statsd_metric_mapper_cache_hits_total 0
# HELP statsd_metric_mapper_cache_length The count of unique metrics currently cached.
# TYPE statsd_metric_mapper_cache_length gauge
statsd_metric_mapper_cache_length 0