clickhouse-backup server consumes memory when idle after first backup

[viktordebulat]

Why clickhouse-backup server container placed as a sidecar inside clickhouse server pod consumes memory when idle after first backup succeed. Using remote backups on gcs. When just started, container consumes tens of Mi. But after backup made and uploaded, it consumes couple Gi of memory whole day being idle.

Image: altinity/clickhouse-backup:2.4.30

Default config overrides:

  general:
    remote_storage: gcs
    backups_to_keep_local: -1
    backups_to_keep_remote: 7
    download_concurrency: 1
    upload_concurrency: 1
    restore_schema_on_cluster: "{cluster}"
  clickhouse:
    max_connections: 2
  gcs:
    path: backup/{replica}-{shard}
    client_pool_size: 3
  api:
    create_integration_tables: true

Other gcs and clickhouse creds related config set over env variables.

Any ideas how to deal with such issue? Whole pod restart after each backup not an option.

[Slach]

which clickhouse-backup version do you use?
please upgrade to 2.4.31
we have some unnecessary memory buffer allocations in some 2.4.x releases

[viktordebulat]

Bumped to 2.4.31 - the same picture. First it consumes almost nothing. Then, when upload started it consumes more (that's logical, ok) but after finished it consumes the same amount of memory, not increasing. The same picture on each node.

[viktordebulat]

@Slach Nope, it proceeded to grow. Meanwhile, on 1 node it grew up to 11 Gi (being idle), and on another node decreased from 2.68 to 1+ Gi

[Slach]

what mean WSS here?
how do you calculate this metric?

[viktordebulat]

WSS - Working Set Size, container metrics from cAdvisor
container_memory_working_set_bytes

[Slach]

most of allocations required during UPLOAD/DOWNLOAD inside remote storage related SDKs
i will appreciate with debuging memory allocations and help how to resolve it

[Slach]

what mean wss here the same as rss?
could you share curl http://127.0.0.1:7171/metrics | grep -i mem inside container which have grow memory related metrics?

[viktordebulat]

# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 2.500536e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 9.150332264e+09
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.548252e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 7.299674e+06
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 3.865024e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 2.500536e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 7.2851456e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 5.070848e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 10180
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 7.2327168e+07
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 7.7922304e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.7084448142689362e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 7.309854e+06
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 9600
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 15600
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 188800
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 310080
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 5.820376e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.49486e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 1.769472e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 1.769472e+06
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 8.6925592e+07
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 3.5500032e+07
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.363279872e+09
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19

[Slach]

process_resident_memory_bytes 3.5500032e+07

it means only 33.86 MiB

could you explain how exactly do you calculate memory on your graph?

[viktordebulat]

cAdvisor - container_memory_working_set_bytes
Now memory usage decreased to 4 Gi on 1 node (where 11 Gi was) and became twice bigger on another (from 1.3 to 2.7). No data transfer from clickhouse-backup container was during that time (backup uploaded long ago)

[Slach]

could you also run
curl -o heap.out http://127.0.0.1:7171//debug/pprof/heap?seconds=120

and share heap.out ?

[viktordebulat]

Failed to open the file heap.out: Permission denied

using inside podTemplate

  securityContext:
    fsGroup: 101
    runAsGroup: 101
    runAsUser: 101

Could it be an issue?

[Slach]

curl -o /tmp/heap.out http://127.0.0.1:7171//debug/pprof/heap?seconds=120

after it use kubectl cp to copy heap.out

[viktordebulat]

No file generated.

$ curl -o /tmp/heap.out http://127.0.0.1:7171//debug/pprof/heap?seconds=120
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

[Slach]

did you wait 120 seconds?

[viktordebulat]

It doesn't generate anything, request is ended right after its execution without response

[nnor1]

Hello everyone,

we currently have the scenario that our memory utilization often shoots up when creating a backup. However, this only happens when uploading the last table (approx. 10 GB), after which the memory utilization remains this high.
Do you have any idea what the reason for this might be?

Setup details:

• Image: altinity/clickhouse-backup:2.6.2
• Node: Memory-optimized | 4 vCPUs | 32 GiB
• DOWNLOAD_CONCURRENCY=2
• UPLOAD_CONCURRENCY=2
• S3_CONCURRENCY=2
• S3_MAX_PARTS_COUNT=5000
• Backup Size 50 GB
• We use Hot and Cold Storage

Thank you

[nnor1]

@Slach

general:
    remote_storage: s3
    max_file_size: 0
    backups_to_keep_local: -1
    backups_to_keep_remote: 30
    log_level: info
    allow_empty_backups: false
    download_concurrency: 2
    upload_concurrency: 2
    upload_max_bytes_per_second: 0
    download_max_bytes_per_second: 0
    object_disk_server_side_copy_concurrency: 32
    allow_object_disk_streaming: false
    use_resumable_state: true
    restore_schema_on_cluster: ""
    upload_by_part: true
    download_by_part: true
    restore_database_mapping: {}
    restore_table_mapping: {}
    retries_on_failure: 3
    retries_pause: 5s
    watch_interval: 1h
    full_interval: 24h
    watch_backup_name_template: shard{shard}-{type}-{time:20060102150405}
    sharded_operation_mode: ""
    cpu_nice_priority: 15
    io_nice_priority: idle
    rbac_backup_always: true
    rbac_conflict_resolution: recreate
    retriesduration: 5s
    watchduration: 1h0m0s
    fullduration: 24h0m0s

clickhouse:
    username: default
    password: ""
    host: localhost
    port: 9000
    disk_mapping: {}
    skip_tables:
        - system.*
        - INFORMATION_SCHEMA.*
        - information_schema.*
        - _temporary_and_external_tables.*
    skip_table_engines: []
    timeout: 30m
    freeze_by_part: false
    freeze_by_part_where: ""
    use_embedded_backup_restore: false
    embedded_backup_disk: ""
    backup_mutations: true
    restore_as_attach: false
    check_parts_columns: true
    secure: false
    skip_verify: false
    sync_replicated_tables: false
    log_sql_queries: true
    config_dir: /etc/clickhouse-server/
    restart_command: exec:systemctl restart clickhouse-server
    ignore_not_exists_error_during_freeze: true
    check_replicas_before_attach: true
    tls_key: ""
    tls_cert: ""
    tls_ca: ""
    max_connections: 2
    debug: false

s3:
    access_key: access_key
    secret_key: secret_key
    bucket: ""
    endpoint: endpoint
    region: us-east-1
    acl: private
    assume_role_arn: ""
    force_path_style: true
    path: backups
    object_disk_path: var/lib/clickhouse/disks/s3
    disable_ssl: false
    compression_level: 1
    compression_format: tar
    sse: ""
    sse_kms_key_id: ""
    sse_customer_algorithm: ""
    sse_customer_key: ""
    sse_customer_key_md5: ""
    sse_kms_encryption_context: ""
    disable_cert_verification: false
    use_custom_storage_class: false
    storage_class: STANDARD
    custom_storage_class_map: {}
    concurrency: 2
    part_size: 0
    max_parts_count: 5000
    allow_multipart_download: false
    object_labels: {}
    request_payer: ""
    check_sum_algorithm: ""
    debug: false

gcs:
    credentials_file: ""
    credentials_json: ""
    credentials_json_encoded: ""
    embedded_access_key: ""
    embedded_secret_key: ""
    skip_credentials: false
    bucket: ""
    path: ""
    object_disk_path: ""
    compression_level: 1
    compression_format: tar
    debug: false
    force_http: false
    endpoint: ""
    storage_class: STANDARD
    object_labels: {}
    custom_storage_class_map: {}
    client_pool_size: 32
    chunk_size: 0

cos:
    url: ""
    timeout: 2m
    secret_id: ""
    secret_key: ""
    path: ""
    object_disk_path: ""
    compression_format: tar
    compression_level: 1
    debug: false

api:
    listen: 0.0.0.0:7171
    enable_metrics: true
    enable_pprof: true
    username: ""
    password: ""
    secure: false
    certificate_file: ""
    private_key_file: ""
    ca_cert_file: ""
    ca_key_file: ""
    create_integration_tables: false
    integration_tables_host: ""
    allow_parallel: false
    complete_resumable_after_restart: true
    watch_is_main_process: false

ftp:
    address: ""
    timeout: 2m
    username: ""
    password: ""
    tls: false
    skip_tls_verify: false
    path: ""
    object_disk_path: ""
    compression_format: tar
    compression_level: 1
    concurrency: 6
    debug: false

sftp:
    address: ""
    port: 22
    username: ""
    password: ""
    key: ""
    path: ""
    object_disk_path: ""
    compression_format: tar
    compression_level: 1
    concurrency: 6
    debug: false

azblob:
    endpoint_schema: https
    endpoint_suffix: core.windows.net
    account_name: ""
    account_key: ""
    sas: ""
    use_managed_identity: false
    container: ""
    path: ""
    object_disk_path: ""
    compression_level: 1
    compression_format: tar
    sse_key: ""
    buffer_size: 0
    buffer_count: 3
    max_parts_count: 256
    timeout: 4h
    debug: false

custom:
    upload_command: ""
    download_command: ""
    list_command: ""
    delete_command: ""
    command_timeout: 4h
    commandtimeoutduration: 4h0m0s

[Slach]

memory allocates inside s3 sdk
could you share?
SELECT database, table, disk_name, formatReadableSize(max(toInt64(bytes_on_disk))) data_by_disk FROM system.parts WHERE active GROUP BY ALL

as some workaround you could
decrease

s3:
 concurrency: 1

how exactly you measure memory usage?
could you share prometheus or other data source metrics expression from shared screenshot?

[nnor1]

Thanks for the help.

We measure our memory usage with the following query:
sum by (container) (container_memory_working_set_bytes{pod=~"$pod", container=~"$container", kubernetes_io_arch=""})

Details

<style> </style>

№	database	table	disk_name	data_by_disk
1	cdp	table 1	s3	41.34 KiB
2	cdp	table 2	default	163.93 KiB
3	cdp	table 3	s3	1.79 KiB
4	cdp	table 4	default	8.55 KiB
5	cdp	table 5	s3	2.06 KiB
6	cdp	table 6	default	2.12 KiB
7	cdp	table 7	s3	279.02 KiB
8	cdp	table 8	s3	2.85 MiB
9	cdp	table 9	s3	40.11 KiB
10	cdp	table 10	s3	316.42 KiB
11	cdp	table 11	default	1.40 KiB
12	cdp	table 12	default	2.07 KiB
13	cdp	table 13	s3	1.68 KiB
14	cdp	table 14	default	8.91 KiB
15	cdp	table 15	s3	36.50 KiB
16	cdp	table 16	default	2.45 MiB
17	cdp	table 17	default	10.48 KiB
18	cdp	table 18	s3	64.33 KiB
19	cdp	table 19	s3	21.93 KiB
20	cdp	table 20	s3	1.43 KiB
21	cdp	table 21	default	2.82 KiB
22	cdp	table 22	s3	52.08 KiB
23	cdp	table 23	default	3.83 KiB
24	cdp	table 24	default	251.65 KiB
25	system	table 25	default	81.06 MiB
26	system	table 26	s3	861.65 KiB
27	cdp	table 27	default	62.02 KiB
28	cdp	table 28	default	2.03 MiB
29	cdp	table 29	default	1.90 MiB
30	cdp	table 30	default	3.78 KiB
31	cdp	table 31	default	84.75 KiB
32	cdp	table 32	default	179.03 KiB
33	cdp	table 33	s3	175.08 KiB
34	cdp	table 34	default	8.55 KiB
35	cdp	table 35	default	4.42 KiB
36	cdp	table 36	default	1.70 KiB
37	cdp	table 37	default	1.75 MiB
38	cdp	table 38	s3	66.60 KiB
39	cdp	table 39	default	1.82 KiB
40	cdp	table 40	default	1.71 KiB
41	cdp	table 41	default	1.42 KiB
42	cdp	table 42	default	1.91 KiB
43	cdp	table 43	s3	25.38 MiB
44	cdp	table 44	default	144.27 KiB
45	cdp	table 45	s3	2.07 MiB
46	cdp	table 46	default	3.74 MiB
47	cdp	table 47	s3	2.28 KiB
48	cdp	table 48	s3	7.01 MiB
49	cdp	table 49	s3	1.77 MiB
50	cdp	table 50	default	6.09 MiB
51	cdp	table 51	s3	1.72 MiB
52	cdp	table 52	default	80.20 KiB
53	cdp	table 53	s3	10.27 KiB
54	cdp	table 54	default	11.30 KiB
55	cdp	table 55	s3	14.21 KiB
56	cdp	table 56	default	297.07 KiB
57	cdp	table 57	default	4.09 MiB
58	cdp	table 58	s3	2.07 KiB
59	cdp	table 59	s3	2.28 MiB
60	cdp	table 60	s3	1.32 KiB
61	cdp	table 61	s3	1.66 KiB
62	cdp	table 62	default	53.11 KiB
63	cdp	table 63	default	37.40 KiB
64	cdp	table 64	default	22.30 KiB
65	cdp	table 65	default	1.55 KiB
66	cdp	table 66	s3	108.54 KiB
67	cdp	table 67	s3	1.37 KiB
68	cdp	table 68	s3	395.30 KiB
69	cdp	table 69	s3	33.82 KiB
70	cdp	table 70	s3	1.37 KiB
71	cdp	table 71	s3	1.25 KiB
72	cdp	table 72	default	8.55 KiB
73	cdp	table 73	default	1.40 KiB
74	cdp	table 74	default	1.71 KiB
75	cdp	table 75	s3	54.36 KiB
76	cdp	table 76	s3	10.72 KiB
77	cdp	table 77	default	3.59 KiB
78	cdp	table 78	default	2.91 KiB
79	cdp	table 79	s3	8.18 MiB
80	cdp	table 80	default	20.63 KiB
81	cdp	table 81	s3	92.62 KiB
82	cdp	table 82	default	70.35 KiB
83	cdp	table 83	s3	4.05 MiB
84	cdp	table 84	s3	3.03 MiB
85	cdp	table 85	default	1.86 KiB
86	cdp	table 86	s3	123.90 KiB
87	cdp	table 87	s3	2.00 KiB
88	cdp	table 88	default	4.93 MiB
89	cdp	table 89	default	4.04 KiB
90	cdp	table 90	default	2.95 KiB
91	cdp	table 91	s3	920.55 KiB
92	cdp	table 92	default	24.97 KiB
93	cdp	table 93	s3	37.86 KiB
94	cdp	table 94	s3	2.35 KiB
95	cdp	table 95	default	2.99 KiB
96	cdp	table 96	s3	7.44 KiB
97	cdp	table 97	s3	9.40 MiB
98	cdp	table 98	default	1.71 KiB
99	cdp	table 99	s3	2.28 KiB
100	cdp	table 100	default	315.11 KiB
101	cdp	table 101	default	233.79 KiB
102	cdp	table 102	default	7.63 KiB
103	cdp	table 103	s3	57.32 KiB
104	cdp	table 104	default	1.44 MiB
105	cdp	table 105	default	701.83 KiB
106	cdp	table 106	s3	2.30 KiB
107	cdp	table 107	s3	1.62 MiB
108	cdp	table 108	default	5.10 KiB
109	cdp	table 109	s3	1.43 MiB
110	cdp	table 110	s3	1.34 KiB
111	cdp	table 111	s3	16.15 KiB
112	cdp	table 112	s3	52.56 KiB
113	cdp	table 113	default	962.48 KiB
114	cdp	table 114	default	81.48 KiB
115	cdp	table 115	default	130.27 KiB
116	cdp	table 116	s3	9.37 KiB
117	cdp	table 117	s3	6.20 MiB
118	cdp	table 118	s3	6.69 KiB
119	cdp	table 119	default	1.73 KiB
120	cdp	table 120	s3	1.75 KiB
121	cdp	table 121	s3	20.44 KiB
122	cdp	table 122	s3	13.28 MiB
123	cdp	table 123	s3	41.80 KiB
124	cdp	table 124	default	53.34 KiB
125	cdp	table 125	s3	126.33 KiB
126	cdp	table 126	default	70.54 KiB
127	cdp	table 127	default	21.71 KiB
128	cdp	table 128	s3	1.98 KiB
129	cdp	table 129	default	4.93 KiB
130	cdp	table 130	default	9.11 KiB
131	cdp	table 131	s3	11.63 KiB
132	cdp	table 132	default	2.12 KiB
133	cdp	table 133	default	1.06 MiB
134	cdp	table 134	s3	563.14 KiB
135	cdp	table 135	default	3.12 KiB
136	cdp	table 136	s3	474.47 KiB
137	cdp	table 137	default	5.01 MiB
138	cdp	table 138	s3	4.59 KiB
139	cdp	table 139	default	431.55 KiB
140	system	table 140	s3	50.08 MiB
141	cdp	table 141	default	2.13 KiB
142	cdp	table 142	default	23.57 KiB
143	cdp	table 143	s3	2.74 KiB
144	cdp	table 144	default	335.20 KiB
145	cdp	table 145	s3	2.51 KiB
146	cdp	table 146	s3	18.42 KiB
147	cdp	table 147	default	3.81 MiB
148	cdp	table 148	s3	1.61 KiB
149	cdp	table 149	s3	165.18 KiB
150	cdp	table 150	s3	1.89 KiB
151	cdp	table 151	default	912.20 KiB
152	cdp	table 152	default	2.33 KiB
153	cdp	table 153	default	75.43 KiB
154	system	table 154	default	16.16 MiB
155	cdp	table 155	default	3.10 KiB
156	cdp	table 156	s3	22.72 KiB
157	cdp	table 157	default	59.78 KiB
158	cdp	table 158	s3	1.99 KiB
159	cdp	table 159	s3	2.64 KiB
160	cdp	table 160	default	1.86 MiB
161	cdp	table 161	default	34.98 KiB
162	cdp	table 162	s3	11.67 KiB
163	cdp	table 163	default	2.05 KiB
164	cdp	table 164	default	453.39 KiB
165	cdp	table 165	s3	1.74 KiB
166	cdp	table 166	default	4.22 KiB
167	cdp	table 167	default	3.04 MiB
168	cdp	table 168	s3	332.01 KiB
169	cdp	table 169	default	4.66 KiB
170	cdp	table 170	default	2.33 MiB
171	cdp	table 171	default	3.39 KiB
172	cdp	table 172	s3	3.16 KiB
173	cdp	table 173	s3	14.07 KiB
174	cdp	table 174	s3	18.31 MiB
175	cdp	table 175	default	80.87 KiB
176	cdp	table 176	default	2.06 KiB
177	cdp	table 177	default	9.88 KiB
178	cdp	table 178	s3	1.60 MiB
179	cdp	table 179	default	1.77 KiB
180	cdp	table 180	default	5.41 KiB
181	cdp	table 181	default	695.00 B
182	cdp	table 182	default	94.43 KiB
183	cdp	table 183	s3	6.29 KiB
184	cdp	table 184	s3	2.34 MiB
185	cdp	table 185	s3	2.08 KiB
186	cdp	table 186	s3	12.64 KiB
187	cdp	table 187	s3	913.97 KiB
188	cdp	table 188	s3	4.52 KiB
189	cdp	table 189	default	6.77 KiB
190	cdp	table 190	default	1.65 KiB
191	cdp	table 191	default	7.32 KiB
192	cdp	table 192	default	823.79 KiB
193	cdp	table 193	default	43.27 KiB
194	cdp	table 194	s3	2.45 KiB
195	cdp	table 195	default	80.30 KiB
196	cdp	table 196	default	6.20 MiB
197	cdp	table 197	default	4.55 KiB
198	system	table 198	s3	2.31 MiB
199	cdp	table 199	default	5.21 KiB
200	cdp	table 200	s3	14.57 KiB
201	cdp	table 201	s3	1.65 KiB
202	cdp	table 202	default	3.91 MiB
203	cdp	table 203	s3	3.46 KiB
204	cdp	table 204	s3	74.45 KiB
205	cdp	table 205	default	7.51 KiB
206	cdp	table 206	default	44.82 KiB
207	cdp	table 207	default	13.71 KiB
208	cdp	table 208	default	1.71 KiB
209	cdp	table 209	default	1.47 KiB
210	cdp	table 210	s3	3.02 KiB
211	cdp	table 211	default	3.87 KiB
212	cdp	table 212	default	617.17 KiB
213	cdp	table 213	default	7.61 MiB
214	cdp	table 214	s3	26.67 KiB
215	cdp	table 215	s3	2.13 MiB
216	cdp	table 216	default	307.44 KiB
217	cdp	table 217	default	2.74 KiB
218	cdp	table 218	s3	16.93 KiB
219	cdp	table 219	s3	363.29 KiB
220	cdp	table 220	s3	4.26 MiB
221	cdp	table 221	s3	2.24 KiB
222	cdp	table 222	default	2.81 KiB
223	cdp	table 223	s3	2.53 KiB
224	cdp	table 224	default	8.61 KiB
225	cdp	table 225	s3	72.80 KiB
226	cdp	table 226	default	17.42 KiB
227	cdp	table 227	s3	1.01 MiB
228	cdp	table 228	default	986.73 KiB
229	cdp	table 229	s3	270.15 KiB
230	cdp	table 230	default	35.62 KiB
231	cdp	table 231	default	1.69 KiB
232	cdp	table 232	s3	5.21 KiB
233	cdp	table 233	default	1.16 MiB
234	cdp	table 234	default	1.20 MiB
235	cdp	table 235	s3	8.28 KiB
236	cdp	table 236	default	195.56 KiB
237	cdp	table 237	s3	5.26 KiB
238	cdp	table 238	default	9.56 KiB
239	cdp	table 239	default	486.98 KiB
240	cdp	table 240	default	136.53 KiB
241	cdp	table 241	default	1.46 KiB
242	cdp	table 242	s3	5.12 KiB
243	cdp	table 243	default	3.31 KiB
244	cdp	table 244	default	3.34 KiB
245	cdp	table 245	s3	188.45 KiB
246	cdp	table 246	default	2.17 KiB
247	cdp	table 247	default	99.07 KiB
248	cdp	table 248	s3	6.95 MiB
249	cdp	table 249	default	22.53 KiB
250	cdp	table 250	default	1.89 KiB
251	cdp	table 251	default	3.02 MiB
252	cdp	table 252	default	17.88 KiB
253	cdp	table 253	default	50.92 KiB
254	cdp	table 254	s3	1.10 MiB
255	cdp	table 255	default	21.23 KiB
256	cdp	table 256	default	14.38 KiB
257	cdp	table 257	default	1.52 KiB
258	cdp	table 258	s3	1.29 KiB
259	cdp	table 259	s3	36.35 KiB
260	cdp	table 260	default	55.56 KiB
261	cdp	table 261	default	3.05 KiB
262	cdp	table 262	s3	21.83 KiB
263	cdp	table 263	s3	3.98 KiB
264	cdp	table 264	s3	29.61 KiB
265	cdp	table 265	default	268.99 KiB
266	cdp	table 266	s3	2.09 KiB
267	cdp	table 267	default	1.96 KiB
268	cdp	table 268	s3	382.63 KiB
269	cdp	table 269	s3	1.90 KiB
270	cdp	table 270	s3	480.24 KiB
271	cdp	table 271	default	19.70 KiB
272	cdp	table 272	s3	2.05 KiB
273	cdp	table 273	default	338.19 KiB
274	cdp	table 274	default	123.79 KiB
275	cdp	table 275	s3	119.21 KiB
276	cdp	table 276	default	741.92 KiB
277	cdp	table 277	s3	148.48 MiB
278	cdp	table 278	default	529.31 KiB
279	cdp	table 279	default	3.90 KiB
280	cdp	table 280	default	3.11 MiB
281	cdp	table 281	default	700.96 KiB
282	cdp	table 282	default	253.77 KiB
283	cdp	table 283	default	5.94 KiB
284	cdp	table 284	s3	49.17 KiB
285	cdp	table 285	default	11.47 KiB
286	cdp	table 286	s3	275.03 KiB
287	cdp	table 287	default	18.13 KiB
288	cdp	table 288	default	2.88 MiB
289	cdp	table 289	default	10.66 KiB
290	cdp	table 290	s3	5.19 KiB
291	cdp	table 291	default	27.25 KiB
292	cdp	table 292	s3	1.09 MiB
293	cdp	table 293	s3	1.18 KiB
294	cdp	table 294	default	19.85 MiB
295	cdp	table 295	default	638.02 KiB
296	cdp	table 296	s3	1.65 KiB
297	cdp	table 297	default	7.00 KiB
298	cdp	table 298	s3	78.11 KiB
299	cdp	table 299	s3	5.09 MiB
300	cdp	table 300	default	1.88 KiB
301	cdp	table 301	default	209.51 KiB
302	cdp	table 302	default	1.20 MiB
303	cdp	table 303	default	1.43 MiB
304	cdp	table 304	default	2.67 KiB
305	cdp	table 305	s3	18.32 MiB
306	cdp	table 306	s3	278.75 KiB
307	cdp	table 307	default	9.94 MiB
308	cdp	table 308	default	3.69 MiB
309	cdp	table 309	default	115.22 KiB
310	cdp	table 310	s3	1.47 KiB
311	cdp	table 311	default	109.90 KiB
312	cdp	table 312	s3	3.03 MiB
313	cdp	table 313	s3	81.57 KiB
314	cdp	table 314	s3	1.79 KiB
315	cdp	table 315	s3	143.14 KiB
316	cdp	table 316	default	2.00 KiB
317	cdp	table 317	s3	1.94 KiB
318	cdp	table 318	s3	4.85 MiB
319	cdp	table 319	s3	7.37 KiB
320	cdp	table 320	default	5.09 KiB
321	cdp	table 321	s3	2.10 KiB
322	cdp	table 322	s3	115.38 KiB
323	cdp	table 323	default	535.00 B
324	cdp	table 324	default	23.30 KiB
325	cdp	table 325	default	982.07 KiB
326	cdp	table 326	default	1.56 KiB
327	cdp	table 327	s3	33.54 KiB
328	cdp	table 328	default	21.99 KiB
329	cdp	table 329	s3	100.64 KiB
330	cdp	table 330	default	22.88 MiB
331	cdp	table 331	default	1.30 MiB
332	cdp	table 332	s3	20.29 KiB
333	cdp	table 333	s3	2.66 KiB
334	cdp	table 334	s3	21.15 KiB
335	cdp	table 335	default	1.89 KiB
336	cdp	table 336	default	39.26 KiB
337	cdp	table 337	default	15.24 KiB
338	cdp	table 338	default	1.88 KiB
339	cdp	table 339	default	556.52 KiB
340	cdp	table 340	s3	59.60 KiB
341	cdp	table 341	s3	81.27 KiB
342	cdp	table 342	default	129.72 KiB
343	cdp	table 343	s3	199.05 KiB
344	cdp	table 344	s3	6.09 KiB
345	cdp	table 345	default	2.77 MiB
346	cdp	table 346	s3	463.89 KiB
347	cdp	table 347	default	58.91 KiB
348	cdp	table 348	default	41.26 KiB
349	cdp	table 349	s3	261.98 KiB
350	cdp	table 350	s3	1.89 KiB
351	cdp	table 351	default	34.97 KiB
352	cdp	table 352	s3	40.91 KiB
353	cdp	table 353	default	7.00 KiB
354	cdp	table 354	default	3.01 MiB
355	cdp	table 355	s3	4.03 KiB
356	cdp	table 356	s3	80.16 KiB
357	cdp	table 357	s3	4.73 KiB
358	cdp	table 358	default	1.31 MiB
359	cdp	table 359	s3	2.09 KiB
360	cdp	table 360	s3	1.37 KiB
361	cdp	table 361	default	4.76 KiB
362	cdp	table 362	s3	2.33 KiB
363	cdp	table 363	default	3.72 MiB
364	cdp	table 364	s3	1.77 KiB
365	cdp	table 365	default	1.53 KiB
366	cdp	table 366	s3	3.02 KiB
367	cdp	table 367	s3	25.82 KiB
368	cdp	table 368	s3	345.02 KiB
369	cdp	table 369	default	124.85 KiB
370	cdp	table 370	s3	1.51 KiB
371	cdp	table 371	default	2.79 KiB
372	cdp	table 372	default	3.52 MiB
373	cdp	table 373	default	7.40 KiB
374	cdp	table 374	default	2.09 MiB
375	cdp	table 375	s3	1.35 KiB
376	cdp	table 376	default	42.41 KiB
377	cdp	table 377	default	1.30 KiB
378	cdp	table 378	default	365.89 KiB
379	cdp	table 379	default	2.71 KiB
380	cdp	table 380	default	3.92 KiB
381	cdp	table 381	default	1.44 MiB
382	cdp	table 382	default	9.14 KiB
383	cdp	table 383	s3	5.50 KiB
384	cdp	table 384	default	14.43 KiB
385	cdp	table 385	s3	532.25 KiB
386	cdp	table 386	s3	75.56 KiB
387	cdp	table 387	s3	2.73 KiB
388	cdp	table 388	default	94.19 KiB
389	cdp	table 389	s3	1.71 KiB
390	cdp	table 390	default	1.73 MiB
391	cdp	table 391	s3	1.88 KiB
392	cdp	table 392	s3	2.00 KiB
393	cdp	table 393	s3	2.18 KiB
394	cdp	table 394	default	6.56 KiB
395	cdp	table 395	s3	2.28 MiB
396	cdp	table 396	default	54.77 KiB
397	cdp	table 397	default	292.01 KiB
398	cdp	table 398	s3	69.37 KiB
399	cdp	table 399	s3	7.34 KiB
400	cdp	table 400	s3	2.47 KiB
401	cdp	table 401	s3	181.26 KiB
402	cdp	table 402	s3	1.79 KiB
403	cdp	table 403	default	268.36 KiB
404	cdp	table 404	default	2.07 KiB
405	cdp	table 405	s3	2.57 KiB
406	cdp	table 406	s3	1.47 KiB
407	cdp	table 407	default	5.26 MiB
408	cdp	table 408	default	4.88 KiB
409	cdp	table 409	default	1.91 KiB
410	cdp	table 410	s3	1.00 KiB
411	system	table 411	s3	125.74 MiB
412	cdp	table 412	s3	15.83 KiB
413	cdp	table 413	s3	2.10 MiB
414	cdp	table 414	s3	19.62 KiB
415	cdp	table 415	default	346.72 KiB
416	cdp	table 416	s3	2.17 KiB
417	cdp	table 417	default	2.81 KiB
418	cdp	table 418	s3	4.16 KiB
419	cdp	table 419	default	5.61 KiB
420	cdp	table 420	default	1.92 KiB
421	cdp	table 421	default	1.48 KiB
422	cdp	table 422	default	124.74 KiB
423	cdp	table 423	s3	6.24 KiB
424	cdp	table 424	s3	191.18 KiB
425	cdp	table 425	s3	628.29 KiB
426	cdp	table 426	default	117.06 MiB
427	cdp	table 427	s3	143.29 KiB
428	cdp	table 428	s3	160.58 KiB
429	cdp	table 429	s3	33.60 KiB
430	cdp	table 430	default	315.81 KiB
431	cdp	table 431	s3	9.94 KiB
432	cdp	table 432	s3	1.30 KiB
433	cdp	table 433	default	1.01 MiB
434	cdp	table 434	s3	14.07 MiB
435	cdp	table 435	default	477.18 KiB
436	cdp	table 436	s3	20.70 MiB
437	cdp	table 437	default	154.58 KiB
438	cdp	table 438	default	199.46 KiB
439	cdp	table 439	s3	18.07 KiB
440	cdp	table 440	s3	77.00 KiB
441	cdp	table 441	default	14.57 KiB
442	cdp	table 442	default	27.70 KiB
443	cdp	table 443	default	1.70 KiB
444	cdp	table 444	default	8.49 KiB
445	cdp	table 445	s3	27.83 KiB
446	cdp	table 446	s3	230.81 KiB
447	cdp	table 447	s3	336.93 KiB
448	cdp	table 448	s3	822.17 KiB
449	cdp	table 449	default	9.90 KiB
450	cdp	table 450	s3	340.19 KiB
451	cdp	table 451	s3	3.33 KiB
452	cdp	table 452	default	2.44 KiB
453	cdp	table 453	default	24.61 MiB
454	cdp	table 454	default	126.31 KiB
455	cdp	table 455	s3	39.78 KiB
456	cdp	table 456	s3	5.79 KiB
457	system	table 457	s3	17.38 MiB
458	cdp	table 458	s3	38.87 KiB
459	cdp	table 459	default	21.26 MiB
460	cdp	table 460	default	1.27 MiB
461	cdp	table 461	default	2.83 KiB
462	cdp	table 462	default	147.85 KiB
463	cdp	table 463	default	4.16 MiB
464	cdp	table 464	s3	3.45 KiB
465	cdp	table 465	default	19.75 KiB
466	system	table 466	default	3.84 MiB
467	cdp	table 467	default	1.65 KiB
468	cdp	table 468	default	1.96 KiB
469	cdp	table 469	s3	358.08 KiB
470	cdp	table 470	s3	1.48 KiB
471	cdp	table 471	s3	1.44 MiB
472	cdp	table 472	s3	4.22 KiB
473	cdp	table 473	default	39.32 KiB
474	cdp	table 474	default	112.81 KiB
475	cdp	table 475	default	58.37 KiB
476	cdp	table 476	s3	2.73 KiB
477	cdp	table 477	s3	862.35 KiB
478	cdp	table 478	default	20.56 MiB
479	cdp	table 479	default	169.87 MiB
480	cdp	table 480	s3	148.40 KiB
481	cdp	table 481	default	1.63 KiB
482	cdp	table 482	default	385.40 KiB
483	cdp	table 483	s3	3.21 KiB
484	cdp	table 484	default	3.79 MiB
485	cdp	table 485	default	7.70 KiB
486	cdp	table 486	s3	26.24 KiB
487	cdp	table 487	s3	23.47 KiB
488	cdp	table 488	s3	66.75 KiB
489	cdp	table 489	s3	1.33 KiB
490	cdp	table 490	default	2.90 KiB
491	cdp	table 491	default	4.42 MiB
492	cdp	table 492	default	65.18 KiB
493	cdp	table 493	s3	2.21 KiB
494	cdp	table 494	default	978.75 KiB
495	cdp	table 495	s3	1.46 KiB
496	cdp	table 496	default	2.20 KiB
497	cdp	table 497	s3	150.84 KiB
498	cdp	table 498	s3	130.45 KiB
499	system	table 499	default	31.02 MiB
500	cdp	table 500	s3	136.16 KiB
501	cdp	table 501	s3	2.11 KiB
502	cdp	table 502	default	57.74 KiB
503	cdp	table 503	default	155.74 KiB
504	cdp	table 504	s3	2.08 KiB
505	cdp	table 505	s3	1.11 MiB
506	cdp	table 506	default	4.66 KiB
507	system	table 507	default	9.34 MiB
508	cdp	table 508	s3	160.28 KiB
509	cdp	table 509	s3	2.07 KiB
510	cdp	table 510	default	10.06 MiB
511	cdp	table 511	s3	9.47 KiB
512	cdp	table 512	s3	2.16 MiB
513	cdp	table 513	default	518.67 KiB
514	cdp	table 514	s3	2.98 KiB
515	cdp	table 515	s3	80.98 KiB
516	cdp	table 516	default	381.24 KiB
517	cdp	table 517	default	2.16 KiB
518	cdp	table 518	default	5.18 MiB
519	cdp	table 519	default	1.66 KiB
520	cdp	table 520	default	113.88 KiB
521	cdp	table 521	default	232.09 KiB
522	cdp	table 522	s3	2.49 KiB
523	cdp	table 523	s3	1.66 KiB
524	cdp	table 524	s3	10.22 KiB
525	cdp	table 525	default	67.97 KiB
526	cdp	table 526	default	1.47 MiB
527	cdp	table 527	default	116.94 KiB
528	cdp	table 528	default	47.93 KiB
529	cdp	table 529	default	1.93 MiB
530	cdp	table 530	s3	1.31 MiB
531	cdp	table 531	default	16.12 KiB
532	cdp	table 532	s3	826.39 KiB
533	cdp	table 533	default	1.45 KiB
534	cdp	table 534	default	4.78 KiB
535	cdp	table 535	default	11.85 KiB

[Slach]

check
sum by (container) (container_memory_rss{pod=~"$pod", container=~"$container", kubernetes_io_arch=""})

working_set this is Resident memory + page cache buffers

clickhouse-backup read whole your data during upload, so it allocates page cache

[nnor1]

Thank you