self hosted sentry clickhouse pod crashing loop backoff

[gauravks20072023]

Issue submitter TODO list

• I've searched for an already existing issues here

Describe the bug (actual behavior)

4. ? @ 0x10d0b117 in /usr/bin/clickhouse

5. ThreadPoolImpl::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0x902f738 in /usr/bin/clickhouse

6. ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl::scheduleImpl(std::__1::function<void ()>, int, std::__1::optional)::'lambda0'()>(void&&, void ThreadPoolImpl::scheduleImpl(std::__1::function<void ()>, int, std::__1::optional)::'lambda0'()&&...)::'lambda'()::operator()() @ 0x90312df in /usr/bin/clickhouse

7. ThreadPoolImplstd::__1::thread::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0x902ca1f in /usr/bin/clickhouse

8. ? @ 0x9030303 in /usr/bin/clickhouse

9. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so

10. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
    (version 21.8.13.6 (official build))
    2024.12.18 12:17:11.768495 [ 132 ] {} auto DB::IBackgroundJobExecutor::execute(DB::JobAndPool)::(anonymous class)::operator()() const: Code: 236, e.displayText() = DB::Exception: Cancelled merging parts, Stack trace (when copying this message, always include the lines below):

11. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x8febd9a in /usr/bin/clickhouse

12. DB::MergeTreeDataMergerMutator::mergePartsToTemporaryPart(DB::FutureMergedMutatedPart const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::MergeListElement&, std::__1::shared_ptrDB::RWLockImpl::LockHolderImpl&, long, std::__1::shared_ptr<DB::Context const>, std::__1::unique_ptr<DB::IReservation, std::__1::default_deleteDB::IReservation > const&, bool, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > > > const&, DB::MergeTreeData::MergingParams const&, DB::IMergeTreeDataPart const*, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&) @ 0x10dea06b in /usr/bin/clickhouse

13. DB::StorageMergeTree::mergeSelectedParts(std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, bool, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > > > const&, DB::StorageMergeTree::MergeMutateSelectedEntry&, std::__1::shared_ptrDB::RWLockImpl::LockHolderImpl&) @ 0x10fb784f in /usr/bin/clickhouse

14. ? @ 0x10fc7637 in /usr/bin/clickhouse

15. ? @ 0x10d0b117 in /usr/bin/clickhouse

16. ThreadPoolImpl::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0x902f738 in /usr/bin/clickhouse

17. ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl::scheduleImpl(std::__1::function<void ()>, int, std::__1::optional)::'lambda0'()>(void&&, void ThreadPoolImpl::scheduleImpl(std::__1::function<void ()>, int, std::__1::optional)::'lambda0'()&&...)::'lambda'()::operator()() @ 0x90312df in /usr/bin/clickhouse

18. ThreadPoolImplstd::__1::thread::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0x902ca1f in /usr/bin/clickhouse

19. ? @ 0x9030303 in /usr/bin/clickhouse

20. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so

21. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
    (version 21.8.13.6 (official build))
    2024.12.18 12:17:11.776217 [ 52 ] {} Application: DB::Exception: Suspiciously many (16) broken parts to remove.: Cannot attach table default.outcomes_raw_local from metadata file /var/lib/clickhouse/store/106/106a431e-d37d-4547-906a-431ed37dd547/outcomes_raw_local.sql from query ATTACH TABLE default.outcomes_raw_local UUID 'a651b480-0ef5-47e3-a651-b4800ef597e3' (org_id UInt64, project_id UInt64, key_id Nullable(UInt64), timestamp DateTime, category UInt8, outcome UInt8, reason LowCardinality(Nullable(String)), quantity UInt32, event_id Nullable(UUID) CODEC(LZ4HC(0)) TTL timestamp + toIntervalDay(30), size UInt32, INDEX minmax_key_id key_id TYPE minmax GRANULARITY 1, INDEX minmax_outcome outcome TYPE minmax GRANULARITY 1) ENGINE = ReplicatedMergeTree('/clickhouse/tables/outcomes/{shard}/default/outcomes_raw_local', '{replica}') PARTITION BY toMonday(timestamp) ORDER BY (org_id, project_id, timestamp) TTL timestamp + toIntervalDay(30) SETTINGS index_granularity = 16384: while loading database default from path /var/lib/clickhouse/metadata/d

Expected behavior

clickhouse should start without issue

values.yaml

clickhouse:
enabled: true
clickhouse:
replicas: "1"
imageVersion: "21.8.13.6"
configmap:
remote_servers:
internal_replication: true
replica:
backup:
enabled: false
zookeeper_servers:
enabled: true
config:
- index: "clickhouse"
hostTemplate: "{{ .Release.Name }}-zookeeper-clickhouse"
port: "2181"
users:
enabled: false
user:
# the first user will be used if enabled
- name: default
config:
password: ""
networks:
- ::/0
profile: default
quota: default

persistentVolumeClaim:
  enabled: true
  dataPersistentVolume:
    enabled: true
    accessModes:
      - "ReadWriteOnce"
    storage: "30Gi"

nodeSelector:
dedicated: user-pool
tolerations:

• key: "kubernetes.azure.com/scalesetpriority"
  operator: "Equal"
  value: "spot"
  effect: "NoSchedule"

Helm chart version

sentry 25.17.0

Steps to reproduce

start clickhouse stateful sets

Screenshots

No response

Logs

2024.12.18 12:17:11.809528 [ 1 ] {} Application: Child process exited normally with code 70.

==> /var/log/clickhouse-server/clickhouse-server.err.log <==
2024.12.18 12:17:11.732404 [ 66 ] {} auto DB::IBackgroundJobExecutor::execute(DB::JobAndPool)::(anonymous class)::operator()() const: Code: 236, e.displayText() = DB::Exception: Cancelled merging parts, Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x8febd9a in /usr/bin/clickhouse

1. DB::MergeTreeDataMergerMutator::mergePartsToTemporaryPart(DB::FutureMergedMutatedPart const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::MergeListElement&, std::__1::shared_ptrDB::RWLockImpl::LockHolderImpl&, long, std::__1::shared_ptr<DB::Context const>, std::__1::unique_ptr<DB::IReservation, std::__1::default_deleteDB::IReservation > const&, bool, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > > > const&, DB::MergeTreeData::MergingParams const&, DB::IMergeTreeDataPart const*, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&) @ 0x10dea25f in /usr/bin/clickhouse

2. DB::StorageMergeTree::mergeSelectedParts(std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, bool, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > > > const&, DB::StorageMergeTree::MergeMutateSelectedEntry&, std::__1::shared_ptrDB::RWLockImpl::LockHolderImpl&) @ 0x10fb784f in /usr/bin/clickhouse

3. ? @ 0x10fc7637 in /usr/bin/clickhouse

4. ? @ 0x10d0b117 in /usr/bin/clickhouse

5. ThreadPoolImpl::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0x902f738 in /usr/bin/clickhouse

6. ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl::scheduleImpl(std::__1::function<void ()>, int, std::__1::optional)::'lambda0'()>(void&&, void ThreadPoolImpl::scheduleImpl(std::__1::function<void ()>, int, std::__1::optional)::'lambda0'()&&...)::'lambda'()::operator()() @ 0x90312df in /usr/bin/clickhouse

7. ThreadPoolImplstd::__1::thread::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0x902ca1f in /usr/bin/clickhouse

8. ? @ 0x9030303 in /usr/bin/clickhouse

9. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so

10. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
    (version 21.8.13.6 (official build))
    2024.12.18 12:17:11.768495 [ 132 ] {} auto DB::IBackgroundJobExecutor::execute(DB::JobAndPool)::(anonymous class)::operator()() const: Code: 236, e.displayText() = DB::Exception: Cancelled merging parts, Stack trace (when copying this message, always include the lines below):

11. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x8febd9a in /usr/bin/clickhouse

12. DB::MergeTreeDataMergerMutator::mergePartsToTemporaryPart(DB::FutureMergedMutatedPart const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::MergeListElement&, std::__1::shared_ptrDB::RWLockImpl::LockHolderImpl&, long, std::__1::shared_ptr<DB::Context const>, std::__1::unique_ptr<DB::IReservation, std::__1::default_deleteDB::IReservation > const&, bool, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > > > const&, DB::MergeTreeData::MergingParams const&, DB::IMergeTreeDataPart const*, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&) @ 0x10dea06b in /usr/bin/clickhouse

13. DB::StorageMergeTree::mergeSelectedParts(std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, bool, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > > > const&, DB::StorageMergeTree::MergeMutateSelectedEntry&, std::__1::shared_ptrDB::RWLockImpl::LockHolderImpl&) @ 0x10fb784f in /usr/bin/clickhouse

14. ? @ 0x10fc7637 in /usr/bin/clickhouse

15. ? @ 0x10d0b117 in /usr/bin/clickhouse

16. ThreadPoolImpl::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0x902f738 in /usr/bin/clickhouse

17. ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl::scheduleImpl(std::__1::function<void ()>, int, std::__1::optional)::'lambda0'()>(void&&, void ThreadPoolImpl::scheduleImpl(std::__1::function<void ()>, int, std::__1::optional)::'lambda0'()&&...)::'lambda'()::operator()() @ 0x90312df in /usr/bin/clickhouse

18. ThreadPoolImplstd::__1::thread::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0x902ca1f in /usr/bin/clickhouse

19. ? @ 0x9030303 in /usr/bin/clickhouse

20. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so

21. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
    (version 21.8.13.6 (official build))
    2024.12.18 12:17:11.776217 [ 52 ] {} Application: DB::Exception: Suspiciously many (16) broken parts to remove.: Cannot attach table default.outcomes_raw_local from metadata file /var/lib/clickhouse/store/106/106a431e-d37d-4547-906a-431ed37dd547/outcomes_raw_local.sql from query ATTACH TABLE default.outcomes_raw_local UUID 'a651b480-0ef5-47e3-a651-b4800ef597e3' (org_id UInt64, project_id UInt64, key_id Nullable(UInt64), timestamp DateTime, category UInt8, outcome UInt8, reason LowCardinality(Nullable(String)), quantity UInt32, event_id Nullable(UUID) CODEC(LZ4HC(0)) TTL timestamp + toIntervalDay(30), size UInt32, INDEX minmax_key_id key_id TYPE minmax GRANULARITY 1, INDEX minmax_outcome outcome TYPE minmax GRANULARITY 1) ENGINE = ReplicatedMergeTree('/clickhouse/tables/outcomes/{shard}/default/outcomes_raw_local', '{replica}') PARTITION BY toMonday(timestamp) ORDER BY (org_id, project_id, timestamp) TTL timestamp + toIntervalDay(30) SETTINGS index_granularity = 16384: while loading database default from path /var/lib/clickhouse/metadata/defa

Additional context

No response

[Mokto]

This issue is stale because it has been open for 30 days with no activity.

[patsevanton]

Please use markdown or we won't be able to figure out your issue.