After 24.10.0 version, I losing reporting of performance and cron

[PhilippeThouvenot]

Self-Hosted Version

24.11.1

CPU Architecture

x86_64

Docker Version

26.1.4

Docker Compose Version

2.27.1

Steps to Reproduce

Upgrade after 24.10.0

I've try to search any configuration do modify, but everything seems to be ok.

If someone have an idea, thank you.

Expected Result

Gathering information for cron and performance

Actual Result

Event ID

No response

[aldy505]

Do you have anything suspicious from your Docker logs? sudo docker compose logs --since 3h -t might be helpful.

[PhilippeThouvenot]

Hi, I have lot of clickhouse error :

clickhouse-1 | 2024-12-10T07:00:07.764106608Z 2024.12.10 07:00:07.763950 [ 398023 ] {} ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 107, Net Exception: Socket is not connected, Stack trace (when copying this message, always include the lines below):
clickhouse-1 | 2024-12-10T07:00:07.764126208Z
clickhouse-1 | 2024-12-10T07:00:07.764139508Z 0. Poco::Net::SocketImpl::error(int, String const&) @ 0x0000000015b3dbf2 in /usr/bin/clickhouse
clickhouse-1 | 2024-12-10T07:00:07.764142108Z 1. Poco::Net::SocketImpl::peerAddress() @ 0x0000000015b40376 in /usr/bin/clickhouse
clickhouse-1 | 2024-12-10T07:00:07.764144608Z 2. DB::HTTPServerRequest::HTTPServerRequest(std::shared_ptrDB::IHTTPContext, DB::HTTPServerResponse&, Poco::Net::HTTPServerSession&) @ 0x0000000013154417 in /usr/bin/clickhouse
clickhouse-1 | 2024-12-10T07:00:07.764147208Z 3. DB::HTTPServerConnection::run() @ 0x0000000013152ba4 in /usr/bin/clickhouse
clickhouse-1 | 2024-12-10T07:00:07.764149708Z 4. Poco::Net::TCPServerConnection::start() @ 0x0000000015b42834 in /usr/bin/clickhouse
clickhouse-1 | 2024-12-10T07:00:07.764152408Z 5. Poco::Net::TCPServerDispatcher::run() @ 0x0000000015b43a31 in /usr/bin/clickhouse
clickhouse-1 | 2024-12-10T07:00:07.764166109Z 6. Poco::PooledThread::run() @ 0x0000000015c7a667 in /usr/bin/clickhouse
clickhouse-1 | 2024-12-10T07:00:07.764168409Z 7. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015c7893c in /usr/bin/clickhouse
clickhouse-1 | 2024-12-10T07:00:07.764170809Z 8. ? @ 0x00007f53681df609 in ?
clickhouse-1 | 2024-12-10T07:00:07.764173309Z 9. ? @ 0x00007f5368104353 in ?
clickhouse-1 | 2024-12-10T07:00:07.764175509Z (version 23.8.11.29.altinitystable (altinity build))

or postgres error, but it seem not that :
postgres-1 | 2024-12-10T05:56:06.170730131Z 2024-12-10 05:56:06.170 UTC [1239613] ERROR: duplicate key value violates unique constraint "sentry_groupsubscription_group_id_user_id_62180a04_uniq"
postgres-1 | 2024-12-10T06:01:01.712447970Z 2024-12-10 06:01:01.712 UTC [1240000] ERROR: duplicate key value violates unique constraint "sentry_grouprelease_group_id_release_id_envi_044354c8_uniq"
postgres-1 | 2024-12-10T06:05:07.222325951Z 2024-12-10 06:05:07.222 UTC [1240301] ERROR: duplicate key value violates unique constraint "sentry_groupsubscription_group_id_user_id_62180a04_uniq"
postgres-1 | 2024-12-10T06:11:05.316538037Z 2024-12-10 06:11:05.316 UTC [1240690] ERROR: duplicate key value violates unique constraint "sentry_groupsubscription_group_id_user_id_62180a04_uniq"
postgres-1 | 2024-12-10T06:21:02.293918255Z 2024-12-10 06:21:02.293 UTC [1241379] ERROR: duplicate key value violates unique constraint "sentry_environmentprojec_project_id_environment_i_91da82f2_uniq"
postgres-1 | 2024-12-10T06:24:04.243944115Z 2024-12-10 06:24:04.243 UTC [1241577] ERROR: duplicate key value violates unique constraint "sentry_grouprelease_group_id_release_id_envi_044354c8_uniq"
postgres-1 | 2024-12-10T06:24:04.351743380Z 2024-12-10 06:24:04.351 UTC [1241578] ERROR: duplicate key value violates unique constraint "sentry_grouprelease_group_id_release_id_envi_044354c8_uniq"
postgres-1 | 2024-12-10T06:26:02.338134272Z 2024-12-10 06:26:02.338 UTC [1241717] ERROR: duplicate key value violates unique constraint "sentry_grouprelease_group_id_release_id_envi_044354c8_uniq"
postgres-1 | 2024-12-10T06:27:01.629699032Z 2024-12-10 06:27:01.629 UTC [1241786] ERROR: duplicate key value violates unique constraint "sentry_grouprelease_group_id_release_id_envi_044354c8_uniq"
postgres-1 | 2024-12-10T06:28:01.639768827Z 2024-12-10 06:28:01.639 UTC [1241848] ERROR: duplicate key value violates unique constraint "sentry_grouprelease_group_id_release_id_envi_044354c8_uniq"
postgres-1 | 2024-12-10T06:30:05.253274603Z 2024-12-10 06:30:05.253 UTC [1242102] ERROR: duplicate key value violates unique constraint "sentry_groupsubscription_group_id_user_id_62180a04_uniq"
postgres-1 | 2024-12-10T06:33:02.112716224Z 2024-12-10 06:33:02.112 UTC [1242355] ERROR: duplicate key value violates unique constraint "sentry_grouprelease_group_id_release_id_envi_044354c8_uniq"
postgres-1 | 2024-12-10T06:33:05.931711254Z 2024-12-10 06:33:05.931 UTC [1242360] ERROR: duplicate key value violates unique constraint "sentry_grouprelease_group_id_release_id_envi_044354c8_uniq"
postgres-1 | 2024-12-10T06:33:06.052912864Z 2024-12-10 06:33:06.052 UTC [1242361] ERROR: duplicate key value violates unique constraint "sentry_grouprelease_group_id_release_id_envi_044354c8_uniq"
postgres-1 | 2024-12-10T06:33:06.185899669Z 2024-12-10 06:33:06.185 UTC [1242370] ERROR: duplicate key value violates unique constraint "sentry_grouprelease_group_id_release_id_envi_044354c8_uniq"
postgres-1 | 2024-12-10T06:33:06.313533386Z 2024-12-10 06:33:06.313 UTC [1242371] ERROR: duplicate key value violates unique constraint "sentry_grouprelease_group_id_release_id_envi_044354c8_uniq"
postgres-1 | 2024-12-10T06:38:06.466197251Z 2024-12-10 06:38:06.466 UTC [1242705] ERROR: duplicate key value violates unique constraint "sentry_environmentprojec_project_id_environment_i_91da82f2_uniq"
postgres-1 | 2024-12-10T06:39:06.860344925Z 2024-12-10 06:39:06.860 UTC [1242772] ERROR: duplicate key value violates unique constraint "sentry_grouprelease_group_id_release_id_envi_044354c8_uniq"
postgres-1 | 2024-12-10T06:40:05.227810362Z 2024-12-10 06:40:05.227 UTC [1242833] ERROR: duplicate key value violates unique constraint "sentry_groupsubscription_group_id_user_id_62180a04_uniq"
postgres-1 | 2024-12-10T06:45:04.219327509Z 2024-12-10 06:45:04.219 UTC [1243245] ERROR: duplicate key value violates unique constraint "sentry_organizationonboa_organization_id_task_e08c2f3d_uniq"
postgres-1 | 2024-12-10T06:45:04.220947536Z 2024-12-10 06:45:04.220 UTC [1243247] ERROR: duplicate key value violates unique constraint "sentry_organizationonboa_organization_id_task_e08c2f3d_uniq"
postgres-1 | 2024-12-10T06:51:02.257320582Z 2024-12-10 06:51:02.257 UTC [1243620] ERROR: duplicate key value violates unique constraint "sentry_environmentprojec_project_id_environment_i_91da82f2_uniq"
postgres-1 | 2024-12-10T06:53:04.515530455Z 2024-12-10 06:53:04.515 UTC [1243760] ERROR: duplicate key value violates unique constraint "sentry_grouprelease_group_id_release_id_envi_044354c8_uniq"
postgres-1 | 2024-12-10T06:56:04.105247170Z 2024-12-10 06:56:04.105 UTC [1243938] ERROR: duplicate key value violates unique constraint "sentry_grouprelease_group_id_release_id_envi_044354c8_uniq"
postgres-1 | 2024-12-10T06:57:01.548894174Z 2024-12-10 06:57:01.548 UTC [1244003] ERROR: duplicate key value violates unique constraint "sentry_environmentprojec_project_id_environment_i_91da82f2_uniq"
postgres-1 | 2024-12-10T07:00:05.123142843Z 2024-12-10 07:00:05.122 UTC [1244323] ERROR: duplicate key value violates unique constraint "sentry_grouprelease_group_id_release_id_envi_044354c8_uniq"
postgres-1 | 2024-12-10T07:01:01.652934990Z 2024-12-10 07:01:01.652 UTC [1244377] ERROR: duplicate key value violates unique constraint "sentry_grouprelease_group_id_release_id_envi_044354c8_uniq"
postgres-1 | 2024-12-10T07:01:05.315759538Z 2024-12-10 07:01:05.315 UTC [1244395] ERROR: duplicate key value violates unique constraint "sentry_groupsubscription_group_id_user_id_62180a04_uniq"
postgres-1 | 2024-12-10T07:01:08.356130834Z 2024-12-10 07:01:08.356 UTC [1244421] ERROR: duplicate key value violates unique constraint "sentry_groupsubscription_group_id_user_id_62180a04_uniq"

[aldy505]

Most likely related to getsentry/snuba#5707

Would you mind trying out this solution getsentry/snuba#5707 (comment) and then re-running ./install.sh script again?

[PhilippeThouvenot]

Hi,
no more errors for clickhouse with consumer.
But the problem is already here.
I can see the call when i load a page but nothing in performance :
nginx-1 | 2024-12-11T07:21:46.768765918Z - - [11/Dec/2024:07:21:46 +0000] "POST /api/9/envelope/ HTTP/1.0" 200 41 "-" "sentry.php.laravel/4.10.1"

[PhilippeThouvenot]

I have found the problem, it occurs when I start the project with .env.custom : docker compose --env-file .env.custom up -d

I tried with no diff between .env and .env.custom and it exactly the same :

The up with .env.custom :
[+] Running 27/27
✔ Network sentry-self-hosted_default Created 0.1s
✔ Container sentry-self-hosted-postgres-1 Healthy 32.8s
✔ Container sentry-self-hosted-smtp-1 Started 2.0s
✔ Container sentry-self-hosted-symbolicator-cleanup-1 Started 1.8s
✔ Container sentry-self-hosted-geoipupdate-1 Started 1.5s
✔ Container sentry-self-hosted-kafka-1 Healthy 35.3s
✔ Container sentry-self-hosted-clickhouse-1 Healthy 11.8s
✔ Container sentry-self-hosted-symbolicator-1 Started 1.3s
✔ Container sentry-self-hosted-redis-1 Healthy 35.3s
✔ Container sentry-self-hosted-memcached-1 Started 2.1s
✔ Container sentry-self-hosted-snuba-subscription-consumer-events-1 Started 32.2s
✔ Container sentry-self-hosted-snuba-outcomes-billing-consumer-1 Started 31.9s
✔ Container sentry-self-hosted-snuba-replacer-1 Started 32.1s
✔ Container sentry-self-hosted-snuba-errors-consumer-1 Started 31.7s
✔ Container sentry-self-hosted-snuba-outcomes-consumer-1 Started 31.8s
✔ Container sentry-self-hosted-snuba-api-1 Started 32.1s
✔ Container sentry-self-hosted-snuba-group-attributes-consumer-1 Started 31.8s
✔ Container sentry-self-hosted-subscription-consumer-events-1 Started 34.3s
✔ Container sentry-self-hosted-worker-1 Started 34.7s
✔ Container sentry-self-hosted-attachments-consumer-1 Started 34.8s
✔ Container sentry-self-hosted-cron-1 Started 34.9s
✔ Container sentry-self-hosted-post-process-forwarder-errors-1 Started 34.4s
✔ Container sentry-self-hosted-web-1 Healthy 75.9s
✔ Container sentry-self-hosted-sentry-cleanup-1 Started 33.9s
✔ Container sentry-self-hosted-events-consumer-1 Started 34.5s
✔ Container sentry-self-hosted-relay-1 Started 76.2s
✔ Container sentry-self-hosted-nginx-1 Started

The up with .env :
[+] Running 58/58
✔ Network sentry-self-hosted_default Created 0.1s
✔ Container sentry-self-hosted-symbolicator-cleanup-1 Started 2.0s
✔ Container sentry-self-hosted-geoipupdate-1 Started 1.7s
✔ Container sentry-self-hosted-clickhouse-1 Healthy 11.6s
✔ Container sentry-self-hosted-postgres-1 Healthy 36.1s
✔ Container sentry-self-hosted-memcached-1 Started 1.6s
✔ Container sentry-self-hosted-kafka-1 Healthy 46.8s
✔ Container sentry-self-hosted-redis-1 Healthy 46.8s
✔ Container sentry-self-hosted-vroom-cleanup-1 Started 1.2s
✔ Container sentry-self-hosted-symbolicator-1 Started 1.5s
✔ Container sentry-self-hosted-smtp-1 Started 1.6s
✔ Container sentry-self-hosted-vroom-1 Started 12.8s
✔ Container sentry-self-hosted-snuba-subscription-consumer-transactions-1 Started 33.9s
✔ Container sentry-self-hosted-snuba-profiling-functions-consumer-1 Started 36.5s
✔ Container sentry-self-hosted-snuba-replays-consumer-1 Started 33.4s
✔ Container sentry-self-hosted-snuba-generic-metrics-sets-consumer-1 Started 35.2s
✔ Container sentry-self-hosted-snuba-replacer-1 Started 33.7s
✔ Container sentry-self-hosted-snuba-subscription-consumer-events-1 Started 32.4s
✔ Container sentry-self-hosted-snuba-transactions-consumer-1 Started 35.0s
✔ Container sentry-self-hosted-snuba-metrics-consumer-1 Started 35.6s
✔ Container sentry-self-hosted-snuba-generic-metrics-counters-consumer-1 Started 36.3s
✔ Container sentry-self-hosted-snuba-profiling-profiles-consumer-1 Started 36.2s
✔ Container sentry-self-hosted-snuba-generic-metrics-distributions-consumer-1 Started 34.8s
✔ Container sentry-self-hosted-snuba-issue-occurrence-consumer-1 Started 33.2s
✔ Container sentry-self-hosted-snuba-spans-consumer-1 Started 37.0s
✔ Container sentry-self-hosted-snuba-generic-metrics-gauges-consumer-1 Started 32.6s
✔ Container sentry-self-hosted-snuba-errors-consumer-1 Started 32.9s
✔ Container sentry-self-hosted-snuba-group-attributes-consumer-1 Started 37.2s
✔ Container sentry-self-hosted-snuba-api-1 Started 35.4s
✔ Container sentry-self-hosted-snuba-outcomes-billing-consumer-1 Started 33.0s
✔ Container sentry-self-hosted-snuba-subscription-consumer-metrics-1 Started 35.6s
✔ Container sentry-self-hosted-snuba-outcomes-consumer-1 Started 32.7s
✔ Container sentry-self-hosted-ingest-occurrences-1 Started 50.5s
✔ Container sentry-self-hosted-ingest-monitors-1 Started 49.3s
✔ Container sentry-self-hosted-monitors-clock-tick-1 Started 47.1s
✔ Container sentry-self-hosted-ingest-replay-recordings-1 Started 46.9s
✔ Container sentry-self-hosted-subscription-consumer-metrics-1 Started 46.9s
✔ Container sentry-self-hosted-ingest-feedback-events-1 Started 47.5s
✔ Container sentry-self-hosted-cron-1 Started 51.6s
✔ Container sentry-self-hosted-subscription-consumer-generic-metrics-1 Started 48.1s
✔ Container sentry-self-hosted-subscription-consumer-events-1 Started 40.1s
✔ Container sentry-self-hosted-billing-metrics-consumer-1 Started 51.9s
✔ Container sentry-self-hosted-subscription-consumer-transactions-1 Started 48.9s
✔ Container sentry-self-hosted-post-process-forwarder-issue-platform-1 Started 45.6s
✔ Container sentry-self-hosted-ingest-profiles-1 Started 43.0s
✔ Container sentry-self-hosted-events-consumer-1 Started 45.9s
✔ Container sentry-self-hosted-sentry-cleanup-1 Started 48.0s
✔ Container sentry-self-hosted-transactions-consumer-1 Started 40.9s
✔ Container sentry-self-hosted-worker-1 Started 48.2s
✔ Container sentry-self-hosted-web-1 Healthy 196.9s
✔ Container sentry-self-hosted-post-process-forwarder-transactions-1 Started 50.3s
✔ Container sentry-self-hosted-attachments-consumer-1 Started 51.9s
✔ Container sentry-self-hosted-post-process-forwarder-errors-1 Started 45.9s
✔ Container sentry-self-hosted-generic-metrics-consumer-1 Started 51.8s
✔ Container sentry-self-hosted-monitors-clock-tasks-1 Started 47.7s
✔ Container sentry-self-hosted-metrics-consumer-1 Started 49.2s
✔ Container sentry-self-hosted-relay-1 Started 197.3s
✔ Container sentry-self-hosted-nginx-1 Started

[aldy505]

Oh I can see why.

Most likely, you don't have this on your .env.custom file. Or.. you have this set as "errors-only".

self-hosted/.env

Line 2 in 1f63d4d

COMPOSE_PROFILES=feature-complete

[PhilippeThouvenot]

In the documentation we can found this :
docker compose --env-file .env --env-file .env.custom up -d

The end of the installer say this :
docker compose --env-file .env.custom up -d

Just update the information, and this will be ok, I guess,