ref(profiles): Remove the ingest-profiles Arroyo consumer#117912
Merged
Conversation
The profiles consumer has been replaced by taskbroker passthrough mode (STREAM-1041), which consumes the profiles topic directly and dispatches process_profile_from_kafka tasks. This removes the now-dead Arroyo consumer: - Drop the ingest-profiles entry from KAFKA_CONSUMERS - Remove ProcessProfileStrategyFactory and the process_message Arroyo wrapper, keeping the shared passthrough logic (renamed _process_profile_message -> process_profile_message) and simplifying header handling to the dict-only path the passthrough uses - Update tests to exercise process_profile_message directly The profiling.killswitch.ingest-profiles option is retained since it is still enforced in the passthrough path. ref STREAM-1194
With the consumer gone, the profiles/consumers package had only the taskbroker-passthrough glue left. Fold it into the task module: - Delete the sentry.profiles.consumers package; move the killswitch and sampling helpers (_should_drop / _is_sampled) into task.py, inlined into process_profile_from_kafka (always inline, no .delay()). - Demote process_profile_task from an @instrumented_task to a plain function. It is no longer dispatched as its own task (only called inline from the passthrough task), so the decorator and the ingest_profiling_tasks namespace are no longer needed. The namespace definition is retained for now since removing it requires coordinated getsentry worker-config changes. - Move the relocated tests into tests/sentry/profiles/test_task.py. ref STREAM-1194
With process_profile_task demoted to a plain function, no tasks remain in the ingest.profiling namespace (the live passthrough task uses ingest.profiling.passthrough). Nothing else references it, so remove it. ref STREAM-1194
evanh
approved these changes
Jun 17, 2026
Contributor
|
PR reverted: 0508e3e |
Contributor
Backend Test FailuresFailures on
|
untitaker
added a commit
that referenced
this pull request
Jun 22, 2026
Re-applies #117912, which was reverted shortly after merging due to an issue in the US2 region. That has since been resolved, so this is safe to land. ## Conflict resolution vs original One conflict in `task.py`: PRs #118091 and #118100 landed after the revert and bumped `processing_deadline_duration` and renamed producers on `process_profile_from_kafka`. Those changes are preserved. The decorator on `process_profile_task` is removed as originally intended. ref STREAM-1210: https://linear.app/getsentry/issue/STREAM-1210/re-apply-the-pr-that-deletes-profiles-consumer --- [View Session in Sentry](https://sentry.sentry.io/traces/?project=4510944073809921&query=gen_ai.conversation.id%3A%22slack%3AD0B0XMY6PHC%3A1781098448.772909%22) Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com>
sehr-m
pushed a commit
that referenced
this pull request
Jun 23, 2026
The `ingest-profiles` Arroyo consumer has been replaced by taskbroker passthrough mode (STREAM-1041), which consumes the `profiles` topic directly and dispatches `process_profile_from_kafka` tasks. This removes the now-dead consumer and the leftover glue around it. ## Changes - Drop the `ingest-profiles` entry from `KAFKA_CONSUMERS`. - Delete the `sentry.profiles.consumers` package entirely (`ProcessProfileStrategyFactory`, the Arroyo `process_message` wrapper, and the killswitch/sampling helpers). - Move the killswitch and sampling helpers (`_should_drop` / `_is_sampled`) into `task.py`, inlined into `process_profile_from_kafka` (always inline — no `.delay()` indirection, `dict`-only headers). - Demote `process_profile_task` from an `@instrumented_task` to a plain function. It is no longer dispatched as its own task (only called inline from the passthrough task). - Remove the now-empty `ingest.profiling` namespace. No tasks remain in it — the live passthrough task uses `ingest.profiling.passthrough` — and nothing else references it. - Update tests: move the relocated cases into `tests/sentry/profiles/test_task.py`, drop the profiles backpressure tests (that step no longer exists), and remove `ingest-profiles` from the no-DLQ exemption list. The `profiling.killswitch.ingest-profiles` option is retained since it is still enforced in `process_profile_from_kafka`. The `devservices` taskbroker passthrough entries (the replacement) are unchanged. ref STREAM-1194
sehr-m
pushed a commit
that referenced
this pull request
Jun 23, 2026
Re-applies #117912, which was reverted shortly after merging due to an issue in the US2 region. That has since been resolved, so this is safe to land. ## Conflict resolution vs original One conflict in `task.py`: PRs #118091 and #118100 landed after the revert and bumped `processing_deadline_duration` and renamed producers on `process_profile_from_kafka`. Those changes are preserved. The decorator on `process_profile_task` is removed as originally intended. ref STREAM-1210: https://linear.app/getsentry/issue/STREAM-1210/re-apply-the-pr-that-deletes-profiles-consumer --- [View Session in Sentry](https://sentry.sentry.io/traces/?project=4510944073809921&query=gen_ai.conversation.id%3A%22slack%3AD0B0XMY6PHC%3A1781098448.772909%22) Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
ingest-profilesArroyo consumer has been replaced by taskbroker passthrough mode (STREAM-1041), which consumes theprofilestopic directly and dispatchesprocess_profile_from_kafkatasks. This removes the now-dead consumer and the leftover glue around it.Changes
ingest-profilesentry fromKAFKA_CONSUMERS.sentry.profiles.consumerspackage entirely (ProcessProfileStrategyFactory, the Arroyoprocess_messagewrapper, and the killswitch/sampling helpers)._should_drop/_is_sampled) intotask.py, inlined intoprocess_profile_from_kafka(always inline — no.delay()indirection,dict-only headers).process_profile_taskfrom an@instrumented_taskto a plain function. It is no longer dispatched as its own task (only called inline from the passthrough task).ingest.profilingnamespace. No tasks remain in it — the live passthrough task usesingest.profiling.passthrough— and nothing else references it.tests/sentry/profiles/test_task.py, drop the profiles backpressure tests (that step no longer exists), and removeingest-profilesfrom the no-DLQ exemption list.The
profiling.killswitch.ingest-profilesoption is retained since it is still enforced inprocess_profile_from_kafka. Thedevservicestaskbroker passthrough entries (the replacement) are unchanged.ref STREAM-1194