Skip to content

ref(profiles): Remove the ingest-profiles Arroyo consumer#117912

Merged
untitaker merged 3 commits into
masterfrom
remove-profiles-consumer
Jun 17, 2026
Merged

ref(profiles): Remove the ingest-profiles Arroyo consumer#117912
untitaker merged 3 commits into
masterfrom
remove-profiles-consumer

Conversation

@untitaker

@untitaker untitaker commented Jun 17, 2026

Copy link
Copy Markdown
Member

The ingest-profiles Arroyo consumer has been replaced by taskbroker passthrough mode (STREAM-1041), which consumes the profiles topic directly and dispatches process_profile_from_kafka tasks. This removes the now-dead consumer and the leftover glue around it.

Changes

  • Drop the ingest-profiles entry from KAFKA_CONSUMERS.
  • Delete the sentry.profiles.consumers package entirely (ProcessProfileStrategyFactory, the Arroyo process_message wrapper, and the killswitch/sampling helpers).
  • Move the killswitch and sampling helpers (_should_drop / _is_sampled) into task.py, inlined into process_profile_from_kafka (always inline — no .delay() indirection, dict-only headers).
  • Demote process_profile_task from an @instrumented_task to a plain function. It is no longer dispatched as its own task (only called inline from the passthrough task).
  • Remove the now-empty ingest.profiling namespace. No tasks remain in it — the live passthrough task uses ingest.profiling.passthrough — and nothing else references it.
  • Update tests: move the relocated cases into tests/sentry/profiles/test_task.py, drop the profiles backpressure tests (that step no longer exists), and remove ingest-profiles from the no-DLQ exemption list.

The profiling.killswitch.ingest-profiles option is retained since it is still enforced in process_profile_from_kafka. The devservices taskbroker passthrough entries (the replacement) are unchanged.

ref STREAM-1194

The profiles consumer has been replaced by taskbroker passthrough mode
(STREAM-1041), which consumes the profiles topic directly and dispatches
process_profile_from_kafka tasks. This removes the now-dead Arroyo
consumer:

- Drop the ingest-profiles entry from KAFKA_CONSUMERS
- Remove ProcessProfileStrategyFactory and the process_message Arroyo
  wrapper, keeping the shared passthrough logic (renamed
  _process_profile_message -> process_profile_message) and simplifying
  header handling to the dict-only path the passthrough uses
- Update tests to exercise process_profile_message directly

The profiling.killswitch.ingest-profiles option is retained since it is
still enforced in the passthrough path.

ref STREAM-1194
@untitaker untitaker requested review from a team as code owners June 17, 2026 13:46
@linear-code

linear-code Bot commented Jun 17, 2026

Copy link
Copy Markdown

STREAM-1194

@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Jun 17, 2026
With the consumer gone, the profiles/consumers package had only the
taskbroker-passthrough glue left. Fold it into the task module:

- Delete the sentry.profiles.consumers package; move the killswitch and
  sampling helpers (_should_drop / _is_sampled) into task.py, inlined
  into process_profile_from_kafka (always inline, no .delay()).
- Demote process_profile_task from an @instrumented_task to a plain
  function. It is no longer dispatched as its own task (only called
  inline from the passthrough task), so the decorator and the
  ingest_profiling_tasks namespace are no longer needed. The namespace
  definition is retained for now since removing it requires coordinated
  getsentry worker-config changes.
- Move the relocated tests into tests/sentry/profiles/test_task.py.

ref STREAM-1194
With process_profile_task demoted to a plain function, no tasks remain
in the ingest.profiling namespace (the live passthrough task uses
ingest.profiling.passthrough). Nothing else references it, so remove it.

ref STREAM-1194
@untitaker untitaker requested a review from a team as a code owner June 17, 2026 14:00
@untitaker untitaker merged commit e583f4f into master Jun 17, 2026
86 checks passed
@untitaker untitaker deleted the remove-profiles-consumer branch June 17, 2026 19:00
@untitaker untitaker added the Trigger: Revert Add to a merged PR to revert it (skips CI) label Jun 18, 2026
@getsentry-bot

Copy link
Copy Markdown
Contributor

PR reverted: 0508e3e

getsentry-bot added a commit that referenced this pull request Jun 18, 2026
…17912)"

This reverts commit e583f4f.

Co-authored-by: untitaker <837573+untitaker@users.noreply.github.com>
@github-actions

github-actions Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Backend Test Failures

Failures on e583f4f in this run:

tests/snuba/api/endpoints/test_organization_group_index_stats.py::GroupListTest::test_platform_tag_collision_badge_resolved_correctlylog
[gw1] linux -- Python 3.13.1 /home/runner/work/sentry/sentry/.venv/bin/python3
tests/snuba/api/endpoints/test_organization_group_index_stats.py:294: in test_platform_tag_collision_badge_resolved_correctly
    assert response.data[0]["filtered"]["count"] == "3"
E   AssertionError: assert '0' == '3'
E     
E     �[0m�[91m- 3�[39;49;00m�[90m�[39;49;00m
E     �[92m+ 0�[39;49;00m�[90m�[39;49;00m
tests/sentry/issues/endpoints/test_group_events.py::GroupEventsTest::test_platform_tag_collision_resolved_correctlylog
[gw0] linux -- Python 3.13.1 /home/runner/work/sentry/sentry/.venv/bin/python3
tests/sentry/issues/endpoints/test_group_events.py:642: in test_platform_tag_collision_resolved_correctly
    assert len(response.data) == 3
E   assert 0 == 3
E    +  where 0 = len([])
E    +    where [] = <Response status_code=200, "application/json">.data
tests/sentry/issues/endpoints/test_group_event_details.py::GroupEventDetailsHelpfulEndpointTest::test_platform_tag_collision_resolved_correctlylog
[gw0] linux -- Python 3.13.1 /home/runner/work/sentry/sentry/.venv/bin/python3
tests/sentry/issues/endpoints/test_group_event_details.py:580: in test_platform_tag_collision_resolved_correctly
    assert response.status_code == 200, response.content
E   AssertionError: b'{"detail":"No matching event found. Try changing the environments, date range, or query."}'
E   assert 404 == 200
E    +  where 404 = <Response status_code=404, "application/json">.status_code

untitaker added a commit that referenced this pull request Jun 22, 2026
Re-applies #117912, which was reverted shortly after merging due to an
issue in the US2 region. That has since been resolved, so this is safe
to land.

## Conflict resolution vs original

One conflict in `task.py`: PRs #118091 and #118100 landed after the
revert and bumped `processing_deadline_duration` and renamed producers
on `process_profile_from_kafka`. Those changes are preserved. The
decorator on `process_profile_task` is removed as originally intended.

ref STREAM-1210:
https://linear.app/getsentry/issue/STREAM-1210/re-apply-the-pr-that-deletes-profiles-consumer

---
[View Session in
Sentry](https://sentry.sentry.io/traces/?project=4510944073809921&query=gen_ai.conversation.id%3A%22slack%3AD0B0XMY6PHC%3A1781098448.772909%22)

Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com>
sehr-m pushed a commit that referenced this pull request Jun 23, 2026
The `ingest-profiles` Arroyo consumer has been replaced by taskbroker
passthrough mode (STREAM-1041), which consumes the `profiles` topic
directly and dispatches `process_profile_from_kafka` tasks. This removes
the now-dead consumer and the leftover glue around it.

## Changes

- Drop the `ingest-profiles` entry from `KAFKA_CONSUMERS`.
- Delete the `sentry.profiles.consumers` package entirely
(`ProcessProfileStrategyFactory`, the Arroyo `process_message` wrapper,
and the killswitch/sampling helpers).
- Move the killswitch and sampling helpers (`_should_drop` /
`_is_sampled`) into `task.py`, inlined into `process_profile_from_kafka`
(always inline — no `.delay()` indirection, `dict`-only headers).
- Demote `process_profile_task` from an `@instrumented_task` to a plain
function. It is no longer dispatched as its own task (only called inline
from the passthrough task).
- Remove the now-empty `ingest.profiling` namespace. No tasks remain in
it — the live passthrough task uses `ingest.profiling.passthrough` — and
nothing else references it.
- Update tests: move the relocated cases into
`tests/sentry/profiles/test_task.py`, drop the profiles backpressure
tests (that step no longer exists), and remove `ingest-profiles` from
the no-DLQ exemption list.

The `profiling.killswitch.ingest-profiles` option is retained since it
is still enforced in `process_profile_from_kafka`. The `devservices`
taskbroker passthrough entries (the replacement) are unchanged.

ref STREAM-1194
sehr-m pushed a commit that referenced this pull request Jun 23, 2026
…17912)"

This reverts commit e583f4f.

Co-authored-by: untitaker <837573+untitaker@users.noreply.github.com>
sehr-m pushed a commit that referenced this pull request Jun 23, 2026
Re-applies #117912, which was reverted shortly after merging due to an
issue in the US2 region. That has since been resolved, so this is safe
to land.

## Conflict resolution vs original

One conflict in `task.py`: PRs #118091 and #118100 landed after the
revert and bumped `processing_deadline_duration` and renamed producers
on `process_profile_from_kafka`. Those changes are preserved. The
decorator on `process_profile_task` is removed as originally intended.

ref STREAM-1210:
https://linear.app/getsentry/issue/STREAM-1210/re-apply-the-pr-that-deletes-profiles-consumer

---
[View Session in
Sentry](https://sentry.sentry.io/traces/?project=4510944073809921&query=gen_ai.conversation.id%3A%22slack%3AD0B0XMY6PHC%3A1781098448.772909%22)

Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components Trigger: Revert Add to a merged PR to revert it (skips CI)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants