feat(taskbroker): port profiles consumer to taskbroker raw mode

[untitaker]

Replaces the dedicated ingest-profiles consumer with taskbroker raw mode. This must be merged before the next sentry release, as the consumer code is already gone.

We're going to tear down the following consumers in sentry:

• profiles
• process-segments
• all subscriptions consumers
• ingest-replay-recordings

they're not all blocking the next release, though expect more PRs soon :)

ref STREAM-1248

[linear-code]

STREAM-1248

[aminvakil]

Failing integration test is not related to this PR.

[aldy505]

I wonder how would you scale this for high throughput ingest traffic?

[aminvakil]

Notes to myself:

This is a fix for another integration test as sentry nightly does not have / support ingest-profiles anymore.

Failing line:

ingest-profiles-1 | Error: No consumer named ingest-profiles in sentry.consumers.KAFKA_CONSUMERS.

But the unhealthy web is still present.

[untitaker]

I wonder how would you scale this for high throughput ingest traffic?

you can horizontally scale taskworker normally. eventually it will overload taskbroker.

scaling taskbroker horizontally is a bit tricky right now (basically, you have to create a new container with its own volumes and grpc port, then point new taskworkers to it), but we're hoping to fix that too in a few months.

[untitaker]

This is a fix for another integration test as sentry nightly does not have / support ingest-profiles anymore.

sorry about that. we were planning to delete the code after migrating self-hosted (although, not waiting for a self-hosted release) -- we only realized today that profiles was never ported in self-hosted

[aldy505]

This is a fix for another integration test as sentry nightly does not have / support ingest-profiles anymore.

sorry about that. we were planning to delete the code after migrating self-hosted (although, not waiting for a self-hosted release) -- we only realized today that profiles was never ported in self-hosted

oh so this was the culprit for the failing e2e tests (slack msg) all along?

[untitaker]

re scaling, it seems we already have scaling docs. fundamentally nothing changes, only the config format. you can split out the topic into a new, separate taskbroker too if you really want (similar to namespaces)

getsentry/sentry-docs#18550

[untitaker]

oh so this was the culprit for the failing e2e tests (slack msg) all along?

getsentry/sentry#117912 was merged on jun 17th, so possibly. but it then got reverted and only reapplied on jun 22nd, so not sure if it explains every failure.

[aminvakil]

getsentry/sentry#117912 was merged on jun 17th, so possibly. but it then got reverted and only reapplied on jun 22nd, so not sure if it explains every failure.

It doesn't seem so, after merging this PR, integration test failed on this exact commit on master.

[aminvakil]

I think we have to give web service more time to get started and opened #4387 .

My plan is to close / open it multiple times to see if it gets passed all the times :)

[untitaker]

i see two failures:

• process-spans failing to start subprocesses
• snuba-api failing to become healthy

i'm adding some debug logs for process-spans: getsentry/sentry#118444

but it's not clear to me that this is the thing that causes the integration tests to fail. it's just something i found in the logs.