Redundancy for a self-host installation running on a linux VM on Azure

[joaoantunessitedocs]

Self-Hosted Version

27.5.1

CPU Architecture

x86_64

Docker Version

27.5.1

Docker Compose Version

v2.32.4

Machine Specification

• My system meets the minimum system requirements of Sentry

Steps to Reproduce

Greetings

I currently have a self-hosted installation running on azure virtual machine with linux OS. I couldn't see any reference about it through the official documentation but I'm wondering if there is any reference architecture and configs for azure that might be use in order to implement redundancy on self-hosted infrastructure, mainly for the production workload.

Thanks for your support!
João Antunes

Expected Result

Scale virtual machines to get more instances available.

Actual Result

Can't scale the virtual machines in order to handle with the high traffic demand.

Event ID

No response

[aldy505]

I'm wondering if there is any reference architecture and configs for azure that might be use in order to implement redundancy on self-hosted infrastructure, mainly for the production workload.

There is not any. But, we can still help you. You will need to be more specific on what you already have and what kind of "production scale" that you want.

I wouldn't recommend the Sentry helm chart, since it easily gets outdated, but I can suggest you a few things (obviously after reading the entire documentation on develop docs):

• It is really recommended to use external storage like S3 and GCS. Sadly Azure Blob Storage is not supported (for obvious reasons). https://develop.sentry.dev/self-hosted/experimental/external-storage/ -- But! There are workarounds: https://github.com/gaul/s3proxy or https://azuremarketplace.microsoft.com/en-us/marketplace/apps/flexify.azure-s3-api?tab=overview (I don't use both, since I only use Azure for specific stuff other than these kind of stuffs).
  • Also use S3 for nodestore, see https://github.com/stayallive/sentry-nodestore-s3 and feat: Use S3 node store with garage #3498 (please do not use Garage directly)
• Put a reverse proxy in front of your entire docker compose mess https://develop.sentry.dev/self-hosted/experimental/reverse-proxy/
• Use an external Kafka Introduce patches with external kafka #3521 -- or swap Kafka with Redpanda

If what I said to you seems vague. Don't hesitate to ask me in great details.

You might also want to read this: https://develop.sentry.dev/application-architecture/overview/

[joaoantunessitedocs]

Thanks for all awesome explanations @aldy505. there is a couple things that i've definitely to check on my side too. At this point, I'm trying to understand if there is a way to set up a full sentry self hosted installation within more than 1 virtual machine instance in order to handle the high volume of traffic.

following some architecture like this:

Thanks in advance.
João Antunes

[aldy505]

Hi. I'll be occupied with my full time job, will give a full reply on Friday, if that's okay with you.

The quick answer: Deploying self-hosted Sentry with more than one VM will be tricky. The "shared space" will be lots of component: Kafka, Postgres, Redis, ClickHouse, Memcached, and s3proxy + Azure Blob. If you are a one-man-show that will manage all this, I'd suggest the regular Docker Compose + external storage. But if you're not, well, I'll draw some alternatives for you.

[aldy505]

Pretty similar to what you've drawn, but I want to highlight that the "shared space" is not that simple. This is the Docker Compose that you want for "Sentry App Virtual Machine" https://gist.github.com/aldy505/50e47b29332f191d539b9ff21e891be3 -- I would also recommend to use Redpanda instead of Kafka, they use the same API: https://docs.redpanda.com/current/home/ and the best thing is: it requires waaaay less resource.

Another option if you want more throughput is to put a distributed Load Balancer that has Relay running in Proxy Mode (https://docs.sentry.io/product/relay/modes/#proxy-mode). This is useful if you have multiple regions (availability zones that are not so close with each other), therefore the client's request would be routed to the closest endpoint. Please also see this: https://develop.sentry.dev/self-hosted/experimental/reverse-proxy/#expose-only-ingest-endpoint-publicly

Let me know if you have further questions.