Skip to content

Commit

Permalink
Merge branch 'master' into patch-1
Browse files Browse the repository at this point in the history
  • Loading branch information
BYK authored Dec 21, 2024
2 parents d262658 + aebe554 commit 0a6ae17
Show file tree
Hide file tree
Showing 37 changed files with 445 additions and 315 deletions.
7 changes: 6 additions & 1 deletion .env
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@
COMPOSE_PROJECT_NAME=sentry-self-hosted
# Set COMPOSE_PROFILES to "feature-complete" to enable all features
# To enable errors monitoring only, set COMPOSE_PROFILES=errors-only
# See https://develop.sentry.dev/self-hosted/experimental/errors-only/
COMPOSE_PROFILES=feature-complete
SENTRY_EVENT_RETENTION_DAYS=90
# You can either use a port number or an IP:PORT combo for SENTRY_BIND
# See https://docs.docker.com/compose/compose-file/#ports for more
Expand All @@ -10,10 +14,11 @@ SNUBA_IMAGE=getsentry/snuba:nightly
RELAY_IMAGE=getsentry/relay:nightly
SYMBOLICATOR_IMAGE=getsentry/symbolicator:nightly
VROOM_IMAGE=getsentry/vroom:nightly
WAL2JSON_VERSION=latest
HEALTHCHECK_INTERVAL=30s
HEALTHCHECK_TIMEOUT=1m30s
HEALTHCHECK_RETRIES=10
# Caution: Raising max connections of postgres increases CPU and RAM usage
# see https://github.com/getsentry/self-hosted/pull/2740 for more information
POSTGRES_MAX_CONNECTIONS=100
# Set SETUP_JS_SDK_ASSETS to 1 to enable the setup of JS SDK assets
# SETUP_JS_SDK_ASSETS=1
18 changes: 16 additions & 2 deletions .github/ISSUE_TEMPLATE/problem-report.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ body:
id: docker_version
attributes:
label: Docker Version
placeholder: 20.10.16 ← should look like this
placeholder: 20.10.16 ← should look like this (docker --version)
description: |
What version of docker are you using to run self-hosted?
e.g: (docker --version)
Expand All @@ -39,6 +39,16 @@ body:
e.g: (docker compose version)
validations:
required: true
- type: checkboxes
id: machine_specification
attributes:
label: Machine Specification
description: Make sure your system meets the [minimum system requirements of Sentry](https://develop.sentry.dev/self-hosted/#required-minimum-system-resources).
options:
- label: My system meets the minimum system requirements of Sentry
required: true
validations:
required: true
- type: textarea
id: repro
attributes:
Expand All @@ -54,6 +64,8 @@ body:
id: expected
attributes:
label: Expected Result
description: |
What did you expect to happen?
validations:
required: true
- type: textarea
Expand All @@ -70,7 +82,7 @@ body:
- logs output
validations:
required: true
- type: textarea
- type: input
id: event_id
attributes:
label: Event ID
Expand All @@ -82,5 +94,7 @@ body:
value: |-
## Thanks
Check our [triage docs](https://open.sentry.io/triage/) for what to expect next.
If you're reporting a security issue, please follow our [security policy](https://github.com/getsentry/.github/blob/main/SECURITY.md) instead.
validations:
required: false
3 changes: 0 additions & 3 deletions .github/ISSUE_TEMPLATE/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,8 @@ body:
[previous YY.M.N](https://github.com/getsentry/self-hosted/issues) | ***YY.M.N*** | [next YY.M.N](https://github.com/getsentry/self-hosted/issues)
- [ ] Release all components (_replace items with [publish repo issue links](https://github.com/getsentry/publish/issues)_).
- [ ] [`develop`](https://github.com/getsentry/develop/actions/workflows/prepare-release.yml)
- [ ] [`relay`](https://github.com/getsentry/relay/actions/workflows/release_binary.yml)
- [ ] [`sentry`](https://github.com/getsentry/sentry/actions/workflows/release.yml)
- [ ] [`sentry-docs`](https://github.com/getsentry/sentry-docs/actions/workflows/prepare-release.yml)
- [ ] [`snuba`](https://github.com/getsentry/snuba/actions/workflows/release.yml)
- [ ] [`symbolicator`](https://github.com/getsentry/symbolicator/actions/workflows/release.yml)
- [ ] [`vroom`](https://github.com/getsentry/vroom/actions/workflows/release.yaml)
Expand All @@ -25,7 +23,6 @@ body:
- [ ] Follow up.
- [ ] [Create the next release issue](https://github.com/getsentry/self-hosted/issues/new?assignees=&labels=&projects=&template=release.yml) and link it from this one.
- _replace with link_
- [ ] Update the [quarterly ticket](https://github.com/getsentry/team-ospo/issues).
- [ ] Update the [release issue template](https://github.com/getsentry/self-hosted/blob/master/.github/ISSUE_TEMPLATE/release.yml).
- [ ] Create a PR to update relocation release tests to add the new version.
- _replace with link_
Expand Down
11 changes: 8 additions & 3 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,15 +19,21 @@ jobs:
runs-on: ubuntu-latest
name: "Release a new version"
steps:
- name: Get auth token
id: token
uses: actions/create-github-app-token@5d869da34e18e7287c1daad50e0b8ea0f506ce69 # v1.11.0
with:
app-id: ${{ vars.SENTRY_RELEASE_BOT_CLIENT_ID }}
private-key: ${{ secrets.SENTRY_RELEASE_BOT_PRIVATE_KEY }}
- uses: actions/checkout@v4
with:
token: ${{ secrets.GH_RELEASE_PAT }}
token: ${{ steps.token.outputs.token }}
fetch-depth: 0
- name: Prepare release
id: prepare-release
uses: getsentry/action-prepare-release@v1
env:
GITHUB_TOKEN: ${{ secrets.GH_RELEASE_PAT }}
GITHUB_TOKEN: ${{ steps.token.outputs.token }}
with:
version: ${{ github.event.inputs.version }}
force: ${{ github.event.inputs.force }}
Expand All @@ -42,7 +48,6 @@ jobs:
steps:
- uses: actions/checkout@v4
with:
token: ${{ secrets.GH_RELEASE_PAT }}
fetch-depth: 0
- uses: getsentry/action-release@v1
env:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,7 @@ jobs:
docker compose logs
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v4
uses: codecov/codecov-action@v5
with:
token: ${{ secrets.CODECOV_TOKEN }}
slug: getsentry/self-hosted
Expand Down
3 changes: 0 additions & 3 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -99,9 +99,6 @@ geoip/GeoIP.conf
geoip/*.mmdb
geoip/.geoipupdate.lock

# wal2json download
postgres/wal2json

# integration testing
_integration-test/custom-ca-roots/nginx/*
sentry/test-custom-ca-roots.py
Expand Down
77 changes: 77 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,82 @@
# Changelog

## 24.12.0

- No documented changes.

## 24.11.2

### Various fixes & improvements

- fix(redis): Actually use custom config (#3459) by @BYK
- feat(release): Replace release bot with GH app (#3458) by @Jeffreyhung
- chore(issue-template): ask for machine specification and provide link to security policy (#3447) by @aldy505
- add sentry/backup.json to gitignore (#3450) by @niklassc7
- ref: remove suggested fix (#3446) by @aldy505

## 24.11.1

### Various fixes & improvements

- fix(redis): Use a safer eviction rule (#3432) by @BYK
- feat: add Redis configuration for improved memory management (#3427) by @Hassanzadeh-sd
- build(deps): bump codecov/codecov-action from 4 to 5 (#3429) by @dependabot

## 24.11.0

### Various fixes & improvements

- feat(healthcheck): Improve redis healthcheck (#3422) by @hubertdeng123
- fix: missing mime types and turning off autoindex for js-sdk endpoint (#3395) by @aldy505
- fix: Use js.sentry-cdn.com for JS SDK downloads (#3417) by @BYK
- fix(loader): provide js sdk assets from 4.x (#3415) by @aldy505
- Revert "Revert "ref(feedback): remove issue platform flags after releasing issue types"" (#3403) by @BYK
- Revert "ref(feedback): remove issue platform flags after releasing issue types" (#3402) by @BYK
- ref(feedback): remove issue platform flags after releasing issue types (#3397) by @aliu39
- fix(sentry-admin): Do not wait for command finish to display output (#3390) by @Makhonya

## 24.10.0

### Various fixes & improvements

- chore: Disable codecov for master/release branches (#3384) by @hubertdeng123
- chore: replace old URLs of the repo with the new docs (#3375) by @victorelec14
- ref: span normalization allowed host config (#3245) by @aldy505
- docs: explicitly specify `mail.use-{tls,ssl}` is mutually exclusive (#3368) by @aldy505
- ref: allow hosted js sdk bundles (#3365) by @aldy505
- fix(clickhouse): Allow nullable key (#3354) by @nikhars

## 24.9.0

### Various fixes & improvements

- docs: link to develop docs (#3307) by @joshuarli
- fix: more leeway for minimum RAM (#3290) by @joshuarli
- Mandate minimum requirements for ram/cpu (#3275) by @hubertdeng123
- ref(feedback): cleanup topic rollout option (#3276) by @aliu39
- Update release template (#3270) by @hubertdeng123

## 24.8.0

### Various fixes & improvements

- Migrate to zookeeper-less kafka (#3263) by @hubertdeng123
- Revert "ref(feedback): cleanup topic rollout option" (#3262) by @aliu39
- ref(feedback): cleanup topic rollout option (#3256) by @aliu39
- Remove cdc and wal2json and use the default postgres entrypoint (#3260) by @beezz
- add `-euo pipefail` to enhance-image.example.sh (#3246) by @asottile-sentry
- remove python-dev (#3242) by @asottile-sentry
- feat: enable user feedback feature (#3193) by @aldy505
- Use CDN by default for JS SDK Loader (#3213) by @stayallive

## 24.7.1

### Various fixes & improvements

- Fix: errors only config flag (#3220) by @hubertdeng123
- Add errors only self-hosted infrastructure (#3190) by @hubertdeng123
- feat(generic-metrics): Add gauges to docker compose, re-try (#3177) by @ayirr7

## 24.7.0

### Various fixes & improvements
Expand Down
12 changes: 9 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,13 @@
## Testing

Validate changes to the setup by running the integration test:
### Running Tests with Pytest

```shell
./integration-test.sh
We use pytest for running tests. To run the tests:

1) Ensure that you are in the root directory of the project.
2) Run the following command:
```bash
pytest
```

This will automatically discover and run all test cases in the project.
82 changes: 3 additions & 79 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,81 +1,5 @@
# Self-Hosted Sentry nightly
# Self-Hosted Sentry

Official bootstrap for running your own [Sentry](https://sentry.io/) with [Docker](https://www.docker.com/).
[Sentry](https://sentry.io/), feature-complete and packaged up for low-volume deployments and proofs-of-concept.

## Requirements

* Docker 19.03.6+
* Compose 2.19.0+
* 4 CPU Cores
* 16 GB RAM
* 20 GB Free Disk Space

## Setup

### Installation

To get started with all the defaults, simply clone the repo and run `./install.sh` in your local check-out. Please also read the section below about monitoring. Sentry uses Python 3 by default since December 4th, 2020 and Sentry 21.1.0 is the last version to support Python 2.

During the install, a prompt will ask if you want to create a user account. If you require that the install goes on without creating a user, run `./install.sh --skip-user-creation`.

Thinking of not managing this yourself? Check out the [SaaS migration docs](https://docs.sentry.io/product/sentry-basics/migration/) or [contact us](https://sentry.io/from/self-hosted) for help.

Please visit [our documentation](https://develop.sentry.dev/self-hosted/) for everything else.

### Customize DotEnv (.env) file

Environment specific configurations can be done in the `.env.custom` file. It will be located in the root directory of the Sentry installation, and if it exists then `.env` will be ignored entirely.

By default, there exists no `.env.custom` file. In this case, you can manually add this file by copying the `.env` file to a new `.env.custom` file and adjust your settings in the `.env.custom` file.

Please keep in mind to check the `.env` file for changes, when you perform an upgrade of Sentry, so that you can adjust your `.env.custom` accordingly, if required, as `.env` is ignored entirely if `.env.custom` is present.

### Enhance Sentry image

To install plugins and their dependencies or make other modifications to the Sentry base image,
copy `sentry/enhance-image.example.sh` to `sentry/enhance-image.sh` and add necessary steps there.
For example, you can use `apt-get` to install dependencies and use `pip` to install plugins.

After making modifications to `sentry/enhance-image.sh`, run `./install.sh` again to apply them.

## Tips & Tricks

### Event Retention

Sentry comes with a cleanup cron job that prunes events older than `90 days` by default. If you want to change that, you can change the `SENTRY_EVENT_RETENTION_DAYS` environment variable in `.env` or simply override it in your environment. If you do not want the cleanup cron, you can remove the `sentry-cleanup` service from the `docker-compose.yml`file.

### Installing a specific SHA

If you want to install a specific release of Sentry, use the tags/releases on this repo.

We continuously push the Docker image for each commit made into [Sentry](https://github.com/getsentry/sentry), and other services such as [Snuba](https://github.com/getsentry/snuba) or [Symbolicator](https://github.com/getsentry/symbolicator) to [our Docker Hub](https://hub.docker.com/u/getsentry) and tag the latest version on master as `:nightly`. This is also usually what we have on sentry.io and what the install script uses. You can use a custom Sentry image, such as a modified version that you have built on your own, or simply a specific commit hash by setting the `SENTRY_IMAGE` environment variable to that image name before running `./install.sh`:

```shell
SENTRY_IMAGE=getsentry/sentry:83b1380 ./install.sh
```

Note that this may not work for all commit SHAs as this repository evolves with Sentry and its satellite projects. It is highly recommended to check out a version of this repository that is close to the timestamp of the Sentry commit you are installing.

### Using Linux

If you are using Linux and you need to use `sudo` when running `./install.sh`, make sure to place the environment variable *after* `sudo`:

```shell
sudo SENTRY_IMAGE=us.gcr.io/sentryio/sentry:83b1380 ./install.sh
```

Where you replace `83b1380` with the sha you want to use.

### Self-Hosted Monitoring

We'd love to catch errors in self-hosted so you don't run into them, and so we can fix them faster! When you run `./install.sh`, you will be prompted to select whether to opt in or out of our monitoring. If you opt into our monitoring, we will send information to our own self-hosted Sentry instance for development and debugging purposes. We may collect:

- OS username
- IP address
- install log
- runtime errors in Sentry
- performance data

Thirty (30) day retention. No marketing. Privacy policy at sentry.io/privacy.

Starting with the 22.10.0 release in October, we will require those running the Sentry installer to choose to opt in or out. If you are running the installer under automation, you may want to set `REPORT_SELF_HOSTED_ISSUES` or pass `--(no-)report-self-hosted-issues` to the installer accordingly.
Documentation [here](https://develop.sentry.dev/self-hosted/).
8 changes: 8 additions & 0 deletions _integration-test/test_run.py
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,14 @@ def test_initial_redirect():
assert initial_auth_redirect.url == f"{SENTRY_TEST_HOST}/auth/login/sentry/"


def test_asset_internal_rewrite():
"""Tests whether we correctly map `/_assets/*` to `/_static/dist/sentry` as
we don't have a CDN setup in self-hosted."""
response = httpx.get(f"{SENTRY_TEST_HOST}/_assets/entrypoints/app.js")
assert response.status_code == 200
assert response.headers["Content-Type"] == "text/javascript"


def test_login(client_login):
client, login_response = client_login
parser = BeautifulSoup(login_response.text, "html.parser")
Expand Down
3 changes: 1 addition & 2 deletions _unit-test/create-docker-volumes-test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,7 @@ sentry-data
sentry-kafka
sentry-postgres
sentry-redis
sentry-symbolicator
sentry-zookeeper"
sentry-symbolicator"

before=$(get_volumes)

Expand Down
32 changes: 32 additions & 0 deletions _unit-test/js-sdk-assets-test.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
#!/usr/bin/env bash

source _unit-test/_test_setup.sh
source install/dc-detect-version.sh
$dcb --force-rm web

export SETUP_JS_SDK_ASSETS=1

source install/setup-js-sdk-assets.sh

sdk_files=$(docker compose run --no-deps --rm -v "sentry-nginx-www:/var/www" nginx ls -lah /var/www/js-sdk/)
sdk_tree=$(docker compose run --no-deps --rm -v "sentry-nginx-www:/var/www" nginx tree /var/www/js-sdk/ | tail -n 1)
non_empty_file_count=$(docker compose run --no-deps --rm -v "sentry-nginx-www:/var/www" nginx find /var/www/js-sdk/ -type f -size +1k | wc -l)

# `sdk_files` should contains 5 lines, '4.*', '5.*', '6.*', `7.*` and `8.*`
echo $sdk_files
total_directories=$(echo "$sdk_files" | grep -c '[45678]\.[0-9]*\.[0-9]*$')
echo $total_directories
test "5" == "$total_directories"
echo "Pass"

# `sdk_tree` should output "5 directories, 17 files"
echo "$sdk_tree"
test "5 directories, 17 files" == "$(echo "$sdk_tree")"
echo "Pass"

# Files should all be >1k (ensure they are not empty)
echo "Testing file sizes"
test "17" == "$non_empty_file_count"
echo "Pass"

report_success
2 changes: 2 additions & 0 deletions clickhouse/config.xml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@
<session_log remove="remove"/>
<part_log remove="remove"/>

<allow_nullable_key>1</allow_nullable_key>

<profiles>
<default>
<log_queries>0</log_queries>
Expand Down
Loading

0 comments on commit 0a6ae17

Please sign in to comment.