Skip to content
82 changes: 63 additions & 19 deletions content/en/monitors/notify/notification_rules.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,23 +14,67 @@
---

## Overview

Monitor notification rules are predefined sets of conditions that automate the process of alerting your team based on predefined conditions and tags. Instead of individually configuring notification recipients and routing for every monitor, notification rules allow you to define the notification logic and recipients in one place and automatically route all monitor events with matching tags to that list of handles.
Monitor notification rules are predefined sets of conditions that automate the process of alerting your team based on tags and rule logic. Instead of individually configuring recipients for every monitor, notification rules let you define once and automatically route all monitor notifications whose notification tagset matches the rule's scope.

Check notice on line 17 in content/en/monitors/notify/notification_rules.md

View workflow job for this annotation

GitHub Actions / vale

Datadog.sentencelength

Suggestion: Try to keep your sentence length to 25 words or fewer.

<div class="alert alert-info">There is a default limit of 1000 rules per organization.</a>.</div>

## Creating notification rules
## Setup

### 1. Create a Monitor Notification Rule in Datadog

Check warning on line 23 in content/en/monitors/notify/notification_rules.md

View workflow job for this annotation

GitHub Actions / vale

Datadog.headings

'1. Create a Monitor Notification Rule in Datadog' should use sentence-style capitalization.

1. Go to [**Notification Rules**][1].
2. Click **New Rule**.

{{< img src="/monitors/notifications/notification_rules/notification_rule_form-light.png" alt="Configuration for a notification rule showing scopes, routing conditions, recipients, and matching monitors" style="width:100%;" >}}

### 2. Choose the scope
Define the required tags for a monitor notification to be routed to this rule. Matching evaluates the notification tagset (the union of monitor tags and the firing group's tags for multi alert monitors). Monitor tags alone can satisfy the scope, and this is still considered a match. Only monitor and group tags participate in matching. Matching is case-insensitive.

**Note**: Any monitor created or edited after the notification rule is saved and is routed to the defined recipients if it matches the scope of the rule.

Check notice on line 33 in content/en/monitors/notify/notification_rules.md

View workflow job for this annotation

GitHub Actions / vale

Datadog.sentencelength

Suggestion: Try to keep your sentence length to 25 words or fewer.

#### Rule scope syntax

The Notification Rule scope query supports Boolean logic and follows the same common [search syntax][3] supported by many other platform products.

##### Boolean operators
- Supported: AND, OR, NOT
- Implicit operator: AND

<div class="alert alert-danger">You must have the <a href="/account_management/rbac/permissions/#monitors"><code>monitor_config_policy_write</code> permission</a> to create a rule.</div>
##### Wildcards
Only `key:*` is supported (for example, `env:*`). Partial wildcards like `env:prod-*` are not supported. `key:*` matches if the key exists anywhere in the notification tagset.

1. Navigate to [**Monitors > Settings > Notification Rules**][1].
1. Click **New Rule**.
1. Add specific tags and values to set the scope for the rule. Notification rules use an AND logic for multiple tags. Both monitor tags and group tags are considered when matching the scope. For an example of this, see [Routing logic](#routing-logic).
1. Define your routing conditions by specifying **when** and **to whom** a notification should be sent. Notifications can be sent to emails, Team channels, or Integration channels. There is a limit of 50 notification recipients per rule. For more information, see [Notifications][2].
1. Add a name for the rule.
1. Click **Create Rule**.
##### Multiple values for the same key
Use either `env:(prod OR staging)` or `env:prod OR env:staging`.

{{< img src="/monitors/notifications/notification_rules/notification_rules_form_with_conditional_recipients.png" alt="Configuration for a notification rule showing tag scopes, routing conditions, recipients, and matching monitors" style="width:100%;" >}}
##### Quoting
Wrap values that contain spaces or special characters in quotes, for example: `team:"data platform"`.

##### Examples

| Notification Rule scope | Explanation |
| ------------------- | ---------------------- |
| `service:web-store` | Route any notification about the `web-store` service. |
| `service:web-store AND env:prod` | Route any notification about the `web-store` service running on the `prod` environment. |
| `service:webstore AND NOT env:staging` | Route any notification about the `web-store` service that is **not** running on the `staging` environment. |
| `env:*` | Route any notification that carry the `env:<value>` tag (either from monitor tags or group). |

For an example of this, see [Routing logic](#routing-logic).

#### Rule scope limitations
The following limitations are **not supported**:

* Keyless tags, such as `prod AND service:(A or B)` or `prod`, aren't supported. Tags need to have a key, in this case for example `env:prod`.
* Partial wildcards (`service:web-*`) and question mark wildcards `service:auth?` are not supported. Wildcard is allowed only if used alone like `service:*`.
* Scope lenght up to 3000 characters.

### 3. Configure the recipients
Define the recipients to notify when a monitor notification matches the rule's scope. You can specify when and to which recipients a notification should be sent. Notifications can be sent to email or any integration channel. There is a limit of 50 notification recipients per rule. For more information, see [Notifications][2].

#### Conditional recipients
You can notify a specific recipient only when certain conditions are met. For example, route alerts to your on-call recipient, and send warn notifications to a Slack channel if they don't require immediate action.

### 4. Name it
Use a clear name so the rule is identifiable on the **Manage** page.

## Managing notification rules

Expand All @@ -49,17 +93,16 @@
Additionally, you can click the vertical three-dot menu on the notification rule to **Edit** or **Delete**.

### From an individual monitor

In your monitor configuration, you can view the notification recipients that are applied to the monitor under **Recipient Summary**. Notification rules automatically add recipients to monitors that match the configured scopes.
In your monitor configuration, the **Recipient Summary** shows recipients that are applied to the monitor by matching notification rules. On the **Monitor** edit page, you may also see rules that _could_ match when new groups report (multi alert monitors). The **Monitor** status page shows rules that match.

{{< img src="/monitors/notifications/notification_rules/monitor_matching_notification_rule.png" alt="Recipient summary field showing the notification recipients applied by notification rules" style="width:100%;" >}}

## Routing logic
## How matching works

Notification rules apply to all monitor notifications that match the scopes defined in the rule configuration.
- Multiple tags apply an AND logic to the scope.
- Tags can be either from monitor tags or from monitor groups
- Multiple rules can match a single monitor notification, and all recipients are added to the monitor alert without duplication.
- Notification tagset is the union of monitor tags and tags of the firing group (for multi alert monitors). If a key has multiple values across monitor/group, all values are considered.
- Currently matches: A rule matches if at least one reporting group, combined with monitor tags, satisfies the scope; or, if the monitor tags alone do. NOT is evaluated per candidate tagset, so a group with a denied value does not match.

Check warning on line 103 in content/en/monitors/notify/notification_rules.md

View workflow job for this annotation

GitHub Actions / vale

Datadog.tense

Avoid temporal words like 'Currently'.
- Could match when new groups report (multi alert monitors, Monitor edit surface): Treat each group-by key as present with any value, constrained by the monitor query's allow/deny filters.
- If multiple rules match a single notification, recipients from all matching rules are merged and de-duplicated.

{{< img src="/monitors/notifications/notification_rules/diagram_notification-rules.png" alt="Flowchart showing how Monitor notification rules match tags, combine recipients from monitors and rules, and remove duplicates before sending alerts" style="width:100%;" >}}

Expand Down Expand Up @@ -150,7 +193,7 @@
<td style="text-align: center">{{< X >}}</td>
<td style="text-align: center">{{< X >}}</td>
<td style="text-align: center"></td>
<td><i>@pagerduty-service1 @slack-service1</i><br><i>@jira-project</i><br><i>@[email protected]</i></td>
<td><i>@slack-channel1</i><br><i>@slack-service1</i><br><i>@jira-project</i><br><i>@[email protected]</i></td>
</tr>
</tbody>
</table>
Expand All @@ -163,3 +206,4 @@

[1]: https://app.datadoghq.com/monitors/settings/notifications
[2]: /monitors/notify/#notifications
[3]: /logs/explorer/search_syntax/
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading