Skip to main content

Grouping rules

When one problem fires many alerts, you want one incident, not fifty. Grouping rules tell Regen which alerts belong together: alerts that match the same rule and share the same values for the labels you choose are attached to a single open incident.

Without grouping rules, each new alert opens its own incident. Repeats of an existing alert are always handled by deduplication first.

Grouping rules are managed through the API

There is no screen for grouping rules in the app yet. You create and change them with the API, using a key that has the grouping_rule:read and grouping_rule:write scopes.:::

How grouping works​

For each new alert that is going to open an incident:

  1. The alert goes through your alert routes first. If a route suppresses it, grouping never sees it.
  2. Grouping rules are checked from lowest priority number to highest. The first enabled rule that matches decides.
  3. From the alert's labels, Regen builds a group key (explained below).
  4. If an incident with that group key is still open (triggered or acknowledged) and was created within the rule's time window, the alert is attached to it. Otherwise a new incident is opened with that group key.
  5. If no rule matches, the alert opens its own incident, as if there were no grouping.

The time window is measured from when the incident was created, not from the last alert. A one-hour window means alerts join an incident for up to an hour after it opened, then start a new one.

Rule settings​

FieldNotes
nameRequired, up to 255 characters.
descriptionOptional, up to 1000 characters.
enabledDefaults to true. Disabled rules are skipped.
priorityRequired, 1 to 1000. Lower numbers are checked first. Each priority can be used once.
match_labelsRequired. Which alerts the rule applies to. See below.
time_window_secondsRequired, 1 to 86400 (24 hours).
cross_source_labelsOptional list of label names that define the group key. See below.

Which alerts a rule applies to​

match_labels is a set of label names and values. An alert matches when it has all of them:

  • {"service": "api", "env": "prod"} matches alerts with exactly those two label values.
  • {"service": "*"} matches any alert that has a service label, whatever its value.
  • {} matches every alert.

Unlike alert routes, grouping rules use exact values, not regular expressions. Only * is special.

What makes two alerts "the same"​

The group key is built from label values:

  • By default, from the labels named in match_labels. With {"service": "*", "env": "*"}, alerts with the same service and env are grouped, and other labels such as instance are ignored.
  • If you set cross_source_labels, those labels are used instead. This lets alerts from different monitoring tools join one incident, for example a Prometheus alert and a Grafana alert for the same service and env.

A label that an alert does not have is left out of its key.

Examples​

One incident per service and environment, within 10 minutes

{
  "name": "Group by service and environment",
  "priority": 10,
  "match_labels": { "service": "*", "env": "*" },
  "time_window_seconds": 600
}

Join Prometheus and Grafana alerts for the same service

{
  "name": "Cross-tool by service",
  "priority": 20,
  "match_labels": { "service": "*" },
  "cross_source_labels": ["service", "env"],
  "time_window_seconds": 900
}

Create a rule with:

curl -X POST https://regen.fluidify.ai/api/v1/grouping-rules \
  -H "Authorization: Bearer regen_live_xxxxxxxxxxxxxxxx" \
  -H "Content-Type: application/json" \
  -d '{"name":"Group by service and environment","priority":10,"match_labels":{"service":"*","env":"*"},"time_window_seconds":600}'

The other endpoints work on /api/v1/grouping-rules: GET to list, GET /{id}, PUT /{id} to change a rule (send only the fields you are changing), and DELETE /{id}.

Tips​

  • Group on labels that identify the problem, such as service and env. Do not group on labels that differ per alert, such as instance or pod, or nothing will ever group.
  • Keep windows short enough that a genuinely new problem does not get attached to an old incident.
  • Give specific rules low priority numbers and a catch-all rule a high one.