# Alerts and webhooks

> Unrent raises alerts from 16 rules and sends them by email to members who opt in and to up to 10 webhooks. Each webhook request is signed: verify `x-unrent-signature` as an HMAC-SHA256 of the raw body with your webhook secret.

URL: https://unrent.dev/docs/alerts/

## Rules

Alert rules

| Rule | Severity | Fires when |
| --- | --- | --- |
| `service_down` | critical | A deployed service’s container has failed and no deploy is in progress |
| `restart_loop` | critical | The container keeps restarting (3 or more in 10 minutes) |
| `server_disconnected` | critical | No connection from the agent for more than 2 minutes |
| `server_silent` | warning | Connected, but no status for more than 5 minutes |
| `disk_usage` | warning / critical | 85 % / 95 % of the disk used |
| `memory_pressure` | warning / critical | 90 % / 97 % of memory used for 2 minutes |
| `backup_late` | warning | No verified backup within the interval plus a grace period (30 hours by default) |
| `backup_failing` | critical | The last backup failed |
| `drill_failed` | critical | A restore drill failed |
| `deploy_failed` | warning | A pull, deploy or rollback failed |
| `config_failed` | warning | Applying config failed |
| `cron_failed` | warning | A scheduled job exited non-zero |
| `ingress_failed` | critical | Routes or certificates could not be applied |
| `cert_expiring` | warning / critical | A certificate expires in under 14 / 3 days |
| `converge_failed` | warning | Applying hardening failed |
| `hardening_drift` | warning | A hardened server fails a check in the nightly run |

One alert per condition: it updates instead of repeating, escalates from warning to critical, reopens if it returns within 15 minutes, and is marked flapping after 4 changes in 30 minutes. Developers can acknowledge alerts and mute a rule for 5 minutes to 7 days. At most 30 notifications per organisation per hour are sent.

## Email

Each member turns on **Settings → Notifications → Email me** for themselves. Messages come from `no-reply@unrent.dev` with the subject `[Unrent] Critical: …`.

## Webhooks

Admins add webhooks: https on port 443, at a public address. The secret (`whsec_…`) is shown once. Each delivery is a `POST` with a JSON body:

Body:

```
{
  "id": "dlv_…",
  "type": "alert.fired",
  "timestamp": 1763030000,
  "org_id": "org_…",
  "suppressed_last_hour": 0,
  "alert": { "id": "alr_…", "rule": "service_down", "severity": "critical", "state": "firing",
             "title": "Service production/web is down", "server_name": "web-1", "url": "https://app.unrent.dev/…" }
}
```

Types: `alert.fired`, `alert.escalated`, `alert.resolved`, `alert.flapping`, `alert.acknowledged` and `webhook.test`. Headers: `x-unrent-event`, `x-unrent-delivery` (the `id`) and `x-unrent-signature: sha256=<hex>`.

## Verify a webhook

1. Compute HMAC-SHA256 of the **raw** request body with your secret, as hex.
2. Compare it with the value after `sha256=` in `x-unrent-signature`, in constant time.
3. Reject a `timestamp` (seconds) older than 5 minutes.
4. Deduplicate by `id`: a retry sends the identical body.

Node.js:

```
import { createHmac, timingSafeEqual } from 'node:crypto';

export function verify(rawBody, header, secret) {
  const expected = Buffer.from('sha256=' + createHmac('sha256', secret).update(rawBody).digest('hex'));
  const given = Buffer.from(header ?? '');
  if (given.length !== expected.length || !timingSafeEqual(given, expected)) return false;
  const { timestamp } = JSON.parse(rawBody);
  return Math.abs(Date.now() / 1000 - timestamp) <= 300;
}
```

Answer with any 2xx. 408, 429, 5xx, timeouts (10 seconds) and connection errors are retried after 30 seconds, 2 minutes, 10 minutes, 1 hour and 6 hours; any other answer, redirects included, is final.