Alert Webhook

Edit alertmanager.yml

Open your Alertmanager configuration file (default path: /etc/alertmanager/alertmanager.yml) and add a new entry under receivers: plus a matching route: block.

global:
  resolve_timeout: 1m

route:
  receiver: cloudops-webhook
  group_by: [alertname]
  group_wait: 0s
  group_interval: 30s
  repeat_interval: 5m

receivers:
  - name: cloudops-webhook
    webhook_configs:
      - url: '<CLOUDOPS_WEBHOOK_URL>'
        send_resolved: true

Replace <CLOUDOPS_WEBHOOK_URL> with the Webhook URL you copied from the CloudOps console.

route:
  receiver: existing-default
  routes:
    - receiver: cloudops-webhook
      matchers:
        - severity =~ "critical|warning"
      continue: false

Reload Alertmanager

Apply the new config without restarting:

# Default signal handler
kill -HUP $(pidof alertmanager)

# Or, if --web.enable-lifecycle was set at start
curl -X POST http://localhost:9093/-/reload

Open Alertmanager UI (http://localhost:9093) and confirm the new receiver appears under [Status > Config].

Edit alert.rules.yml

Add or extend a rule group file (example path: /etc/prometheus/alert.rules.yml):

groups:
  - name: cloudops-integration
    interval: 1m
    rules:
      - alert: CloudOpsHighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05
        for: 2m
        labels:
          severity: critical
          team: platform
        annotations:
          summary: "High 5xx error rate on {{ $labels.service }}"
          description: "5xx error rate exceeded 5% for 2 minutes"

labels become the alert’s labels in Alertmanager (used by route / routes: matching). annotations become the human-readable summary CloudOps displays.

Register the rule file in prometheus.yml

Open prometheus.yml (default: /etc/prometheus/prometheus.yml) and add the rule file path under rule_files, plus an alerting.alertmanagers block if not already present:

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093   # adjust to your Alertmanager host:port

rule_files:
  - /etc/prometheus/alert.rules.yml

Reload Prometheus

# Default signal handler
kill -HUP $(pidof prometheus)

# Or, if --web.enable-lifecycle was set at start
curl -X POST http://localhost:9090/-/reload

Open Prometheus UI (http://localhost:9090) and go to [Status > Rules] — the new rule group should be listed with state: ok.

Force a permanent firing for a quick check

Temporarily replace the expr in your rule with vector(1) — this constant always evaluates true, forcing the rule to fire on the next evaluation:

- alert: CloudOpsE2ETrigger
  expr: vector(1)
  for: 0s
  labels:
    severity: critical
  annotations:
    summary: "CloudOps webhook integration test"

Reload Prometheus and wait one evaluation cycle (default: 1 minute).

Verify on CloudOps

Open the CloudOps console and check that the test event arrived under the webhook source you registered. The Requests (24h) counter on the webhook row should increment within ~1 minute.

If it does not arrive:

• Confirm the URL in webhook_configs.url matches the value issued by CloudOps exactly.
• Confirm the route (or a child routes: entry) directs to the new receiver — check Alertmanager UI [Status > Config].
• Check Alertmanager UI [Status > Alerts] for the firing alert; if absent, the rule did not fire — see Prometheus UI [Alerts].
• Check Alertmanager logs for notify delivery errors.

Verify the RECOVERY flow (optional)

To exercise the resolved path, flip the test rule’s expr from vector(1) to vector(0) and reload Prometheus. Alertmanager will dispatch a resolved status webhook within repeat_interval, and CloudOps will record it as event_type=RECOVERY.

Restore the original expression

After verification, restore your real expr and reload Prometheus.