← Back to Dashboard
JAMSHelp Center

Grafana & Prometheus Alertmanager

Page your on-call team when a Grafana alert fires. Grafana sends the alert to JAMS Incident Management, which calls, texts or pushes your recipients until someone acknowledges. When the alert resolves in Grafana, the alerts stop.

It works with self-hosted Grafana, Grafana Cloud and Prometheus Alertmanager.

Plan availability: The Grafana integration is included on the Professional plan and up. See Plans & Billing.

Two parts. Receiving alerts (Steps 1–3) needs nothing from Grafana but a contact point. Updating Grafana when someone acknowledges — a note on the graph, and optionally a silence — is an optional extra step that needs a Grafana service-account key. See Update Grafana when acknowledged.

Do I need an Agent (the JAMS Incident Connector)? Never to receive alerts: Grafana sends them out to JAMS Incident Management, so that works behind a firewall as long as Grafana can reach the internet over HTTPS. Only the optional update Grafana step needs JAMS Incident Management to reach your Grafana — Grafana Cloud always works; a self-hosted Grafana must be reachable from the internet. Quick test: can you open your Grafana from your phone, off Wi-Fi and off VPN?

What you'll need

  • Grafana Cloud or self-hosted Grafana, with permission to edit contact points and notification policies — or a Prometheus Alertmanager you can configure.
  • For the signed setup below, a Grafana whose webhook contact point has an HMAC Signature section (Grafana Cloud and current self-hosted releases do; tested on Grafana 13). On an older Grafana without it, use Authorization Header instead: scheme Bearer, credentials = your JAMS Incident Management secret.
  • Admin access in JAMS Incident Management.

Step 1 — Create a Grafana connection in JAMS Incident Management

  1. Open Integrations → Grafana & Prometheus (under Monitoring in the sidebar).
  2. Enter a Connection name — naming it after the team it pages (e.g. Platform team) keeps things tidy.
  3. Choose who gets notified (people, groups, on-call schedules or Teams/Slack channels) and the channels to use.
  4. Set the escalation interval — how long to wait for each person before moving to the next.
  5. Click Save Connection.

JAMS Incident Management then shows your webhook URL and secret. You can find them again any time under Show the Grafana / Alertmanager setup on the connection.

Keep the secret private. Anyone with the URL and secret can raise alerts on this connection. If it is shared by mistake, click Rotate secret and paste the new one into Grafana.

Step 2 — Add a contact point in Grafana

  1. In Grafana, open Alerting → Contact points (in Grafana Cloud: Alerts & IRM → Alerting → Contact points) and click + Add contact point.
  2. Name it (e.g. JAMS Incident Management) and set Integration to Webhook.
  3. Paste your JAMS Incident Management webhook URL into URL.
  4. Open Optional Webhook settings → HMAC Signature:
    • Secret: paste your JAMS Incident Management secret
    • Header: leave as X-Grafana-Alerting-Signature
    • Timestamp header: type exactly X-Grafana-Alerting-Signature-Timestamp
  5. Click Test, then Send test notification.

The connection in JAMS Incident Management turns green within a few seconds and says test received. Nobody is paged for a test.

The timestamp header must be typed exactly. Grafana has no default for it. If it's missing or misspelled, JAMS Incident Management rejects the alert and the connection shows the reason.

  1. Click Save contact point.

Step 3 — Send alerts to the contact point

A contact point only receives the alerts you route to it. Either:

  • Use a notification policy (best for many rules): open Alerting → Notification policies and set the contact point on the default policy, or add a child policy that matches the alerts that should page (for example team = platform).
  • Or choose it per rule: when editing an alert rule, under Configure notifications, select the contact point.

To page different people for different alerts, create one JAMS Incident Management connection per team, one contact point for each, and a child policy per team.

Prometheus Alertmanager

Alertmanager can't sign requests, so it sends the secret as a Bearer token instead. JAMS Incident Management shows a ready-made receiver on the Prometheus Alertmanager tab of the setup:

receivers:
  - name: jams
    webhook_configs:
      - url: <your JAMS Incident Management webhook URL>
        send_resolved: true
        http_config:
          authorization:
            type: Bearer
            credentials: <your JAMS Incident Management secret>
  1. Add the receiver to alertmanager.yml, keeping your existing receivers.
  2. Point a route at it — set route.receiver: jams, or add a child route matching the alerts that should page.
  3. Keep send_resolved: true, or alerts never stop in JAMS Incident Management when they resolve.
  4. Reload Alertmanager (POST /-/reload, or restart it).

Alertmanager has no test button. The connection turns green when the first alert arrives.

What your team receives

  • The alert says where it came from — Grafana Alert: <alert name>: <summary> (or Grafana Cloud Alert / Prometheus Alert) — with the severity, the alert's description, the values that triggered it and its labels. That's what email, Teams, Slack, WhatsApp and the app show. Teams and Slack show Runbook, Dashboard, Graph and Source as buttons.
  • The graph itself appears in Teams, Slack, email and the app when Grafana includes a screenshot that JAMS Incident Management can reach (Grafana Cloud with screenshots turned on, or a self-hosted Grafana whose images are on the internet). Otherwise the Graph button links to it.
  • By SMS: just "URGENT: Grafana alert. <alert name>. Severity: …" and the reply menu — no links. Carriers' security scanners open links in text messages, so SMS never carries them. The details are in the other channels and on the Incidents page.
  • On a phone call: the same one line, then "Press 1 or say acknowledge…".
  • Acknowledge (press 1, reply 1, or the button in Teams, Slack, the app or email) stops the escalation for everyone on that alert.
  • When the alert resolves in Grafana, any escalation still running stops, and Teams/Slack cards show Closed in Grafana.

Each Grafana alert is tracked on its own, even when Grafana sends several in one batch. While an alert stays open, Grafana's repeat notifications don't page anyone again. If it resolves and later fires again, that's a new incident.

Severity

JAMS Incident Management reads the alert's severity label:

Label valueSeverity in JAMS Incident Management
critical, p1, pageCritical
error, high, major, p2Error
warning, medium, minor, p3Warning
info, low, p4, p5Informational

With no severity label, the alert arrives as Warning. Add the label to your alert rules so JAMS Incident Management can tell a page-worthy alert from a heads-up.

Tips

  • Flapping alerts: Grafana can hold an alert in Recovering instead of resolving it straight away. Set Keep firing for (and a recovery threshold) on the rule, so a value hovering around the threshold doesn't page repeatedly.
  • Max Alerts: if the contact point's Max Alerts is set, alerts beyond it are never sent. Leave it at 0 (unlimited) or set it high.
  • Grafana sends updates every few minutes, not instantly. Grafana batches notifications (every 5 minutes by default), so an alert can reach JAMS Incident Management a little after it fires in Grafana, and its resolve a little after it resolves.

Troubleshooting

The connection's status line in JAMS Incident Management shows the last thing that happened, including why a request was rejected.

StatusWhat to do
No X-Grafana-Alerting-Signature-Timestamp headerSet Timestamp header in the contact point's HMAC Signature to exactly X-Grafana-Alerting-Signature-Timestamp.
Signature does not matchThe secret in Grafana is different from the one in JAMS Incident Management. Copy it again from the connection (the copy button avoids stray spaces).
More than 5 minutes from our clockYour Grafana server's clock is wrong. Fix its time sync (NTP).
Not signed and has no Bearer tokenHMAC Signature isn't turned on in the contact point, or the Alertmanager receiver has no authorization block.
Nothing received yetGrafana couldn't reach JAMS Incident Management. Check that the Grafana server can make outbound HTTPS requests, and that the URL was pasted in full.
Test works, but real alerts never arriveThe alert isn't routed to the contact point. Check your notification policies, or the rule's Configure notifications.
Alerts never stop in JAMS Incident ManagementFor Alertmanager, make sure send_resolved: true is set. For Grafana, check Disable resolved message is off in the contact point.
This key cannot add annotations, create silences…The service account doesn't have the Editor role. Change its role in Grafana, then click Re-test.
This address is not on the public internet… connect it through an Agent (the JAMS Incident Connector)JAMS Incident Management can't reach your Grafana. Alerts still work; the update Grafana step needs an Agent (the JAMS Incident Connector) inside your network.
No note appears on the graphThe alert rule isn't linked to a dashboard panel. In Grafana, edit the rule and use Link dashboard and panel.

Update Grafana when acknowledged

Optional. When someone acknowledges or closes an alert in JAMS Incident Management, it adds a note on the alert's graph in Grafana — "Acknowledged by Kevin Casey via Voice call — JAMS Incident Management" — so anyone looking at the dashboard sees it's being handled. You can also have JAMS Incident Management silence the alert in Grafana while it's being worked on.

  1. In Grafana, open Administration → Users and access → Service accounts and click Add service account. Name it JAMS Incident Management and give it the Editor role.
  2. Click Add service account token, generate it, and copy it.
  3. In JAMS Incident Management, open the connection and expand Update Grafana when acknowledged. Choose Directly, enter your Grafana address (for example https://yourcompany.grafana.net, including any sub-path such as /grafana) and paste the token.
  4. Choose what JAMS Incident Management should do:
    • Add a note to the alert's graph when it's acknowledged or closed — on by default.
    • Silence the alert in Grafana when it's acknowledged — off by default; choose 30 minutes to 4 hours.
  5. Click Save & test. JAMS Incident Management checks what the key is allowed to do — without changing anything in Grafana — and tells you if it's missing a permission.

When silencing on every acknowledgement is off, calls and texts offer it as a choice instead: "Press 2 or say silence to acknowledge it and silence it in Grafana for 1 hour."

Silences and resolves. While an alert is silenced, Grafana sends nothing about it — not even that it resolved. So while JAMS Incident Management holds a silence, it checks the alert in Grafana every minute and closes the incident itself when the alert stops firing. When the incident closes, JAMS Incident Management ends the silence. Silences always end on their own at the time you chose.

Notes:

  • The note needs the alert rule to be linked to a dashboard panel (Grafana's Link dashboard and panel option on the rule). Rules without a panel are still silenced and paged; they just have nowhere to draw the note.
  • Prometheus Alertmanager alerts are never silenced from JAMS Incident Management — this step is for Grafana-managed alert rules.
  • The token is stored encrypted and is never shown again. To change it, paste a new one and save.

Grafana inside your network

If JAMS Incident Management can't reach your Grafana from the internet, Save & test says so and switches to Through an Agent. The Agent is the JAMS Incident Connector, a small service you install on any Windows or Linux server that can reach Grafana. It only connects out, and Grafana's token stays on that server.

  1. Create the service account and token in Grafana as in steps 1–2 above.

  2. On the Grafana integration page, under Agents for Grafana inside your network, name the Agent and click Add an Agent. Copy the pairing token; it's shown only once.

  3. Download the Agent from the same panel and install it:

    • Windows: run the installer and paste the pairing token. Setup recognises it as a Grafana Agent and asks for Grafana's address and the service-account token.
    • Linux: install the package, run sudo JamsConnector setup, and paste the pairing token.

    Either way, setup checks the token's permissions in Grafana before it saves.

  4. Back on the connection, open Update Grafana when acknowledged, choose Through an Agent, pick your Agent and click Save & test.

Everything above works the same through the Agent. It can also fetch the alert's graph from your Grafana, so the graph appears in Teams, Slack, email and the app; JAMS Incident Management keeps that image for 30 days. Untick Show the graph in alerts to keep only the Graph link.

If the Agent stops, alerts still page as normal. Only the updates to Grafana wait until it's back.

Questions

  • Does acknowledging silence the alert in Grafana? Only if you set up Update Grafana when acknowledged and turn silencing on — or the responder chooses silence on the call or text.
  • Can I close an incident from JAMS Incident Management? Yes, from Incidents (Close in JAMS Incident Management). That stops paging and, with the update step set up, adds a Closed note to the graph and ends any silence JAMS Incident Management created. The alert itself still resolves in Grafana on its own.
  • Does it work with Grafana OnCall or Grafana IRM? You don't need either. JAMS Incident Management takes their place for paging, phone calls and SMS.