Webhooks

The AI Engine API can deliver run lifecycle events to a callback_url you provide on POST /runs. Every webhook is signed with HMAC-SHA256 so you can verify it is genuine. This page tells you everything you need to receive webhooks safely.

Overview

Webhooks are fire-and-forget from your perspective: include callback_url on a run, and Havilo POSTs a signed JSON payload to that URL when each eligible event fires. Step-level events (step.started, step.completed, etc.) are not delivered by webhook — fetch them via polling or SSE.

Eligible event types

Only the two terminal outcomes are delivered. A run that is created, paused or canceled emits the event on the run's own event log — read it with GET /runs/{id}/events or the SSE stream — but nothing is POSTed to your callback for it.

Receiving a webhook

Webhooks arrive as POST requests to your callback_url with the following headers:

POST /your/webhook/path HTTP/1.1
Content-Type: application/json
X-Havilo-Signature: t=1714500000,v1=811900d36a6227fa2d87b81913f19cf42f25100c569b9ad405a4003b1d82a0fb
X-Havilo-Event-Id: evt_01hxbexample
X-Havilo-Api-Version: 2026-04-22
User-Agent: havilo-webhook/1

Body shape:

{
  "event_id": "evt_01hxbexample",
  "event_type": "run.succeeded",
  "occurred_at": 1714500000,
  "api_version": "2026-04-22",
  "run_id": "550e8400-e29b-41d4-a716-446655440000",
  "workspace_id": "8b3d0c7a-4ef5-4f11-9e8a-8d3b08c29c10",
  "data": {
    "id": "550e8400-e29b-41d4-a716-446655440000",
    "object": "run",
    "status": "succeeded",
    "created_at": 1714499820000,
    "completed_at": 1714500000000
  }
}

occurred_at is Unix epoch seconds (matches the timestamp in the signature header). created_at and completed_at inside data are Unix epoch milliseconds (consistent with the run envelope).

The data field is intentionally minimal. To get full run details, including artifacts, steps, and the canonical model-call usage projection, call GET /runs/{id} after receiving the webhook.

Your signing secret

Your webhook signing secret is the SHA-256 hex digest of the API key that created the run. There is no separate secret to fetch or rotate: if you hold the key, you can derive it.

import { createHash } from 'crypto'

const secret = createHash('sha256').update(process.env.HAVILO_API_KEY).digest('hex')

import hashlib, os

secret = hashlib.sha256(os.environ["HAVILO_API_KEY"].encode()).hexdigest()

Two consequences worth planning for:

• Rotating your API key changes the signing secret. When you rotate, update your webhook verifier in the same change as the caller. Until you do, deliveries keep arriving and fail verification — a loud mismatch, not a silent drop.
• The run is signed by the key that created it. POST /runs/{id}/resume continues the same run and keeps the original key's secret. POST /runs/{id}/rerun-from starts a new run and uses the key that asked for it.

Derive the secret once at startup rather than per request, and keep it out of logs — it is a deterministic function of your credential.

Verifying the signature

The signature header has the form:

X-Havilo-Signature: t=<unix_seconds>,v1=<hex_sha256>

The signed payload is ${timestamp}.${raw_body} — the unix-seconds timestamp, a literal dot, and the raw HTTP request body (no whitespace normalization, no JSON re-serialization).

HMAC-SHA256(secret, "${timestamp}.${raw_body}")

Use a constant-time comparison (timingSafeEqual in Node, hmac.compare_digest in Python) to prevent timing oracle attacks.

Test vector

To confirm your verification code is correct, run it against this fixed vector:

If your implementation produces a different hex string for these inputs, the bug is in your code, not in the contract.

Node.js verification

import { createHmac, timingSafeEqual } from 'crypto'

export function verifyWebhook(rawBody: string, header: string, secret: string): boolean {
  const match = header.match(/^t=(\d+),v1=([0-9a-f]{64})$/)
  if (!match) return false

  const timestamp = parseInt(match[1], 10)
  const received = match[2]

  const now = Math.floor(Date.now() / 1000)
  if (Math.abs(now - timestamp) > 300) return false

  const expected = createHmac('sha256', secret).update(`${timestamp}.${rawBody}`).digest('hex')

  const a = Buffer.from(expected, 'utf8')
  const b = Buffer.from(received, 'utf8')
  if (a.length !== b.length) return false

  return timingSafeEqual(a, b)
}

Python verification

import hmac
import hashlib
import re
import time

_HEADER = re.compile(r'^t=(\d+),v1=([0-9a-f]{64})$')

def verify_webhook(raw_body: bytes, header: str, secret: str) -> bool:
    m = _HEADER.match(header)
    if not m:
        return False

    timestamp = int(m.group(1))
    received = m.group(2)

    if abs(int(time.time()) - timestamp) > 300:
        return False

    expected = hmac.new(
        secret.encode("utf-8"),
        f"{timestamp}.".encode("utf-8") + raw_body,
        hashlib.sha256,
    ).hexdigest()

    return hmac.compare_digest(expected, received)

In your framework of choice, capture the raw request body before any JSON parsing — JSON.parse followed by JSON.stringify is not idempotent across implementations, and the resulting body will fail signature verification.

Replay protection

Reject webhooks whose t= timestamp differs from the current time by more than 5 minutes. The reference implementations above default to a 300-second tolerance.

This guards against:

• Replay attacks — an attacker who captures a valid webhook cannot replay it 6 minutes later.
• Out-of-sync clocks — if your server clock drifts, a strict 0-second tolerance would spuriously reject legitimate webhooks.

If your clock is more than ~30s out of sync with NTP, the 5-minute window will absorb the drift but you should still fix the clock.

Retry behavior

Each webhook is delivered up to 3 times.

Havilo retries on:

• Connection errors (DNS failure, refused, timeout)
• HTTP 408 Request Timeout
• HTTP 429 Too Many Requests
• HTTP 5xx Server Error

Havilo does not retry on:

• HTTP 2xx (success — no retry needed)
• HTTP 3xx redirects (treated as a misconfiguration)
• HTTP 4xx other than 408 / 429 (your code is rejecting the request; retrying won't help)

After 3 failed attempts, delivery is recorded as failed for support investigation. Store the event_id in your logs so support can correlate the failed delivery.

Timeouts

Each delivery attempt has a hard 10-second timeout. If your handler does not return a 2xx response within 10 seconds, the request is aborted and counted as a failure.

Best practice: acknowledge the webhook with a 2xx response immediately, then process it asynchronously (e.g., enqueue to a background job). If processing fails, do not return a 5xx — that triggers a retry that you don't need. Instead, return 2xx and handle the failure in your background queue.

Handling duplicates and ordering

Webhooks may arrive more than once. Havilo retries on transient failures, and a slow consumer that takes 11 seconds to respond will see the same event twice (once retried, once your delayed response).

Use event_id as an idempotency key in your consumer:

async function handleWebhook(event: WebhookEvent) {
  const seen = await db.webhookSeen.findUnique({ where: { event_id: event.event_id } })
  if (seen) return // already processed

  await processEvent(event)
  await db.webhookSeen.create({ data: { event_id: event.event_id } })
}

Webhooks may also arrive out of order. If you depend on ordering (e.g., run.succeeded must be processed after run.created), use occurred_at rather than the order in which webhooks arrive at your endpoint.

Consumer checklist

Before shipping a webhook integration, confirm each item:

• Verify X-Havilo-Signature on every request — never skip.
• Reject if |now - t| > 300 seconds (replay protection).
• Use timingSafeEqual (or hmac.compare_digest) — never ==.
• Capture the raw request body before parsing.
• Return 2xx within 10 seconds; move processing to a background queue if needed.
• Deduplicate using event_id.
• Order events by occurred_at, not arrival time.
• Log event_id and request_id so support can correlate with server-side logs.
• Have a plan for the failed-delivery case; contact support with the event_id if all retries fail.

Related

• Events — full event catalog and the polling/SSE alternatives to webhooks.
• Runs — callback_url field on POST /runs.
• Reliability — overall idempotency, retry, and limits guidance.