AI Agent Observability

Tracewell

See the tool-call timeline, prompt diff, and token cost behind a failed run.

Buyer & pain

Applied AI Engineer / AI Solutions Architect / Forward Deployed Engineer — agents fail in production at 3 AM, and current tooling (Datadog, LangSmith, n8n run history) can't tell you which tool call consumed the budget, what the prompt looked like at the exact failing step, or how it differed from the last run that worked.

Problem

Nobody renders an agent run as a clickable, replayable, diff-able timeline the way Sentry renders a stacktrace. Infrastructure monitoring knows the Lambda timed out; it doesn't know the prompt drifted 4,400 tokens between Monday and Wednesday. That's the white space.

Before

A wall of JSON. The failure message says context_length_exceeded. Nobody knows why tonight's run is 4,000 tokens heavier than Monday's. The postmortem is a meeting.

After

One click opens the run inspector. The tool-call tree draws itself. The failing node glows amber. "Compare to last success" splits the prompt diff — a single 4,390-token compliance_policy_v3 → v4 injection added Tuesday. Postmortem done in three clicks.

What the 30-second demo proves

  1. Failure-mode vocabulary — context_overflow, guardrail_reject, tool_timeout: you name these only after you've been paged for them.
  2. Hierarchical data as legible UI — a nested tool-call tree with per-node token cost rendered in one screen; the exact take-home FDE and Solutions Architect roles assign.
  3. Root-cause thinking, not log-tailing — failure-mode clustering surfaces one root cause behind six incidents across 50 synthetic runs.
  4. Observability designed in — token spend, latency, and replay affordance per step; matches the n8n/StockX "instrument for auditability" ask verbatim.

How it works

  1. Timeline view — 50 synthetic agent runs (3 agents, 7 days) with status, failure-mode badges, and token totals. Failure-mode sidebar clusters runs by signature: FP-01 · 6 runs · contract-review-agent · context_overflow.
  2. Run inspector — click any row to open a split-panel. Left: tool-call tree with per-node token cost and latency. Right: prompt snapshot, model output, and a character-level diff against the last successful run of the same step.
  3. Signature moment — run_8f2a1c · contract-review-agent · 03:14 · FAILED. Click the red node. Click "Compare to last run that worked." The diff shows a single contiguous 4,390-token block added at inject_context. That's the regression. Three clicks, full postmortem.
  4. Synthetic replay — the replay tab accepts edits and re-runs against a recorded response fixture. Labeled clearly: "Replay is synthetic — edits route to a recorded response fixture."
  5. Analyze your own trace (live backend) — paste or upload a recorded agent-run trace and a real Cloudflare Function (/demos/tracewell/analyze) classifies each run's failure mode, pins the root-cause step, and returns a fix — computed entirely server-side from your trace. The detection rules are real; the data is yours.

Integration surface

Astro site plus a Cloudflare Pages Function for trace analysis. The shipped 50 runs are synthetic and pre-recorded; the /demos/tracewell/analyze endpoint diagnoses an uploaded trace server-side with deterministic rules — no API keys and no live model calls (the analyzer is rule-based, and "edit & replay" routes to a bundled fixture). Diff rendering via diff-match-patch (4 kb gzipped). Deploys as a Cloudflare Pages subpath.

Skills proven

Roles this maps to

Try the live demo → dallascrilley.com