DeCeroAI Solutions

Specialized AI services firm

AI Agent Observability

Built operational observability for an LLM agent in production: heartbeat, users served, top tools, spend, errors, and deploys. Most teams ship the agent and then discover they have no idea what it is doing.

Headline number
7-day operational dashboard
Role
Built
Category
Security & Operations
Updated

Architecture

  1. Agent runtimestructured log lines
  2. Log Analytics
  3. KQL queriesheartbeat, users, tools, spend
  4. 7-day HTML dashboard
    On-demand digest tool
Structured logs from the agent flow into Log Analytics. KQL queries feed a static dashboard and an in-product digest tool.

Key takeaways

  • Structured logs flow to Log Analytics and are queried with KQL
  • Heartbeat health, unique callers, top tools, error rates, deploys, and spend by user on one page
  • Static HTML render so the dashboard opens locally and shares without a server
  • An in-product digest tool, so the agent can report on itself when asked

The situation

Observability for a non-deterministic system is its own discipline. Request counts and latency tell you almost nothing about whether an agent is helping, looping, or quietly spending money. The team needed to see what the agent was doing, for whom, and at what cost.

What I built

Every tool call, model call, error, and deploy emits a structured log line. Those stream into Log Analytics, and a set of KQL queries turns them into a seven-day picture: heartbeat with per-tick execution detail, unique callers enriched to human-readable names, top tools, error rates correlated with deploys, and model spend broken down by user.

A Python renderer produces a standalone HTML dashboard that opens from a file and can be shared in a message. Inside the product, a digest tool lets anyone ask the agent for the same summary on demand.

The outcome

When something regresses after a deploy, the correlation is on the page within the hour. Spend conversations have numbers in them. The agent is a system the team operates rather than a black box they hope about.

Stack

Azure Kubernetes ServiceLog AnalyticsKQLPythonSlack API

Related work

Same discipline, different system.

Data Platforms

Data Observability: Feed Health at Scale

Every external feed, sweep, and scheduled job leaves evidence. A sampler judges each one every five minutes against its own freshness rule, draws 90 days of history, and emails once when a feed goes red and once when it recovers.

Read the case study
AI Agents & Runtimes

Multi-Tool AI Agent Platform

Shipped an autonomous AI operator that lives in a company's Slack workspace, built on the open-source OpenClaw harness. 84 tools across 13 SaaS surfaces: it reads context, takes action, and reports back.

Read the case study
Security & Operations

Agent Security Architecture

Designed and documented the security model for an AI agent with private data access, external egress, and untrusted input, the worst-case surface for prompt injection. Controls that hold up.

Read the case study

Need this built inside your organization?

One person, architecture to deployment. Tell me what is stuck and I will tell you where I would start.

Book a callAll work →