Ep. 6: Observability & SLOs for AI agent workloads with Honeycomb

Back when programs were deterministic, we could hope to predict production behavior based on tests. With AI involved, this is impossible. There's a nondeterministic black box in our system; we need to find out what went in, what went out, and what the consequences were. Observability is made for this. Distributed tracing across an application provides the picture of how agent-related software works in each unique case. Meanwhile, SLOs help monitor performance without knowing everything we need to watch out for. In this session, we'll see the innards of an AI agent through tracing.

You'll come away with an understanding of:

• SLOs for monitoring agent workloads
• LLM evaluation techniques
• Instrumentation with OpenTelemetry

Be prepared to tackle the challenge of understanding production behavior enough to improve it, lowering costs and boosting user experience.