Tracing, debugging, evaluation

Already exporting OpenTelemetry to Application Insights and scoring answers? Skip to the next lesson.

Your agent gives a wrong answer. Where do you even look? A model call is not a crash. There is no stack trace. This lesson is about seeing inside.

Tracing

Tracing records what happened during a run: which model calls were made, which tools ran, how long each took, what came back.

OpenTelemetry is the standard way of recording that, used across the industry and not specific to Azure. Application Insights is where those records land inside Azure, and where you can search them afterwards.

You have been using a trace all along. The panel on the right of every lesson is one. Now you wire the real thing.

The failure that looks like success

Here is the one worth remembering.

Your tool runs. It returns successfully. Your dashboard is green. And the model's answer ignores what the tool said completely.

That happens when the result never made it back into the conversation, which is the mistake Module 2 warned about. Every metric says fine. The answer is wrong. A green tool call proves the tool ran, and nothing more.

Evaluation

Evaluation is scoring answers instead of eyeballing them.

Groundedness is the most useful score to start with: how much of the answer is actually supported by the sources retrieved. It turns "seems alright" into a number you can set a floor under and refuse to ship below.

What to write

1. OTEL_ENABLED = True
2. APPINSIGHTS_CONNECTION, a connection string starting InstrumentationKey=
3. EVAL_GROUNDEDNESS = 4 or higher
4. Dispatch lookup_policy and feed its result back before the next chat

The point

You cannot debug what you cannot see, and a successful tool call does not mean the result was used.

Break it on purpose

Leave the tool result out of messages. The trace shows the tool ran fine. The check still fails. Sit with that for a second: that is a bug that would pass every dashboard you own.