LLM & Agent Evaluation: Trajectories, RAGAS, LLM-as-Judge, and Hallucination Mitigation

Evaluating a standard LLM is relatively simple: give it an input, score its output against a reference answer or rubric. Accuracy on a benchmark is a single number.

Evaluating an agent is fundamentally different because agents produce trajectories — sequences of tool calls, reasoning steps, observations, and decisions — not single outputs. This creates evaluation problems that have no equivalent in standard model evaluation.

Correct answer, wrong path: An agent that solves a math problem by hallucinating an intermediate step and happening to get the right final answer will score 100% on outcome evaluation. But its reasoning is brittle — change the numbers slightly and it fails. A trajectory evaluator would catch that the intermediate step was wrong, flagging this agent as unreliable despite its apparent success.

Wrong answer, correct reasoning: An agent might follow exactly the right approach but fail because a retrieval tool returned outdated data. Outcome evaluation marks this as a failure; trajectory evaluation reveals the failure was in the environment (tool reliability), not the agent's reasoning. This distinction matters enormously for diagnosis and improvement.

The evaluation environment problem (the insight most candidates miss): Agents look better or worse depending on tool reliability. A flaky web search API that returns timeouts on 20% of calls will make the agent look unintelligent — it retries, gets inconsistent results, and eventually gives a confused answer. The failure is infrastructure, not reasoning. Production evaluation must measure agent quality holding environment quality constant.

Outcome evaluation asks a binary question: did the agent complete the task? Score = 1 if the final state matches the goal, 0 otherwise. This is easy to implement and aligns with user value (users care about whether the task got done), but it's a coarse signal that misses reasoning quality.

Trajectory evaluation evaluates each step in the agent's reasoning chain:

• Did the agent identify the right sub-tasks to solve?
• Did it call the right tools with the right parameters at each step?
• Did it correctly interpret tool outputs?
• Did it make a coherent next decision given the observation?
• Did it terminate at the right point (not too early, not too late)?

Trajectory evaluation requires a reference trajectory or a rubric that specifies what the optimal trajectory looks like. For complex open-ended tasks, generating reference trajectories is expensive — often requiring senior engineers to manually trace optimal paths. This is why trajectory evaluation is common in research (WebArena, GAIA have reference trajectories) but underused in production.

The hybrid approach used in practice: Run outcome evaluation on all queries. For queries where the outcome fails, run trajectory evaluation to diagnose why it failed (wrong tool? wrong parameter? ignored observation? premature termination?). This targets the expensive trajectory evaluation where it provides the most signal.

Tool call evaluation is the most granular form of trajectory evaluation. For each step in the agent's execution, you measure:

Tool selection precision: Of the tool calls the agent made, what fraction were the correct tool for the situation? A coding agent that calls web_search instead of run_tests to verify its code has low tool selection precision.

Tool selection recall: Of the tool calls the agent should have made, what fraction did it actually make? An agent that solves a problem without calling retrieve_context when context was clearly needed has low tool selection recall.

Parameter correctness: Even when the right tool is selected, were the parameters correct? vector_search(query="Python syntax") when the correct query is vector_search(query="Python asyncio event loop") selects the right tool but with a poorly specified parameter. This is measured by semantic similarity between the agent's parameter and the optimal parameter.

WebArena (Zhou et al., 2023) is the canonical benchmark that measures these metrics on web navigation tasks. Agents are scored on task completion rate, but the paper's analysis breaks down failures by: wrong element clicked (tool selection error), wrong text typed (parameter error), correct action but wrong timing (sequencing error).

Production implementation: Log all tool calls with their arguments. Build an automated evaluator that checks each log against a set of rules: "for query type X, tool Y should be called before tool Z", "parameter of type P should match schema S". Flag deviations for human review.

GAIA (General AI Assistants benchmark, Mialon et al., 2023, Meta AI + HuggingFace) is the most revealing benchmark for production agent capability. It contains 466 real-world tasks requiring multi-step reasoning and tool use: web search, code execution, file reading, and mathematical reasoning. Tasks range from simple one-step queries to complex multi-day research tasks.

The numbers that matter:

• Human baseline: 92% success rate
• GPT-4 with tools (2023): ~30%
• Best agents as of 2024: ~50–60% (AutoGPT-style, GPT-4o with code interpreter)
• Human vs best agent gap: ~32–42 percentage points

This enormous gap reveals that current agents are not approaching human performance on general multi-step tasks — despite performing near-human on narrow benchmarks like HumanEval (code generation). The GAIA failures cluster around: finding obscure information requiring multiple search reformulations, composing tools in non-obvious sequences, and handling ambiguous task specifications that humans resolve with common sense.

Why GAIA is more honest than HumanEval or GSM8K: Those benchmarks test the LLM's parametric knowledge. GAIA tests the agent's ability to use tools to find information it doesn't know. It's the difference between a test where you can use the internet vs. a closed-book exam. Real-world agent deployments face GAIA-style tasks, not HumanEval-style tasks.

LLM-as-judge (Zheng et al., 2023) uses a powerful LLM (typically GPT-4o or Claude 3.5 Sonnet) to evaluate another LLM's outputs. It's cheaper than human annotation (~$0.01–0.05 per evaluation vs. ~$1–5 for human) and faster (seconds vs. hours). Agreement with human ratings is ~0.85 on factual tasks and ~0.7 on creative/open-ended tasks.

Where it works well: Factual accuracy evaluation, format compliance, safety policy violations, response coherence. Any task with clear, objective criteria that can be specified in a rubric.

The four failure modes that matter in production:

Positional bias: LLMs rate the first response in a side-by-side comparison higher than the second, regardless of quality. In a study evaluating GPT-4-judge on response pairs, swapping the order changed the winner ~27% of the time. Mitigation: always evaluate in both orders; require the judge to justify its ranking; penalize positional wins without substantive explanation.

Self-preference bias (self-serving bias): GPT-4-judge rates GPT-4-generated content higher; Claude-judge rates Claude-generated content higher. The magnitude is ~5–15% on pairwise comparisons. Mitigation: use a different model family as judge than as generator. If your agent uses Claude 3.5, evaluate with GPT-4o as judge.

Verbosity bias: Longer, more detailed responses are rated higher even when shorter responses are more accurate. LLM judges are not immune to being fooled by confident, detailed-sounding responses. Mitigation: add explicit rubric criteria that penalize unnecessary verbosity; include a "conciseness" dimension with equal weight.

Score inflation over time: When LLM-as-judge is used to generate training data for RLHF or DPO, the resulting model learns to generate outputs that LLM-judge scores highly — including surface features like polite phrasing that judges reward but users don't value. Mitigation: periodically recalibrate judge scores against fresh human annotations.

SWE-bench (Jimenez et al., 2023) measures agents' ability to fix real GitHub issues from popular Python repositories. It became the de facto benchmark for coding agents. But by late 2024, several labs reported inflated performance because their training data included GitHub issue discussions, commit messages, and PR descriptions that referenced the exact issues in the benchmark.

The contamination mechanism: An agent trained on data that includes "Issue #1234: Fix the IndexError in list_comprehension.py — Solved by removing the off-by-one in line 42" effectively memorizes the solution. On evaluation, it appears to "reason" its way to the answer but is actually recalling training data. This isn't fabricated — it happens naturally when training corpora include GitHub at scale.

GAIA's contamination resistance: GAIA tasks involve finding specific obscure facts (e.g., "What is the highest mountain in the country that borders both X and Y?") that require combining retrieved information, not recalling memorized answers. Contamination is harder because the task can't be solved by memorizing a fixed answer.

What this means for your evaluation design: When building internal benchmarks, source tasks from your actual production data, not from public benchmarks. Public benchmarks are subject to contamination if your training data includes the internet. Your internal regression suite (curated from real failures) is contamination-immune because it was generated after your training cutoff.

For agents that retrieve and ground responses (Agentic RAG, retrieval-heavy assistants), evaluation must measure retrieval quality and generation grounding separately — a correct final answer can come from broken retrieval (model fell back on parametric memory) and a wrong final answer can come from good retrieval (model ignored the context).

The RAGAS framework (Es et al., 2023) defines four production metrics:

• Faithfulness: fraction of claims in the response that are entailed by retrieved context. Detects hallucinated facts even when retrieval was good.
• Answer relevance: how well the response addresses the user's actual question. Detects topic drift and verbose-but-off-target answers.
• Context precision: fraction of retrieved chunks that are actually useful for answering. Low precision means noisy retrieval polluting context.
• Context recall: whether all needed facts are present in the retrieved context. Low recall means key documents weren't surfaced.

Run RAGAS metrics alongside outcome and trajectory evaluators. Diagnose by metric: low faithfulness + high context recall → generation problem (model ignored context); low context recall → retrieval problem (chunks missing or ranked wrong).

BLEU/ROUGE caveat: classical NLP metrics reward string overlap with a reference. Modern LLM outputs can be correct with very different wording, so BLEU/ROUGE under-score good answers and sometimes over-score wrong-but-similar text. Reject them as primary metrics.

Hallucination is a pipeline problem, not just a model problem: weak retrieval, noisy context, permissive prompts, and missing output checks all contribute. The interviewer trap is proposing a silver bullet ("just use lower temperature" or "just add RAG"). Production systems need layered defense where each layer catches a different failure class:

1. Grounding layer — retrieval + citations + context pruning. Pulls the right evidence and removes noise.
2. Generation layer — policy-aware prompt + structured output (JSON schema, claim IDs). Constrains the model from free-form drift.
3. Verification layer — citation span check, NLI/entailment, claim-to-evidence alignment. Independent of the generator.
4. Fallback layer — if any gate fails, return "I don't know" or HITL escalation rather than a speculative answer.

Each layer is independent; failure in one is caught by the next. Constitutional AI (Bai et al., 2022) belongs in layer 2 as a policy shaper — it improves behavior under self-critique and RLAIF, but it does not replace the verification layer. Treating Constitutional AI as your full reliability layer is a senior-level mistake.