magnus919_agent-skills/llamaindex/references/evaluation-workflow.md

Evaluation Workflow — ParamTuner and Metrics

This reference shows how to set up systematic evaluation for a LlamaIndex RAG pipeline, including parameter tuning, evaluator configuration, and batch scoring.

Setup

from llama_index.core.evaluation import (
    FaithfulnessEvaluator,
    RelevancyEvaluator,
    SemanticSimilarityEvaluator,
    BatchEvalRunner,
)
from llama_index.llms.openai import OpenAI

gpt4 = OpenAI(model="gpt-4o")

faith_evaluator = FaithfulnessEvaluator(llm=gpt4)
rel_evaluator = RelevancyEvaluator(llm=gpt4)
sim_evaluator = SemanticSimilarityEvaluator(llm=gpt4)

Single Query Evaluation

response = query_engine.query("What is the rate limit for the API?")

faith_result = faith_evaluator.evaluate_response(response=response)
print(f"Faithfulness: {faith_result.score} — {faith_result.feedback}")
# Example output: Faithfulness: 0.92 — The answer is grounded in the provided context

rel_result = rel_evaluator.evaluate_response(
    response=response,
    question="What is the rate limit for the API?"
)
print(f"Relevancy: {rel_result.score} — {rel_result.feedback}")
# Example output: Relevancy: 0.88 — The answer addresses the core question

Batch Evaluation

eval_questions = [
    "How do I authenticate?",
    "What are the rate limits?",
    "How do I paginate results?",
    "What error codes exist?",
    "How do I handle webhooks?",
]

# Get responses
responses = [query_engine.query(q) for q in eval_questions]

# Batch evaluate
runner = BatchEvalRunner(
    {
        "faithfulness": FaithfulnessEvaluator(),
        "relevancy": RelevancyEvaluator(),
    },
    workers=4,
)

results = runner.evaluate_responses(eval_questions, responses)

for metric_name, metric_results in results.items():
    scores = [r.score for r in metric_results]
    print(f"{metric_name}: mean={sum(scores)/len(scores):.2f}, "
          f"min={min(scores):.2f}, max={max(scores):.2f}")
# Example output:
# faithfulness: mean=0.91, min=0.78, max=1.00
# relevancy: mean=0.85, min=0.72, max=0.94

ParamTuner — Systematic Optimization

from llama_index.core import VectorStoreIndex
from llama_index.core.param_tuner.base import ParamTuner
from llama_index.core.evaluation import SemanticSimilarityEvaluator
import numpy as np

def build_and_evaluate(params):
    chunk_size = params["chunk_size"]
    top_k = params["top_k"]

    # Build index with these parameters
    index = VectorStoreIndex.from_documents(
        documents,
        transformations=[SentenceSplitter(chunk_size=chunk_size)]
    )

    # Query
    query_engine = index.as_query_engine(similarity_top_k=top_k)
    responses = [query_engine.query(q) for q in eval_questions]

    # Evaluate
    evaluator = SemanticSimilarityEvaluator(llm=gpt4)
    scores = []
    for i, resp in enumerate(responses):
        result = evaluator.evaluate_response(
            response=resp,
            reference=reference_answers[i]
        )
        scores.append(result.score)

    return np.mean(scores)

param_tuner = ParamTuner(
    param_fn=build_and_evaluate,
    param_dict={
        "chunk_size": [256, 512, 1024],
        "top_k": [2, 5, 10],
    },
    fixed_param_dict={
        "documents": documents,
        "eval_questions": eval_questions[:3],
    },
)

results = param_tuner.tune()
best = results.best_run_result
print(f"Best: chunk_size={best.params['chunk_size']}, "
      f"top_k={best.params['top_k']}, score={best.score:.3f}")
# Example output: Best: chunk_size=512, top_k=5, score=0.894

Seven RAG Measurement Aspects

Based on (Gao, Yunfan et al., 2023), evaluate across these dimensions:

Aspect	What it measures	Evaluator
Answer Relevance	Does the answer address the question?	RelevancyEvaluator
Context Relevance	Is the retrieved context on-topic?	RelevancyEvaluator (on context)
Faithfulness	Is the answer grounded in context?	FaithfulnessEvaluator
Context Recall	Are all needed chunks retrieved?	Custom (check coverage)
Context Precision	Are irrelevant chunks excluded?	Custom (check rank order)
Noise Sensitivity	Does noise degrade answers?	Compare with/without noise
Answer Correctness	Are facts correct?	SemanticSimilarityEvaluator

Evaluation Best Practices

1. Use held-out queries — never tune on the same questions you evaluate on
2. Run evaluation in the same process — span-attached scoring preserves trace context
3. Monitor in production — offline notebook scoring catches known issues; production monitoring catches novel ones
4. Combine multiple metrics — faithfulness alone misses relevancy failures and vice versa
5. Track over time — regressions are easier to catch when you have a baseline