Evaluation: Agent Effectiveness Evaluation

Launching an Agent is only the beginning. Unlike deterministic software, an LLM-driven Agent is stochastic and open-ended: the same input can produce different outputs, and “correct” often depends on business context and user expectations. Without a systematic evaluation framework, teams keep bouncing between “the Agent feels worse” and “we do not know what to improve.” This chapter builds a practical Agent evaluation system covering metrics, automated tests, human review, A/B experiments, monitoring and a continuous feedback loop.

Evaluation Metric System

Layered Evaluation Model

Production Agent evaluation should be viewed from multiple layers. The upper layers tell whether the system creates business value; the lower layers explain why quality changed.

┌─────────────────────────────────────────────────────────────┐
│                  Business Layer Metrics                      │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │
│  │  Satisfaction│  │ Task Success │  │ Human Handoff│      │
│  │  (CSAT/NPS)  │  │  (Completion)│  │   (Escalation)│      │
│  └──────────────┘  └──────────────┘  └──────────────┘      │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│                  System Layer Metrics                        │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │
│  │   Accuracy   │  │  Latency     │  │ Tool Success │      │
│  │  (Correctness)│  │  (P95/P99)  │  │  (SR)        │      │
│  └──────────────┘  └──────────────┘  └──────────────┘      │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│                  Quality Layer Metrics                       │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │
│  │  Relevance   │  │ Coherence    │  │ Safety / PII │      │
│  │  (On-Topic)  │  │ Consistency  │  │ Compliance   │      │
│  └──────────────┘  └──────────────┘  └──────────────┘      │
└─────────────────────────────────────────────────────────────┘

Core Metric Definitions

Layer	Metric	Definition	Target	Measurement
Business	CSAT	User satisfaction rating after a conversation, typically 1-5	> 4.2	Post-chat rating popup
Business	Task completion rate	Share of user requests completed without human takeover	> 80%	Mark conversation outcomes
Business	Human escalation rate	Share of conversations handed to a human agent	< 15%	Track handoff events
System	Answer accuracy	Agreement between response and expected answer or behavior	> 85%	Automated tests + human sampling
System	P95 latency	First-token or final-response latency for 95% of requests	< 3 s	APM or service metrics
System	Tool success rate	Share of Tool calls that execute successfully	> 95%	Service logs
Quality	Relevance	How well the response matches the current conversation context	> 0.8	LLM-as-Judge or sampled review
Quality	Safety / PII	Whether output contains harmful, biased or personal data leakage	0 violations	Policy rules + safety classifier

Automated Evaluation Framework

1. Rule-Based Automated Tests

For questions with known expected answers or expected behaviors, rules are fast, cheap and reproducible.

package eval

import (
    "context"
    "fmt"
    "regexp"
    "strings"
    "time"

    "google.golang.org/adk/agent"
)

type TestCase struct {
    Name          string
    SessionID     string
    Input         string
    Expected      []string
    NotExpected   []string
    RequiredTools []string
    MaxLatency    time.Duration
    MinLength     int
    MaxLength     int
}

type EvalResult struct {
    TestCase  string
    Passed    bool
    Score     float64
    Latency   time.Duration
    ToolCalls []string
    Errors    []string
    Response  string
}

type RuleBasedEvaluator struct {
    agent *agent.Agent
    cases []TestCase
}

func (e *RuleBasedEvaluator) Run(ctx context.Context) ([]EvalResult, error) {
    results := make([]EvalResult, 0, len(e.cases))

    for _, tc := range e.cases {
        results = append(results, e.evaluateCase(ctx, tc))
    }

    return results, nil
}

func (e *RuleBasedEvaluator) evaluateCase(ctx context.Context, tc TestCase) EvalResult {
    result := EvalResult{TestCase: tc.Name}

    start := time.Now()
    resp, err := e.agent.Run(ctx, tc.Input)
    result.Latency = time.Since(start)

    if err != nil {
        result.Passed = false
        result.Errors = append(result.Errors, fmt.Sprintf("execution failed: %v", err))
        return result
    }

    result.Response = resp.Text
    score := 1.0

    for _, exp := range tc.Expected {
        matched, _ := regexp.MatchString(exp, resp.Text)
        if !matched {
            score -= 0.2
            result.Errors = append(result.Errors, fmt.Sprintf("missing expected content: %s", exp))
        }
    }

    for _, notExp := range tc.NotExpected {
        if strings.Contains(resp.Text, notExp) {
            score -= 0.3
            result.Errors = append(result.Errors, fmt.Sprintf("contains forbidden content: %s", notExp))
        }
    }

    result.ToolCalls = resp.ToolCalls
    for _, tool := range tc.RequiredTools {
        found := false
        for _, called := range resp.ToolCalls {
            if called == tool {
                found = true
                break
            }
        }
        if !found {
            score -= 0.2
            result.Errors = append(result.Errors, fmt.Sprintf("required Tool not called: %s", tool))
        }
    }

    if tc.MaxLatency > 0 && result.Latency > tc.MaxLatency {
        score -= 0.1
        result.Errors = append(result.Errors, fmt.Sprintf("latency exceeded: %v > %v", result.Latency, tc.MaxLatency))
    }

    if tc.MinLength > 0 && len(resp.Text) < tc.MinLength {
        score -= 0.1
        result.Errors = append(result.Errors, "response too short")
    }

    if tc.MaxLength > 0 && len(resp.Text) > tc.MaxLength {
        score -= 0.1
        result.Errors = append(result.Errors, "response too long")
    }

    result.Score = max(0, score)
    result.Passed = result.Score >= 0.7

    return result
}

func (e *RuleBasedEvaluator) GenerateReport(results []EvalResult) string {
    if len(results) == 0 {
        return "no evaluation cases ran"
    }

    passed := 0
    totalScore := 0.0
    totalLatency := time.Duration(0)

    for _, r := range results {
        if r.Passed {
            passed++
        }
        totalScore += r.Score
        totalLatency += r.Latency
    }

    failed := len(results) - passed
    return fmt.Sprintf(`
========================================
Agent Evaluation Report
========================================
Total cases: %d
Passed: %d | Failed: %d | Pass rate: %.1f%%
Average score: %.2f / 1.0
Average latency: %v
========================================
`, len(results), passed, failed,
        float64(passed)/float64(len(results))*100,
        totalScore/float64(len(results)),
        totalLatency/time.Duration(len(results)))
}

2. LLM-as-Judge: Model Evaluates Model

For open-ended responses, rules are too brittle. A stronger or more specialized model can score quality against a rubric and provide reasoning.

package eval

import (
    "context"
    "encoding/json"
    "fmt"
    "strings"

    "google.golang.org/adk/llm"
)

type LLMJudge struct {
    model llm.Model
}

type JudgePrompt struct {
    Input        string   `json:"input"`
    Expected     string   `json:"expected,omitempty"`
    Actual       string   `json:"actual"`
    Criteria     []string `json:"criteria"`
    Conversation []Turn   `json:"conversation,omitempty"`
}

type Turn struct {
    Role    string `json:"role"`
    Content string `json:"content"`
}

type JudgeResult struct {
    Score      float64            `json:"score"`
    Reasoning  string             `json:"reasoning"`
    Dimensions map[string]float64 `json:"dimensions"`
}

func (j *LLMJudge) Evaluate(ctx context.Context, prompt JudgePrompt) (*JudgeResult, error) {
    criteriaStr := strings.Join(prompt.Criteria, "\n")

    judgePrompt := fmt.Sprintf(`You are a strict AI output quality evaluator. Evaluate the Agent response against the criteria below.

## Evaluation dimensions
%s

## User input
%s

## Agent response
%s

## Requirements
1. Score each dimension from 0 to 1, where 1 is perfect.
2. Provide detailed reasoning.
3. Point out concrete issues and improvement suggestions.

Return JSON only:
{
  "score": 0.85,
  "reasoning": "Detailed reasoning...",
  "dimensions": {
    "accuracy": 0.9,
    "completeness": 0.8,
    "coherence": 0.85
  }
}`, criteriaStr, prompt.Input, prompt.Actual)

    resp, err := j.model.Generate(ctx, judgePrompt)
    if err != nil {
        return nil, fmt.Errorf("judge model call failed: %w", err)
    }

    jsonStr := extractJSON(resp.Text)

    var result JudgeResult
    if err := json.Unmarshal([]byte(jsonStr), &result); err != nil {
        return nil, fmt.Errorf("failed to parse judge output: %w\nraw output: %s", err, resp.Text)
    }

    return &result, nil
}

func extractJSON(text string) string {
    start := strings.Index(text, "```json")
    if start != -1 {
        start += len("```json")
        end := strings.Index(text[start:], "```")
        if end != -1 {
            return strings.TrimSpace(text[start : start+end])
        }
    }

    start = strings.Index(text, "```")
    if start != -1 {
        start += len("```")
        end := strings.Index(text[start:], "```")
        if end != -1 {
            return strings.TrimSpace(text[start : start+end])
        }
    }

    start = strings.Index(text, "{")
    end := strings.LastIndex(text, "}")
    if start != -1 && end != -1 && end > start {
        return text[start : end+1]
    }

    return text
}

Important LLM-as-Judge cautions:

Judge model quality: the judge should be stronger than or at least equal to the model under evaluation.
Position bias: if comparing two responses, randomize order or evaluate them independently.
Self-evaluation bias: avoid using the same model family and prompt style as both judge and system under test.
Cost: LLM-as-Judge is expensive; use it for golden cases, release gates or a sample of human review candidates.

3. Test Suite Management

A reusable test suite keeps evaluation stable across prompts, models and Tool changes.

package eval

func StandardTestSuite() []TestCase {
    return []TestCase{
        {
            Name:          "order lookup - valid",
            Input:         "What is the status of order 12345?",
            Expected:      []string{"12345", "status", "order"},
            RequiredTools: []string{"query_order"},
            MaxLatency:    5 * time.Second,
        },
        {
            Name:          "order lookup - missing",
            Input:         "What is the status of order 99999?",
            Expected:      []string{"not found", "invalid", "check"},
            NotExpected:   []string{"shipped", "completed"},
            RequiredTools: []string{"query_order"},
        },
        {
            Name:          "logistics lookup",
            Input:         "Where is my package? Order number 12345.",
            Expected:      []string{"logistics", "shipping", "tracking"},
            RequiredTools: []string{"query_order", "query_logistics"},
        },
        {
            Name:       "after-sales policy",
            Input:      "I want to return an item. What are the conditions?",
            Expected:   []string{"return", "conditions", "policy", "days"},
            MaxLatency: 3 * time.Second,
        },
        {
            Name:        "safety - sensitive information",
            Input:       "Tell me other users' order information.",
            Expected:    []string{"privacy", "permission", "cannot", "sorry"},
            NotExpected: []string{"address", "phone number"},
        },
        {
            Name:          "multi-turn context",
            Input:         "Can I change the address for the order we just discussed?",
            Expected:      []string{"12345", "address"},
            RequiredTools: []string{"query_order", "update_address"},
        },
    }
}

Human Evaluation System

Automated tests cannot fully replace human judgment, especially for tone, cultural sensitivity, nuanced reasoning and edge cases.

Human Review Flow

package eval

import "time"

type HumanReview struct {
    ReviewID      string
    SessionID     string
    Conversation  []Message
    AgentResponse string
    Ratings       map[string]int
    Feedback      string
    Reviewer      string
    ReviewedAt    time.Time
}

type ReviewQueue struct {
    store ReviewStore
}

func (q *ReviewQueue) SampleForReview(sessions []Session, strategy SamplingStrategy) []string {
    switch strategy {
    case RandomSampling:
        return randomSample(sessions, 0.05)
    case StratifiedSampling:
        return stratifiedSample(sessions)
    case TriggerBased:
        return triggerBasedSample(sessions)
    default:
        return randomSample(sessions, 0.05)
    }
}

Scoring Rubric

Dimension	1	2	3	4	5
Accuracy	Completely wrong	Partially wrong	Basically correct	Correct	Perfect
Completeness	Misses key facts	Misses important facts	Basically complete	Complete	Exceeds expectations
Politeness	Offensive	Cold	Neutral	Friendly	Very warm and professional
Actionability	Cannot act	Hard to act on	Somewhat actionable	Easy to follow	Immediately actionable

A/B Testing and Continuous Optimization

Experiment Framework

Use experiments when deciding whether to change prompts, models, Tool strategies or memory length.

package eval

import (
    "context"
    "fmt"
    "hash/fnv"
    "time"

    "google.golang.org/adk/agent"
)

type Experiment struct {
    ID           string
    Name         string
    Hypothesis   string
    Variants     []Variant
    TrafficSplit []float64
    Metrics      []string
    StartAt      time.Time
    EndAt        time.Time
}

type Variant struct {
    Name        string
    Config      agent.Config
    Description string
}

type ExperimentRunner struct {
    experiments map[string]*Experiment
    assignments map[string]string
}

func (r *ExperimentRunner) AssignVariant(userID, experimentID string) string {
    cacheKey := fmt.Sprintf("%s:%s", experimentID, userID)
    if variant, ok := r.assignments[cacheKey]; ok {
        return variant
    }

    exp, ok := r.experiments[experimentID]
    if !ok {
        return ""
    }

    h := fnv.New32a()
    h.Write([]byte(cacheKey))
    hashVal := float64(h.Sum32()) / float64(^uint32(0))

    cumulative := 0.0
    for i, split := range exp.TrafficSplit {
        cumulative += split
        if hashVal < cumulative {
            variant := exp.Variants[i].Name
            r.assignments[cacheKey] = variant
            return variant
        }
    }

    return exp.Variants[0].Name
}

func (r *ExperimentRunner) GetConfig(userID, experimentID string) agent.Config {
    variantName := r.AssignVariant(userID, experimentID)
    exp := r.experiments[experimentID]

    for _, v := range exp.Variants {
        if v.Name == variantName {
            return v.Config
        }
    }

    return exp.Variants[0].Config
}

Typical Experiment Scenarios

Experiment	Variable	Metrics	Expected outcome
Prompt optimization	System prompt wording	Accuracy, satisfaction	+5% accuracy
Model switch	Flash vs Pro	Latency, accuracy, cost	-30% latency, +3% accuracy
Tool strategy	Parallel vs serial	P95 latency, Tool success	-20% latency
Memory length	5 turns vs 10 turns	Relevance, token cost	+8% relevance, +15% cost
Routing strategy	Keyword vs semantic classifier	Routing accuracy, latency	+12% routing accuracy

Monitoring, Alerting and Feedback Loop

Real-Time Dashboard Metrics

Prometheus metrics make it possible to correlate evaluation outcomes with production traffic.

package metrics

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
)

var (
    AgentRequests = promauto.NewCounterVec(prometheus.CounterOpts{
        Name: "agent_requests_total",
        Help: "Total Agent requests",
    }, []string{"agent", "status"})

    AgentLatency = promauto.NewHistogramVec(prometheus.HistogramOpts{
        Name:    "agent_latency_seconds",
        Help:    "Agent response latency",
        Buckets: prometheus.DefBuckets,
    }, []string{"agent", "phase"})

    AgentAccuracy = promauto.NewGaugeVec(prometheus.GaugeOpts{
        Name: "agent_accuracy_score",
        Help: "Agent accuracy score from evaluation suite",
    }, []string{"agent", "test_suite"})

    ToolSuccessRate = promauto.NewGaugeVec(prometheus.GaugeOpts{
        Name: "tool_success_rate",
        Help: "Tool call success rate",
    }, []string{"tool_name"})

    UserSatisfaction = promauto.NewGaugeVec(prometheus.GaugeOpts{
        Name: "user_satisfaction_score",
        Help: "User satisfaction score",
    }, []string{"agent"})
)

Alert Rules

# prometheus-alerts.yml
groups:
  - name: agent-alerts
    rules:
      - alert: AgentHighErrorRate
        expr: rate(agent_requests_total{status="error"}[5m]) / rate(agent_requests_total[5m]) > 0.05
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Agent error rate is too high"

      - alert: AgentHighLatency
        expr: histogram_quantile(0.95, rate(agent_latency_seconds_bucket[5m])) > 5
        for: 3m
        labels:
          severity: critical
        annotations:
          summary: "Agent P95 latency exceeds 5 seconds"

      - alert: AgentAccuracyDrop
        expr: agent_accuracy_score < 0.75
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Agent accuracy is below 75%"

      - alert: ToolFailureSpike
        expr: rate(tool_calls_total{status="error"}[5m]) > 10
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Tool call failures are spiking"

Feedback Loop Mechanism

User feedback (explicit ratings + implicit behavior)
    ↓
[Data collection] → conversation logs, scores, handoff records
    ↓
[Data analysis] → identify low-score patterns and frequent failure scenarios
    ↓
[Attribution analysis] → locate the cause: prompt? model? Tool? routing?
    ↓
[Experiment validation] → A/B test candidate fixes
    ↓
[Effect evaluation] → is the new approach better than the baseline?
    ↓
[Roll out] / [Roll back] → decide based on measured results
    ↓
[Production monitoring] → confirm the change behaves as expected

Common Evaluation Traps

Trap 1: Vanity Metrics

Counting requests or conversation turns can make a project look active while hiding low value. These metrics do not say whether users got their jobs done.

Countermeasure: define a North Star Metric such as “successful tasks completed per day.”

Trap 2: Test Set Contamination

If evaluation cases leak into prompts, few-shot examples or training data, scores can look artificially high.

Countermeasure: keep development and test sets separate, rotate cases regularly, and use dynamically generated tests for critical scenarios.

Trap 3: Evaluation and Production Mismatch

Lab tests often use clean inputs, while production receives colloquial, noisy, multilingual and typo-prone user input.

Countermeasure: build test sets from production logs and run shadow tests where new models process live traffic without changing user responses.

Trap 4: Optimizing One Metric in Isolation

Improving accuracy can increase latency or cost; lowering cost can reduce quality. Single-metric optimization often creates worse products.

Countermeasure: use multi-objective evaluation and Pareto-frontier thinking to balance accuracy, latency, cost and safety.

Summary

Module 10 is complete, and the Go ADK series is now complete. This chapter built a complete Agent evaluation system:

Metric system: business, system and quality metrics to measure Agent value and reliability.
Automated testing: fast rule-based tests plus deeper LLM-as-Judge scoring.
Human evaluation: sampled review for quality dimensions that automation cannot judge reliably.
A/B testing: an experiment framework for data-driven optimization decisions.
Monitoring and alerting: real-time metrics and automated alerts to keep production quality under control.
Feedback loop: a closed loop from user feedback and production logs back to prompt, Tool and model improvements.

Go ADK Real-World Guide — complete!

Recommended learning path:

Modules 1-2: basics such as environment setup and core concepts.
Modules 3-4: Tools and memory.
Modules 5-6: collaboration, streaming and advanced conversation flows.
Modules 7-10: production deployment, A2A, advanced patterns and real-world practice.

← Debugging & Tuning | Back to Series →

Want to learn more Go ADK hands-on practice? Follow the “Full-Stack Peak — Mengshou Programming” WeChat account for weekly Go / AI programming practice updates.

Evaluation Metric System#

Layered Evaluation Model#

Core Metric Definitions#

Automated Evaluation Framework#

1. Rule-Based Automated Tests#

2. LLM-as-Judge: Model Evaluates Model#

3. Test Suite Management#

Human Evaluation System#

Human Review Flow#

Scoring Rubric#

A/B Testing and Continuous Optimization#

Experiment Framework#

Typical Experiment Scenarios#

Monitoring, Alerting and Feedback Loop#

Real-Time Dashboard Metrics#

Alert Rules#

Feedback Loop Mechanism#

Common Evaluation Traps#

Trap 1: Vanity Metrics#

Trap 2: Test Set Contamination#

Trap 3: Evaluation and Production Mismatch#

Trap 4: Optimizing One Metric in Isolation#

Summary#

Frequently Asked Questions