梦兽编程™
AI_SUITE

Debugging Notes and Performance Tuning: Common Issues and Solutions

Summarize common issues, debugging notes, and performance tuning experiences in Go ADK development—real project experiences.

May 31, 2026 · 11 min · 2266 words · Mengshou Programming -

This chapter summarizes the typical problems, debugging notes, and performance tuning patterns encountered while building production Agent systems with Go ADK. The material is drawn from multiple real projects and covers the full lifecycle from local development and debugging to production operations. Each issue includes root-cause analysis, practical fixes, and preventive measures so that the same mistake does not repeat.

Common API Layer Issues

Error 1: UNAUTHORIZED — Authentication Failure

Symptoms: The Agent returns UNAUTHORIZED during startup or at runtime, indicating that the API key is invalid.

Root causes:

  1. Environment variable not loaded: .env exists but is not loaded into the process environment.
  2. Malformed key: The copied key contains extra spaces, newlines or quotes.
  3. Insufficient permissions: The API key has not been granted access to the model or service.
  4. Expired or revoked key: The key was revoked in the console or exceeded its validity period.
  5. Project configuration conflict: GOOGLE_APPLICATION_CREDENTIALS and GOOGLE_API_KEY point to different projects.

Fix: Validate and normalize the key early before any model call.

package main

import (
    "fmt"
    "log"
    "os"
    "strings"
)

func validateAPIKey() error {
    key := os.Getenv("GOOGLE_API_KEY")

    if key == "" {
        return fmt.Errorf("GOOGLE_API_KEY is not set")
    }

    key = strings.TrimSpace(key)

    if len(key) < 20 {
        return fmt.Errorf("GOOGLE_API_KEY looks invalid: length %d", len(key))
    }

    if strings.ContainsAny(key, "\n\r\t") {
        return fmt.Errorf("GOOGLE_API_KEY contains invalid whitespace")
    }

    os.Setenv("GOOGLE_API_KEY", key)
    return nil
}

func main() {
    if err := loadEnvFile(".env"); err != nil {
        log.Printf("failed to load .env file; continuing if variables exist elsewhere: %v", err)
    }

    if err := validateAPIKey(); err != nil {
        log.Fatalf("API key validation failed: %v", err)
    }

    // Initialize ADK and other services after the key is known to be present.
}

func loadEnvFile(path string) error {
    data, err := os.ReadFile(path)
    if err != nil {
        return err
    }

    for _, line := range strings.Split(string(data), "\n") {
        line = strings.TrimSpace(line)
        if line == "" || strings.HasPrefix(line, "#") {
            continue
        }

        parts := strings.SplitN(line, "=", 2)
        if len(parts) != 2 {
            continue
        }

        key := strings.TrimSpace(parts[0])
        value := strings.TrimSpace(parts[1])
        value = strings.Trim(value, `"'`)

        if key != "" {
            os.Setenv(key, value)
        }
    }

    return nil
}

Prevention: add API key validation to CI/CD startup checks, store secrets in a Secret Manager rather than in plain text, and expose a health-check endpoint that verifies key validity without leaking the key.

Error 2: RESOURCE_EXHAUSTED — Quota Exhausted

Symptoms: Requests return RESOURCE_EXHAUSTED, rate-limit exceeded or quota reached.

Root causes:

  1. RPM exceeded: traffic spikes exceed Requests Per Minute.
  2. TPD exceeded: daily token usage reaches the quota.
  3. Concurrency exceeded: too many simultaneous connections or streams.
  4. Default quota too low: the project quota was not raised for production traffic.

Fix: use adaptive throttling and backoff so traffic degrades gracefully under quota pressure.

package ratelimit

import (
    "context"
    "log"
    "sync"
    "time"

    "golang.org/x/time/rate"
)

// AdaptiveRateLimiter reduces pressure when quota errors appear and recovers slowly after success.
type AdaptiveRateLimiter struct {
    limiter      *rate.Limiter
    mu           sync.RWMutex
    currentRPM   int
    minRPM       int
    maxRPM       int
    backoffUntil time.Time
}

func NewAdaptiveRateLimiter(minRPM, maxRPM int) *AdaptiveRateLimiter {
    return &AdaptiveRateLimiter{
        limiter:    rate.NewLimiter(rate.Every(time.Minute/time.Duration(maxRPM)), maxRPM),
        currentRPM: maxRPM,
        minRPM:     minRPM,
        maxRPM:     maxRPM,
    }
}

func (a *AdaptiveRateLimiter) Wait(ctx context.Context) error {
    a.mu.RLock()
    backoff := a.backoffUntil
    a.mu.RUnlock()

    if time.Now().Before(backoff) {
        select {
        case <-time.After(time.Until(backoff)):
        case <-ctx.Done():
            return ctx.Err()
        }
    }

    return a.limiter.Wait(ctx)
}

func (a *AdaptiveRateLimiter) OnSuccess() {
    a.mu.Lock()
    defer a.mu.Unlock()

    if a.currentRPM < a.maxRPM {
        a.currentRPM = min(a.currentRPM+1, a.maxRPM)
        a.limiter.SetLimit(rate.Every(time.Minute / time.Duration(a.currentRPM)))
    }
}

func (a *AdaptiveRateLimiter) OnRateLimit() {
    a.mu.Lock()
    defer a.mu.Unlock()

    a.currentRPM = max(a.currentRPM/2, a.minRPM)
    a.limiter.SetLimit(rate.Every(time.Minute / time.Duration(a.currentRPM)))
    a.backoffUntil = time.Now().Add(30 * time.Second)

    log.Printf("rate limit triggered, lowered to %d RPM with 30s backoff", a.currentRPM)
}

func callWithRateLimit(ctx context.Context, limiter *AdaptiveRateLimiter, fn func() error) error {
    if err := limiter.Wait(ctx); err != nil {
        return err
    }

    err := fn()
    if err != nil {
        if isRateLimitError(err) {
            limiter.OnRateLimit()
        }
        return err
    }

    limiter.OnSuccess()
    return nil
}

func isRateLimitError(err error) bool {
    return err != nil && strings.Contains(err.Error(), "RESOURCE_EXHAUSTED")
}

Longer-term strategy: request quota increases in the cloud console, add multi-layer caching, and automatically fall back to lighter models during traffic peaks.

Error 3: context deadline exceeded — Request Timeout

Symptoms: The request is cancelled by context timeout before completion.

Root causes:

  1. Slow model inference: complex prompts or long context increase inference time.
  2. Blocking Tool execution: database or external API calls lack timeouts.
  3. Cascade timeout: outer timeout is shorter than the sum of inner operation timeouts.
  4. Network instability: latency spikes occur between the service and model API.

Fix: make timeouts layered and deliberate. The outer timeout should cover model time, Tool time and serialization time with headroom.

package timeout

import (
    "context"
    "fmt"
    "time"
)

type LayeredTimeout struct {
    TotalTimeout       time.Duration
    ModelTimeout       time.Duration
    ToolTimeout        time.Duration
    StreamChunkTimeout time.Duration
}

func DefaultLayeredTimeout() LayeredTimeout {
    return LayeredTimeout{
        TotalTimeout:       30 * time.Second,
        ModelTimeout:       20 * time.Second,
        ToolTimeout:        5 * time.Second,
        StreamChunkTimeout: 2 * time.Second,
    }
}

func (lt LayeredTimeout) Execute(ctx context.Context, fn func(context.Context) error) error {
    ctx, cancel := context.WithTimeout(ctx, lt.TotalTimeout)
    defer cancel()

    done := make(chan error, 1)
    go func() {
        done <- fn(ctx)
    }()

    select {
    case err := <-done:
        return err
    case <-ctx.Done():
        return fmt.Errorf("total request timeout (%v): %w", lt.TotalTimeout, ctx.Err())
    }
}

func CallToolWithTimeout(ctx context.Context, toolName string, timeout time.Duration, fn func(context.Context) (string, error)) (string, error) {
    ctx, cancel := context.WithTimeout(ctx, timeout)
    defer cancel()

    result, err := fn(ctx)
    if err != nil {
        if ctx.Err() == context.DeadlineExceeded {
            return fmt.Sprintf("[%s timed out, please try again later]", toolName), nil
        }
        return "", err
    }

    return result, nil
}

Rule of thumb: keep TotalTimeout > ModelTimeout + ToolTimeout * MaxToolCalls, and leave roughly 20% buffer for serialization, network jitter and retry overhead.

Performance Diagnosis and Tuning

Issue 1: High Response Latency

Diagnostic flow:

User perceives high latency
    ↓
[Break down latency] Use distributed tracing such as OpenTelemetry to separate phase costs
    ↓
├─ Network transfer?      → Check CDN, regional deployment, connection pool reuse
├─ Model inference?       → Model fallback, prompt simplification, context caching
├─ Tool execution?        → Parallelize Tools, cache repeated results, set per-Tool timeouts
├─ Routing decision?      → Simplify classifier, cache routing decisions
└─ Serialization?         → Prefer smaller payloads and efficient codecs where applicable

Model inference optimization: cache repeated prefix context and choose models according to query complexity.

func (a *Agent) buildPromptWithCache(ctx context.Context, userInput string) (string, error) {
    cacheKey := a.config.Instruction + a.session.GetSummary()

    if cached, ok := a.promptCache.Get(cacheKey); ok {
        return cached.(string) + "\nUser: " + userInput, nil
    }

    fullPrompt := a.config.Instruction + "\n" + a.session.GetHistory() + "\nUser: " + userInput
    a.promptCache.Set(cacheKey, fullPrompt, 10*time.Minute)
    return fullPrompt, nil
}

func selectModel(query string, history []Message) string {
    if isSimpleQuery(query) && len(history) < 3 {
        return "gemini-2.0-flash"
    }

    if requiresComplexReasoning(query) {
        return "gemini-2.0-pro"
    }

    return "gemini-2.0-flash"
}

Parallel Tool calls: independent Tools should not wait for each other.

func (a *Agent) callToolsParallel(ctx context.Context, toolCalls []ToolCall) []ToolResult {
    var wg sync.WaitGroup
    results := make([]ToolResult, len(toolCalls))

    for i, tc := range toolCalls {
        wg.Add(1)
        go func(index int, call ToolCall) {
            defer wg.Done()

            result, err := a.executeTool(ctx, call)
            results[index] = ToolResult{
                ToolName: call.Name,
                Result:   result,
                Error:    err,
            }
        }(i, tc)
    }

    wg.Wait()
    return results
}

Issue 2: Memory Spike and OOM

Root causes:

  1. Session accumulation: expired or abandoned sessions are not evicted.
  2. State inflation: full conversation history or large artifacts are stored in Session state.
  3. Goroutine leaks: streaming or background tasks are not closed cleanly.
  4. Unbounded cache: caches grow without size or TTL limits.

Fix: add active memory monitoring, forced cleanup and explicit per-session state caps.

package memory

import (
    "encoding/json"
    "log"
    "runtime"
    "sync"
    "time"

    "github.com/shirou/gopsutil/mem"
)

type MemoryGuard struct {
    maxMemoryMB  uint64
    gcThreshold  float64
    sessionLimit int
    checkInterval time.Duration
}

func (g *MemoryGuard) Start(sessionManager *SessionManager) {
    ticker := time.NewTicker(g.checkInterval)
    go func() {
        for range ticker.C {
            v, _ := mem.VirtualMemory()
            if v.UsedPercent > g.gcThreshold {
                log.Printf("memory usage %.1f%% exceeds threshold %.1f%%", v.UsedPercent, g.gcThreshold)

                runtime.GC()

                evicted := sessionManager.EvictExpired(30 * time.Minute)
                log.Printf("evicted %d expired sessions", evicted)

                if v, _ := mem.VirtualMemory(); v.UsedPercent > g.gcThreshold {
                    lruEvicted := sessionManager.EvictLRU(g.sessionLimit / 2)
                    log.Printf("LRU evicted %d sessions", lruEvicted)
                }
            }
        }
    }()
}

type SessionManager struct {
    sessions    map[string]*Session
    mu          sync.RWMutex
    maxSize     int
    maxStateSize int
}

func (sm *SessionManager) SetState(sessionID string, key string, value interface{}) error {
    sm.mu.Lock()
    defer sm.mu.Unlock()

    session, ok := sm.sessions[sessionID]
    if !ok {
        return fmt.Errorf("session not found: %s", sessionID)
    }

    data, _ := json.Marshal(value)
    if len(data) > sm.maxStateSize {
        return fmt.Errorf("state too large: %d bytes, maximum allowed %d", len(data), sm.maxStateSize)
    }

    session.State[key] = value
    session.LastActive = time.Now()
    return nil
}

Issue 3: Goroutine Leak

A common leak is a streaming goroutine that never exits when the consumer disconnects.

func badStream() <-chan string {
    ch := make(chan string)
    go func() {
        for {
            msg := <-someSource
            ch <- msg // may block forever if the consumer has gone away
        }
    }()
    return ch
}

func goodStream(ctx context.Context) <-chan string {
    ch := make(chan string)
    go func() {
        defer close(ch)
        for {
            select {
            case msg := <-someSource:
                select {
                case ch <- msg:
                case <-ctx.Done():
                    return
                }
            case <-ctx.Done():
                return
            }
        }
    }()
    return ch
}

Detection: monitor goroutine count and dump stacks when it grows significantly above baseline.

import "runtime"

func monitorGoroutines() {
    ticker := time.NewTicker(time.Minute)
    go func() {
        baseline := runtime.NumGoroutine()
        for range ticker.C {
            current := runtime.NumGoroutine()
            if current > baseline*2 {
                log.Printf("[ALERT] goroutine count spike: %d (baseline: %d)", current, baseline)
                buf := make([]byte, 1<<20)
                n := runtime.Stack(buf, true)
                log.Printf("goroutine stacks:\n%s", buf[:n])
            }
        }
    }()
}

Debugging Skills and Tools

Structured Logging

Structured logs are much easier to search and aggregate than plain text logs, especially when correlating requests, Agents and Tool calls.

package logging

import (
    "context"
    "encoding/json"
    "log"
    "os"
    "time"
)

type LogLevel int

const (
    DEBUG LogLevel = iota
    INFO
    WARN
    ERROR
)

func (l LogLevel) String() string {
    switch l {
    case DEBUG:
        return "debug"
    case INFO:
        return "info"
    case WARN:
        return "warn"
    case ERROR:
        return "error"
    default:
        return "unknown"
    }
}

type AgentLogger struct {
    logger *log.Logger
    level  LogLevel
}

type LogEntry struct {
    Timestamp time.Time              `json:"timestamp"`
    Level     string                 `json:"level"`
    Agent     string                 `json:"agent"`
    SessionID string                 `json:"session_id"`
    Phase     string                 `json:"phase"`
    Message   string                 `json:"message"`
    Duration  int64                  `json:"duration_ms,omitempty"`
    Error     string                 `json:"error,omitempty"`
    Metadata  map[string]interface{} `json:"metadata,omitempty"`
}

func (l *AgentLogger) Log(ctx context.Context, level LogLevel, phase, message string, metadata map[string]interface{}) {
    if level < l.level {
        return
    }

    entry := LogEntry{
        Timestamp: time.Now(),
        Level:     level.String(),
        Agent:     ctx.Value("agent_name").(string),
        SessionID: ctx.Value("session_id").(string),
        Phase:     phase,
        Message:   message,
        Metadata:  metadata,
    }

    if duration, ok := metadata["duration"]; ok {
        entry.Duration = duration.(int64)
        delete(metadata, "duration")
    }

    if err, ok := metadata["error"]; ok {
        if e, ok := err.(error); ok {
            entry.Error = e.Error()
        }
        delete(metadata, "error")
    }

    data, _ := json.Marshal(entry)
    l.logger.Println(string(data))
}

Usage example: log before/after each Agent phase and attach request IDs, session IDs and elapsed time so problems can be found quickly in production.

Local Debugging Configuration

pprof can be enabled behind a build tag so it is available during local development but not shipped by default.

// debug.go
//go:build debug

package main

import (
    "log"
    "net/http"
    _ "net/http/pprof"
)

func init() {
    go func() {
        log.Println("pprof debug server listening at http://localhost:6060")
        log.Fatal(http.ListenAndServe("localhost:6060", nil))
    }()
}

Run the debug build with:

go run -tags debug ./cmd/server

Request Tracing Middleware

Trace every request from entry to exit with a stable request ID and duration.

func traceMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        requestID := r.Header.Get("X-Request-ID")
        if requestID == "" {
            requestID = generateRequestID()
        }

        ctx := context.WithValue(r.Context(), "request_id", requestID)
        ctx = context.WithValue(ctx, "start_time", time.Now())

        w.Header().Set("X-Request-ID", requestID)

        next.ServeHTTP(w, r.WithContext(ctx))

        duration := time.Since(ctx.Value("start_time").(time.Time))
        log.Printf("[%s] %s %s - %v", requestID, r.Method, r.URL.Path, duration)
    })
}

Deployment and Operations Pitfalls

Pitfall: Configuration Drift

Development, staging and production use different configuration mechanisms, leading to “it works on my machine” failures.

Fix: use one explicit configuration object across environments.

type Config struct {
    Environment    string `env:"ENV" envDefault:"development"`
    Port           int    `env:"PORT" envDefault:"8080"`
    GoogleAPIKey   string `env:"GOOGLE_API_KEY" required:"true"`
    RedisURL       string `env:"REDIS_URL" envDefault:"redis://localhost:6379"`
    LogLevel       string `env:"LOG_LEVEL" envDefault:"info"`
    MaxConcurrency int    `env:"MAX_CONCURRENCY" envDefault:"100"`

    Model         string `env:"MODEL" envDefault:"gemini-2.0-flash"`
    ModelTimeout  int    `env:"MODEL_TIMEOUT_SEC" envDefault:"30"`
    RateLimitRPM  int    `env:"RATE_LIMIT_RPM" envDefault:"60"`
}

func LoadConfig() (*Config, error) {
    var cfg Config
    if err := env.Parse(&cfg); err != nil {
        return nil, fmt.Errorf("failed to parse config: %w", err)
    }
    return &cfg, nil
}

Pitfall: Missing Graceful Shutdown

If the Agent service is killed abruptly during deploys, in-flight conversations and streams can be interrupted.

Fix: listen for termination signals and shut down with a timeout.

func main() {
    // ... initialize config, Agent, plugins and router ...

    srv := &http.Server{
        Addr:    fmt.Sprintf(":%d", cfg.Port),
        Handler: router,
    }

    go func() {
        if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
            log.Fatalf("server start failed: %v", err)
        }
    }()

    quit := make(chan os.Signal, 1)
    signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
    <-quit

    log.Println("gracefully shutting down...")

    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()

    if err := srv.Shutdown(ctx); err != nil {
        log.Printf("server shutdown error: %v", err)
    }

    for _, plugin := range plugins {
        plugin.Shutdown(ctx)
    }

    log.Println("server closed safely")
}

Summary

Debugging and performance tuning are complete. The core lessons from this chapter are:

  1. API layer: authentication, quota and timeout errors are the most common failures; handle them with startup validation, quota-aware retries and layered timeouts.
  2. Performance layer: latency tuning requires end-to-end tracing across network, model inference, Tool execution and serialization.
  3. Memory layer: proactively monitor memory, cap Session state and cache size, and evict expired data before OOM occurs.
  4. Debugging layer: structured logs, request tracing and pprof are the core tools for production troubleshooting.
  5. Operations layer: unified configuration and graceful shutdown are basic production requirements.

Next comes the final topic: Evaluation — measuring whether the Agent is actually effective.

← End-to-End Project | Evaluation →

Want to learn more Go ADK hands-on practice? Follow the “Full-Stack Peak — Mengshou Programming” WeChat account for weekly Go / AI programming practice updates.

Frequently Asked Questions