梦兽编程™
AI_SUITE

Agent Routing Deep Dive: Dynamic Dispatch Strategies, Load Balancing, and Intelligent Scheduling

In-depth analysis of the ADK Go Agent routing mechanism design, covering rule-based routing, LLM routing, hybrid routing strategies, load-aware scheduling, and production-grade traffic management.

May 30, 2026 · 12 min · 2381 words · Mengshou Programming -

Agent routing is the “traffic command center” of a multi-Agent system—it decides which Agent should handle a user request. A well-designed routing system can significantly improve throughput, reduce latency, and increase accuracy. This post dives into the architecture design, strategy selection, and production practice of routing mechanisms in ADK Go.

Architectural Position of the Routing System

Position in the System

                    User Request
                        │
                        ▼
                ┌───────────────┐
                │     Router     │ ← Routing layer (focus of this post)
                │ (Dispatch Hub) │
                └───────┬───────┘
                        │
        ┌───────────────┼───────────────┐
        │               │               │
        ▼               ▼               ▼
   ┌─────────┐    ┌─────────┐    ┌─────────┐
   │ Agent A │    │ Agent B │    │ Agent C │
   │(Weather)│    │(Order)  │    │(General)│
   └─────────┘    └─────────┘    └─────────┘

Core responsibilities of the routing system:

  1. Intent recognition: understand the type of user request.
  2. Agent matching: find the most suitable Agent to handle it.
  3. Load balancing: avoid overloading a single point.
  4. Degradation and fault tolerance: switch quickly on failure.
  5. Cache optimization: avoid repeated routing decisions.

Routing Strategy System

1. Rule-Based Routing

Match based on predefined rules, with the highest determinism:

// Rule-based routing engine
type RuleRouter struct {
    rules        []RoutingRule
    defaultAgent *agent.Agent
    cache        *RouteCache
    metrics      *RouterMetrics
}

type RoutingRule struct {
    Name          string
    Priority      int                    // higher number = higher priority
    Matcher       RuleMatcher            // matcher
    TargetAgent   *agent.Agent           // target Agent
    Preprocessors []Preprocessor         // preprocessor chain
    Transformers  []Transformer          // input transformers
}

// Rule matcher interface
type RuleMatcher interface {
    Match(input string, context *RoutingContext) (bool, float64)
}

// Keyword matcher
type KeywordMatcher struct {
    Keywords      []string
    MatchMode     MatchMode // ANY | ALL | EXACT
    CaseSensitive bool
}

func (m *KeywordMatcher) Match(input string, context *RoutingContext) (bool, float64) {
    inputLower := strings.ToLower(input)
    matched := 0

    for _, keyword := range m.Keywords {
        kw := keyword
        if !m.CaseSensitive {
            kw = strings.ToLower(kw)
        }

        if strings.Contains(inputLower, kw) {
            matched++
            if m.MatchMode == MatchModeAny {
                return true, 1.0
            }
        }
    }

    switch m.MatchMode {
    case MatchModeAll:
        return matched == len(m.Keywords), float64(matched) / float64(len(m.Keywords))
    case MatchModeExact:
        return matched == 1 && len(m.Keywords) == 1, float64(matched)
    default:
        return matched > 0, float64(matched) / float64(len(m.Keywords))
    }
}

// Regex matcher
type RegexMatcher struct {
    Patterns []*regexp.Regexp
    Logic    LogicMode // AND | OR
}

func (m *RegexMatcher) Match(input string, context *RoutingContext) (bool, float64) {
    matched := 0

    for _, pattern := range m.Patterns {
        if pattern.MatchString(input) {
            matched++
            if m.Logic == LogicModeOR {
                return true, 1.0
            }
        }
    }

    if m.Logic == LogicModeAND {
        return matched == len(m.Patterns), float64(matched) / float64(len(m.Patterns))
    }

    return matched > 0, float64(matched) / float64(len(m.Patterns))
}

// Semantic matcher (based on embeddings)
type SemanticMatcher struct {
    embeddings map[string][]float64    // semantic vectors of Agents
    threshold  float64                 // similarity threshold
    model      EmbeddingModel          // embedding model
}

func (m *SemanticMatcher) Match(input string, context *RoutingContext) (bool, float64) {
    inputVec, err := m.model.Embed(input)
    if err != nil {
        return false, 0
    }

    bestScore := 0.0
    for _, agentVec := range m.embeddings {
        score := cosineSimilarity(inputVec, agentVec)
        if score > bestScore {
            bestScore = score
        }
    }

    return bestScore >= m.threshold, bestScore
}

2. LLM-Based Routing

Use the LLM’s semantic understanding for intelligent routing:

type LLMRouter struct {
    model          agent.Model
    agentRegistry  map[string]*AgentDescriptor
    promptTemplate string
    cache          *RouteCache
}

type AgentDescriptor struct {
    Agent        *agent.Agent
    Name         string
    Description  string
    Capabilities []string
    Examples     []string
}

func (r *LLMRouter) Route(ctx context.Context, input string) (*agent.Agent, error) {
    // 1. Check cache
    if cached := r.cache.Get(input); cached != nil {
        return cached.(*agent.Agent), nil
    }

    // 2. Build routing prompt
    prompt := r.buildRoutingPrompt(input)

    // 3. Call LLM for decision
    response, err := r.model.GenerateContent(ctx, prompt)
    if err != nil {
        return nil, fmt.Errorf("llm routing failed: %w", err)
    }

    // 4. Parse decision result
    decision, err := r.parseRoutingDecision(response)
    if err != nil {
        return nil, fmt.Errorf("parse routing decision: %w", err)
    }

    // 5. Get target Agent
    target, ok := r.agentRegistry[decision.AgentName]
    if !ok {
        return nil, fmt.Errorf("unknown agent: %s", decision.AgentName)
    }

    // 6. Cache decision
    r.cache.Set(input, target.Agent, 5*time.Minute)

    return target.Agent, nil
}

func (r *LLMRouter) buildRoutingPrompt(input string) string {
    var sb strings.Builder
    sb.WriteString("You are an intelligent routing system. Choose the most suitable Agent to handle the user input.\n\n")
    sb.WriteString("Available Agents:\n")

    for name, desc := range r.agentRegistry {
        sb.WriteString(fmt.Sprintf("\n[%s]\n", name))
        sb.WriteString(fmt.Sprintf("Description: %s\n", desc.Description))
        sb.WriteString(fmt.Sprintf("Capabilities: %s\n", strings.Join(desc.Capabilities, ", ")))
        if len(desc.Examples) > 0 {
            sb.WriteString(fmt.Sprintf("Examples: %s\n", strings.Join(desc.Examples, "; ")))
        }
    }

    sb.WriteString(fmt.Sprintf("\nUser input: %s\n", input))
    sb.WriteString("\nPlease output in JSON format: {\"agent\": \"AgentName\", \"confidence\": 0.95, \"reason\": \"reason for selection\"}")

    return sb.String()
}

3. Hybrid Routing

Combine the speed of rule-based routing with the accuracy of LLM routing:

type HybridRouter struct {
    ruleRouter     *RuleRouter      // fast path
    llmRouter      *LLMRouter       // intelligent path
    fallbackAgent  *agent.Agent     // fallback Agent

    // Strategy configuration
    ruleThreshold    float64       // rule match confidence threshold
    llmThreshold     float64       // LLM routing confidence threshold
    useLLMForUnknown bool          // use LLM for unknown inputs
}

func (r *HybridRouter) Route(ctx context.Context, input string) (*agent.Agent, error) {
    // Layer 1: rule-based routing (fast path)
    ruleResult, confidence, err := r.ruleRouter.Match(input)
    if err == nil && confidence >= r.ruleThreshold {
        return ruleResult, nil
    }

    // Layer 2: LLM routing (intelligent path)
    if r.useLLMForUnknown || confidence > 0 {
        llmResult, err := r.llmRouter.Route(ctx, input)
        if err == nil {
            return llmResult, nil
        }
    }

    // Fallback
    if r.fallbackAgent != nil {
        return r.fallbackAgent, nil
    }

    return nil, fmt.Errorf("no suitable agent found for input: %s", input)
}

Load-Aware Scheduling

Dynamic Load Balancing

type LoadAwareRouter struct {
    agents        []*LoadAwareAgent
    strategy      LoadBalanceStrategy
    healthChecker *HealthChecker
}

type LoadAwareAgent struct {
    Agent       *agent.Agent
    CurrentLoad float64           // current load (0-1)
    AvgLatency  time.Duration     // average latency
    ErrorRate   float64           // error rate
    LastChecked time.Time         // last check time
    Weight      float64           // weight
}

type LoadBalanceStrategy interface {
    Select(agents []*LoadAwareAgent, input string) *LoadAwareAgent
}

// Weighted round-robin
func (s *WeightedRoundRobin) Select(agents []*LoadAwareAgent, input string) *LoadAwareAgent {
    var best *LoadAwareAgent
    bestScore := -1.0

    for _, a := range agents {
        // Skip unhealthy or overloaded Agents
        if a.ErrorRate > 0.5 || a.CurrentLoad > 0.9 {
            continue
        }

        // Composite score: weight / (load + normalized latency + error rate)
        latencyNorm := float64(a.AvgLatency.Milliseconds()) / 1000.0
        score := a.Weight / (a.CurrentLoad + latencyNorm + a.ErrorRate + 0.1)

        if score > bestScore {
            bestScore = score
            best = a
        }
    }

    return best
}

// Consistent hashing (ensure same input routes to same Agent)
type ConsistentHashStrategy struct {
    ring *consistent.Consistent
}

func (s *ConsistentHashStrategy) Select(agents []*LoadAwareAgent, input string) *LoadAwareAgent {
    agentName, err := s.ring.Get(input)
    if err != nil {
        return nil
    }

    for _, a := range agents {
        if a.Agent.Name() == agentName {
            return a
        }
    }

    return nil
}

// Least connections
func (s *LeastConnections) Select(agents []*LoadAwareAgent, input string) *LoadAwareAgent {
    var best *LoadAwareAgent
    minConnections := int(^uint(0) >> 1) // MaxInt

    for _, a := range agents {
        connections := a.Agent.ActiveConnections()
        if connections < minConnections && a.CurrentLoad < 0.9 {
            minConnections = connections
            best = a
        }
    }

    return best
}

Health Checks and Auto-Eviction

type HealthChecker struct {
    checkInterval time.Duration
    agents        map[string]*AgentHealth
}

type AgentHealth struct {
    State               HealthState
    LastCheck           time.Time
    SuccessCount        int
    FailureCount        int
    ConsecutiveFailures int
}

type HealthState int
const (
    HealthStateHealthy HealthState = iota
    HealthStateDegraded
    HealthStateUnhealthy
)

func (hc *HealthChecker) Start(ctx context.Context) {
    ticker := time.NewTicker(hc.checkInterval)
    defer ticker.Stop()

    for {
        select {
        case <-ctx.Done():
            return
        case <-ticker.C:
            hc.checkAll()
        }
    }
}

func (hc *HealthChecker) checkAll() {
    for id, health := range hc.agents {
        // Perform health check
        err := hc.performCheck(id)

        if err != nil {
            health.FailureCount++
            health.ConsecutiveFailures++

            if health.ConsecutiveFailures >= 3 {
                health.State = HealthStateUnhealthy
            } else if health.ConsecutiveFailures >= 1 {
                health.State = HealthStateDegraded
            }
        } else {
            health.SuccessCount++
            health.ConsecutiveFailures = 0
            health.State = HealthStateHealthy
        }

        health.LastCheck = time.Now()
    }
}

Routing Cache and Optimization

Multi-Level Cache Architecture

type MultiLevelRouteCache struct {
    l1 *LRUCache        // L1: in-memory cache, O(1) access
    l2 *RedisCache      // L2: distributed cache, shared across instances
    l3 *PersistentCache // L3: persistent cache, survives restart
}

func (c *MultiLevelRouteCache) Get(input string) (*agent.Agent, error) {
    // L1 lookup
    if agent := c.l1.Get(input); agent != nil {
        return agent, nil
    }

    // L2 lookup
    if agent := c.l2.Get(input); agent != nil {
        c.l1.Set(input, agent, time.Minute) // backfill L1
        return agent, nil
    }

    // L3 lookup
    if agent := c.l3.Get(input); agent != nil {
        c.l2.Set(input, agent, time.Hour)   // backfill L2
        c.l1.Set(input, agent, time.Minute) // backfill L1
        return agent, nil
    }

    return nil, fmt.Errorf("cache miss")
}

func (c *MultiLevelRouteCache) Set(input string, agent *agent.Agent, ttl time.Duration) {
    c.l1.Set(input, agent, ttl/10)        // L1 caches 1/10 TTL
    c.l2.Set(input, agent, ttl)           // L2 caches full TTL
    c.l3.Set(input, agent, ttl*24)        // L3 caches 24x TTL
}

Cache Preheating

func (r *Router) PreheatCache(ctx context.Context, sampleInputs []string) error {
    for _, input := range sampleInputs {
        agent, err := r.Route(ctx, input)
        if err != nil {
            log.Printf("Preheating failed for '%s': %v", input, err)
            continue
        }

        // Pre-cache result
        r.cache.Set(input, agent, 24*time.Hour)
    }

    return nil
}

Hands-On Scenario: Multi-Function Intelligent Assistant

Full Implementation

package main

import (
    "context"
    "fmt"
    "log"
    "regexp"
    "strings"
    "time"

    "github.com/google/adk-go/agent"
    "github.com/google/adk-go/team"
    "github.com/google/adk-go/tool"
)

// IntelligentAssistant multi-function intelligent assistant
type IntelligentAssistant struct {
    router *team.HybridRouter
}

func NewIntelligentAssistant(model agent.Model) (*IntelligentAssistant, error) {
    // 1. Create domain-specific Agents

    // Weather Agent
    weatherAgent, err := agent.New(agent.Config{
        Name:        "weather-expert",
        Model:       model,
        Instruction: `You are a weather expert. Provide accurate weather forecasts, clothing advice, and travel reminders.`,
        Tools: []tool.Tool{
            tool.NewWeatherQueryTool(),
            tool.NewAirQualityTool(),
        },
        Timeout: 10 * time.Second,
    })
    if err != nil {
        return nil, err
    }

    // Order Agent
    orderAgent, err := agent.New(agent.Config{
        Name:        "order-expert",
        Model:       model,
        Instruction: `You are an order processing expert. Help users query, modify, and cancel orders.`,
        Tools: []tool.Tool{
            tool.NewOrderQueryTool(),
            tool.NewOrderModifyTool(),
        },
        Timeout: 15 * time.Second,
    })
    if err != nil {
        return nil, err
    }

    // Logistics Agent
    logisticsAgent, err := agent.New(agent.Config{
        Name:        "logistics-expert",
        Model:       model,
        Instruction: `You are a logistics query expert. Track packages and estimate delivery time.`,
        Tools: []tool.Tool{
            tool.NewPackageTrackingTool(),
        },
        Timeout: 10 * time.Second,
    })
    if err != nil {
        return nil, err
    }

    // Writing Agent
    writingAgent, err := agent.New(agent.Config{
        Name:        "writing-expert",
        Model:       model,
        Instruction: `You are a writing assistant. Help users draft and polish various copy.`,
        Timeout: 20 * time.Second,
    })
    if err != nil {
        return nil, err
    }

    // General Agent
    generalAgent, err := agent.New(agent.Config{
        Name:        "general-assistant",
        Model:       model,
        Instruction: `You are a general assistant. Answer users' various questions in a friendly and professional manner.`,
        Timeout: 15 * time.Second,
    })
    if err != nil {
        return nil, err
    }

    // 2. Build rule-based router
    ruleRouter := team.NewRuleRouter()

    // Weather rule (high priority)
    ruleRouter.AddRule(team.RoutingRule{
        Name:     "weather-rule",
        Priority: 100,
        Matcher: team.NewKeywordMatcher([]string{
            "weather", "temperature", "rain", "snow", "air quality", "clothing", "umbrella",
        }, team.MatchModeAny, false),
        TargetAgent: weatherAgent,
    })

    // Order rule
    ruleRouter.AddRule(team.RoutingRule{
        Name:     "order-rule",
        Priority: 90,
        Matcher: team.NewCompositeMatcher(team.LogicModeOR,
            team.NewKeywordMatcher([]string{"order", "purchase", "payment", "refund", "cancel"}, team.MatchModeAny, false),
            team.NewRegexMatcher([]*regexp.Regexp{
                regexp.MustCompile(`order number[:]?\\s*\\d+`),
            }, team.LogicModeOR),
        ),
        TargetAgent: orderAgent,
    })

    // Logistics rule
    ruleRouter.AddRule(team.RoutingRule{
        Name:     "logistics-rule",
        Priority: 90,
        Matcher: team.NewKeywordMatcher([]string{
            "express", "logistics", "package", "shipped", "delivered", "where",
        }, team.MatchModeAny, false),
        TargetAgent: logisticsAgent,
    })

    // Writing rule
    ruleRouter.AddRule(team.RoutingRule{
        Name:     "writing-rule",
        Priority: 80,
        Matcher: team.NewKeywordMatcher([]string{
            "write", "create", "polish", "revise", "copy", "article", "email",
        }, team.MatchModeAny, false),
        TargetAgent: writingAgent,
    })

    // 3. Build LLM router (for fuzzy matching)
    llmRouter := team.NewLLMRouter(model, map[string]*team.AgentDescriptor{
        "weather-expert": {
            Agent:        weatherAgent,
            Description:  "Handle weather-related queries",
            Capabilities: []string{"weather forecast", "air quality", "clothing advice"},
            Examples:     []string{"How is the weather today", "Do I need an umbrella tomorrow"},
        },
        "order-expert": {
            Agent:        orderAgent,
            Description:  "Handle order-related operations",
            Capabilities: []string{"order query", "order modification", "refund processing"},
            Examples:     []string{"Check my order", "I want to cancel my order"},
        },
        // ... other Agents
    })

    // 4. Build hybrid router
    hybridRouter := team.NewHybridRouter(
        team.WithRuleRouter(ruleRouter),
        team.WithLLMRouter(llmRouter),
        team.WithFallbackAgent(generalAgent),
        team.WithRuleThreshold(0.8),
        team.WithLLMThreshold(0.7),
        team.WithCache(team.NewMultiLevelRouteCache()),
    )

    return &IntelligentAssistant{router: hybridRouter}, nil
}

func (a *IntelligentAssistant) Handle(ctx context.Context, userInput string) (string, error) {
    // Route to suitable Agent
    selectedAgent, err := a.router.Route(ctx, userInput)
    if err != nil {
        return "", fmt.Errorf("routing failed: %w", err)
    }

    log.Printf("[Router] Input routed to %s", selectedAgent.Name())

    // Execute Agent
    result, err := selectedAgent.Run(ctx, userInput)
    if err != nil {
        return "", fmt.Errorf("agent execution failed: %w", err)
    }

    return result, nil
}

func main() {
    ctx := context.Background()

    assistant, err := NewIntelligentAssistant(model)
    if err != nil {
        log.Fatalf("Failed to create assistant: %v", err)
    }

    // Test different scenarios
    testCases := []string{
        "How is the weather in Beijing today",
        "Check my order 12345",
        "Where is my package",
        "Help me write a resignation letter",
        "What is quantum computing",
    }

    for _, input := range testCases {
        response, err := assistant.Handle(ctx, input)
        if err != nil {
            log.Printf("Error: %v", err)
            continue
        }
        fmt.Printf("\nUser: %s\nAssistant: %s\n", input, response)
    }
}

Observability of Routing Decisions

Tracing and Auditing

type RoutingDecision struct {
    RequestID     string
    Timestamp     time.Time
    Input         string
    InputHash     string
    SelectedAgent string
    RouteType     string // "rule" | "llm" | "fallback"
    Confidence    float64
    LatencyMs     int64
    RulesChecked  []string
    CacheHit      bool
}

func (r *Router) logDecision(decision *RoutingDecision) {
    // Log to console
    logData, _ := json.Marshal(decision)
    log.Printf("[RoutingDecision] %s", logData)

    // Record metrics
    r.metrics.RoutingLatency.WithLabelValues(decision.RouteType).Observe(float64(decision.LatencyMs))
    r.metrics.RoutingCounter.WithLabelValues(decision.SelectedAgent, decision.RouteType).Inc()

    if decision.CacheHit {
        r.metrics.CacheHitCounter.Inc()
    } else {
        r.metrics.CacheMissCounter.Inc()
    }
}

Common Questions in Depth

Q: How do I choose between rule-based and LLM routing?

A: Decision matrix:

DimensionRule-Based RoutingLLM Routing
Latency<1ms500ms-2s
AccuracyHigh (deterministic)High (semantic)
CostLowHigh (per LLM call)
MaintenanceRequires rule updatesSelf-adapting
Suitable forClear classificationFuzzy semantics

Production recommendation: use a hybrid approach, with rule-based routing handling 80% of clear requests and LLM routing handling 20% of fuzzy requests.

Q: How long does the routing cache last?

A: Multi-level expiration strategy:

  • L1 memory: 1-5 minutes, based on access frequency
  • L2 Redis: 1 hour, based on time
  • L3 persistent: 24 hours, based on LRU

For dynamic content (e.g., “weather today”), disable caching or use a very short TTL.

Q: How do I handle routing errors (wrong Agent selected)?

A: Three layers of protection:

  1. Confidence threshold: return the general Agent when confidence is low.
  2. User confirmation: ask the user for intent on fuzzy requests.
  3. Auto-correction: automatically try other Agents when an Agent fails.
func (r *Router) RouteWithFallback(ctx context.Context, input string) (*agent.Agent, error) {
    agent, confidence, err := r.route(ctx, input)
    if err != nil {
        return r.fallbackAgent, nil
    }

    if confidence < 0.5 {
        // Low confidence, may need user confirmation
        return r.clarificationAgent, nil
    }

    return agent, nil
}

Next Steps

You now have a deep understanding of Agent routing dynamic dispatch and intelligent scheduling. Module 5 is complete; next, enter Module 6: Streaming—Streaming principles, event handling, and real-time communication.

← Custom Workflow | Streaming Principles →

Want to learn more Go ADK hands-on? Follow the “Mengshou Programming” channel for weekly updates on Go and AI programming.

Frequently Asked Questions

What is the role of Agent routing?

The routing system decides which Agent should handle a user request. It is the 'traffic command center' that improves throughput, reduces latency, and increases accuracy.

How do I choose between rule-based and LLM routing?

Rule-based routing has low latency and strong determinism, suitable for clear classification; LLM routing has strong semantic understanding, suitable for fuzzy requests. Production environments should use a hybrid approach.

How long does the routing cache last?

L1 in-memory cache 1-5 minutes, L2 Redis cache 1 hour, L3 persistent cache 24 hours; dynamic content should use a shorter TTL or disable caching.

How do I handle routing errors (wrong Agent selected)?

Set confidence thresholds, ask the user for confirmation on fuzzy requests, and automatically try other Agents as a correction when an Agent fails.

How does the routing system implement load balancing?

Through dynamic load balancing algorithms, health checks that automatically remove abnormal instances, and hash-based routing according to input features to disperse traffic.