Multimodal Streaming: Audio, Image, and Video Processing

Text-only chat is no longer enough for modern AI applications. Users paste screenshots, upload voice messages, and share video clips. ADK Go can handle these inputs, but multimodal streaming adds complexity: the media must be received, prepared, sent to the model, and rendered without blocking the typewriter-style response.

← Event Handling | Streaming UI →

Why Multimodal Streaming Is Different

A text message can be sent almost immediately. A photo may need resizing, a voice clip may need transcription, and a video may need frame extraction. The product goal is to preserve the streaming experience while doing that work in the background.

Never wait for expensive media processing to start visible output.

Unified Media Event Model

Multimodal inputs should be represented as first-class events so the frontend, logs, and retry logic can understand them.

type MediaType string

const (
    MediaTypeImage MediaType = "image"
    MediaTypeAudio MediaType = "audio"
    MediaTypeVideo MediaType = "video"
)

type MediaEvent struct {
    Type       MediaType
    URL        string
    MIMEType   string
    Sequence   int64
    Chunk      []byte
    ChunkIndex int
    Complete   bool
    Metadata   map[string]string
}

A good event model distinguishes partial uploads from final content. That distinction makes it possible to show a thumbnail immediately, continue accepting chunks, and only invoke the model after the media is complete.

Image Streaming

Images are common and usually manageable, but a full-resolution photo can be several megabytes. Sending it directly to a model can create both latency and token cost. The production pattern is: accept chunks from the client, return a placeholder or low-resolution preview, reconstruct and validate the image server-side, resize or encode a model-friendly version, attach it to the agent request, and continue streaming normal text events.

func handleImageUpload(w http.ResponseWriter, r *http.Request) {
    writer, err := storage.CreateUploadedFile(r.Context(), "image")
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }
    defer writer.Close()

    _, err = io.Copy(writer, r.Body)
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }

    prepared, err := image.PrepareForModel(writer.FilePath(), image.PrepareModelOptions{
        MaxWidth:  1024,
        MaxHeight: 1024,
        Format:    image.JPEG,
        Quality:   85,
    })
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }

    writeJSON(w, http.StatusOK, map[string]string{
        "media_url":    prepared.URL,
        "content_type": prepared.MIMEType,
        "size_bytes":   strconv.Itoa(prepared.Size),
    })
}

A useful rule of thumb is to keep model inputs small unless the task truly needs detail. A 1024x1024 JPEG is often enough for document review, screenshots, and product photos.

Audio Streaming

Audio usually arrives as a voice message or live recording. The user cares about whether the system heard the message and how fast the assistant starts replying. The backend can stream raw audio chunks into a buffer while a transcription job runs asynchronously. When the first transcript segment is available, emit a TranscriptDelta event so the UI can show progress.

type AudioProcessor struct {
    buffer     *chunk.Buffer
    transcript chan string
    done       chan error
}

func (p *AudioProcessor) Start(ctx context.Context, source io.ReadCloser) {
    go func() {
        defer source.Close()
        defer close(p.done)
        p.buffer = chunk.NewBuffer(4 * 1024 * 1024)

        _, err := io.Copy(p.buffer, source)
        if err != nil {
            p.done <- err
            return
        }

        segments, err := transcriber.StreamTranscribe(ctx, p.buffer.Bytes())
        if err != nil {
            p.done <- err
            return
        }

        for _, segment := range segments {
            select {
            case p.transcript <- segment.Text:
            case <-ctx.Done():
                return
            }
        }
    }()
}

For long voice messages, partial transcripts are more useful than waiting for the full file. The UI can show “listening”, then “heard: …”, then let the assistant stream a response after transcription completes.

Video Streaming

Video is the most expensive modality because each frame can become visual tokens. Do not send every raw frame to the model. Instead, use a staged pipeline: accept chunks, extract metadata, select key frames based on scene changes or fixed intervals, downsample frames, send a compact frame set plus an optional audio transcript, and stream analysis while the full media remains available for replay.

func ExtractKeyFrames(ctx context.Context, path string, maxFrames int) ([]MediaFrame, error) {
    meta, err := media.Metadata(ctx, path)
    if err != nil {
        return nil, err
    }

    frames, err := media.DetectScenes(ctx, path, maxFrames)
    if err != nil {
        frames, err = media.UniformFrames(ctx, path, meta.Duration, maxFrames)
    }
    if err != nil {
        return nil, err
    }

    for i := range frames {
        frames[i] = media.ResizeFrame(frames[i], image.PrepareModelOptions{
            MaxWidth:  768,
            MaxHeight: 768,
            Format:    image.JPEG,
            Quality:   80,
        })
    }
    return frames, nil
}

For a one-minute video, ten frames plus an audio transcript is often a better prompt than dozens of full-resolution frames.

Real-time Synchronization

Multimodal streams need synchronization because text, media, and tool events can arrive independently. Use a single ordering key across all outputs.

type UnifiedEvent struct {
    Kind       EventType
    Sequence   int64
    MediaID    string
    Payload    any
    ReceivedAt time.Time
}

The frontend should render based on Sequence, not arrival time. If the first image chunk arrives before the first text delta, the UI can still show the placeholder image without breaking the assistant message.

Production Media Pipeline

A production pipeline separates concerns: an upload service accepts chunks and validates MIME types; a preparation service resizes images, transcribes audio, and extracts frames; an agent service composes prompts and runs ADK streaming; an event service serializes text and media events to the client.

func RunMultimodalChat(ctx context.Context, req ChatRequest) {
    preps := []PreparedMedia{}
    for _, item := range req.MediaItems {
        prep, err := media.Prepare(ctx, item)
        if err != nil {
            emitError(ctx, err)
            continue
        }
        preps = append(preps, prep)
    }

    stream, err := agent.RunStream(ctx, buildPrompt(req.Message, preps))
    if err != nil {
        emitError(ctx, err)
        return
    }
    defer stream.Close()

    for {
        event, err := stream.Recv()
        if err == io.EOF {
            emitDone(ctx)
            return
        }
        if err != nil {
            emitError(ctx, err)
            return
        }
        emit(ctx, event)
    }
}

Cost and Latency Controls

Multimodal prompts can become expensive quickly. Keep the following controls in place: compress images to roughly 512x512 or 1024x1024 depending on precision, transcribe audio to text unless audio content is essential, extract fewer than ten frames for short clips, cache prepared media for replay or retry, and record token usage per modality.

Common Pitfalls

Multimodal streaming is easy to overbuild. Watch out for waiting for full upload before sending any feedback, sending full-resolution media to the model, treating every video frame as equally useful, blocking SSE delivery while audio transcription finishes, and forgetting that media URLs may expire or require signed access.

Next Step

You now have a model for handling non-text inputs without losing the streaming experience: staged preparation, partial feedback, and unified event ordering. The next tutorial turns all of this into a real frontend interaction: SSE or WebSocket chat with incremental rendering and production connection management.

Want to keep learning hands-on Go ADK? Follow the “Mengshou Programming” channel for weekly Go and AI programming posts.