Building a Distributed Chat Room in Rust: Making Actors Talk Across Nodes

A while back, I was working on a microservices architecture at work when I suddenly realized something: our services were like a bunch of introverted programmers, each working alone in their own little rooms, barely talking to each other. When they occasionally needed to collaborate, they had to go through complex HTTP calls - it was painfully inefficient.

Later, I discovered Rust’s Actor model, and it felt like finding a new continent. Actors are like independent little assistants that can receive messages, handle tasks, and send messages to other Actors. But here’s the problem: if these Actors are distributed across different servers, how do they chat with each other?

Today, I want to share my journey of implementing cross-node Actor communication in Rust, complete with all the pitfalls I encountered. I guarantee that after reading this, you’ll be able to build your own distributed Actor system.

First, What Is an Actor System?

Imagine you’re managing a large shipping company. Each employee (Actor) has their own job responsibilities:

Package Receivers: Specialized in receiving customer packages
Sorters: Categorize packages by address
Delivery Drivers: Handle the last-mile delivery

Each person has their own “inbox” (message queue), and they process tasks from their inbox when working. This is the core idea of the Actor model: each Actor is an independent entity that collaborates through message passing.

In Rust, a simple Actor looks like this:

// This is like a delivery driver's job description
struct DeliveryActor {
    pub name: String,
}

#[async_trait::async_trait]
impl Actor for DeliveryActor {
    type Message = String;

    async fn handle(&mut self, package: String) {
        println!("📦 {} received package: {}", self.name, package);
        // Simulate delivery process
        tokio::time::sleep(Duration::from_millis(100)).await;
        println!("✅ {} delivered successfully!", self.name);
    }
}

A single-machine Actor system is like a small shipping station where everyone works in the same office - you can just shout across the room to communicate. But here’s where the real challenge begins: if we have multiple shipping stations (nodes) distributed across different cities, how do they collaborate?

The Real Challenges of Cross-Node Communication

Initially, I naively thought that just establishing WebSocket connections between each node would solve everything. I quickly discovered that things were far more complicated:

Addressing Problem: If Alice at Station A wants to send a message to Bob at Station B, how does she know where Bob is?
Routing Problem: What path should the message take? Direct connection or through a relay?
Fault Tolerance: What happens if Station B goes offline?

Just like in real shipping systems, we need an intelligent “sorting center” to solve these problems.

ClusterRouter: Our Intelligent Package Sorting Center

After some exploration, I designed something called ClusterRouter. It’s like FedEx’s sorting center that can automatically determine which route a package should take:

#[derive(Clone)]
pub struct ClusterRouter {
    pub self_id: String,           // Current station ID, like "Beijing Station"
    pub registry: Registry,        // Local employee directory
    pub cluster: ClusterClient,    // Cross-station communication device
}

The core logic of this router is particularly interesting, like an experienced package sorter:

impl ClusterRouter {
    pub async fn send(&self, address: &str, message: &str) {
        // Parse address format: employee_name@station_name
        if let Some((actor_name, node_id)) = address.split_once('@') {

            if node_id == self.self_id {
                // Local delivery: same station, find person directly
                println!("📍 Local delivery to: {}", actor_name);
                let registry = self.registry.read().unwrap();

                if let Some(actor_addr) = registry.get(actor_name) {
                    // Found them, deliver directly
                    actor_addr.send(message.to_string()).await;
                } else {
                    println!("❌ Employee not found at this station: {}", actor_name);
                }
            } else {
                // Remote delivery: needs to be sent over network to other stations
                println!("🚀 Remote delivery to: {}", address);
                self.cluster
                    .send_to_remote(address, message)
                    .await;
            }
        } else {
            println!("⚠️ Invalid address format: {}", address);
        }
    }
}

What’s great about this design? Actors sending messages don’t need to care where the recipient is! It’s like sending a package - you just write the correct address, and the shipping company handles the rest.

Real-World Example: Building a Cross-Station Printing Service

Let me show you the power of this system with a concrete example. Suppose we want to build a cross-station document printing service:

Scenario Setup:

Beijing station has a RouterActor (dispatcher)
Shanghai station has a PrinterActor (printer)
The Beijing dispatcher needs the Shanghai printer to print documents

First, create a dispatcher with routing capabilities:

struct RouterActor {
    pub router: ClusterRouter,
}

#[async_trait::async_trait]
impl Actor for RouterActor {
    type Message = String;

    async fn handle(&mut self, document: String) {
        println!("📋 Dispatcher received document: {}", document);

        // Send to Shanghai station's printer
        self.router
            .send("printer@shanghai", &document)
            .await;

        println!("✉️ Forwarded to Shanghai printer");
    }
}

Then register this dispatcher at Beijing station:

// Create local employee directory
let registry = new_registry();

// Create cross-station communication device
let cluster_client = ClusterClient::new("shanghai");

// Create router
let router = ClusterRouter {
    self_id: "beijing".into(),
    registry: registry.clone(),
    cluster: cluster_client.clone(),
};

// Start dispatcher and register
let router_actor = spawn_actor(RouterActor { router });
registry.write().unwrap().insert("router".into(), router_actor);

Network Message Format: Our “Shipping Label”

To ensure different stations can correctly understand messages, I designed a unified “shipping label” format:

#[derive(Serialize, Deserialize, Debug)]
pub struct NetworkMessage {
    pub to: String,       // Recipient address: "printer@shanghai"
    pub from: String,     // Sender address: "router@beijing"
    pub payload: String,  // Package contents (serialized message)
}

This is like the information on a real shipping label: recipient, sender, package contents - crystal clear.

Test Results: Let’s See Our Achievement

When the system runs, you’ll see logs like this:

🔗 Beijing station connected to Shanghai station @ ws://127.0.0.1:9000
📋 Dispatcher received document: important-contract.pdf
📍 Detected remote address, preparing cross-station transmission...
🚀 Remote delivery to: printer@shanghai
📦 Shanghai station received: NetworkMessage {
    to: "printer",
    from: "router@beijing",
    payload: "important-contract.pdf"
}
🖨️ Shanghai printer: Printing important-contract.pdf
✅ Print completed!

Seeing this output, I was so excited I almost jumped up. This meant we had successfully achieved:

Transparent Routing: Actors don’t need to know if message delivery is local or remote
Automatic Discovery: The system automatically determines the target Actor’s location
Error Handling: Clear error messages when Actors can’t be found

Some Pitfalls I’ve Encountered

During actual development, I ran into quite a few issues. Here are some typical ones I’d like to share:

Pitfall 1: Serialization Issues

Initially, I tried to be lazy and directly pass complex data structures, only to find that serialized data often got corrupted during transmission between different nodes. I learned my lesson and now consistently use String as the payload, serializing complex data to JSON when needed.

Pitfall 2: Network Disconnection Handling

WebSocket connections sometimes drop for mysterious reasons, and if you don’t handle reconnection, the entire cluster becomes useless. Now I always add automatic reconnection mechanisms:

// Simplified reconnection logic
impl ClusterClient {
    async fn ensure_connected(&mut self) {
        if !self.is_connected() {
            println!("🔄 Detected disconnection, reconnecting...");
            self.reconnect().await;
        }
    }
}

Pitfall 3: Message Ordering Issues

In high-concurrency scenarios, message arrival order might not match sending order. If your business is sensitive to order, remember to add message sequence numbers or timestamps.

What’s Next

This system can already handle static Actor communication quite well, but there are some cooler features on the way:

Dynamic Service Discovery: Let Actors automatically discover other Actors in the network
Load Balancing: Intelligently distribute tasks to the least busy nodes
Failover: Automatically migrate tasks to other nodes when a node goes down

Summary

Looking back, implementing cross-node Actor communication is like building a modern shipping network:

Actor = Shipping employees
ClusterRouter = Intelligent sorting center
NetworkMessage = Standardized shipping labels
WebSocket = High-speed transportation network

Through this system, we’ve achieved true distributed Actor communication. The code is natural to write and not complex to maintain. Most importantly, it allows us to easily scale system size, just like adding new stations to a shipping network.

If you’re also exploring distributed programming in Rust, I recommend building such a system yourself. Trust me, the sense of achievement when you see Actors from different nodes start “chatting” is indescribable.

If you found this article helpful, feel free to follow my tech column where I’ll continue sharing more Rust practical experiences. Feel free to discuss any questions in the comments - I’ll try my best to answer everyone’s queries. Next time I’m planning to talk about service discovery and dynamic scaling topics, stay tuned!

Building a Distributed Chat Room in Rust: Making Actors Talk Across Nodes#

First, What Is an Actor System?#

The Real Challenges of Cross-Node Communication#

ClusterRouter: Our Intelligent Package Sorting Center#

Real-World Example: Building a Cross-Station Printing Service#

Network Message Format: Our “Shipping Label”#

Test Results: Let’s See Our Achievement#

Some Pitfalls I’ve Encountered#

Pitfall 1: Serialization Issues#

Pitfall 2: Network Disconnection Handling#

Pitfall 3: Message Ordering Issues#

What’s Next#

Summary#