Python Took 14 Seconds, Rust Took 0.7 — Here's What Changed

My Coffee Wasn’t Even Cold

Yesterday a data processing job took 14 seconds. Today I swapped one library and it finished in 0.7 seconds.

I thought the program had crashed. Checked the output file — everything was there, perfectly formatted. Same computer, same data, same logic. The only difference: Python pandas replaced with Rust’s Polars.

It’s like upgrading from a bicycle to a delivery truck. Same route, same packages, but while you’re still pedaling, the truck’s already unloading.

Let’s talk about why this Rust library called Polars is so ridiculously fast.

Why Python Struggles with Speed

First things first — Python isn’t bad. It’s quick to write, easy to learn, and has an incredible ecosystem. It’s the darling of data science for good reason. But it was never designed for raw speed.

Think of Python pandas as a diligent chef who does everything personally: chopping, cooking, plating — one dish at a time. The problem? When dinner rush hits and you need 500 dishes out the door, that single chef bottlenecks the whole kitchen.

Specifically, pandas has these limitations:

1. Interpreted execution: Python code runs line-by-line through an interpreter, not as compiled machine code — overhead is built in
2. Object overhead: Every cell in a pandas DataFrame is a Python object, with memory and CPU constantly dealing with the wrapper
3. Single-core bound: By default, pandas uses one CPU core while others sit idle
4. Memory layout: Data scattered throughout memory means CPU caches go underutilized

Enter Polars. Written in Rust, a compiled language that’s naturally fast. But the real secret is its completely different approach to data.

Why Polars Is 20x Faster

Continuing the kitchen analogy: Polars isn’t one chef cooking solo. It’s a full assembly line — chopping, cooking, plating, all happening simultaneously. And each station has professional-grade equipment: industrial choppers, commercial burners, automated platers.

Technically, Polars gets four things right:

1. Columnar Storage

Traditional pandas is “row-oriented” — think of your Excel spreadsheet, where each row is a complete record. Polars uses “columnar storage” — all values from the same column sit next to each other in memory.

Why is this faster? Modern CPUs excel at processing contiguous memory. Like reading a book — powering through 100 consecutive pages is way faster than jumping around finding 100 scattered page numbers.

2. Lazy Execution + Query Optimization

Polars has a “lazy mode” — it doesn’t execute commands immediately. Instead, it records your requests, analyzes the full pipeline, optimizes where possible, then runs everything in one go.

For example, if you say: filter for China data, calculate price times quantity, then group by date and sum — pandas processes step by step. But Polars thinks: wait, I can read only the needed columns and push down the filter, avoiding all that useless data movement. It’s like a smart shopper who makes one consolidated trip instead of running back and forth to the store.

3. Default Parallelism

Polars is multithreaded by design. Got 8 CPU cores? It spawns 8 workers simultaneously. Pandas? One worker grunting away while 7 cores do nothing.

4. SIMD Vectorization

This gets technical, but basically: CPUs can process batches of data in one go, not item by item. Polars leans heavily into this capability.

Real Numbers: Not Small Potatoes

Enough talk — let’s look at actual test results.

Test setup:

• 8-core CPU, 32GB RAM, NVMe SSD
• 10 million rows, 12 columns, realistic mix of integers, floats, strings
• Task: filter, compute new column, group by three keys, aggregate two metrics, write Parquet

Results:

20x — not 20%, but 20 times.

If you run this job 100 times daily, pandas needs 23 minutes. Polars takes just over a minute. That’s two coffees worth of time saved every single day.

What Does the Code Look Like?

Polars code is actually quite clean:

use polars::prelude::*;

fn main() -> PolarsResult<()> {
    // Read CSV
    let df = CsvReadOptions::default()
        .with_has_header(true)
        .try_into_reader_with_file_path(Some("events.csv".into()))?
        .finish()?;

    // Data processing: filter, compute, group, aggregate
    let result = df
        .lazy()  // Enable lazy execution
        .filter(col("country").eq(lit("CN")))  // Filter China data
        .with_column((col("price") * col("qty")).alias("amount"))  // Calculate amount
        .group_by([col("day"), col("channel"), col("category")])  // Group by three columns
        .agg([
            col("amount").sum().alias("revenue"),  // Sum
            col("id").count().alias("orders"),     // Count
        ])
        .sort(["revenue"], SortMultipleOptions::default().with_order_descending(true))  // Sort
        .collect()?;  // Execute optimized plan

    // Write Parquet
    ParquetWriter::new(std::fs::File::create("output.parquet")?)
        .with_compression(ParquetCompression::Zstd(None))
        .finish(&mut result.clone())?;

    Ok(())
}

Compare with Python pandas:

import pandas as pd

df = pd.read_csv("events.csv")
df = df[df["country"] == "CN"].copy()
df["amount"] = df["price"] * df["qty"]

result = (
    df.groupby(["day", "channel", "category"], as_index=False)
      .agg(revenue=("amount", "sum"), orders=("id", "count"))
      .sort_values("revenue", ascending=False)
)

result.to_parquet("output.parquet", compression="zstd")

Similar amount of code, but one runs in 14 seconds, the other in 0.7.

When Should You Use Polars?

After all this praise, let’s be fair about when Polars doesn’t make sense:

Good fit for Polars:

• Large datasets — millions to tens of millions of rows
• Batch processing jobs running on schedules
• Hard performance requirements — like sub-5-second SLAs
• Willingness to learn some Rust (or use Polars’ Python bindings)

Stick with pandas when:

• Exploratory data analysis, just poking around
• Small datasets, thousands to tens of thousands of rows
• Deep integration with pandas-specific libraries
• Team lacks Rust experience, learning curve too steep

Side note: Polars has a Python version too. Not as fast as native Rust, but still significantly faster than pandas. A solid middle ground if Rust isn’t in the cards.

How to Migrate Safely

If you’re ready to make the switch, here’s a sensible approach:

1. Pick one heavy job to start — don’t rewrite everything at once
2. Recreate the exact logic in Rust and verify outputs match
3. Run both paths in parallel for a week with a feature flag
4. Cut over once results align and the clock proves the win

Track these four metrics:

• Wall time
• CPU time
• Peak memory usage
• Output file size

When these numbers check out and time savings are real, you’re good to go.

TL;DR

Three things: Polars beats pandas by 20x thanks to columnar storage, query optimization, and multithreading — not magic. Not every scenario needs switching — pandas remains great for small data and exploration. Focus on hot paths — swap out the most time-consuming task first, that’s where the payoff is.

If your data pipeline has a job that keeps you waiting, give Polars a shot. You might just find yourself with coffee still hot and work already done.

If you found this helpful, give it a like to spread the word. Share it with coworkers still watching pandas spin. Follow Dream Beast Programming for more “20x with one change” technical insights. And hey — how long do your data processing tasks usually take? Drop a comment, let’s chat. See you in the next one.