Polars vs. pandas: a speed comparison, through an Apple Silicon lens

When I move and reshape data in a migration – locally or in a cloud warehouse – execution speed isn't a vanity metric. It shows up in three concrete places:

• Cost. In most cloud billing models, compute time is the bill. Faster processing can also let you drop to a cheaper service tier when resources are pre-allocated.
• Freshness. A stream that takes five minutes to process means the dashboards are at least five minutes stale. Even for hourly batch jobs, two minutes beats twenty.
• Feedback loop. A job that finishes in a minute lets me catch logic errors fast, and makes the work less painful.

The question this comparison tackles: across common data-engineering tasks, does Polars actually outrun pandas – and if so, where?

The claim being tested

Polars advertises gains of more than 30x over pandas, backed by its own open source benchmark suite. Publishing those benchmarks is to its credit, but there's an obvious conflict of interest when a tool's authors design the test for both their tool and the competition. Not necessarily dishonest – just prone to unconsciously favourable framing. So this exercise has two goals:

1. Sanity-check the 30x claim against ordinary, realistic tasks.
2. Locate the gains – show which operations get faster, so anyone weighing a switch knows what they're buying.

A few deliberate limits: this isn't a tutorial, it doesn't touch PySpark (a distributed model makes apples-to-apples awkward), and it isn't a fully reproducible scientific study – it's a practical guide aimed at rough, decision-useful estimates.

Setup

Datasets (both from Kaggle):

• Small / CSV: the classic Iris dataset – 5 columns, 150 rows. The kind of tiny file that pours out of Excel exports and database dumps. This stresses start-up overhead.
• Large / Parquet: the Financial Transactions Dataset for Fraud Detection – 18 columns, 5 million rows. About 796 MB as CSV, compressed to roughly 217 MB as Parquet with zstd. This stresses scalability. It sits at the low end of "big data", which is fine for our purposes.

Five tasks – two I/O, three processing:

1. Read each file with default settings (read_csv / read_parquet).
2. Write each file back out with defaults.
3. Compute a numeric expression as a new column.
4. Filter rows by condition(s).
5. Group by a column and aggregate.

Test rig: Apple MacBook Air (M4), 16 GB RAM, Python 3.14, Polars 1.41.2, pandas 3.0.3. Parquet written with zstd compression. Timing via timeit, median of 10 runs. No external acceleration for pandas – both libraries measured out of the box, the way most teams actually run them.

Results

The picture is more nuanced than the marketing suggests. Polars wins most tasks, and the lead is widest exactly where it matters – the heavy operations on the large dataset. But it isn't a clean sweep: on light element-wise compute and on reads it's roughly a wash, and on a couple of the tiny-data operations pandas even edges ahead.

Task	Dataset	pandas	Polars	Polars advantage
Read	Iris	0.20 ms	0.06 ms	~3.1x
Read	Transactions	166.87 ms	139.94 ms	~1.2x
Write	Iris	0.30 ms	0.21 ms	~1.4x
Write	Transactions	2.12 s	0.48 s	~4.4x
Numeric expr	Iris	0.11 ms	0.01 ms	~7.5x
Numeric expr	Transactions	2.43 ms	2.74 ms	~0.9x (tie)
Filter	Iris	0.10 ms	0.07 ms	~1.5x
Filter	Transactions	133.97 ms	21.71 ms	~6.2x
Group by	Iris	0.12 ms	0.14 ms	~0.8x (pandas ahead)
Group by	Transactions	87.98 ms	7.77 ms	~11.3x

The standouts: group-by on large data (~11.3x) and filtering large data (~6.2x), with writing Parquet a solid third (~4.4x). The surprises cut the other way too: reading the large Parquet is basically a tie (~1.2x), and the simple numeric expression is a wash – on the small frame pandas even wins the group-by outright. So the gains are real but concentrated: they live in the parallel-friendly heavy lifting, not in I/O or trivial element-wise math.

Why Apple Silicon shapes these numbers

These timings come straight off the M4, and a few of the chip's traits explain the shape of the table:

• Both libraries run native on Apple Silicon (arm64). Polars and pandas both ship native ARM wheels, so there's no Rosetta penalty – the absolute times benefit from modern cores, high memory bandwidth and a very fast SSD.
• Polars uses every core; pandas mostly doesn't. Polars is multithreaded in Rust and spreads parallel-friendly work across all of the M4's cores, while pandas runs these operations largely single-threaded out of the box. That's exactly why filtering and group-by show the widest gaps above – they parallelise beautifully.
• Reads barely move, and that makes sense. Reading the Parquet is dominated by I/O and by Arrow's decode path, which pandas already drives through a multithreaded pyarrow reader. With both libraries hitting the same fast Arrow machinery, the M4's SSD and memory bandwidth are the bottleneck, not the dataframe library – so the gap collapses to ~1.2x. The big win on write is the flip side: there Polars' Rust writer pulls clear of pandas.
• Unified memory helps the big-data case. The M4's high-bandwidth unified memory is a real asset for the Transactions-style workload, where throughput matters more than raw clock speed.
• RAM is the variable to watch. The ~217 MB Parquet file decompresses and expands in memory, and intermediate copies during transforms add up. It fits comfortably in 16 GB for this dataset, but scale to several files or much wider tables and 16 GB is where you start to feel pressure – exactly where Polars' lazy API (below) earns its keep.

The asterisk: Polars is being sandbagged here

Every test above measures a single operation in isolation. That's not how real pipelines work – you usually chain many transforms together. Polars has a lazy API that looks at the whole chain and optimises it before executing a single row. Because this comparison only timed one-shot operations, that advantage never showed up. In realistic multi-step pipelines the gap would likely be larger, not smaller – and on a memory-constrained machine, lazy evaluation also helps you avoid materialising intermediate copies you don't need.

Verdict

• Ingestion: writing Parquet is clearly faster on Polars (~4.4x on the large file), but reading is essentially a tie (~1.2x) – both libraries lean on the same fast Arrow reader, so I/O, not the dataframe engine, sets the pace.
• Processing: this is where Polars earns its reputation – group-by (~11.3x) and filter (~6.2x) on the large dataset are decisive wins. But a plain element-wise numeric expression is a wash (~0.9x), and on tiny frames the differences are noise – pandas even takes the small group-by.
• Overall: Polars wins the operations that dominate real ETL cost – grouping, filtering, writing – often by a wide margin. It is not uniformly faster: on reads and trivial compute the two are level.

And – nothing here comes close to the advertised 30x. Even the best case tops out around 11x, on one operation. On ordinary tasks and realistic hardware, that headline number looks optimistic. The honest takeaway is more grounded: Polars is meaningfully faster where it counts, no slower anywhere that matters, and it isn't harder to work with than pandas.

Practical recommendation: for your next data-engineering project where the data fits in memory – which, on a modern 16 GB laptop, covers a lot of ground – default to Polars. The heavy operations that drive your runtime get materially faster, the rest is at least as fast, and the lazy API gives you headroom as pipelines and data sizes grow.

Reproduce it yourself

Want to run the same five tasks on your own machine? Grab the script – it's the exact code behind this article:

polars_vs_pandas_benchmark.py

It synthesises stand-in datasets by default, so it runs with no downloads – or point it at the real Kaggle files to reproduce the table above. Parquet is written with zstd; the script auto-detects which columns to compute and group on (or you can name them):

pip install numpy pandas polars pyarrow

# Synthetic data, out of the box:
python polars_vs_pandas_benchmark.py

# The real datasets used here (Iris CSV + the Transactions Parquet):
python polars_vs_pandas_benchmark.py \
    --iris iris.csv \
    --transactions financial_fraud_detection_dataset.parquet \
    --tx-num-col amount --tx-group-col merchant_category \
    --compression zstd

Here's the actual run behind the table above – the real Kaggle data, median of 10 runs via timeit, zstd Parquet, on the M4:

pandas 3.0.3  ·  polars 1.41.2  ·  numpy 2.4.6
Timing: median of 10 runs via timeit  ·  Parquet codec: zstd

=== Small / CSV (Iris) ===
Task                        pandas      Polars   Advantage
----------------------------------------------------------
Read (read_csv)            0.20 ms     0.06 ms        3.1x
Write                      0.30 ms     0.21 ms        1.4x
Numeric expr               0.11 ms     0.01 ms        7.5x
Filter                     0.10 ms     0.07 ms        1.5x
Group by                   0.12 ms     0.14 ms        0.8x

=== Large / Parquet (Transactions) ===
  rows=5,000,000  cols=18  numeric='amount'  group='merchant_category'
Read (read_parquet)      166.87 ms   139.94 ms        1.2x
Write                       2.12 s   484.49 ms        4.4x
Numeric expr               2.43 ms     2.74 ms        0.9x
Filter                   133.97 ms    21.71 ms        6.2x
Group by                  87.98 ms     7.77 ms       11.3x

A note on the numbers: timings shift a little run to run, and they depend on your hardware, your data's shape and the Parquet codec – on a smaller machine, or with a wider table, the absolute times move. What stays stable is the shape of the result: Polars pulls away on group-by, filter and write, while reads and trivial element-wise compute stay close.

Curious what this looks like on your own data and hardware? Get in touch – or just run the script above and see your actual times.