FireDucks Beats Pandas by Up to 20.77x on 10M-Row Benchmarks

Pandas is the default starting point for tabular data work in Python, but its eager execution model—running each operation the moment it is called—becomes a measurable liability once row counts climb into the millions. FireDucks, a compiler-accelerated DataFrame library developed by NEC, attacks that bottleneck without requiring a rewrite: swap the import statement, keep your existing pandas syntax, and the library's lazy execution engine handles the rest. A benchmark published September 2, 2026 puts hard numbers behind that claim, and the margins are large enough that any engineer running multi-million-row pipelines should pay attention.

How FireDucks Executes Differently

Where pandas materialises each operation immediately, FireDucks defers execution. It accumulates a sequence of DataFrame operations, constructs an optimised execution plan, and only triggers evaluation when the result is actually required—such as at a print() call or an explicit ._evaluate() call. That plan is then dispatched across multiple CPU cores simultaneously. Redundant intermediate computations are eliminated before any CPU cycles are spent on them, and the remaining work is parallelised.

The API compatibility is high but not total. FireDucks DataFrames are distinct internal objects, so some pandas features and third-party library integrations may behave differently or fail outright. Installation is a single pip command: pip install -U fireducks, and the entry point is import fireducks.pandas as pd.

Benchmark Methodology

The benchmark used a reproducible 10-million-row primary dataset and a separate 2-million-row lookup table. Seven workloads were tested: Parquet reading, column filtering, low-cardinality groupby, high-cardinality groupby, full-dataset sorting, joining against the lookup table, and a chained multi-step pipeline. Each workload ran one warm-up pass followed by five measured passes, with execution order alternated between the two libraries and median time reported. Both libraries were verified to produce equivalent outputs.

The test environment was CPU-only on Linux: 9 AMD EPYC 9V74 cores, 15.93 GiB RAM, Python 3.12.13, pandas 2.3.3, FireDucks 1.4.4, NumPy 2.5.2, and PyArrow 21.0.0.

Benchmark Results

FireDucks was faster across all seven workloads. The peak gain was 20.77× on the full-dataset sort, where pandas spent 15.2674 seconds against FireDucks' 0.7352 seconds. Low-cardinality groupby followed at 15.44×, and column filtering at 11.63×. The chained pipeline—the workload most representative of production ETL code—finished 5.94× faster. The geometric mean speedup across all seven tests was 7.28×.

Real-World Corroboration and Practical Limits

Independent results extend beyond this single benchmark. Toyota Technical Development Corporation tested FireDucks inside its internal AI framework and recorded a 60% reduction in data-analysis time alongside a 76% decrease in analysis-PC operating time. Data science writer Avi Chawla ran FireDucks on Google Colab and observed a drop from 12.3 seconds to 3.5 seconds on a pandas workload—roughly a 4× improvement, consistent with the lower end of the structured benchmark range.

The 20.77× figure is real but operation-specific. Gains depend on data cardinality, the mix of operations, and how much opportunity the lazy planner finds to eliminate intermediate steps. Join performance yielded only 3.20×—still meaningful but well below the sort ceiling. Engineers should profile against their actual workload mix rather than assuming peak-case acceleration. As covered in our analysis of pipeline architecture choices that drive AI performance gains, the composition of operations matters as much as the tool selected.

FireDucks' near-zero migration cost—a single import change—and consistent wins across all seven tested workload types make it one of the more actionable performance tools available to Python data engineers, particularly in cost-sensitive environments where compute time maps directly to infrastructure spend.