Polars vs. Pandas — An Independent Speed Comparison

Overview

1. Introduction — Purpose and Reasons
2. Datasets, Tasks, and Settings
3. Results
4. Conclusions
5. Wrapping Up

Introduction — Objective and Causes

Pace is essential when coping with massive quantities of information. If you’re dealing with information in a cloud information warehouse or comparable, then the pace of execution on your information ingestion and processing impacts the next:

• Cloud prices: That is most likely the largest issue. Extra compute time equals extra prices in most billing fashions. In different billing primarily based on a specific amount of preallocated sources, you might have chosen a decrease service degree if the pace of your ingestion and processing was greater.
• Information timeliness: In case you have a real-time stream that takes 5 minutes to course of information, then your customers could have a lag of a minimum of 5 minutes when viewing the info by way of e.g. a Energy BI rapport. This distinction is usually a lot in sure conditions. Even for batch jobs, the info timeliness is essential. If you’re operating a batch job each hour, it’s a lot higher if it takes 2 minutes fairly than 20 minutes.
• Suggestions loop: In case your batch job takes solely a minute to run, then you definately get a really fast suggestions loop. This most likely makes your job extra gratifying. As well as, it lets you discover logical errors extra shortly.

As you’ve most likely understood from the title, I’m going to offer a pace comparability between the 2 Python libraries Polars and Pandas. If you realize something about Pandas and Polars from earlier than, then you realize that Polars is the (comparatively) new child on the block proclaiming to be a lot quicker than Pandas. You most likely additionally know that Polars is applied in Rust, which is a development for a lot of different fashionable Python instruments like uv and Ruff.

There are two distinct causes that I need to do a pace comparability take a look at between Polars and Pandas:

Purpose 1 — Investigating Claims

Polars boasts on its web site with the next declare: In comparison with pandas, it (Polars) can obtain greater than 30x efficiency features.

As you may see, you may observe a link to the benchmarks that they’ve. It’s commendable that they’ve pace checks open supply. However in case you are writing the comparability checks for each your individual software and a competitor’s software, then there may be a slight battle of curiosity. I’m not right here saying that they’re purposefully overselling the pace of Polars, however fairly that they may have unconsciously chosen for favorable comparisons.

Therefore the primary cause to do a pace comparability take a look at is solely to see whether or not this helps the claims offered by Polars or not.

Purpose 2 — Larger granularity

Another excuse for doing a pace comparability take a look at between Polars and Pandas is to make it barely extra clear the place the efficiency features may be.

This may be already clear in case you’re an skilled on each libraries. Nevertheless, pace checks between Polars and Pandas are principally of curiosity to these contemplating switching up their software. In that case, you may not but have performed round a lot with Polars since you are not sure whether it is price it.

Therefore the second cause to do a pace comparability is solely to see the place the pace features are situated.

I need to take a look at each libraries on completely different duties each inside information ingestion and Data Processing. I additionally need to contemplate datasets which can be each small and huge. I’ll keep on with frequent duties inside information engineering, fairly than esoteric duties that one seldom makes use of.

What I cannot do

• I cannot give a tutorial on both Pandas or Polars. If you wish to study Pandas or Polars, then an excellent place to begin is their documentation.
• I cannot cowl different frequent information processing libraries. This may be disappointing to a fan of PySpark, however having a distributed compute mannequin makes comparisons a bit tougher. You may discover that PySpark is faster than Polars on duties which can be very simple to parallelize, however slower on different duties the place conserving all the info in reminiscence reduces journey instances.
• I cannot present full reproducibility. Since that is, in humble phrases, solely a weblog submit, then I’ll solely clarify the datasets, duties, and system settings that I’ve used. I cannot host an entire operating setting with the datasets and bundle every part neatly. This isn’t a exact scientific experiment, however fairly a information that solely cares about tough estimations.

Lastly, earlier than we begin, I need to say that I like each Polars and Pandas as instruments. I’m not financially or in any other case compensated by any of them clearly, and don’t have any incentive apart from being inquisitive about their efficiency ☺️

Datasets, Duties, and Settings

Let’s first describe the datasets that I might be contemplating, the duties that the libraries will carry out, and the system settings that I might be operating them on.

Datasets

A most firms, you will want to work with each small and (comparatively) massive datasets. For my part, an excellent information processing software can sort out each ends of the spectrum. Small datasets problem the start-up time of duties, whereas bigger datasets problem scalability. I’ll contemplate two datasets, each will be discovered on Kaggle:

• A small dataset on the format CSV: It’s no secret that CSV recordsdata are all over the place! Typically they’re fairly small, coming from Excel recordsdata or database dumps. What higher instance of this than the classical iris dataset (licensed with CC0 1.0 Universal License) with 5 columns and 150 rows. The iris model I linked to on Kaggle has 6 columns, however the classical one doesn’t have a operating index column. So take away this column if you would like exactly the identical dataset as I’ve. The iris dataset is definitely small information by any stretch of the creativeness.
• A big dataset on the format Parquet: The parquet format is tremendous helpful for giant information because it has built-in compression column-wise (together with many different advantages). I’ll use the Transaction dataset (licensed with Apache License 2.0) representing monetary transactions. The dataset has 24 columns and seven 483 766 rows. It’s shut to three GB in its CSV format discovered on Kaggle. I used Pandas & Pyarrow to transform this to a parquet file. The ultimate result’s solely 905 MB because of the compression of the parquet file format. That is on the low finish of what individuals name large information, however it’s going to suffice for us.

Duties

I’ll do a pace comparability on 5 completely different duties. The primary two are I/O duties, whereas the final three are frequent duties in information processing. Particularly, the duties are:

1. Studying information: I’ll learn each recordsdata utilizing the respective strategies read_csv() and read_parquet() from the 2 libraries. I cannot use any non-compulsory arguments as I need to examine their default habits.
2. Writing information: I’ll write each recordsdata again to an identical copies as new recordsdata utilizing the respective strategies to_csv() and to_parquet() for Pandas and write_csv() and write_parquet() for Polars. I cannot use any non-compulsory arguments as I need to examine their default habits.
3. Computing Numeric Expressions: For the iris dataset I’ll compute the expression SepalLengthCm ** 2 + SepalWidthCm as a brand new column in a replica of the DataFrame. For the transactions dataset, I’ll merely compute the expression (quantity + 10) ** 2 as a brand new column in a replica of the DataFrame. I’ll use the usual option to transform columns in Pandas, whereas in Polars I’ll use the usual features all(), col(), and alias() to make an equal transformation.
4. Filters: For the iris dataset, I’ll choose the rows equivalent to the standards SepalLengthCm >= 5.0 and SepalWidthCm <= 4.0. For the transactions dataset, I’ll choose the rows equivalent to the specific standards merchant_category == 'Restaurant'. I’ll use the usual filtering methodology primarily based on Boolean expressions in every library. In pandas, that is syntax equivalent to df_new = df[df['col'] < 5], whereas in Polars that is given equally by the filter() operate together with the col() operate. I’ll use the and-operator & for each libraries to mix the 2 numeric circumstances for the iris dataset.
5. Group By: For the iris dataset, I’ll group by the Species column and calculate the imply values for every species of the 4 columns SepalLengthCm, SepalWidthCm, PetalLengthCm, and PetalWidthCm. For the transactions dataset, I’ll group by the column merchant_category and depend the variety of situations in every of the courses inside merchant_category. Naturally, I’ll use the groupby() operate in Pandas and the group_by() operate in Polars in apparent methods.

Settings

• System Settings: I’m operating all of the duties regionally with 16GB RAM and an Intel Core i5–10400F CPU with 6 Cores (12 logical cores by way of hyperthreading). So it’s not state-of-the-art by any means, however adequate for easy benchmarking.
• Python: I’m operating Python 3.12. This isn’t probably the most present steady model (which is Python 3.13), however I believe this can be a good factor. Generally the newest supported Python model in cloud information warehouses is one or two variations behind.
• Polars & Pandas: I’m utilizing Polars model 1.21 and Pandas 2.2.3. These are roughly the most recent steady releases to each packages.
• Timeit: I’m utilizing the usual timeit module in Python and discovering the median of 10 runs.

Particularly attention-grabbing might be how Polars can reap the benefits of the 12 logical cores by way of multithreading. There are methods to make Pandas reap the benefits of a number of processors, however I need to examine Polars and Pandas out of the field with none exterior modification. In spite of everything, that is most likely how they’re operating in most firms around the globe.

Outcomes

Right here I’ll write down the outcomes for every of the 5 duties and make some minor feedback. Within the subsequent part I’ll attempt to summarize the details right into a conclusion and level out an obstacle that Polars has on this comparability:

Process 1 — Studying information

The median run time over 10 runs for the studying process was as follows:

# Iris Dataset
Pandas: 0.79 milliseconds
Polars: 0.31 milliseconds

# Transactions Dataset
Pandas: 14.14 seconds
Polars: 1.25 seconds

For studying the Iris dataset, Polars was roughly 2.5x quicker than Pandas. For the transactions dataset, the distinction is even starker the place Polars was 11x quicker than Pandas. We are able to see that Polars is far quicker than Pandas for studying each small and huge recordsdata. The efficiency distinction grows with the scale of the file.

Process 2— Writing information

The median run time in seconds over 10 runs for the writing process was as follows:

# Iris Dataset
Pandas: 1.06 milliseconds
Polars: 0.60 milliseconds

# Transactions Dataset
Pandas: 20.55 seconds
Polars: 10.39 seconds

For writing the iris dataset, Polars was round 75% quicker than Pandas. For the transactions dataset, Polars was roughly 2x as quick as Pandas. Once more we see that Polars is quicker than Pandas, however the distinction right here is smaller than for studying recordsdata. Nonetheless, a distinction of near 2x in efficiency is an enormous distinction.

Process 3 —Computing Numeric Expressions

The median run time over 10 runs for the computing numeric expressions process was as follows:

# Iris Dataset
Pandas: 0.35 milliseconds
Polars: 0.15 milliseconds

# Transactions Dataset
Pandas: 54.58 milliseconds
Polars: 14.92 milliseconds

For computing the numeric expressions, Polars beats Pandas with a charge of roughly 2.5x for the iris dataset, and roughly 3.5x for the transactions dataset. This can be a fairly huge distinction. It needs to be famous that computing numeric expressions is quick in each libraries even for the massive dataset transactions.

Process 4 — Filters

The median run time over 10 runs for the filters process was as follows:

# Iris Dataset
Pandas: 0.40 milliseconds
Polars: 0.15 milliseconds

# Transactions Dataset
Pandas: 0.70 seconds
Polars: 0.07 seconds

For filters, Polars is 2.6x quicker on the iris dataset and 10x as quick on the transactions dataset. That is most likely probably the most stunning enchancment for me since I suspected that the pace enhancements for filtering duties wouldn’t be this huge.

Process 5 — Group By

The median run time over 10 runs for the group by process was as follows:

# Iris Dataset
Pandas: 0.54 milliseconds
Polars: 0.18 milliseconds

# Transactions Dataset
Pandas: 334 milliseconds 
Polars: 126 milliseconds

For the group-by process, there’s a 3x pace enchancment for Polars within the case of the iris dataset. For the transactions dataset, there’s a 2.6x enchancment of Polars over Pandas.

Conclusions

Earlier than highlighting every level under, I need to level out that Polars is considerably in an unfair place all through my comparisons. It’s typically that a number of information transformations are carried out after each other in apply. For this, Polars has the lazy API that optimizes this earlier than calculating. Since I’ve thought of single ingestions and transformations, this benefit of Polars is hidden. How a lot this is able to enhance in sensible conditions isn’t clear, however it might most likely make the distinction in efficiency even larger.

Information Ingestion

Polars is considerably quicker than Pandas for each studying and writing information. The distinction is largest in studying information, the place we had an enormous 11x distinction in efficiency for the transactions dataset. On all measurements, Polars performs considerably higher than Pandas.

Information Processing

Polars is considerably quicker than Pandas for frequent information processing duties. The distinction was starkest for filters, however you may a minimum of anticipate a 2–3x distinction in efficiency throughout the board.

Ultimate Verdict

Polars persistently performs quicker than Pandas on all duties with each small and huge information. The enhancements are very important, starting from a 2x enchancment to a whopping 11x enchancment. In terms of studying massive parquet recordsdata or performing filter statements, Polars is leaps and sure in entrance of Pandas.

Nevertheless…Nowhere right here is Polars remotely near performing 30x higher than Pandas, as Polars’ benchmarking suggests. I might argue that the duties that I’ve offered are commonplace duties carried out on sensible {hardware} infrastructure. So I believe that my conclusions give us some room to query whether or not the claims put ahead by Polars give a practical image of the enhancements that you may anticipate.

However, I’m in little doubt that Polars is considerably quicker than Pandas. Working with Polars isn’t extra sophisticated than working with Pandas. So on your subsequent information engineering undertaking the place the info matches in reminiscence, I might strongly recommend that you simply go for Polars fairly than Pandas.

Wrapping Up

I hope this weblog submit gave you a unique perspective on the pace distinction between Polars and Pandas. Please remark when you’ve got a unique expertise with the efficiency distinction between Polars and Pandas than what I’ve offered.

If you’re all for AI, Data Science, or information engineering, please observe me or join on LinkedIn.

Like my writing? Take a look at a few of my different posts: