# Colab setup ------------------
import os, sys, subprocess
if "google.colab" in sys.modules:
cmd = "pip install --upgrade polars watermark"
process = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = process.communicate()
data_path = "https://s3.amazonaws.com/bebi103.caltech.edu/data/"
else:
data_path = "../data/"import numpy as np
import polars as pl
import polars.selectors as cs
import pandas as pdAs of September 2025, Pandas is far and away the most widely used data frame package for Python. We are using Polars primarily because, in my opinion, the API is more intuitive and therefore easier for beginners and experts alike to use. It is also faster, sometimes much faster. It is, however, important to know about Pandas and how to use it because many of your colleagues use it and many packages you may use do, too.
Therefore, in this part of the lesson, I discuss how to convert a Polars data frame to Pandas, and vice versa. I also provide syntax for doing common tasks in Polars and Pandas. It is also worth reading the section of the Polars user guide comparing Pandas and Polars.
For ease of discussion and comparison, we will use a simple data frame that has two categorical columns, 'c1', and 'c2', two quantitative columns as floats, 'q1', and 'q2', and a column, 'i1' of integer values. It also has an identity column, a unique identifier for each row that is useful when converting the data frame to tall format. Note that 'q1' has a null value and 'q2' has a NaN value.
data = dict(
id=list(range(1, 9)),
c1=['a']*4 + ['b']*4,
c2=['c', 'd'] * 4,
q1=[1.1, 2.2, 3.1, None, 2.9, 1.7, 3.0, 7.3],
q2=[4.5, 2.3, np.nan, 1.1, 7.8, 2.3, 1.1, 0.8],
i1=[5, 3, 0, 2, 4, 3, 4, 1],
)
df = pl.DataFrame(data)
# Take a look
df| id | c1 | c2 | q1 | q2 | i1 |
|---|---|---|---|---|---|
| i64 | str | str | f64 | f64 | i64 |
| 1 | "a" | "c" | 1.1 | 4.5 | 5 |
| 2 | "a" | "d" | 2.2 | 2.3 | 3 |
| 3 | "a" | "c" | 3.1 | NaN | 0 |
| 4 | "a" | "d" | null | 1.1 | 2 |
| 5 | "b" | "c" | 2.9 | 7.8 | 4 |
| 6 | "b" | "d" | 1.7 | 2.3 | 3 |
| 7 | "b" | "c" | 3.0 | 1.1 | 4 |
| 8 | "b" | "d" | 7.3 | 0.8 | 1 |
If you have a Polars data frame, you can directly convert it to a Pandas data frame using the to_pandas(), method. Let’s do that for our data frame.
df.to_pandas()| id | c1 | c2 | q1 | q2 | i1 | |
|---|---|---|---|---|---|---|
| 0 | 1 | a | c | 1.1 | 4.5 | 5 |
| 1 | 2 | a | d | 2.2 | 2.3 | 3 |
| 2 | 3 | a | c | 3.1 | NaN | 0 |
| 3 | 4 | a | d | NaN | 1.1 | 2 |
| 4 | 5 | b | c | 2.9 | 7.8 | 4 |
| 5 | 6 | b | d | 1.7 | 2.3 | 3 |
| 6 | 7 | b | c | 3.0 | 1.1 | 4 |
| 7 | 8 | b | d | 7.3 | 0.8 | 1 |
Note that the null value becomes a NaN. All missing data in Pandas are NaN. (Well, not really. You can have an object data type for a column that permits None values. However, when Pandas reads in data, when there are missing data, it assigns it to be NaN by default.)
Note also that Pandas has an index displayed on the left side of the data frame. In general, we will not use Pandas indexes.
Similarly, if you have a data frame in Pandas, you can convert it to a Polars data frame using the pl.from_pandas() function.
pl.from_pandas(pd.DataFrame(data))| id | c1 | c2 | q1 | q2 | i1 |
|---|---|---|---|---|---|
| i64 | str | str | f64 | f64 | i64 |
| 1 | "a" | "c" | 1.1 | 4.5 | 5 |
| 2 | "a" | "d" | 2.2 | 2.3 | 3 |
| 3 | "a" | "c" | 3.1 | null | 0 |
| 4 | "a" | "d" | null | 1.1 | 2 |
| 5 | "b" | "c" | 2.9 | 7.8 | 4 |
| 6 | "b" | "d" | 1.7 | 2.3 | 3 |
| 7 | "b" | "c" | 3.0 | 1.1 | 4 |
| 8 | "b" | "d" | 7.3 | 0.8 | 1 |
Below is a table listing common tasks with data frame done using Polars and Pandas.
| Description | Pandas | Polars |
|---|---|---|
| Convert dictionary to df | pd.DataFrame(data) |
pl.DataFrame(data) |
| Make 2D Numpy array into df | pd.DataFrame(my_ar, columns=['col1', 'col2', 'col3']) |
pl.DataFrame(my_ar, schema=['col1', 'col2', 'col3'], orient='row') |
| Read from CSV | pd.read_csv(file_name) |
pl.read_csv(file_name) |
| Lazily read CSV | — | pl.scan_csv(file_name) |
| Read from Excel | pd.read_excel(file_name) |
pl.read_excel(file_name) |
| Read from JSON | pd.read_json(file_name) |
pl.read_json(file_name) |
| Read from HDF5 | pd.read_hdf(file_name) |
— |
| Write CSV | df.to_csv(fname, index=False) |
df.write_csv(fname) |
| Rename columns | df.rename(columns={'c1': 'cat1', 'c2': 'cat2'}) |
df.rename({'c1': 'cat1', 'c2': 'cat2'}) |
Get column 'q1' as series |
df['q1'] or df.get_column('q1') or df.loc[:,'q1'] |
df['q1'] or df.get_column('q1') |
Get column 'q1' as data frame |
df[['q1']] or df.loc[:,['q1']] |
df.select('q1') |
Get columns 'c1' and 'q2' |
df[['c1', 'q2']] or df.loc[:, ['c1', 'q2']] |
df.select('c1', 'q2') |
| Get columns containing floats | df.select_dtypes(float) |
df.select(cs.float()) |
Get row 4 |
df.loc[4, :] |
df.row(4) or df.row(4, named=True) |
Get row 4 as data frame |
df.loc[[4], :] |
df[4] |
Get row where i1 is 2 |
df.loc[df['i1']==2, :] |
df.row(by_predicate=pl.col('i1')==2) or df.filter(pl.col('i1')==2) |
Sub df with rows where c1 is 'a' |
df.loc[df['c1']=='a', :] |
df.filter(pl.col('c1')=='a') |
Sub df where c1 is 'a' and c2 is 'd' |
df.loc[(df['c1']=='a') & (df['c2']=='d'), :] |
df.filter((pl.col('c1')=='a') & (pl.col('c2')=='d')) |
| Iterate over columns of df | for col, s in df.items() |
for s in df |
| Iterate over rows of df | for row_ind, row in df.iterrows() |
for r in df.iter_rows(named=True) |
| Group by single column | df.groupby('c1') |
df.group_by('c1') |
| Group by maintaining order | df.groupby('c1', sort=False) |
df.group_by('c1', maintain_order=True) |
| Group by multiple columns | df.groupby(['c1', 'c2']) |
df.group_by(['c1', 'c2']) |
| Iterate over groups | for group, subdf in df.groupby('c1') |
for (group,), subdf in df.group_by('c1') |
| Iterate over nested groups | for (g1, g2), subdf in df.groupby(['c1', 'c2']) |
for (g1, g2), subdf in df.group_by(['c1', 'c2']) |
| Group by and apply mean¹ | df.groupby('c1').mean(numeric_only=True) |
df.group_by('c1').mean() |
| Group by and apply median to one column | df.groupby('c1')['q1'].median() |
df.group_by('c1').agg(pl.col('q1').median()) |
| Group by and apply mean to two columns | df.groupby('c1')[['q1', 'q2']].mean() |
df.group_by('c1').agg(pl.col('q1', 'q2').mean()) |
| Group by and apply custom func to col² | df.groupby('c1')['q1'].apply(my_fun) |
df.group_by('c1').agg(pl.col('q1').map_batches(my_fun, return_dtype=float, returns_scalar=True)) |
| Group by and apply custom func to 2 cols³ | df.groupby('c1')[['q1', 'q2']].apply(my_fun) |
df.group_by('c1').agg(pl.struct(['q1', 'q2']).map_batches(my_fun, return_dtype=float, returns_scalar=True)) |
| Group by and rank within each group | df.groupby('c1')['q1'].rank() |
df.select(pl.col('q1').rank().over('c1')) |
| Convert to tall format | df.melt(value_name='value', var_name='var', id_vars='id') |
df.unpivot(value_name='value', variable_name='var', index='id') |
| Pivot tall result above | df_tall.pivot(columns='var', index='id').reset_index() |
df_tall.pivot(on='var', index='id') |
| Select columns with string in name | df.filter(regex='q') or df[df.columns[df.columns.str.contains('q')]] |
df.select(cs.contains('q')) |
| Add column of zeros to data frame | df['new_col'] = 0 or df.assign(new_col=0) |
df.with_columns(pl.lit(0).alias('new_col')) |
| Add a Numpy array as column | df['new_col'] = my_array or df.assign(new_col=my_array) |
df.with_columns(pl.Series(my_array).alias('new_col')) |
| Multiply two columns; make new column | df['q1q2'] = df['q1'] * df['q2'] or df.assign(q1q2=df['q1'] * df['q2'] |
df.with_columns((pl.col('q1') * pl.col('q2')).alias('q1q2')) |
| Apply a function to each row making new col⁴ | df.assign(new_col=my_fun) |
df.with_columns(pl.struct(pl.all()).map_elements(my_fun, return_dtype=float).alias('new_col')) |
| Drop rows with missing data | df.dropna() |
df.drop_nulls() |
| Sort according to a column | df.sort_values(by='i1') |
df.sort(by='i1') |
| Inner join two data frames⁵ | pd.merge(df, df2, on=shared_columns) |
df.join(df2, on=shared_columns) |
| Concatenate data frames vertically | pd.concat((df, df2)) |
pl.concat((df, df2), how='diagonal') |
| Concatenate data frames horizontally | pd.concat((df, df2), axis=1) |
pl.concat((df, df2), how='horizontal') |
Footnotes
nulls are not included.my_fun must take an array_like data type (list, Numpy array, Pandas Series, etc.) as input. For Polars, the function my_fun must take a Polars Series as input. It is wise to specify the data type of the output of the function (shown as float in the above example, but can be whatever type my_fun returns). A Pandas example: my_fun = lambda x: np.sum(np.sin(x)). A Polars example: my_fun = lambda s: s.exp().sum().my_fun = lambda df: (np.sin(df['q1']) * df['q2']).sum(). A Polars example: my_fun = lambda s: (s.struct.field('q1').sin() * s.struct.field('q2')).sum()my_fun must take as its argument a Pandas Series with an index containing the names of the columns of the original data frame. For Polars, my_fun must take as its argument a dictionary with keys given by the names of the columns of the original data frame. The functions may then have the same syntax (though possibly with different type hints). An example: my_fun = lambda r: r['i1'] * np.sin(r['q2']). However, note that in Polars, a null value is treated as None, which means you cannot apply a function to it, multiply by it, etc.on kwarg for df.join() is required. With Pandas, which columns to join on are inferred based on like-names of columns.Pandas supports hierarchical indexes, called MultiIndexes. This is not supported by Polars. Polars will not read a CSV file with hierarchical indexes. If you have a data set in a CSV file with hierarchical indexes, you can convert it to a CSV file in tall format where the MultiIndex has been converted to columns using the bebi103.utils.unpivot_csv() function. This operation is akin to a df.melt() operation on a data frame with a hierarchical index. You can then read the converted CSV file into Polars and begin working with it.
%load_ext watermark
%watermark -v -p numpy,pandas,polars,jupyterlabPython implementation: CPython
Python version : 3.13.7
IPython version : 9.5.0
numpy : 2.2.6
pandas : 2.3.2
polars : 1.33.1
jupyterlab: 4.4.7