Nested Tables in Python Polars - python-polars

Question

One of the key data analytics outputs is tables. And to analyse large research databases, we frequently create nested tables with two or more levels of nesting in rows and / or columns. I could create the nested tables in Pandas, but don't know if they can be created in Polars.
I am using the database from Kaggle called 'Home Loan Approval'. It's URL is https://www.kaggle.com/datasets/rishikeshkonapure/home-loan-approval.
I use the following code in Pandas to create a nested table with 'Gender' and 'Married' in rows and 'Property_Area' and 'Self_Employed' in columns
(
pd.read_csv('loan_sanction_test.csv')
.groupby(['Gender', 'Married', 'Property_Area', 'Self_Employed'])['LoanAmount'].sum()
.unstack([2,3])
)
In Polars, we don't have multi-indexes like Pandas. Polars pivot function only pivots to a single level. For e.g. if I use ['Property_Area' and 'Self_Employed'] in columns, pivot will reproduce these variables one after another and not nest them. Here's the code which illustrates this (using Polars version 0.16):
(
pl.read_csv('loan_sanction_test.csv')
.groupby(['Gender', 'Married', 'Property_Area', 'Self_Employed']).agg(pl.col('LoanAmount').sum())
.pivot(index=['Gender', 'Married'], columns=['Property_Area', 'Self_Employed'], values='LoanAmount')
)
We frequently use three level deep nesting in rows as well as columns. Is there a way to generate nested tables in Polars like Pandas example above?

Answer 1

One thing you can try when nesting columns is concat_str. For example: (My firewall blocks Kaggle downloads so I've simulated data.)
(
df
.sort(['Property_Area', 'Self_Employed'])
.with_columns(
pl.concat_str(['Property_Area', 'Self_Employed'], "|").alias('PA_SE')
)
.pivot(
index=["Gender", "Married"],
columns=["PA_SE"],
values="LoanAmount",
aggregate_fn="count",
)
)
shape: (4, 8)
+--------+---------+----------+-----------+--------------+---------------+----------+-----------+
| Gender | Married | Rural|No | Rural|Yes | Semiurban|No | Semiurban|Yes | Urban|No | Urban|Yes |
| --- | --- | --- | --- | --- | --- | --- | --- |
| str | str | u32 | u32 | u32 | u32 | u32 | u32 |
+===============================================================================================+
| Female | No | 38 | 35 | 54 | 37 | 40 | 41 |
| Female | Yes | 46 | 45 | 38 | 45 | 45 | 38 |
| Male | Yes | 40 | 39 | 47 | 44 | 39 | 40 |
| Male | No | 34 | 48 | 44 | 48 | 45 | 30 |
+--------+---------+----------+-----------+--------------+---------------+----------+-----------+
It's not as slick as Pandas, but it does get you to an answer.

Answer 2

I don't think you can make nested tables, but you could do something kinda similar with
(
df.groupby(
[
"Gender",
"Married",
"Property_Area",
"Self_Employed",
]
)
.agg(pl.col("LoanAmount").sum())
.pivot(
index=["Gender", "Married", "Self_Employed"],
columns=["Property_Area"],
values="LoanAmount",
aggregate_fn="sum",
)
)
which would give you
shape: (15, 6)
┌────────┬─────────┬───────────────┬───────┬───────┬───────────┐
│ Gender ┆ Married ┆ Self_Employed ┆ Rural ┆ Urban ┆ Semiurban │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ str ┆ str ┆ i64 ┆ i64 ┆ i64 │
╞════════╪═════════╪═══════════════╪═══════╪═══════╪═══════════╡
│ Female ┆ No ┆ Yes ┆ 389 ┆ null ┆ 295 │
│ Female ┆ Yes ┆ No ┆ 596 ┆ 1610 ┆ 1195 │
│ Male ┆ Yes ┆ Yes ┆ 1692 ┆ 1011 ┆ 1073 │
│ null ┆ Yes ┆ No ┆ null ┆ null ┆ 270 │
│ ... ┆ ... ┆ ... ┆ ... ┆ ... ┆ ... │
│ Male ┆ Yes ┆ null ┆ 465 ┆ 1437 ┆ 626 │
│ null ┆ No ┆ Yes ┆ null ┆ 138 ┆ null │
│ null ┆ Yes ┆ Yes ┆ 68 ┆ null ┆ null │
│ Female ┆ Yes ┆ null ┆ null ┆ null ┆ 94 │
└────────┴─────────┴───────────────┴───────┴───────┴───────────┘