Python Polars: Unique values in each row - python-polars

Question

I have a Polars dataframe with 150 columns of currency codes. I can identify them with a regex expression df.select(pl.col('^*cur$')). I am trying to determine the unique set of currency codes in each row. Nulls should be ignored.
df = pl.DataFrame(
{
"col1cur": ["EUR", "EUR", "EUR"],
"col2cur": [None, "EUR", None],
"col3cur": ["EUR", None, None],
"col4cur": ["EUR", "GBP", None],
"target": [["EUR"], ["EUR", "GBP"], ["EUR"]]
}
)
In pandas, I would do this. Can anyone help, on how I would approach this in Polars?
pandas_df = pd.DataFrame(
{
"col1cur": ["EUR", "EUR", "EUR"],
"col2cur": [None, "EUR", None],
"col3cur": ["EUR", None, None],
"col4cur": ["EUR", "GBP", None],
},
dtype="string",
)
pandas_df["target"] = pandas_df.apply(
lambda x: pd.Series(x.dropna().unique()).to_list(), axis=1
)

Answer 1

Since polars isn't really good at row operations, I'd start off with a melt.
df.drop('target').with_row_count('i').join(
df.drop('target').with_row_count('i').melt('i').filter(~pl.col('value').is_null()) \
.groupby('i').agg(pl.col('value').unique()),
on='i'
).sort('i').drop('i')
We just do a with_row_count to create an index to maintain the identity of the original rows, then filter out the nulls, then groupby what was previously each row, aggregate to unique, and lastly wrap it in a join with the original columns by the row index.
shape: (3, 5)
┌─────────┬─────────┬─────────┬─────────┬────────────────┐
│ col1cur ┆ col2cur ┆ col3cur ┆ col4cur ┆ value │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ str ┆ str ┆ str ┆ list[str] │
╞═════════╪═════════╪═════════╪═════════╪════════════════╡
│ EUR ┆ null ┆ EUR ┆ EUR ┆ ["EUR"] │
│ EUR ┆ EUR ┆ null ┆ GBP ┆ ["GBP", "EUR"] │
│ EUR ┆ null ┆ null ┆ null ┆ ["EUR"] │
└─────────┴─────────┴─────────┴─────────┴────────────────┘

Answer 2

Here's the closest I could get:
In [39]: df.with_columns(pl.concat_list(pl.col('*')).arr.unique().alias('target'))
Out[39]:
shape: (3, 5)
┌─────────┬─────────┬─────────┬─────────┬──────────────────────┐
│ col1cur ┆ col2cur ┆ col3cur ┆ col4cur ┆ target │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ str ┆ str ┆ str ┆ list[str] │
╞═════════╪═════════╪═════════╪═════════╪══════════════════════╡
│ EUR ┆ null ┆ EUR ┆ EUR ┆ [null, "EUR"] │
│ EUR ┆ EUR ┆ null ┆ GBP ┆ ["EUR", null, "GBP"] │
│ EUR ┆ null ┆ null ┆ null ┆ [null, "EUR"] │
└─────────┴─────────┴─────────┴─────────┴──────────────────────┘
I'll update the answer if/when I find a way to exclude nulls
Slower solution, but which excludes nulls:
In [44]: df.with_columns(pl.concat_list(pl.col('*')).apply(lambda x: list(set(i for i in x if i is not None))).alias('target'))
Out[44]:
shape: (3, 5)
┌─────────┬─────────┬─────────┬─────────┬────────────────┐
│ col1cur ┆ col2cur ┆ col3cur ┆ col4cur ┆ target │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ str ┆ str ┆ str ┆ list[str] │
╞═════════╪═════════╪═════════╪═════════╪════════════════╡
│ EUR ┆ null ┆ EUR ┆ EUR ┆ ["EUR"] │
│ EUR ┆ EUR ┆ null ┆ GBP ┆ ["EUR", "GBP"] │
│ EUR ┆ null ┆ null ┆ null ┆ ["EUR"] │
└─────────┴─────────┴─────────┴─────────┴────────────────┘

Answer 3

I would propose the concat_list with arr_eval.
df.drop('target').with_columns(
pl.concat_list('*').arr.eval(pl.element().unique().drop_nulls(), parallel=True).alias('target'))
This is similar to what #jcurious proposed in his comment.
Here is the result:
shape: (3, 5)
┌─────────┬─────────┬─────────┬─────────┬────────────────┐
│ col1cur ┆ col2cur ┆ col3cur ┆ col4cur ┆ target │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ str ┆ str ┆ str ┆ list[str] │
╞═════════╪═════════╪═════════╪═════════╪════════════════╡
│ EUR ┆ null ┆ EUR ┆ EUR ┆ ["EUR"] │
│ EUR ┆ EUR ┆ null ┆ GBP ┆ ["EUR", "GBP"] │
│ EUR ┆ null ┆ null ┆ null ┆ ["EUR"] │
└─────────┴─────────┴─────────┴─────────┴────────────────┘
Edit: added parallel=True to arr.eval, to run the evaluation in parallel. (suggestion of #jqurious)