I have a dataset similar to this one
exp
pid
mat
pskey
order
1
CR
P
1-CR-P
1
1
M
C
1-M-C
2
1
CR
C
1-CR-C
3
1
PP
C
1-PP-C
4
2
CR
P
2-CR-P
1
2
CR
P
2-CR-P
1
2
M
C
2-M-C
2
2
CR
C
2-CR-C
3
2
CR
C
2-CR-C
3
2
CR
C
2-CR-C
3
2
CR
C
2-CR-C
3
2
CR
C
2-CR-C
3
2
PP
C
2-PP-C
4
2
PP
C
2-PP-C
4
2
PP
C
2-PP-C
4
2
PP
C
2-PP-C
4
2
PP
C
2-PP-C
4
3
M
C
3-M-C
2
4
CR
P
4-CR-P
1
4
M
C
4-M-C
2
4
CR
C
4-CR-C
3
4
PP
C
4-PP-C
4
What I need is to get pskey of the predecessor for the same exp giving the following relation:
order 1 -> no predecessor
order 2 -> no predecessor
order 3 -> [1,2]
order 4 -> [3]
And add those values to a new column called predecessor
The expected result would be like:
+---+---+---+------+-----+----------------------------------------+
|exp|pid|mat|pskey |order|predecessor |
+---+---+---+------+-----+----------------------------------------+
|1 |CR |P |1-CR-P|1 |null |
|1 |M |C |1-M-C |2 |null |
|1 |CR |C |1-CR-C|3 |[1-CR-P, 1-M-C ] |
|1 |PP |C |1-PP-C|4 |[1-CR-C] |
|3 |M |C |3-M-C |2 |null |
|2 |CR |P |2-CR-P|1 |null |
|2 |CR |P |2-CR-P|1 |null |
|2 |M |C |2-M-C |2 |null |
|2 |CR |C |2-CR-C|3 |[2-CR-P, 2-M-C] |
|2 |CR |C |2-CR-C|3 |[2-CR-P, 2-M-C] |
|2 |CR |C |2-CR-C|3 |[2-CR-P, 2-M-C] |
|2 |CR |C |2-CR-C|3 |[2-CR-P, 2-M-C] |
|2 |CR |C |2-CR-C|3 |[2-CR-P, 2-M-C] |
|2 |PP |C |2-PP-C|4 |[2-CR-C] |
|2 |PP |C |2-PP-C|4 |[2-CR-C] |
|2 |PP |C |2-PP-C|4 |[2-CR-C] |
|2 |PP |C |2-PP-C|4 |[2-CR-C] |
|2 |PP |C |2-PP-C|4 |[2-CR-C] |
|4 |CR |P |4-CR-P|1 |null |
|4 |M |C |4-M-C |2 |null |
|4 |CR |C |4-CR-C|3 |[4-CR-P, 4-M-C] |
|4 |PP |C |4-PP-C|4 |[4-CR-C] |
+---+---+---+------+-----+----------------------------------------+
I am quite new to pyspark so I have no clue how to manage it.
Differents cases on order are handled with when. You aggregate the values with a collect_set to get unic identifiers:
from pyspark.sql import functions as F, Window
df2 = df.withColumn(
"predecessor",
F.when(
F.col("order") == 3,
F.collect_set(F.col("pskey")).over(
Window.partitionBy("exp").orderBy("order").rangeBetween(-2, -1)
),
).when(
F.col("order") == 4,
F.collect_set(F.col("pskey")).over(
Window.partitionBy("exp").orderBy("order").rangeBetween(-1, -1)
),
),
)
the result :
df2.show(truncate=False)
+---+---+---+------+-----+----------------+
|exp|pid|mat|pskey |order|predecessor |
+---+---+---+------+-----+----------------+
|1 |CR |P |1-CR-P|1 |null |
|1 |M |C |1-M-C |2 |null |
|1 |CR |C |1-CR-C|3 |[1-CR-P, 1-M-C ]|
|1 |PP |C |1-PP-C|4 |[1-CR-C] |
|3 |M |C |3-M-C |2 |null |
|2 |CR |P |2-CR-P|1 |null |
|2 |CR |P |2-CR-P|1 |null |
|2 |M |C |2-M-C |2 |null |
|2 |CR |C |2-CR-C|3 |[2-CR-P, 2-M-C ]|
|2 |CR |C |2-CR-C|3 |[2-CR-P, 2-M-C ]|
|2 |CR |C |2-CR-C|3 |[2-CR-P, 2-M-C ]|
|2 |CR |C |2-CR-C|3 |[2-CR-P, 2-M-C ]|
|2 |CR |C |2-CR-C|3 |[2-CR-P, 2-M-C ]|
|2 |PP |C |2-PP-C|4 |[2-CR-C] |
|2 |PP |C |2-PP-C|4 |[2-CR-C] |
|2 |PP |C |2-PP-C|4 |[2-CR-C] |
|2 |PP |C |2-PP-C|4 |[2-CR-C] |
|2 |PP |C |2-PP-C|4 |[2-CR-C] |
|4 |CR |P |4-CR-P|1 |null |
|4 |M |C |4-M-C |2 |null |
+---+---+---+------+-----+----------------+
only showing top 20 rows
Related
I have a dataset like:
a
c
c
d
b
a
a
d
d
c
c
b
a
b
I want to add a column that looks like the one below. When 'c' is reached, the new column will be zero and then be increased by one. Is there a way we can do this using pyspark?
a 1
c 0
c 0
d 2
b 2
a 2
a 2
d 2
d 2
c 0
c 0
b 3
a 3
b 3
I have tried the below code but it is not working.
from pyspark.sql.functions import col, when, lag, sum
s = df.filter(col("col") == 'c')
df = df.withColumn("new", when(s.neq(lag("s", 1).over()), sum("s").over(Window.orderBy("index"))).otherwise(0))
The following solution uses PySpark SQL functions to implement the logic requested above.
Set-Up
Create a DataFrame to mimic the example provided
df = spark.createDataFrame(
[('a',),
('c',),
('c',),
('d',),
('b',),
('a',),
('a',),
('d',),
('d',),
('c',),
('c',),
('b',),
('a',),
('b',),],
['id',])
Output
+---+
|id |
+---+
|a |
|c |
|c |
|d |
|b |
|a |
|a |
|d |
|d |
|c |
|c |
|b |
|a |
|b |
+---+
Logic
Calculate row number (reference logic for row_num here)
df = df.withColumn("row_num", F.row_number().over(Window.orderBy(F.monotonically_increasing_id())))
Use row number to determine the preceding id value (the lag). There is no preceding id for the first row so the lag results in a null - set this missing value to "c".
df = df.withColumn("lag_id", F.lag("id",1).over(Window.orderBy("row_num")))
df = df.na.fill(value="c", subset=['lag_id'])
output
+---+--------------+------+
|id | row_num |lag_id|
+---+--------------+------+
|a |1 |c |
|c |2 |a |
|c |3 |c |
|d |4 |c |
|b |5 |d |
|a |6 |b |
|a |7 |a |
|d |8 |a |
|d |9 |d |
|c |10 |d |
|c |11 |c |
|b |12 |c |
|a |13 |b |
|b |14 |a |
+---+--------------+------+
Determine order (sequence) for rows that immediately follow a row where id = "c"
df_sequence = df.filter((df.id != "c") & (df.lag_id == "c"))
df_sequence = df_sequence.withColumn("sequence", F.row_number().over(Window.orderBy("row_num")))
output
+---+--------------+------+--------+
|id | row_num |lag_id|sequence|
+---+--------------+------+--------+
|a |1 |c |1 |
|d |4 |c |2 |
|b |12 |c |3 |
+---+--------------+------+--------+
Join the sequence DF to the original DF
df_joined = df.alias("df1").join(df_sequence.alias("df2"),
on="row_num",
how="leftouter")\
.select(df["*"],df_sequence["sequence"])
)
Set sequence to 0 when id = "c"
df_joined = df_joined.withColumn('sequence', F.when(df_joined.id == "c", 0)
.otherwise(df_joined.sequence)
output
+---+--------------+------+--------+
|id | row_num |lag_id|sequence|
+---+--------------+------+--------+
|a |1 |c |1 |
|c |2 |a |0 |
|c |3 |c |0 |
|d |4 |c |2 |
|b |5 |d |null |
|a |6 |b |null |
|a |7 |a |null |
|d |8 |a |null |
|d |9 |d |null |
|c |10 |d |0 |
|c |11 |c |0 |
|b |12 |c |3 |
|a |13 |b |null |
|b |14 |a |null |
+---+--------------+------+--------+
Forward fill sequence values (reference the forward fill logic here)
df_final = df_joined.withColumn('sequence', F.last('sequence', ignorenulls=True).over(Window.orderBy("row_num")
Final Output
+---+--------------+------+--------+
|id | row_num |lag_id|sequence|
+---+--------------+------+--------+
|a |1 |c |1 |
|c |2 |a |0 |
|c |3 |c |0 |
|d |4 |c |2 |
|b |5 |d |2 |
|a |6 |b |2 |
|a |7 |a |2 |
|d |8 |a |2 |
|d |9 |d |2 |
|c |10 |d |0 |
|c |11 |c |0 |
|b |12 |c |3 |
|a |13 |b |3 |
|b |14 |a |3 |
+---+--------------+------+--------+
I have two dataframes spark(scala):
First:
+-------------------+------------------+-----------------+----------+-----------------+
|id |zone |zone_father |father_id |country |
+-------------------+------------------+-----------------+----------+-----------------+
|2 |1 |123 |1 |0 |
|2 |2 |123 |1 |0 |
|3 |3 |1 |2 |0 |
|2 |4 |123 |1 |0 |
|3 |5 |2 |2 |0 |
|3 |6 |4 |2 |0 |
|3 |7 |19 |2 |0 |
+-------------------+------------------+-----------------+----------+-----------------+
Second:
+-------------------+------------------+-----------------+-----------------+
|country |id |zone |zone_value |
+-------------------+------------------+-----------------+-----------------+
|0 |2 |1 |7 |
|0 |2 |2 |7 |
|0 |2 |4 |8 |
|0 |0 |0 |2 |
+-------------------+------------------+-----------------+-----------------+
Then I need following logic:
1 -> If => first.id = second.id && first.zone = second.zone
2 -> Else if => first.father_id = second.id && first.zone_father = second.zone
3 -> If neither the first nor the second is true, follow the latter => first.country = second.zone
And the expected result would be:
+-------------------+------------------+-----------------+----------+-----------------+-----------------+
|id |zone |zone_father |father_id |country |zone_value |
+-------------------+------------------+-----------------+----------+-----------------+-----------------+
|2 |1 |123 |1 |0 |7 |
|2 |2 |123 |1 |0 |7 |
|3 |3 |1 |2 |0 |7 |
|2 |4 |123 |1 |0 |8 |
|3 |5 |2 |2 |0 |7 |
|3 |6 |4 |2 |0 |8 |
|3 |7 |19 |2 |0 |2 |
+-------------------+------------------+-----------------+----------+-----------------+-----------------+
I tried to join both dataframes, but due "or" operation, two results for each row is returned, because the last premise returns true regardless of the result of the other two.
I have a dataframe with 2 columns "Id" and "category". For each category, I want to label encode the column "Id", so the expected outcome will be the column "Enc_id" like this
Id Category Enc_id
a1 A 0
a2 A 1
b1 B 0
c1 C 0
c2 C 1
a3 A 2
b2 B 1
b3 B 2
b4 B 3
b4 B 3
b3 B 2
Here, the Id may not be unique, so that there may be duplicated rows. I thought of creating a window to partitionBy(category), then apply the label encoding (StringIndexer) over this window but it didn't work. Any hint, please?
You can use the window function with substring function with and calculate the rank
val window = Window.partitionBy($"Category", substring($"Id", 1,1)).orderBy("Id")
df.withColumn("Enc_id", rank().over(window) - 1) // -1 to start the rank from 0
.show(false)
Output:
+---+--------+------+
|Id |Category|Enc_id|
+---+--------+------+
|a1 |A |0 |
|a2 |A |1 |
|a3 |A |2 |
|c1 |C |0 |
|c2 |C |1 |
|b1 |B |0 |
|b2 |B |1 |
|b3 |B |2 |
|b4 |B |3 |
+---+--------+------+
Update1:
for the updated case with duplicate id
df1.groupBy("Id", "Category")
.agg(collect_list("Category") as "list_category")
.withColumn("Enc_id", rank().over(window) - 1)
.withColumn("Category", explode($"list_category"))
.drop("list_category")
.show(false)
Output:
+---+--------+------+
|Id |Category|Enc_id|
+---+--------+------+
|a1 |A |0 |
|a2 |A |1 |
|a3 |A |2 |
|c1 |C |0 |
|c2 |C |1 |
|b1 |B |0 |
|b2 |B |1 |
|b3 |B |2 |
|b3 |B |2 |
|b4 |B |3 |
|b4 |B |3 |
+---+--------+------+
I want to merge 2 columns or 2 dataframes like
df1
+--+
|id|
+--+
|1 |
|2 |
|3 |
+--+
df2 --> this one can be a list as well
+--+
|m |
+--+
|A |
|B |
|C |
+--+
I want to have as resulting table
+--+--+
|id|m |
+--+--+
|1 |A |
|1 |B |
|1 |C |
|2 |A |
|2 |B |
|2 |C |
|3 |A |
|3 |B |
|3 |C |
+--+--+
def crossJoin(right: org.apache.spark.sql.Dataset[_]): org.apache.spark.sql.DataFrame
Using crossJoin function you can get same result. Please check code below.
scala> dfa.show
+---+
| id|
+---+
| 1|
| 2|
| 3|
+---+
scala> dfb.show
+---+
| m|
+---+
| A|
| B|
| C|
+---+
scala> dfa.crossJoin(dfb).orderBy($"id".asc).show(false)
+---+---+
|id |m |
+---+---+
|1 |B |
|1 |A |
|1 |C |
|2 |A |
|2 |B |
|2 |C |
|3 |C |
|3 |B |
|3 |A |
+---+---+
In Spark 1.6.0 / Scala, is there an opportunity to get collect_list("colC") or collect_set("colC").over(Window.partitionBy("colA").orderBy("colB")?
Given that you have dataframe as
+----+----+----+
|colA|colB|colC|
+----+----+----+
|1 |1 |23 |
|1 |2 |63 |
|1 |3 |31 |
|2 |1 |32 |
|2 |2 |56 |
+----+----+----+
You can Window functions by doing the following
import org.apache.spark.sql.functions._
import org.apache.spark.sql.expressions._
df.withColumn("colD", collect_list("colC").over(Window.partitionBy("colA").orderBy("colB"))).show(false)
Result:
+----+----+----+------------+
|colA|colB|colC|colD |
+----+----+----+------------+
|1 |1 |23 |[23] |
|1 |2 |63 |[23, 63] |
|1 |3 |31 |[23, 63, 31]|
|2 |1 |32 |[32] |
|2 |2 |56 |[32, 56] |
+----+----+----+------------+
Similar is the result for collect_set as well. But the order of elements in the final set will not be in order as with collect_list
df.withColumn("colD", collect_set("colC").over(Window.partitionBy("colA").orderBy("colB"))).show(false)
+----+----+----+------------+
|colA|colB|colC|colD |
+----+----+----+------------+
|1 |1 |23 |[23] |
|1 |2 |63 |[63, 23] |
|1 |3 |31 |[63, 31, 23]|
|2 |1 |32 |[32] |
|2 |2 |56 |[56, 32] |
+----+----+----+------------+
If you remove orderBy as below
df.withColumn("colD", collect_list("colC").over(Window.partitionBy("colA"))).show(false)
result would be
+----+----+----+------------+
|colA|colB|colC|colD |
+----+----+----+------------+
|1 |1 |23 |[23, 63, 31]|
|1 |2 |63 |[23, 63, 31]|
|1 |3 |31 |[23, 63, 31]|
|2 |1 |32 |[32, 56] |
|2 |2 |56 |[32, 56] |
+----+----+----+------------+
I hope the answer is helpful
Existing answer is valid, just adding here a different style of writting window functions:
import org.apache.spark.sql.expressions.Window
val wind_user = Window.partitionBy("colA", "colA2").orderBy("colB", "colB2".desc)
df.withColumn("colD_distinct", collect_set($"colC") over wind_user)
.withColumn("colD_historical", collect_list($"colC") over wind_user).show(false)