I was looking at the DataFrame API, i can see two different methods doing the same functionality for removing duplicates from a data set.
I can understand dropDuplicates(colNames) will remove duplicates considering only the subset of columns.
Is there any other differences between these two methods?
The main difference is the consideration of the subset of columns which is great!
When using distinct you need a prior .select to select the columns on which you want to apply the duplication and the returned Dataframe contains only these selected columns while dropDuplicates(colNames) will return all the columns of the initial dataframe after removing duplicated rows as per the columns.
Let's assume we have the following spark dataframe
+---+------+---+
| id| name|age|
+---+------+---+
| 1|Andrew| 25|
| 1|Andrew| 25|
| 1|Andrew| 26|
| 2| Maria| 30|
+---+------+---+
distinct() does not accept any arguments which means that you cannot select which columns need to be taken into account when dropping the duplicates. This means that the following command will drop the duplicate records taking into account all the columns of the dataframe:
df.distinct().show()
+---+------+---+
| id| name|age|
+---+------+---+
| 1|Andrew| 26|
| 2| Maria| 30|
| 1|Andrew| 25|
+---+------+---+
Now in case you want to drop the duplicates considering ONLY id and name you'd have to run a select() prior to distinct(). For example,
>>> df.select(['id', 'name']).distinct().show()
+---+------+
| id| name|
+---+------+
| 2| Maria|
| 1|Andrew|
+---+------+
But in case you wanted to drop the duplicates only over a subset of columns like above but keep ALL the columns, then distinct() is not your friend.
dropDuplicates() will drop the duplicates detected over the provided set of columns, but it will also return all the columns appearing in the original dataframe.
df.dropDuplicates().show()
+---+------+---+
| id| name|age|
+---+------+---+
| 1|Andrew| 26|
| 2| Maria| 30|
| 1|Andrew| 25|
+---+------+---+
dropDuplicates() is thus more suitable when you want to drop duplicates over a selected subset of columns, but also want to keep all the columns:
df.dropDuplicates(['id', 'name']).show()
+---+------+---+
| id| name|age|
+---+------+---+
| 2| Maria| 30|
| 1|Andrew| 25|
+---+------+---+
For more details refer to the article distinct() vs dropDuplicates() in Python
From javadoc, there is no difference between distinc() and dropDuplicates().
dropDuplicates
public DataFrame dropDuplicates()
Returns a new DataFrame that contains only the unique rows from this
DataFrame. This is an alias for distinct.
dropDuplicates() was introduced in 1.4 as a replacement for distinct(), as you can use it's overloaded methods to get unique rows based on subset of columns.
Related
I have data like this
I want to unpivot the dataframe by calling the columns dynamically without hardcoding.
How do I achieve this?
Create an array of struct column that combines the columns and column values using list comprehension. Explode the struct column using inline.
df =spark.createDataFrame([
('78','20','19','90'),
('78','20','19','&')
],
('Machines', 'Books', 'Vehicles', 'Plants'))
df.show()
df.withColumn('tab', F.array(*[F.struct(lit(x).alias('Fields'), col(x).alias('Count')).alias(x) for x in df.columns])).selectExpr('inline(tab)').show()
+---+------+------+
| Id| Date|Amount|
+---+------+------+
| 1|202201| 50|
| 1|202202| 150|
| 1|202203| 100|
| 2|202201| 10|
| 2|202202| |
| 2|202203| 50|
| 3|202201| 20|
| 3|202202| 10|
| 3|202203| |
+---+------+------+
In PySpark, there's the concept of coalesce(colA, colB, ...) which will, per row, take the first non-null value it encounters from those columns. However, I want coalesce(rowA, rowB, ...) i.e. the ability to, per column, take the first non-null value it encounters from those rows. I want to coalesce all rows within a group or window of rows.
For example, given the following dataset, I want to coalesce rows per category and ordered ascending by date.
+---------+-----------+------+------+
| category| date| val1| val2|
+---------+-----------+------+------+
| A| 2020-05-01| null| 1|
| A| 2020-05-02| 2| null|
| A| 2020-05-03| 3| null|
| B| 2020-05-01| null| null|
| B| 2020-05-02| 4| null|
| C| 2020-05-01| 5| 2|
| C| 2020-05-02| null| 3|
| D| 2020-05-01| null| 4|
+---------+-----------+------+------+
What I should get as the output is...
+---------+-----------+------+------+
| category| date| val1| val2|
+---------+-----------+------+------+
| A| 2020-05-01| 2| 1|
| B| 2020-05-01| 4| null|
| C| 2020-05-01| 5| 2|
| D| 2020-05-01| null| 4|
+---------+-----------+------+------+
First, I'll give the answer. Then, I'll point out the important bits.
from pyspark.sql import Window
from pyspark.sql.functions import col, dense_rank, first
df = ... # dataframe from question description
window = (
Window
.partitionBy("category")
.orderBy(col("date").asc())
)
window_unbounded = (
window
.rangeBetween(Window.unboundedPreceding, Window.unboundedFollowing)
)
cols_to_merge = [col for col in df.columns if col not in ["category", "date"]]
merged_cols = [first(col, True).over(window_unbounded).alias(col) for col in cols_to_merge]
df_merged = (
df
.select([col("category"), col("date")] + merged_cols)
.withColumn("rank_col", dense_rank().over(window))
.filter(col("rank_col") == 1)
.drop("rank_col")
)
The row-wise analogue to coalesce is the aggregation function first. Specifically, we use first with ignorenulls = True so that we find the first non-null value.
When we use first, we have to be careful about the ordering of the rows it's applied to. Because groupBy doesn't allow us to maintain order within the groups, we use a Window.
The window itself must be unbounded on both ends rather than the default unbounded preceding to current row, else we'll end up with the first aggregation potentially running on subsets of our groups.
After we aggregate over the window, we alias the column back to its original name to keep the column names consistent.
We use a single select statement of cols rather than a for loop with df.withColumn(col, ...) because the select statement greatly reduces the query plan depth. Should you use the looped withColumn, you might hit a stack overflow error if you have too many columns.
Finally, we run a dense_rank over our window --- this time using the window with the default range --- and filter to only the first ranked rows. We use dense rank here, but we could use any ranking function, whatever fits our needs.
So i have a dataframe as shown below, that has been stored as a temporary view by the name mean_value_gn5 so that i can query using sql(), whenever i need to fetch the data.
+-------+----+
|Species|Avgs|
+-------+----+
| NO2| 43|
| NOX| 90|
| NO| 31|
+-------+----+
This dataframe stores the categorical average of the 'Species' rounded off to the nearest whole number using the ceil() function. I need to use these categorical averages to impute the missing values of column Value in my dataframe of interest clean_gn5. I created a new column Value_imp which would hold my final column with the imputed values.
I made an attempt to do so as follows:
clean_gn5 = clean_gn5.withColumn("Value_imp",
when($"Value".isNull, sql("Select Avgs from mean_value_gn5 where Species = "+$"Species").select("Avgs").head().getLong(0).toInt)
.otherwise($"Value"))
The above mentioned code runs, but the values are getting incorrectly imputed i.e. for the row containing Species as NO the value getting imputed is 43 instead of 31.
Prior to doing this I first checked if i was able to fetch the values correctly by executing the following:
println(sql("Select Avgs from mean_value_gn5 where Species = 'NO'").select("Avgs").head().getLong(0))
I am able to fetch the value correctly after hardcoding the Species and as per my understanding $"Species" should help me fetch the value corresponding to the Species column for that particular row.
Also I thought that probably i was missing the single quotes around the hardcoded Species value i.e. 'NO'. So I tried doing the following
clean_gn5 = clean_gn5.withColumn("Value_imp",
when($"Value".isNull, sql("Select Avgs from mean_value_gn5 where Species = '"+$"Species"+"'").select("Avgs").head().getLong(0).toInt)
.otherwise($"Value"))
But that resulted in the following exception.
Exception in thread "main" java.util.NoSuchElementException: next on empty iterator
I am fairly new to Spark and Scala.
Let's assume clean_gn5 contains the data
+-------+-----+
|Species|Value|
+-------+-----+
| NO2| 2.3|
| NOX| 1.1|
| NO| null|
| ABC| 4.0|
| DEF| null|
| NOX| null|
+-------+-----+
Joining clean_gn5 with mean_value_gn5 using a left join will result in
+-------+-----+----+
|Species|Value|Avgs|
+-------+-----+----+
| NO2| 2.3| 43|
| NOX| 1.1| 90|
| NO| null| 31|
| ABC| 4.0|null|
| DEF| null|null|
| NOX| null| 90|
+-------+-----+----+
On this dataframe you can apply per row the logic you have already given in your question and the result is (after dropping the Avgs column):
+-------+-----+---------+
|Species|Value|Value_imp|
+-------+-----+---------+
| NO2| 2.3| 2.3|
| NOX| 1.1| 1.1|
| NO| null| 31.0|
| ABC| 4.0| 4.0|
| DEF| null| null|
| NOX| null| 90.0|
+-------+-----+---------+
The code:
clean_gn5.join(mean_value_gn5, Seq("Species"), "left")
.withColumn("Value_imp", when('value.isNull, 'Avgs).otherwise('value))
.drop("Avgs")
.show()
df1
+-------+-------+-----+
| ID | Score| hits|
+-------+-------+-----+
| 01| 100| Null|
| 02| Null| 80|
| 03| spark| 1|
| 04| 300| 1|
+-------+-------+-----+
after casting Score to int and hits to float I get the below dataframe:
df2
+-------+-------+-----+
| ID | Score| hits|
+-------+-------+-----+
| 01| 100| Null|
| 02| Null| 80.0|
| 03| Null| 1.0|
| 04| 300| 1.0|
+-------+-------+-----+
Now I want to extract only the bad records , bad records mean that null produced after casting.
I want to do the operations only on existing dataframe. Please help me out if there is any build-in way to get the bad records after casting.
Please also consider this is sample dataframe. The solution should solve for any number of columns and any scenario.
I tried by separating the null records from both dataframes and compare them. Also i have thought of adding another column with number of nulls and then compare the both dataframes if number of nulls is grater in df2 than in df1 then those are bad one. But i think these solutions are pretty old school.
I would like to know the better way to resolve it.
You can use a custom function/udf to convert string to integer and map non integer values to specific number eg. -999999999.
Later you can filter on -999999999 to identify originally non integer records.
def udfInt(value):
if value is None:
return None
elif value.isdigit():
return int(value)
else:
return -999999999
spark.udf.register('udfInt', udfInt)
df.selectExpr("*",
"udfInt(Score) AS new_Score").show()
#+---+-----+----+----------+
#| ID|Score|hits| new_Score|
#+---+-----+----+----------+
#| 01| 100|null| 100|
#| 02| null| 80| null|
#| 03|spark| 1|-999999999|
#| 04| 300| 1| 300|
#+---+-----+----+----------+
Filter on -999999999 to identify non integer (bad records)
df.selectExpr("*","udfInt(Score) AS new_Score").filter("new_score == -999999999").show()
#+---+-----+----+----------+
#| ID|Score|hits| new_Score|
#+---+-----+----+----------+
#| 03|spark| 1|-999999999|
#+---+-----+----+----------+
The same way you can have customized udf for float conversion.
I'm quite new both Spark and Scale and could really need a hint to solve my problem. So I have two DataFrames A (columns id and name) and B (columns id and text) would like to join them, group by id and combine all rows of text into a single String:
A
+--------+--------+
| id| name|
+--------+--------+
| 0| A|
| 1| B|
+--------+--------+
B
+--------+ -------+
| id| text|
+--------+--------+
| 0| one|
| 0| two|
| 1| three|
| 1| four|
+--------+--------+
desired result:
+--------+--------+----------+
| id| name| texts|
+--------+--------+----------+
| 0| A| one two|
| 1| B|three four|
+--------+--------+----------+
So far I'm trying the following:
var C = A.join(B, "id")
var D = C.groupBy("id", "name").agg(collect_list("text") as "texts")
This works quite well besides that my texts column is an Array of Strings instead of a String. I would appreciate some help very much.
I am just adding some minor functions in yours to give the right solution, which is
A.join(B, Seq("id"), "left").orderBy("id").groupBy("id", "name").agg(concat_ws(" ", collect_list("text")) as "texts")
It's quite simple:
val bCollected = b.groupBy('id).agg(collect_list('text).as("texts")
val ab = a.join(bCollected, a("id") == bCollected("id"), "left")
First DataFrame is immediate result, b DataFrame that has texts collected for every id. Then you are joining it with a. bCollected should be smaller that b itself, so it will probably get better shuffle time