Scala dataframe: replace spaces with null value using regexp_replace - scala

Question

I am trying to replace white-spaces with a null value using regexp_replace in Scala. However, all variations I have tried do not arrive at the expected output:
+---+-----+
| Id|col_1|
+---+-----+
| 0| null|
| 1| null|
+---+-----+
I had a go at it which looks like this:
import org.apache.spark.sql.functions._
val df = spark.createDataFrame(Seq(
(0, " "),
(1, null),
(2, "hello"))).toDF("Id", "col_1")
val test = df.withColumn("col_1", regexp_replace(df("col_1"), "^\\s*", lit(Null)))
test.filter("col_1 is null").show()

Answer 1

The way you use regexp_replace won't work as the result will simply be a string with the matched substring replaced with another provided substring. You can use regexp_extract instead for a regex equality check in a when/other clause as shown below:
import org.apache.spark.sql.functions._
val df = Seq(
(0, " "),
(1, null),
(2, "hello"),
(3, "")
).toDF("Id", "col_1")
df.withColumn("col_1",
when($"col_1" === regexp_extract($"col_1", "(^\\s*$)", 1), null).
otherwise($"col_1")
).show
// +---+-----+
// | Id|col_1|
// +---+-----+
// | 0| null|
// | 1| null|
// | 2|hello|
// | 3| null|
// +---+-----+