Apache Spark SQL dataframe filter multi-rules by string - scala

Question

I would like touse Spark dataframe to search contents by 'like'
and　we can use 'or' function do like SQL '||' to filter like this.
voc_0201.filter(
col("contents").like("intel").or(col("contents").like("apple"))
).count
But I have to filter a lot of Strings, how could I filter the String list or array to the dataframe?
Thanks

Answer 1

Let's first define our patterns :
val patterns = Seq("foo", "bar")
and create an example DataFrame:
val df = Seq((1, "bar"), (2, "foo"), (3, "xyz")).toDF("id", "contents")
One simple solution is to fold over patterns:
val expr = patterns.foldLeft(lit(false))((acc, x) =>
acc || col("contents").like(x)
)
df.where(expr).show
// +---+--------+
// | id|contents|
// +---+--------+
// | 1| bar|
// | 2| foo|
// +---+--------+
Another one is to build regular expression and use rlike:
val expr = patterns.map(p => s"^$p$$").mkString("|")
df.where(col("contents").rlike(expr)).show
// +---+--------+
// | id|contents|
// +---+--------+
// | 1| bar|
// | 2| foo|
// +---+--------+
PS: the above solution may not work if this is not simple literal.
Finally for simple patterns you can use isin:
df.where(col("contents").isin(patterns: _*)).show
// +---+--------+
// | id|contents|
// +---+--------+
// | 1| bar|
// | 2| foo|
// +---+--------+
It is also possible to join:
val patternsDF = patterns.map(Tuple1(_)).toDF("contents")
df.join(broadcast(patternsDF), Seq("contents")).show
// +---+--------+
// | id|contents|
// +---+--------+
// | 1| bar|
// | 2| foo|
// +---+--------+