Stream stream joins without equality predicate is not supported - scala

Question

I am using Spark 2.3 and am trying to join two streams of data. My left and my right stream both have an array. I want to join the two streams only when the right stream array is a subset of the left stream array.
For instance, my streamA looks like this:
StreamA:
|---|------|---------------------|-----------|
|id | dept | employeesInMeetings | DateTime |
|---|------|---------------------|-----------|
| 1 | sales| [John] | 7/2 14:00 |
| 2 | mktg | [Adam, Mike] | 7/2 12:30 |
| 3 | hr | [Rick, Jill, Andy] | 7/2 14:00 |
|---|------|---------------------|-----------|
and my streamB looks as follows:
StreamB:
|--------------|--------------|----------|
|employees | confRooms | DateTime |
|--------------|--------------|----------|
| [John, Jane] | A | 7/2 14:00|
| [Adam, Mike] | C | 7/2 12:30|
| [Jill, Andy] | B | 7/2 14:00|
|--------------|--------------|----------|
I only care about employees from the same department that are in the same meeting. Hence, as a result of the intersection, my resulting stream needs to look like:
|---|------|---------------------|-----------|----------|
|id | dept | employeesInMeetings | DateTime | confRoom |
|---|------|---------------------|-----------|----------|
| 2 | mktg | [Adam, Mike] | 7/2 12:30 | C |
| 3 | hr | [Rick, Jill, Andy] | 7/2 14:00 | B |
|---|------|---------------------|-----------|----------|
I created a UDF to do the intersect:
val arrayIntersect = udf((leftArr: Array[String], rightArr: Array[String]) => {
import spark.implicits._
if(leftArr.intersect(rightArr.toSeq).length == rightArr.size){
true
} else {
false
}
})
And tried to use it as follows:
streamA.joinWith(streamB, expr("arrayIntersect(leftArr, rightArr) AND streamA.DateTime BETWEEN streamB.DateTime and streamB.DateTime + INTERVAL 12 hours"))
However, I get the error:
org.apache.spark.sql.AnalysisException: Stream stream joins without equality predicate is not supported;
Does anybody know if there is a workaround here? Any help will be appreciated! Thanks!

Answer 1

Unfortunately, there is no workaround for this in stream-stream joins :(
We really need an equality predicate because we use that to perform the join using a streaming symmetric hash join algorithm -- both the streams are partitioned using the common key so that the related records from both streams end up in the same partition.

Answer 2

First convert your array into string and then search you right Array String in left Array String.
val arrayToString = udf{arr: Seq[String] => arr.sorted.map(_.trim.toLowerCase).mkString(",")}
streamA.withColumn("leftArrStr", arrayToString(col("leftArr"))).joinWith(
streamB.withColumn("rightArrStr", arrayToString(col("rightArr")))
, expr("instr(leftArrStr, rightArrStr) != 0 " +
"AND streamA.DateTime BETWEEN streamB.DateTime and streamB.DateTime + INTERVAL 12 hours"))