PySpark sql for complex sql with !=NULL & NOT IN - pyspark

Question

I have a spark dataframe like that:
sc = CassandraSparkContext(conf=conf)
sql = SQLContext(sc)
log = sc.cassandraTable("test","log_a")\
.select("m_date","userid","fsa","fsid").toDF()
sql.registerDataFrameAsTable(log, "log")
I can query easily with range in m_date like that:
query_str = ("select * from log where m_date >= %s and m_date < %s" %(1497052766,1498059766))
temp=sql.sql(query_str)
temp.show()
Every thing is ok with that simple query. But I have issue with this more complex query like that:
query_str = "select * from log "\
"where userid != NULL "\
"or fsa not in ("\
"select fsa from log where userid is not null)"
query_str = query_str+ ("and m_date > %s and m_date < %s" %(1497052766,1498059766))
temp=sql.sql(query_str)
And I meet this issue:
Py4JJavaError Traceback (most recent call last)
C:\opt\spark\spark-2.2.1-bin-hadoop2.7\python\lib\pyspark.zip\pyspark\sql\utils.py in deco(*a, **kw)
62 try:
---> 63 return f(*a, **kw)
64 except py4j.protocol.Py4JJavaError as e:
C:\opt\spark\spark-2.2.1-bin-hadoop2.7\python\lib\py4j-0.10.4-src.zip\py4j\protocol.py in get_return_value(answer, gateway_client, target_id, name)
318 "An error occurred while calling {0}{1}{2}.\n".
--> 319 format(target_id, ".", name), value)
320 else:
Py4JJavaError: An error occurred while calling o25.sql.
: org.apache.spark.sql.AnalysisException: Null-aware predicate sub-queries cannot be used in nested conditions: (NOT (userid#1 = null) || ((NOT fsa#2 IN (list#62 []) && (m_date#0L > cast(1497052766 as bigint))) && (m_date#0L < cast(1498059766 as bigint))));;
Project [m_date#0L, userid#1, fsa#2, fsid#3]
+- Filter (NOT (userid#1 = null) || ((NOT fsa#2 IN (list#62 []) && (m_date#0L > cast(1497052766 as bigint))) && (m_date#0L < cast(1498059766 as bigint))))
: +- Project [fsa#2]
: +- Filter isnotnull(userid#1)
: +- SubqueryAlias log
: +- LogicalRDD [m_date#0L, userid#1, fsa#2, fsid#3]
+- SubqueryAlias log
+- LogicalRDD [m_date#0L, userid#1, fsa#2, fsid#3]
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.failAnalysis(CheckAnalysis.scala:39)
at org.apache.spark.sql.catalyst.analysis.Analyzer.failAnalysis(Analyzer.scala:91)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:207)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:78)
at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:127)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:126)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:126)
at scala.collection.immutable.List.foreach(List.scala:381)
at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:126)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:78)
at org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:91)
at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:52)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:67)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:632)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:280)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Unknown Source)
During handling of the above exception, another exception occurred:
AnalysisException Traceback (most recent call last)
E:\FPT\project-spark-streaming\spark-calculate-newuser-daily.py in <module>()
76 "select fsa from log where userid is not null)"
77 query_str=query_str+ ("and m_date > %s and m_date < %s" %(1497052766,1498059766))
---> 78 temp=sql.sql(query_str)
79 pass
C:\opt\spark\spark-2.2.1-bin-hadoop2.7\python\lib\pyspark.zip\pyspark\sql\context.py in sql(self, sqlQuery)
382 [Row(f1=1, f2=u'row1'), Row(f1=2, f2=u'row2'), Row(f1=3, f2=u'row3')]
383 """
--> 384 return self.sparkSession.sql(sqlQuery)
385
386 #since(1.0)
C:\opt\spark\spark-2.2.1-bin-hadoop2.7\python\lib\pyspark.zip\pyspark\sql\session.py in sql(self, sqlQuery)
601 [Row(f1=1, f2=u'row1'), Row(f1=2, f2=u'row2'), Row(f1=3, f2=u'row3')]
602 """
--> 603 return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
604
605 #since(2.0)
C:\opt\spark\spark-2.2.1-bin-hadoop2.7\python\lib\py4j-0.10.4-src.zip\py4j\java_gateway.py in __call__(self, *args)
1131 answer = self.gateway_client.send_command(command)
1132 return_value = get_return_value(
-> 1133 answer, self.gateway_client, self.target_id, self.name)
1134
1135 for temp_arg in temp_args:
C:\opt\spark\spark-2.2.1-bin-hadoop2.7\python\lib\pyspark.zip\pyspark\sql\utils.py in deco(*a, **kw)
67 e.java_exception.getStackTrace()))
68 if s.startswith('org.apache.spark.sql.AnalysisException: '):
---> 69 raise AnalysisException(s.split(': ', 1)[1], stackTrace)
70 if s.startswith('org.apache.spark.sql.catalyst.analysis'):
71 raise AnalysisException(s.split(': ', 1)[1], stackTrace)
AnalysisException: 'Null-aware predicate sub-queries cannot be used in nested conditions: (NOT (userid#1 = null) || ((NOT fsa#2 IN (list#62 []) && (m_date#0L > cast(1497052766 as bigint))) && (m_date#0L < cast(1498059766 as bigint))));;\nProject [m_date#0L, userid#1, fsa#2, fsid#3]\n+- Filter (NOT (userid#1 = null) || ((NOT fsa#2 IN (list#62 []) && (m_date#0L > cast(1497052766 as bigint))) && (m_date#0L < cast(1498059766 as bigint))))\n : +- Project [fsa#2]\n : +- Filter isnotnull(userid#1)\n :
+- SubqueryAlias log\n : +- LogicalRDD [m_date#0L, userid#1, fsa#2, fsid#3]\n +- SubqueryAlias log\n +- LogicalRDD [m_date#0L, userid#1, fsa#2, fsid#3]\n'
17/12/24 20:53:17 WARN SparkEnv: Exception while deleting Spark temp dir: C:\Users\hptphuong\AppData\Local\Temp\spark-c9fd644d-de1a-47c9-9e19-cbd0b01df138\userFiles-412a0e89-c56f-4897-98e7-05cd6114855f
java.io.IOException: Failed to delete: C:\Users\hptphuong\AppData\Local\Temp\spark-c9fd644d-de1a-47c9-9e19-cbd0b01df138\userFiles-412a0e89-c56f-4897-98e7-05cd6114855f
at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:1031)
at org.apache.spark.SparkEnv.stop(SparkEnv.scala:103)
at org.apache.spark.SparkContext$$anonfun$stop$11.apply$mcV$sp(SparkContext.scala:1944)
at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1317)
at org.apache.spark.SparkContext.stop(SparkContext.scala:1943)
at org.apache.spark.SparkContext$$anonfun$2.apply$mcV$sp(SparkContext.scala:581)
at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:216)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1948)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
at scala.util.Try$.apply(Try.scala:192)
at org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178)
at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
17/12/24 20:53:17 ERROR ShutdownHookManager: Exception while deleting Spark temp dir: C:\Users\hptphuong\AppData\Local\Temp\spark-c9fd644d-de1a-47c9-9e19-cbd0b01df138\userFiles-412a0e89-c56f-4897-98e7-05cd6114855f
I try to separate of two condition with or, It's will be ok. But when I combine It appear this issue. I tried to replace "or" by union two table. It works. But it looks ridiculous.
Please show me how to fix it.
Thank a lot
#AKSW: soorr for lack the information about the issue. I updated my issue. Please help me.

Answer 1

One obvious error I see in your code is the != NULL comparison. When you want to check if something is or is not null, you should use IS NULL or IS NOT NULL respectively.
Another issue I see is not grouping the conditions using round brackets, but I'll assume that you know what you're doing with the logic.
I would suggest rewriting the query as follows and see if it works for you:
query_str = '''
SELECT *
FROM log
WHERE (m_date > {0} AND m_date < {1})
AND (userid IS NOT NULL
OR fsa NOT IN (
SELECT fsa FROM log WHERE userid IS NOT NULL
)
)'''.format(1497052766, 1498059766)
temp=sql.sql(query_str)
However, I should add a note (as already made in the comments above) that SQL support in Spark is not complete and if this works or not really depends on the Spark version and if you the columns you're using in the query are nullable columns. If that's the case, you'll have to write separate queries and join them based on your logic.

Answer 2

Not IN (Subquery) have some limitations in Spark 2.0 (SEE THIS).
You can still use EXISTS / NOT EXISTS.
PS: Please specify your spark version in order to help others having the same issue