How Can I find all the Null fields in Dataframe - pyspark

Question

I am trying to find out the null fields present in a dataframe and concatenate all the fields in a new field in the same dataframe.
Input dataframe looks like this
name
state
number
James
CA
100
Julia
Null
Null
Null
CA
200
Expected Output
name
state
number
Null Fields
James
CA
100
Julia
Null
Null
state,number
Null
CA
200
name
My code looks like this but it is failing. Please help me here.
from pyspark.sql import functions as F
from pyspark.sql.types import StructType,StructField, StringType, IntegerType
data2 = [("James","CA","100"),
("Julia",None,None),
(None,"CA","200")]
schema = StructType([ \
StructField("name",StringType(),True), \
StructField("state",StringType(),True), \
StructField("number",StringType(),True)
])
df = spark.createDataFrame(data=data2,schema=schema)
cols = ["name","state","number"]
df.show()
def null_constraint_check(df,cols):
df_null_identifier = df.withColumn("NULL Fields",\
[F.count(F.when(F.col(c).isNull(), c)) for c in cols])
return df_null_identifier
df1 = null_constraint_check(df,cols)
Error I am getting
AssertionError: col should be Column

Answer 1

Your approach is correct, you only have to make a small change in null_constraint_check:
[F.count(...)] is a list of columns and withColumn expects a single column as second parameter. One way to get there is to concatenate all elements of the list using concat_ws:
def null_constraint_check(df,cols):
df_null_identifier = df.withColumn("NULL Fields",
F.concat_ws(",",*[F.when(F.col(c).isNull(), c) for c in cols]))
return df_null_identifier
I have also removed the F.count because your question says that you want the names of the null columns.
The result is:
+-----+-----+------+------------+
| name|state|number| NULL Fields|
+-----+-----+------+------------+
|James| CA| 100| |
|Julia| null| null|state,number|
| null| CA| 200| name|
+-----+-----+------+------------+