Lateral view explode strange behaviour - pyspark

Question

I am concatenating two array columns and convert them back to array. Now when I apply explode, nothing is happening. Using Spark 2.3. Is there anything weird here?
df = spark.createDataFrame([(1,25,['A','B','B','C'],['A','B','B','C']),(1,20,['A','A','B','C'],['A','B','B','C']),(1,20,['A','C','B','C'],['A','B','B','C']),(2,26,['X','Y','Z','C'],['A','B','B','C'])],['id','age','one','two'])
+---+---+------------+------------+
| id|age| one| two|
+---+---+------------+------------+
| 1| 25|[A, B, B, C]|[A, B, B, C]|
| 1| 20|[A, A, B, C]|[A, B, B, C]|
| 1| 20|[A, C, B, C]|[A, B, B, C]|
| 2| 26|[X, Y, Z, C]|[A, B, B, C]|
+---+---+------------+------------+
>>> df.createOrReplaceTempView('df')
>>> df2 = spark.sql('''select id,age, array(concat_ws(',', one, two)) as three from df''')
>>> df2.show()
+---+---+-----------------+
| id|age| three|
+---+---+-----------------+
| 1| 25|[A,B,B,C,A,B,B,C]|
| 1| 20|[A,A,B,C,A,B,B,C]|
| 1| 20|[A,C,B,C,A,B,B,C]|
| 2| 26|[X,Y,Z,C,A,B,B,C]|
+---+---+-----------------+
>>> df2.createOrReplaceTempView('df2')
>>> spark.sql('''select id, age, four from df2 lateral view explode(three) tbl as four''').show() //not exploding
+---+---+---------------+
| id|age| four|
+---+---+---------------+
| 1| 25|A,B,B,C,A,B,B,C|
| 1| 20|A,A,B,C,A,B,B,C|
| 1| 20|A,C,B,C,A,B,B,C|
| 2| 26|X,Y,Z,C,A,B,B,C|
+---+---+---------------+
Please note that I can make it work by
>>> df2 = spark.sql('''select id,age, split(concat_ws(',', one, two),',') as three from df''')
But just wondering why the first approach is not working.

Answer 1

concat_ws creates a single string column and not an array:
df.select(F.size(df.one)).show()
df2.select(F.size(df2.three)).show()
Output:
+---------+
|size(one)|
+---------+
| 4|
| 4|
| 4|
| 4|
+---------+
+-----------+
|size(three)|
+-----------+
| 1|
| 1|
| 1|
| 1|
+-----------+
That means your array has just one element:
df2.select(df2.three.getItem(0)).show()
df2.select(df2.three.getItem(1)).show()
df2.printSchema()
Output:
+---------------+
| three[0]|
+---------------+
|A,B,B,C,A,B,B,C|
|A,A,B,C,A,B,B,C|
|A,C,B,C,A,B,B,C|
|X,Y,Z,C,A,B,B,C|
+---------------+
+--------+
|three[1]|
+--------+
| null|
| null|
| null|
| null|
+--------+
root
|-- id: long (nullable = true)
|-- age: long (nullable = true)
|-- three: array (nullable = false)
| |-- element: string (containsNull = false)
So what you actually should use, is concat on spark >= 2.4:
df3 = spark.sql('''select id,age, concat(one, two) as three from df''')
df3.show(truncate=False)
df3.printSchema()
df3.select(df3.three.getItem(0)).show()
df3.select(df3.three.getItem(1)).show()
Output:
+---+---+------------------------+
|id |age|three |
+---+---+------------------------+
|1 |25 |[A, B, B, C, A, B, B, C]|
|1 |20 |[A, A, B, C, A, B, B, C]|
|1 |20 |[A, C, B, C, A, B, B, C]|
|2 |26 |[X, Y, Z, C, A, B, B, C]|
+---+---+------------------------+
root
|-- id: long (nullable = true)
|-- age: long (nullable = true)
|-- three: array (nullable = true)
| |-- element: string (containsNull = true)
+--------+
|three[0]|
+--------+
| A|
| A|
| A|
| X|
+--------+
+--------+
|three[1]|
+--------+
| B|
| A|
| C|
| Y|
+--------+
Concating two arrays with spark < 2.4 requires an udf (check this answer for an example).

Answer 2

Example way to do this using a UDF:
arraycat = F.udf(lambda x,y : x + y, ArrayType(StringType()))
df = df.withColumn("combined", arraycat("one", "two"))
df = df.withColumn("combined", F.explode("combined"))