Unclosed character in matching string for splitting operation in pyspark - pyspark

Question

I have the following information in a pyspark data frame column:
[["A"],["B"]]
and
["A","B"]
I would like to split the column where the values appear as per the first instance, and leave the values in the second instance intact.
However, upon trying to do this via the split operation:
df = df.selectExpr("split(col, '],[') col")
I recieve the following error:
'Unclosed character class near index...'
I have also tried to replace the actual characters with their ascii equivalent:
df = df.selectExpr("split(col, '\x5D\2C\x5B') col")
But it resulted in the same error as above.
Any suggestions are welcome. Tnx.

Answer 1

there is obviously something very strange when you use the char [. If you remove this char, there is no error anymore.
But you can also use the function version of split and it solves the issue :
from pyspark.sql import functions as F
df.select(F.split(F.col("col"), '],\[')).show()
+----------------+
|split(col, ],\[)|
+----------------+
| [[["A", "B"]]]|
| [["A","B"]]|
+----------------+
If we do not consider your actual problem with split but the real use case you're facing, from_json is probably a better idea :
schema_1 = T.ArrayType(T.StringType())
schema_2 = T.ArrayType(T.ArrayType(T.StringType()))
df2 = df.select(
"col",
F.from_json("col", schema_1),
F.from_json("col", schema_2)
)
df2.show()
+-------------+------------------+------------------+
| col|jsontostructs(col)|jsontostructs(col)|
+-------------+------------------+------------------+
|[["A"],["B"]]| [["A"], ["B"]]| [[A], [B]]|
| ["A","B"]| [A, B]| null|
+-------------+------------------+------------------+
df2.printSchema()
root
|-- col: string (nullable = true)
|-- jsontostructs(col): array (nullable = true)
| |-- element: string (containsNull = true)
|-- jsontostructs(col): array (nullable = true)
| |-- element: array (containsNull = true)
| | |-- element: string (containsNull = true)