Spark 2.x inconsistent filter behaviour - scala

Question

I noticed an inconsistent behaviour in Juptyter Notebook's spark scala, although I believe it's rather a Spark native behaviour. I was hoping that someone has seen this behaviour before and knows how the read and filter the data correctly.
In the below example I am reading data from an S3 json source. The filter condition is that the cId field contains at least a value of length 2, so that both null values and the value "0" are filtered out. The values in cId contain both numeric and alphanumeric values, like in the example.
When I apply the filter to the json source (after converting it to StringType) it returns an empty data set. When I create a small dataframe and apply the filter, all rows are displayed correctly.
Once treated, the dataframe is exported to parquet format.
I am trying to correctly display the values from the json source. Here are some of the things I have tried:
Use filter() vs where() function. Even if they're almost aliases I wanted to exclude any method errors
turn of the default conversion behaviour from spark (convertMetastoreParquet)
analyze the output plan (see below)
== Physical Plan ==
Format: JSON, Location: InMemoryFileIndex[s3a://s3pathredacted..., PartitionFilters: [], PushedFilters: [IsNotNull(evaluationType), IsNotNull(requestPayload), EqualTo(evaluationType,inbound_acquiring)], ReadSchema: struct<actionRecommended:string,dateInserted:string,evaluationType:string,requestPayload:struct<b...
: +- BroadcastExchange HashedRelationBroadcastMode(List(input[0, string, true]))
: +- *(1) Project [customerid#116 AS cId#118]
: +- *(1) Filter ((length(customerid#116) > 1) && isnotnull(customerid#116))
: +- *(1) FileScan csv [customerid#116] Batched: false, Format: CSV, Location: InMemoryFileIndex[s3://folder1/folder2/folder3], PartitionFilters: [], PushedFilters: [IsNotNull(customerid)], ReadSchema: struct<customerid:string>
s3MLDataSource = "s3Path"
var dfJson = (spark.read.json(s3MLDataSource)
var dfSeq= Seq("12345","67890","1a2b3c4d").toDF("customerId")
dfJson = dfJson.select(col("customerId").cast(StringType).as("cId"))
dfJson = dfJson.where("length(cId)>1")
dfJson.show(false)
+--------------+
|cId |
+--------------+
| |
+--------------+
//Session settings
%%configure -f
{"executorMemory": "4G","driverMemory": "2G","executorCores": 8,
"conf": { "spark.dynamicAllocation.enabled": "true",
"spark.dynamicAllocation.shuffleTracking.enabled": "true",
"spark.shuffle.service.enabled": "true",
"spark.dynamicAllocation.minExecutors": "2",
"spark.sql.hive.convertMetastoreParquet": "false",
"spark.sql.hive.metastorePartitionPruning": "false",
"spark.dynamicAllocation.maxExecutors": "20",
"spark.sql.autoBroadcastJoinThreshold": "26214400",
"spark.sql.cbo.enabled": "true",
"spark.ui.showConsoleProgress": "false",
"spark.serializer": "org.apache.spark.serializer.KryoSerializer"}}