Write AVRO from spark-shell in Spark 2.4 - scala

Question

Spark 2.4.0 on Java 1.8.0_161 (Scala 2.11.12)
Run command: spark-shell --jars=spark-avro_2.11-2.4.0.jar
Currently working on some POC using small avro files, I want to be able to read in a (single) AVRO file, make a change, then write it back out.
Reading is fine:
val myAv = spark.read.format("avro").load("myAvFile.avro")
However, I am getting this error when trying to write back out (even before making any changes):
scala> myAv.write.format("avro").save("./output-av-file.avro")
org.apache.spark.sql.AnalysisException:
Datasource does not support writing empty or nested empty schemas.
Please make sure the data schema has at least one or more column(s).
;
at org.apache.spark.sql.execution.datasources.DataSource$.org$apache$spark$sql$execution$datasources$DataSource$$validateSchema(DataSource.scala:733)
at org.apache.spark.sql.execution.datasources.DataSource.planForWriting(DataSource.scala:523)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:281)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:270)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:228)
... 49 elided
I've tried specifying the schema of the dataframe manually, but to no avail:
.write.option("avroSchema", c_schema.toString).format("avro") ...

Answer 1

Reason is quite obvious schema is coming as empty. see here from code
if (hasEmptySchema(schema)) {
throw new AnalysisException(
s"""
|Datasource does not support writing empty or nested empty schemas.
|Please make sure the data schema has at least one or more column(s).
""".stripMargin)
}