ParquetProtoWriters creates an unreadable parquet file - scala

Question

My .proto file contains one field of map type.
Message Foo {
...
...
map<string, uint32> fooMap = 19;
}
I'm consuming messages from Kafka source and trying to write the messages as a parquet file to S3 bucket.
The relevant part of the code looks like this:
val basePath = "s3a:// ..."
env
.fromSource(source, WatermarkStrategy.noWatermarks(), "source")
.map(x => toJavaProto(x))
.sinkTo(
FileSink
.forBulkFormat(basePath, ParquetProtoWriters.forType(classOf(Foo)))
.withOutputFileConfig(
OutputFileConfig
.builder()
.withPartPrefix("foo")
.withPartSuffix(".parquet")
.build()
)
.build()
)
.setParallelism(1)
env.execute()
The result is that a parquet file was actually written for S3, but the file appears to be corrupted. When I try to read the file using the Avro / Parquet Viewer plugin I can see this error:
Unable to process file
.../Downloads/foo-9366c15f-270e-4939-ad88-b77ee27ddc2f-0.parquet
java.lang.UnsupportedOperationException: REPEATED not supported
outside LIST or MAP. Type: repeated group fooMap = 19 { optional
binary key (STRING) = 1; optional int32 value = 2; } at
org.apache.parquet.avro.AvroSchemaConverter.convertFields(AvroSchemaConverter.java:277)
at
org.apache.parquet.avro.AvroSchemaConverter.convert(AvroSchemaConverter.java:264)
at
org.apache.parquet.avro.AvroReadSupport.prepareForRead(AvroReadSupport.java:134)
at
org.apache.parquet.hadoop.InternalParquetRecordReader.initialize(InternalParquetRecordReader.java:185)
at
org.apache.parquet.hadoop.ParquetReader.initReader(ParquetReader.java:156)
at
org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:135)
at
uk.co.hadoopathome.intellij.viewer.fileformat.ParquetFileReader.getRecords(ParquetFileReader.java:99)
at
uk.co.hadoopathome.intellij.viewer.FileViewerToolWindow$2.doInBackground(FileViewerToolWindow.java:193)
at
uk.co.hadoopathome.intellij.viewer.FileViewerToolWindow$2.doInBackground(FileViewerToolWindow.java:184)
at java.desktop/javax.swing.SwingWorker$1.call(SwingWorker.java:304)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.desktop/javax.swing.SwingWorker.run(SwingWorker.java:343) at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Flink version 1.15
proto 2

Answer 1

There are some breaking changes in parquet-format and parquet-mr. I'm not familiar with Flink, but I guess you have to configure org.apache.flink.formats.parquet.protobuf.ParquetProtoWriters to generate a correct format.
I used parquet-mr directly and encountered the same issue. An avro parquet reader is unable to read the parquet file generated by the following code:
import org.apache.parquet.proto.ProtoParquetWriter;
import org.apache.parquet.proto.ProtoWriteSupport;
...
var conf = new Configuration();
ProtoWriteSupport.setWriteSpecsCompliant(conf, false);
var builder = ProtoParquetWriter.builder(file)
.withMessage(Xxx.class)
.withCompressionCodec(CompressionCodecName.GZIP)
.withWriteMode(Mode.OVERWRITE)
.withConf(conf);
try (var writer = builder.build()) {
writer.write(pb.toBuilder());
}
If the config value is changed to true, it will succeed:
ProtoWriteSupport.setWriteSpecsCompliant(conf, true);
By looking its source code, we can know this function is for setting the boolean value of parquet.proto.writeSpecsCompliant in the config.
In ParquetProtoWriters.forType's source code, it create a factory with builder class ParquetProtoWriterBuilder, which uses org.apache.parquet.proto.ProtoWriteSupport internally. I guess you can somehow assign it with a correctly configured ProtoWriteSupport to it.
I also install this Python tool: https://pypi.org/project/parquet-tools/ to inspect parquet files. List fields generated in older format will be like:
...
############ Column(f2) ############
name: f2
path: f1.f2
...
and in newer format will be like:
...
############ Column(element) ############
name: element
path: f1.f2.list.element
...
Hope this article could give you some direction.
References
https://github.com/apache/parquet-format/blob/54e53e5d7794d383529dd30746378f19a12afd58/LogicalTypes.md#nested-types
https://github.com/apache/parquet-format/pull/51
https://github.com/apache/parquet-mr/pull/463