Structured Streaming - Could not use FileContext API for managing metadata log files on AWS S3 - scala

Question

I have a StreamingQuery in Spark(v2.2.0), i.e.,
val df = spark
.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("subscribe", "test")
.load()
val query = df
.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
.writeStream
.format("parquet")
.option("checkpointLocation", "s3n://bucket/checkpoint/test")
.option("path", "s3n://bucket/test")
.start()
When I am running query then data does get save on AWS S3 and checkpoints are created at s3n://bucket/checkpoint/test. But, I am also receiving following WARNING in the logs:
WARN [o.a.s.s.e.streaming.OffsetSeqLog] Could not use FileContext API for managing metadata log files at path s3n://bucket/checpoint/test/offsets. Using FileSystem API instead for managing log files. The log may be inconsistent under failures.
I am not able to understand as to why this WARNING is coming. Also, will my checkpoints be inconsistent in case of any failure?
Can anyone help me in resolving it?

Answer 1

Looking at the source code, this error comes from the HDFSMetadataLog class. A comment in the code states that:
Note: [[HDFSMetadataLog]] doesn't support S3-like file systems as they don't guarantee listing files in a directory always shows the latest files.
So the problem is due to using AWS S3 and it will force you to use the FileSystemManager API. Checking the comment for that class, we see that,
Implementation of FileManager using older FileSystem API. Note that this implementation cannot provide atomic renaming of paths, hence can lead to consistency issues. This should be used only as a backup option, when FileContextManager cannot be used.
Hence, some issues can come up when multiple writers want to concurrently do rename operations. There is a related ticket here, however, it has been closed since the issue can't be fixed in Spark.
Some things to consider if you need to checkpoint on S3:
To aviod the warning and potential trouble, checkpoint to HDFS and then copy over the results
Checkpoint to S3, but have a long gap between checkpoints.

Answer 2

Nobody should be using S3n as the connector. It is obsolete and removed from Hadoop 3. If you have the Hadoop 2.7.x JARs on the classpath, use s3a
The issue with rename() is not just the consistency, but the bigger the file, the longer it takes.
Really checkpointing to object stores needs to be done differently. If you look closely, there is no rename(), yet so much existing code expects it to be an O(1) atomic operation.