Decrypting PGP file which is on HDFS - scala

Question

We are decrypting PGP file with the help of "com.didisoft.pgp.PGPLib" in scala.
this is working fine with local files but when we run it for HDFS files we are facing issue like "File not found exception for securingkey"
Even while trying the same thing with unix utility for gpg we faced a file not found issue when path of HDFS file is passed.
Below is sample code for local files thats working fine:
val decryptionPassword = "xxxx"
val sec = "C:/Users/path/secring.gpg"
val originalFileName =pgp.decryptFile("C:/Users/path/pgp_sample_file.PGP",sec,
decryptionPassword ,"C:/Users/path/opfile/PGP.txt")
How can we use these utilities for decrypting our files lying on the HDFS?

Answer 1

You can't access hdfs like a normal file system. You need to either download the file to your local system then use the local file, or open a stream or load the file into memory then decrypt that.
To use gpg from the command line
hdfs dfs -cat <hdfs_file_path> | gpg --batch --yes --passphrase <passphrase> -d
I can't answer how to do it with the Java library (it seems to be proprietary), but there is probably a way to accept an inputstream instead of a filename.
To get an InputStream from an hdfs file, you need to use the hadoop fs api
val fs = org.apache.hadoop.fs.FileSystem.get(new org.apache.hadoop.conf.Configuration())
val inputStream = fs.open(new org.apache.hadoop.fs.Path(<filepath>))

Answer 2

Based on the sample code from puhlen, I can suggest you to try this:
val pgp = new com.didisoft.pgp.PGPLib()
val decryptionPassword = "xxxx"
val fs = org.apache.hadoop.fs.FileSystem.get(new org.apache.hadoop.conf.Configuration())
val keysStream = fs.open(new org.apache.hadoop.fs.Path("hdfs://.../secring.gpg"))
val ks = new com.didisoft.pgp.KeyStore()
ks.importKeyRing(keysStream)
val inputData = fs.open(new org.apache.hadoop.fs.Path("hdfs://.../pgp_sample_file.PGP"))
val outputData = fs.create(new org.apache.hadoop.fs.Path("hdfs://.../PGP.txt"))
val originalFileName = pgp.decryptStream(inputData, ks,
decryptionPassword, outputData)
(don't forget to replace the dots with the correct HDFS paths)