Load XML file from HDFS in Scala - scala

Question

I want to load a XML file from HDFS using XML Scala API. I am trying as follows but its not recognizing the path. Could anyone let me know how we can load file from HDFS by using Scala?
import scala.xml.{NodeSeq, XML}
val xml_load = XML.loadFile("hdfs:////user/np.user/raw/xmlfile.xml")

Answer 1

I assume you're using Scala 2.12.x; I also assume those four slashes in hdfs:////user... are typo.
You're using method XML.loadFile(name: String); it internally uses FileInputStream. It's not possible to open an HDFS file with a plain FileInputStream. You need an input stream which supports HDFS. You can find it in org.apache.hadoop:hadoop-hdfs library.
The code then looks like this:
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs._
// configure properly so the code knows which Hadoop cluster to connect to
// https://hadoop.apache.org/docs/r3.2.0/api/org/apache/hadoop/conf/Configuration.html
val conf = new Configuration()
// obtain input stream instance
val hdfsPath: Path = new Path("hdfs://user/np.user/raw/xmlfile.xml")
val fs: FileSystem = hdfsPath.getFileSystem(conf)
val inputStream: FSDataInputStream = fs.open(hdfsPath)
// load XML
try {
val xml_load = XML.load(inputStream)
} finally {
// close resources; of course, this will silently swallow any exception in close() methods
inputStream.close()
fs.close()
}