Error while moving a file into HDFS from Local using Scala - scala

Question

I have a Scala list: fileNames, that contains the file names present on a local directory.
Ex:
fileNames(2)
res0: String = file:///tmp/audits/xx_user.log
I am trying to move a file from the list: fileNames, from local to HDFS using Scala. In order to do that, I followed the below steps:
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.commons.io.IOUtils;
val hadoopconf = new Configuration();
hadoopconf.addResource(new Path("/etc/hadoop/conf/core-site.xml"));
val fs = FileSystem.get(hadoopconf);
val outFileStream = fs.create(new Path("hdfs://mydev/user/devusr/testfolder"))
The code works fine until here. When I try to add the inputStream, I am getting error message as below:
val inStream = fs.open(new Path(fileNames(2)))
java.lang.IllegalArgumentException: Wrong FS: file:/tmp/audits/xx_user.log, expected: hdfs://mergedev
I also tried by specifying the file name directly and result is same:
val inStream = fs.open(new Path("file:///tmp/audits/xx_user.log"))
java.lang.IllegalArgumentException: Wrong FS: file:/tmp/audits/xx_user.log, expected: hdfs://mergedev
But when I try to load the file directly into spark, it is working fine:
val localToSpark = spark.read.text(fileNames(2))
localToSpark: org.apache.spark.sql.DataFrame = [value: string]
localToSpark.collect
res1: Array[org.apache.spark.sql.Row] = Array([[Wed Dec 20 06:18:02 UTC 2017] INFO: ], [*********************************************************************************************************], [ ], [[Wed Dec 20 06:18:02 UTC 2017] INFO: Diagnostic log for xx_user.]
Could anyone tell me what is the mistake I am doing at this point:
val inStream = fs.open(new Path(fileNames(2))) for which I am getting the error.

Answer 1

For small files, copyFromLocalFile() is sufficient:
fs.copyFromLocalFile(new Path(localFileName), new Path(hdfsFileName))
For large files, it's more efficient to use Apache Commons-IO:
IOUtils.copyLarge(
new FileInputStream(new File(localFileName)), fs.create(new Path(hdfsFileName)))
Keep in mind that the local file name should not include the protocol (so no file:/// in there)