I have spark output in a s3 folders and I want to move all s3 files from that output folder to another location ,but while moving I want to rename the files .
For example I have files in S3 folders like below
Now I want to rename all files and put into another directory,but the name of the files would be like below
Fundamental.FinancialStatement.FinancialStatementLineItems.Japan.1971-BAL.1.2017-10-18-0439.Full.txt
Fundamental.FinancialStatement.FinancialStatementLineItems.Japan.1971-BAL.2.2017-10-18-0439.Full.txt
Fundamental.FinancialStatement.FinancialStatementLineItems.Japan.1971-BAL.3.2017-10-18-0439.Full.txt
Here Fundamental.FinancialStatementis constant in all the files 2017-10-18-0439 current date time .
This is what I have tried so far but not able to get folder name and loop through all files
import org.apache.hadoop.fs._
val src = new Path("s3://trfsmallfffile/Segments/output")
val dest = new Path("s3://trfsmallfffile/Segments/Finaloutput")
val conf = sc.hadoopConfiguration // assuming sc = spark context
val fs = src.getFileSystem(conf)
//val file = fs.globStatus(new Path("src/DataPartition=Japan/part*.gz"))(0).getPath.getName
//println(file)
val status = fs.listStatus(src)
status.foreach(filename => {
val a = filename.getPath.getName.toString()
println("file name"+a)
//println(filename)
})
This gives me below output
file nameDataPartition=Japan
file nameDataPartition=SelfSourcedPrivate
file nameDataPartition=SelfSourcedPublic
file name_SUCCESS
This gives me folders details not files inside the folder.
Reference is taken from here Stack Overflow Refrence
You are getting directory because you have sub dir level in s3 .
/*/* to go in subdir .
Try this
import org.apache.hadoop.fs._
val src = new Path("s3://trfsmallfffile/Segments/Output/*/*")
val dest = new Path("s3://trfsmallfffile/Segments/FinalOutput")
val conf = sc.hadoopConfiguration // assuming sc = spark context
val fs = src.getFileSystem(conf)
val file = fs.globStatus(new Path("s3://trfsmallfffile/Segments/Output/*/*"))
for (urlStatus <- file) {
//println("S3 FILE PATH IS ===:" + urlStatus.getPath)
val partitioName=urlStatus.getPath.toString.split("=")(1).split("\\/")(0).toString
val finalPrefix="Fundamental.FinancialLineItem.Segments."
val finalFileName=finalPrefix+partitioName+".txt"
val dest = new Path("s3://trfsmallfffile/Segments/FinalOutput"+"/"+finalFileName+ " ")
fs.rename(urlStatus.getPath, dest)
}
This has worked for me in past
import org.apache.hadoop.fs.{FileSystem, Path}
import org.apache.hadoop.conf.Configuration
val path = "s3://<bucket>/<directory>"
val fs = FileSystem.get(new java.net.URI(path), spark.sparkContext.hadoopConfiguration)
fs.listStatus(new Path(path))
The list status provides all the files in the s3 directory
Related
For a project I am currently working on with Scala and Spark, I have to make a code that checks if the hdfs directory I am working on is empty, and if it is not, I have to remove every files from the directory.
Before I deploy my code into Azur, I am testing it with a local directory from my computer.
I am starting with: making a method to delete every files from this directory. This is what I have for now :
object DirectoryCleaner {
val spark:SparkSession = SparkSession.builder()
.master("local[3]")
.appName("SparkByExamples.com")
.getOrCreate()
val fs = FileSystem.get(spark.sparkContext.hadoopConfiguration)
val srcPath=new Path("C:\\Users\\myuser\\Desktop\\test_dir\\file1.csv")
def deleFilesDir(): Unit = {
if(fs.exists(srcPath) && fs.isFile(srcPath))
fs.delete(srcPath, true)
}
}
With this code, I am able to delete a single file (file1.csv). I would like to be able to define my path this way val srcPath=new Path("C:\\Users\\myuser\\Desktop\\test_dir") (without specifying any filename), and just delete every files from the test_dir directory. Any idea on how I could do that ?
Thank's for helping
Use fs.listFiles to get all the files in a directory and then loop through them while deleting them. Also, set the recursive flag to false, so you don't recurse into directories.
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.{FileSystem, Path}
def deleteAllFiles(directoryPath: String, fs: FileSystem): Unit = {
val path = new Path(directoryPath)
// get all files in directory
val files = fs.listFiles(path, false)
// print and delete all files
while (files.hasNext) {
val file = files.next()
fs.delete(file.getPath, false)
}
}
// Example for local, non HDFS path
val directoryPath = "file:///Users/m_vemuri/project"
val fs = FileSystem.get(new Configuration())
deleteAllFiles(directoryPath, fs)
I need to write a Spark/Scala function in Apache Zeppelin that simply puts some files that are already present in an HDFS folder into a zip or gzip archive (or some common archive format that is easy to extract in Windows) in the same folder. How would I do this please? Would it be a Java call? I see there's something called ZipOutputStream, is that the right approach? Any tips appreciated.
Thanks
Spark does not support reading/writing from zip directly, so using the ZipOutputStream is basically the only approach.
Here's the code I used to compress my existing data via spark. It recursively lists directory for files and then proceeds to compress them. This code does not preserve directory structure, but keeps file names.
Input directory:
unzipped/
├── part-00001
├── part-00002
└── part-00003
0 directories, 3 files
Output directory:
zipped/
├── part-00001.zip
├── part-00002.zip
└── part-00003.zip
0 directories, 3 files
ZipPacker.scala:
package com.haodemon.spark.compression
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.{FileSystem, Path}
import org.apache.hadoop.io.IOUtils
import org.apache.spark.sql.SparkSession
import org.apache.spark.{SparkConf, SparkContext}
import java.io.FileOutputStream
import java.util.zip.{ZipEntry, ZipOutputStream}
object ZipPacker extends Serializable {
private def getSparkContext: SparkContext = {
val conf: SparkConf = new SparkConf()
.setAppName("local")
.setMaster("local[*]")
SparkSession.builder().config(conf).getOrCreate().sparkContext
}
// recursively list files in a filesystem
private def listFiles(fs: FileSystem, path: Path): List[Path] = {
fs.listStatus(path).flatMap(p =>
if (p.isDirectory) listFiles(fs, p.getPath)
else List(p.getPath)
).toList
}
// zip compress file one by one in parallel
private def zip(inputPath: Path, outputDirectory: Path): Unit = {
val outputPath = {
val name = inputPath.getName + ".zip"
outputDirectory + "/" + name
}
println(s"Zipping to $outputPath")
val zipStream = {
val out = new FileOutputStream(outputPath)
val zip = new ZipOutputStream(out)
val entry = new ZipEntry(inputPath.getName)
zip.putNextEntry(entry)
// max compression
zip.setLevel(9)
zip
}
val conf = new Configuration
val uncompressedStream = inputPath.getFileSystem(conf).open(inputPath)
val close = true
IOUtils.copyBytes(uncompressedStream, zipStream, conf, close)
}
def main(args: Array[String]): Unit = {
val input = new Path(args(0))
println(s"Using input path $input")
val sc = getSparkContext
val uncompressedFiles = {
val conf = sc.hadoopConfiguration
val fs = input.getFileSystem(conf)
listFiles(fs, input)
}
val rdd = sc.parallelize(uncompressedFiles)
val output = new Path(args(1))
println(s"Using output path $output")
rdd.foreach(unzipped => zip(unzipped, output))
}
}
I have multiple Csv files in hdfs and some of them not in good format, I would like to read the directory of csv files and then if successfully move the files to other directory. How can I achieve this using spark-scala
You need something like that :
import org.apache.hadoop.fs.{FileSystem, Path}
import org.apache.spark.SparkContext
val conf = sc.hadoopConfiguration
val fs = FileSystem.get(conf)
val srcPath = "dbfs:/src/"
val dest = "dbfs:/dest/"
val ls = fs.listStatus(new Path(srcPath))
ls.foreach{ p => {
if(true) spark.read.csv(p.getPath.toString).write.csv(dest + p.getName)
else println("File ${p.getName} got wrong format")
}}
I'm working in a cluster. I need to run the same spark operation for each text file contained in HDFS. But I want to do that without submitting spark job shell-command for each file from shell command line, because the number of files is 90.
How can I do that?
My code for one file is structured as follow:
object SparkGraphGen{
def main(args: Array[String]){
val conf = new SparkConf()
.setMaster("yarn")
.setAppName("dataset")
val sc = new SparkContext(conf)
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
import sqlContext.implicits._
val peopleRDD = sc.textFile("file1.csv")
...
do stuff
...
sc.stop()
}}
update:
how about foreach loop:
val sc = new SparkContext(conf)
//val files = new File("Data\\files\\").listFiles.map(_.getAbsolutePath).toList
val files = new File("Data\\files\\").listFiles.map(_.getName).toList
files.foreach { file =>
//val lines = sc.textFile(file)
val lines = sc.textFile("Data\\files\\" + file)
println("total lines in file " + file + " " + lines.count())
//do more stuf... for each file
lines.saveAsTextFile("Data\\output\\" + file + "_output")
}
sc.stop()
output:
total lines in file C:\Users\rpatel\workspaces\Spark\Data\files\file1.txt 4
total lines in file C:\Users\rpatel\workspaces\Spark\Data\files\file2.txt 4
you can also write same for loop in shell-script
#!/bin/bash
for file in $(hadoop fs -ls /hdfs/path/to/files/|awk -F '|' '{print $NF}')
do
#run spark for each file
spark-submit <options> $file /path/output/$file
done
or process all files in one shot....
you can put all files in one directory and pass only full directory path to spark context, spark will process all files in that directory:
val peopleRDD = sc.textFile("/path/to/csv_files/")
you can also combine RDDs like:
val file1RDD = sc.textFile("file1.csv")
val file2RDD = sc.textFile("file2.csv")
val allFileRDD = file1RDD ++ file2RDD // ++ nRDD
but with 90 files, I would put all files in one directory and use directory path to process all in one job...
I want to loop through all text files in a Hadoop dir and count all the occurrences of the word "error". Is there a way to do a hadoop fs -ls /users/ubuntu/ to list all the files in a dir with the Apache Spark Scala API?
From the given first example, the spark context seems to only access files individually through something like:
val file = spark.textFile("hdfs://target_load_file.txt")
In my problem, I do not know how many nor the names of the files in the HDFS folder beforehand. Looked at the spark context docs but couldn't find this kind of functionality.
You can use a wildcard:
val errorCount = sc.textFile("hdfs://some-directory/*")
.flatMap(_.split(" ")).filter(_ == "error").count
import org.apache.hadoop.fs.{FileSystem, FileUtil, Path}
import scala.collection.mutable.Stack
val fs = FileSystem.get( sc.hadoopConfiguration )
var dirs = Stack[String]()
val files = scala.collection.mutable.ListBuffer.empty[String]
val fs = FileSystem.get(sc.hadoopConfiguration)
dirs.push("/user/username/")
while(!dirs.isEmpty){
val status = fs.listStatus(new Path(dirs.pop()))
status.foreach(x=> if(x.isDirectory) dirs.push(x.getPath.toString) else
files+= x.getPath.toString)
}
files.foreach(println)
For a local installation, (the hdfs default path fs.defaultFS can be found by reading /etc/hadoop/core.xml):
For instance,
import org.apache.hadoop.fs.{FileSystem, Path}
val conf = sc.hadoopConfiguration
conf.set("fs.defaultFS", "hdfs://localhost:9000")
val hdfs: org.apache.hadoop.fs.FileSystem = org.apache.hadoop.fs.FileSystem.get(conf)
val fileStatus = hdfs.listStatus(new Path("hdfs://localhost:9000/foldername/"))
val fileList = fileStatus.map(x => x.getPath.toString)
fileList.foreach(println)