I am using Scala and Apache Flink to build an ETL that reads all the files under a directory in my local file system periodically and write the result of processing each file in a single output file under another directory.
So an example of this is would be:
/dir/to/input/files/file1
/dir/to/intput/files/fil2
/dir/to/input/files/file3
and the output of the ETL would be exactly:
/dir/to/output/files/file1
/dir/to/output/files/file2
/dir/to/output/files/file3
I have tried various approaches including reducing the parallel processing to one when writing to the dataSink but I still can't achieve the required result.
This is my current code:
val path = "/path/to/input/files/"
val format = new TextInputFormat(new Path(path))
val socketStream = env.readFile(format, path, FileProcessingMode.PROCESS_CONTINUOUSLY, 10)
val wordsStream = socketStream.flatMap(value => value.split(",")).map(value => WordWithCount(value,1))
val keyValuePair = wordsStream.keyBy(_.word)
val countPair = keyValuePair.sum("count")
countPair.print()
countPair.writeAsText("/path/to/output/directory/"+
DateTime.now().getHourOfDay.toString
+
DateTime.now().getMinuteOfHour.toString
+
DateTime.now().getSecondOfMinute.toString
, FileSystem.WriteMode.NO_OVERWRITE)
// The first write method I trid:
val sink = new BucketingSink[WordWithCount]("/path/to/output/directory/")
sink.setBucketer(new DateTimeBucketer[WordWithCount]("yyyy-MM-dd--HHmm"))
// The second write method I trid:
val sink3 = new BucketingSink[WordWithCount]("/path/to/output/directory/")
sink3.setUseTruncate(false)
sink3.setBucketer(new DateTimeBucketer("yyyy-MM-dd--HHmm"))
sink3.setWriter(new StringWriter[WordWithCount])
sink3.setBatchSize(3)
sink3.setPendingPrefix("file-")
sink3.setPendingSuffix(".txt")
Both writing methods fail in producing the wanted result.
Can some with experience with Apache Flink guide me to the write approach please.
I solved this issue importing the next dependencies to run on local machine:
hadoop-aws-2.7.3.jar
aws-java-sdk-s3-1.11.183.jar
aws-java-sdk-core-1.11.183.jar
aws-java-sdk-kms-1.11.183.jar
jackson-annotations-2.6.7.jar
jackson-core-2.6.7.jar
jackson-databind-2.6.7.jar
joda-time-2.8.1.jar
httpcore-4.4.4.jar
httpclient-4.5.3.jar
You can review it on :
https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/aws.html
Section "Provide S3 FileSystem Dependency"
Related
The folder of hdfs is like :
/test/data/2020-03-01/{multiple inside files csv}
/test/data/2020-03-02/{multiple files csv}
/test/data/2020-03-03/{multiple files csv }
i want to read data inside folder one by one not whole by
spark.read.csv("/test/data/*") //not in such manner
Not in above manner , i want to read file one by one; so that i can make the log entry in some database that date folder is read ; so that on next time i can skip that folder in next day or same day if program run accidentally:
val conf = new Configuration()
val iterate = org.apache.hadoop.fs.FileSystem.get(new URI(strOutput), conf).listLocatedStatus(new org.apache.hadoop.fs.Path(strOutput))
while (iterate.hasNext) {
val pathStr = iterate.next().getPath.toString
println("log---->"+pathStr)
val df = spark.read.text(pathStr)
}
Try something like above and read as data frame, if you want you can union new date df with old df.
I'm trying to combine 3 files present in HDFS through scala. All the 3 files are present in the HDFS location srcPath as mentioned in the code below.
Created a function as below:
def mergeFiles(conf: Configuration, fs: FileSystem, srcPath: Path, dstPath: String, finalFileName: String): Unit {
val localfs = FileSystem.getLocal(conf)
val status = fs.listStatus(srcPath)
status.foreach(x =>
FileUtil.copyMerge(fs, x.getPath, localfs, new Path(dstPath.toString), false, conf, null)
}
I tried executing this, no result, no error, also no file gets created even.
I verified that I'm passing all the arguments properly.
Any clues?
The second argument of copyMerge is a directory, not an individual file.
This should work:
FileUtil.copyMerge(fs, srcPath, localfs, new Path(dstPath.toString), false, conf, null)
Usually reading the source code is the best way to debug such issues.
FileUtil#copyMerge method has been removed. See details for the major change:
https://issues.apache.org/jira/browse/HADOOP-12967
https://issues.apache.org/jira/browse/HADOOP-11392
You can use getmerge
Usage: hadoop fs -getmerge [-nl]
I am trying to Load data in HBase using BulkLoad. I am also using Scala and Spark to write the code. But every time data is loading on only one single region. I need to load this into multiple region. I have used below code -
Hbase Configuration:
def getConf: Configuration = {
val hbaseSitePath = "/etc/hbase/conf/hbase-site.xml"
val conf = HBaseConfiguration.create()
conf.addResource(new Path(hbaseSitePath))
conf.setInt("hbase.mapreduce.bulkload.max.hfiles.perRegion.perFamily", 100)
conf
}
I can load 80GB of Data in only one single region using above mentioned configuration.
But when I am trying load the same amount of data in multiple region with below mentioned configuration getting exception
java.io.IOException: Trying to load more than 32 hfiles to one family
of one region
Updated Configuration -
def getConf: Configuration = {
val conf = HBaseConfiguration.create()
conf.addResource(new Path(hbaseSitePath))
conf.setInt("hbase.mapreduce.bulkload.max.hfiles.perRegion.perFamily", 32)
conf.setLong("hbase.hregion.max.filesize", 107374182)
conf.set("hbase.regionserver.region.split.policy","org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy")
conf
}
For saving records I am using below code -
val kv = new KeyValue(Bytes.toBytes(key), columnFamily.getBytes(),
columnName.getBytes(), columnValue.getBytes())
(new ImmutableBytesWritable(Bytes.toBytes(key)), kv)
rdd.saveAsNewAPIHadoopFile(pathToHFile, classOf[ImmutableBytesWritable], classOf[KeyValue],
classOf[HFileOutputFormat2], conf) //Here rdd is the input
val loadFiles = new LoadIncrementalHFiles(conf)
loadFiles.doBulkLoad(new Path(pathToHFile), hTable)
Need Help on this.
You are getting issue because 32 is default value per region. You should define KeyPrefixRegionSplitPolicy to split your files and you can increase increase hbase.mapreduce.bulkload.max.hfiles.perRegion.perFamily as below
conf.setInt("hbase.mapreduce.bulkload.max.hfiles.perRegion.perFamily", 1024)
Try also to increase file size as
conf.setLong("hbase.hregion.max.filesize", 107374182)
I'm trying to read and execute a sql in SPARK SQL.
sqlContext.sql(scala.io.Source.fromInputStream(getClass.getResourceAsStream("/" + "dq.sql")).getLines.mkString(" ").stripMargin).take(1)
My sql is very long. When I run it straight way in spark shell , it runs fine. When I try to read this using getResourcesAsStream - I'm hitting
java.lang.RuntimeException: [1.10930] failure: end of input
A simple solution could be reading the sql at driver (using any file utility) and pass the variable like ssc.sql(sqlvar)
val stream : InputStream = getClass.getResourceAsStream("/filename.txt")
val readFile = scala.io.Source.fromInputStream( stream ).getLines
val spa = readFile.map(line => " " + line)
val spl = spa.mkString.split(";")
for (m1 <- spl) {
sqlContext.sql(m1)
}
I have 20 million files in S3 spanning roughly 8000 days.
The files are organized by timestamps in UTC, like this: s3://mybucket/path/txt/YYYY/MM/DD/filename.txt.gz. Each file is UTF-8 text containing between 0 (empty) and 100KB of text (95th percentile, although there are a few files that are up to several MBs).
Using Spark and Scala (I'm new to both and want to learn), I would like to save "daily bundles" (8000 of them), each containing whatever number of files were found for that day. Ideally I would like to store the original filenames as well as their content. The output should reside in S3 as well and be compressed, in some format that is suitable for input in further Spark steps and experiments.
One idea was to store bundles as a bunch of JSON objects (one per line and '\n'-separated), e.g.
{id:"doc0001", meta:{x:"blah", y:"foo", ...}, content:"some long string here"}
{id:"doc0002", meta:{x:"foo", y:"bar", ...}, content: "another long string"}
Alternatively, I could try the Hadoop SequenceFile, but again I'm not sure how to set that up elegantly.
Using the Spark shell for example, I saw that it was very easy to read the files, for example:
val textFile = sc.textFile("s3n://mybucket/path/txt/1996/04/09/*.txt.gz")
// or even
val textFile = sc.textFile("s3n://mybucket/path/txt/*/*/*/*.txt.gz")
// which will take for ever
But how do I "intercept" the reader to provide the file name?
Or perhaps I should get an RDD of all the files, split by day, and in a reduce step write out K=filename, V=fileContent?
You can use this
First You can get a Buffer/List of S3 Paths :
import scala.collection.JavaConverters._
import java.util.ArrayList
import com.amazonaws.services.s3.AmazonS3Client
import com.amazonaws.services.s3.model.ObjectListing
import com.amazonaws.services.s3.model.S3ObjectSummary
import com.amazonaws.services.s3.model.ListObjectsRequest
def listFiles(s3_bucket:String, base_prefix : String) = {
var files = new ArrayList[String]
//S3 Client and List Object Request
var s3Client = new AmazonS3Client();
var objectListing: ObjectListing = null;
var listObjectsRequest = new ListObjectsRequest();
//Your S3 Bucket
listObjectsRequest.setBucketName(s3_bucket)
//Your Folder path or Prefix
listObjectsRequest.setPrefix(base_prefix)
//Adding s3:// to the paths and adding to a list
do {
objectListing = s3Client.listObjects(listObjectsRequest);
for (objectSummary <- objectListing.getObjectSummaries().asScala) {
files.add("s3://" + s3_bucket + "/" + objectSummary.getKey());
}
listObjectsRequest.setMarker(objectListing.getNextMarker());
} while (objectListing.isTruncated());
//Removing Base Directory Name
files.remove(0)
//Creating a Scala List for same
files.asScala
}
Now Pass this List object to the following piece of code, note : sc is an object of SQLContext
var df: DataFrame = null;
for (file <- files) {
val fileDf= sc.textFile(file)
if (df!= null) {
df= df.unionAll(fileDf)
} else {
df= fileDf
}
}
Now you got a final Unified RDD i.e. df
Optional, And You can also repartition it in a single BigRDD
val files = sc.textFile(filename, 1).repartition(1)
Repartitioning always works :D
have you tried something along the lines of sc.wholeTextFiles?
It creates an RDD where the key is the filename and the value is the byte array of the whole file. You can then map this so the key is the file date, and then groupByKey?
http://spark.apache.org/docs/latest/programming-guide.html
At your scale, elegant solution would be a stretch.
I would recommend against using sc.textFile("s3n://mybucket/path/txt/*/*/*/*.txt.gz") as it takes forever. What you can do is use AWS DistCp or something similar to move files into HDFS. Once its in HDFS, spark is quite fast in ingesting the information in whatever way suits you.
Note that most of these processes require some sort of file list so you'll need to generate that somehow. for 20 mil files, this creation of file list will be a bottle neck. I'd recommend creating a file that get appended with the file path, every-time a file gets uploaded to s3.
Same for output, put into hdfs and then move to s3 (although direct copy might be equally efficient).