I was trying to set maximum filesize using bsub -F option. But there is no manual suggesting max value.
Can some body please help in setting maximum value for filesize limit.
As Squirrel comments, this is an application specific choice. This is an optional input and I think the idea here is to prevent a runaway job from filling up storage space. If a job exceeds this limit, LSF will be kill that job.
Quote from documentation:
-F file_limit
Sets a per-process (soft) file size limit for each of the processes that belong to the job (see getrlimit(2)). The limit is
specified in KB.
If a job process attempts to write to a file that exceeds the file
size limit, then that process is sent a SIGXFSZ signal. The SIGXFSZ
signal normally terminates the process.
Related
I'm using pg_restore to restore a database to its original state for load testing. I see it has a --jobs=number-of-jobs option.
How can I get a ballpark estimate of what this value should be on my machine? I know this is dependent on a bunch of factors, e.g. machine, dataset, but it would be great to get a conceptual starting point.
I'm on a MacBook Pro so maybe I can use the number of physical CPU cores:
sysctl -n hw.physicalcpu
# 10
If there is no concurrent activity at all, don't exceed the number of parallel I/O requests your disk can handle. This applies to the COPY part of the dump, which is likely I/O bound. But a restore also creates indexes, which uses CPU time (in addition to I/O). So you should also not exceed the number of CPU cores available.
I'm using SingleStore to load events from Kafka. I created a Kafka pipeline with the following script:
create pipeline `events_stream`
as load data kafka 'kafka-all-broker:29092/events_stream'
batch_interval 10000
max_partitions_per_batch 6
into procedure `proc_events_stream`
fields terminated by '\t' enclosed by '' escaped by '\\'
lines terminated by '\n' starting by '';
And SingleStore failing with OOM error like the following:
Memory used by MemSQL (4537.88 Mb) has reached the 'maximum_memory' setting (4915 Mb) on this node. Possible causes include (1) available query execution memory has been used up for table memory (in use table memory: 71.50 Mb) and (2) the query is large and complex and requires more query execution memory than is available
I'm a quite confused why 4Gb is not enough to read Kafka by batches....
Is it possible to configure batch_size for the pipeline to avoid memory issues and make the pipeline more predictable?
unfortunately, current version of Singlestore's pipelines only has a global batch size and can not be set individually in stored procedures.
However, In general, each pipeline batch has some overhead, so running 1 batch for 10000 messages should be better in terms of total resources than 100 or even 10 batches for the same number of messages. If the stored procedure is relatively light, the time delay you are experiencing is likely dominated by the network download of 500mb.
It is worth checking if 10000 large messages are arriving on the same partition or several. In Singlestore's pipelines, each database partition downloads messages from a different kafka partition, and that helps parallelize the workload. However, if the messages arrive on just one partition, then you are not getting the benefits of parallel execution.
Data was read from postgres table & written to file using Talend. Table size is 1.8GB with 1,050,000 records and has around 125 columns.
Assigned JVM as -Xms256M -Xmx1024M. The job failed due to being out of memory. Postgres keeps the result set in physical memory until the query completes. So the entire JVM was occupied and getting an out of memory issue. Please correct me if my understanding is wrong.
Enabled Cursor option and kept the value as 100,000 and JVM as -Xms256M -Xmx1024M. Job failed with java.lang.OutOfMemoryError: Java heap space
I don't understand the concept here. Cursor used here denotes the fetch size of rows. In my case, 100,000 was set. So 100,000 will be fetched and stored in physical memory and it will be pushed to file. Then, the occupied memory will be released and the next batch will be fetched. Please correct me if I'm wrong.
Considering my case, with 1,050,000 records it occupies 1.8GB. Each record occupies 1.8KB of size. 100,000 * 1.8 = 180,000KB. So entire size is just 175MB. Why is the job not running with a 1GB JVM? Someone please help me with, how does this process work?
Some records got dropped after setting the cursor option in talend. Cannot trace the problem in that.
Had the same problem with a tPostgresInput component. Disabling the Auto Commit setting in the tPostgresConnection component fixed the problem for me!
I have a cluster and I execute wholeTextFiles which should pull about a million text files who sum up to approximately 10GB total
I have one NameNode and two DataNode with 30GB of RAM each, 4 cores each. The data is stored in HDFS.
I don't run any special parameters and the job takes 5 hours to just read the data. Is that expected? are there any parameters that should speed up the read (spark configuration or partition, number of executors?)
I'm just starting and I've never had the need to optimize a job before
EDIT: Additionally, can someone explain exactly how the wholeTextFiles function works? (not how to use it, but how it was programmed). I'm very interested in understand the partition parameter, etc.
EDIT 2: benchmark assessment
So I tried repartition after the wholeTextFile, the problem is the same because the first read is still using the pre-defined number of partitions, so there are no performance improvements. Once the data is loaded the cluster performs really well... I have the following warning message when dealing with the data (for 200k files), on the wholeTextFile:
15/01/19 03:52:48 WARN scheduler.TaskSetManager: Stage 0 contains a task of very large size (15795 KB). The maximum recommended task size is 100 KB.
Would that be a reason of the bad performance? How do I hedge that?
Additionally, when doing a saveAsTextFile, my speed according to Ambari console is 19MB/s. When doing a read with wholeTextFiles, I am at 300kb/s.....
It seems that by increase the number of partitions in wholeTextFile(path,partitions), I am getting better performance. But still only 8 tasks are running at the same time (my number of CPUs). I'm benchmarking to observe the limit...
To summarize my recommendations from the comments:
HDFS is not a good fit for storing many small files. First of all, NameNode stores metadata in memory so the amount of files and blocks you might have is limited (~100m blocks is a max for typical server). Next, each time you read file you first query NameNode for block locations, then connect to the DataNode storing the file. Overhead of this connections and responses is really huge.
Default settings should always be reviewed. By default Spark starts on YARN with 2 executors (--num-executors) with 1 thread each (--executor-cores) and 512m of RAM (--executor-memory), giving you only 2 threads with 512MB RAM each, which is really small for the real-world tasks
So my recommendation is:
Start Spark with --num-executors 4 --executor-memory 12g --executor-cores 4 which would give you more parallelism - 16 threads in this particular case, which means 16 tasks running in parallel
Use sc.wholeTextFiles to read the files and then dump them into compressed sequence file (for instance, with Snappy block level compression), here's an example of how this can be done: http://0x0fff.com/spark-hdfs-integration/. This will greatly reduce the time needed to read them with the next iteration
in fact the following is what i really want to ask:
the set of chunk size is 1M. when a config server down, the whole config servers are readonly. current cluster have only a chunk, if I want to insert a lot of data to this cluster, the capacity of these data is more than 1M, Can I successfully insert these data?
if yes, do it describe that the real chunk size can more than the set of chunk size?
Thanks!
Short answer: yes you can, but you should fix your config server(s) asap to avoid unbalancing your shards for long.
Chunks are split automatically when they reach their size threshold - stated here.
However, during a config server failure, chunks cannot be split. Even if just one server fails. See here.
Edits
As stated by Sergio Tulentsev, you should fix your config server(s) before performing your insert. The system's metadata will continue to be readonly until then.
As Adam C's link points out, your shard will become unbalanced if you were to perform an insert like you describe before fixing your config server(s).