I have implemented my first GridGain application and am not getting the performance improvements I expected. Sadly it is slower. I would like some help in improving my implementation so it can be faster.
The gist of my application is I am doing a brute force optimization with millions of possible parameters that take a fraction of a second for each function evaluation. I have implemented this by dividing up the millions of iterations into a few groups, and each group is executed as one job.
The relevant piece of code is below. the function maxAppliedRange calls function foo for every value in the range x, and returns the maximum, and the result becomes the maximum of all the maximums found by each job.
scalar {
result = grid !*~
(for (x <- (1 to threads).map(i => ((i - 1) * iterations / threads, i * iterations / threads)))
yield () => maxAppliedRange(x, foo), (s: Seq[(Double, Long)]) => s.max)
}
My code can chose between a multi-threaded execution on one machine or use several GridGain nodes using the code above. When I run the gridgain version it starts out like it is going to be faster, but then a few things always happen:
One of the nodes (on a different machine) misses a heartbeat, causing the node on my main computer to give up on that node and to start executing the job a second time.
The node that missed a heartbeat continues doing the same job. Now I have two nodes doing the same thing.
Eventually, all jobs are being executed on my main machine, but since some of the jobs started later, it takes way longer for everything to finish.
Sometimes an exception gets thrown by GridGain because a node timed out and the whole task gets failed.
I get annoyed.
I tried setting it up to have many jobs so if one failed then it wouldn't be as big of a deal, but when I do this I end up with many jobs being executed on each node. That puts a much bigger burden on each machine making it more likely for a node to miss a heartbeat, causing everything to go downhill faster. If I have one job per CPU then if one job fails, a different node has to start over from the beginning. Either way I can't win.
What I think would work best is if I could do two things:
Increase the timeout for heartbeats
Throttle each node so that it only does one job at a time.
If I could do this, I could divide up my task into many jobs. Each node would do one job at a time and no machine would become overburdened to cause it to miss a heartbeat. If a job failed then little work would be lost and recovery would be quick.
Can anyone tell me how to do this? What should I be doing here?
I figured it out.
First, there is an xml configuration file that controls the details of how the grid nodes operate. The default configuration file is in GRIDGAIN_HOME/config/default-spring.xml. I could either edit this or copy it and pass the new file to ggstart.sh when I start the grid node. The two things I needed to add are:
<property name="networkTimeout" value="25000"/>
which sets the timeout for network messages to 25 seconds, and
<property name="executorService">
<bean class="org.gridgain.grid.thread.GridThreadPoolExecutor">
<constructor-arg type="int" value="1"/>
<constructor-arg type="int" value="1"/>
<constructor-arg type="long">
<util:constant static-field="java.lang.Long.MAX_VALUE"/>
</constructor-arg>
<constructor-arg type="java.util.concurrent.BlockingQueue">
<bean class="java.util.concurrent.LinkedBlockingQueue"/>
</constructor-arg>
</bean>
</property>
The first two constructor arguments are for 1 thread to start and a max thread size of 1. The executor service controls the threadpool that executes the gridgain jobs. The default is 100, which is why my application was being overwhelmed and the heartbeats were being timed out.
The other change I had to make to my code is:
scalar.apply("/path/to/gridgain home/config/custom-spring.xml") {
result = grid !*~
(for (x <- (1 to threads).map(i => ((i - 1) * iterations / threads, i * iterations / threads)))
yield () => maxAppliedRange(x, kalmanBruteForceObj.performKalmanIteration), (s: Seq[(Double, Long)]) => s.max)
}
Because without the .apply statement it starts a grid node with all default options, not the configuration file with the above edits, which is what I want.
Now it works exactly as I need it to. I can divide up the task into small pieces and even my weakest and slowest computer can make a contribution to this effort.
Now I have it working correctly. In my situation for my application I am getting about a 50% speed-up over a multithreaded application on one machine, but that is not the best that I can do. More work is to be done.
To use gridgain, it seems that the configuration file is critical to getting everything working. This is where node behavior is set and must match your application's needs.
One thing I needed in my xml configuration file is this:
<property name="discoverySpi">
<bean class="org.gridgain.grid.spi.discovery.multicast.GridMulticastDiscoverySpi">
<property name="maxMissedHeartbeats" value="20"/>
<property name="leaveAttempts" value="10"/>
</bean>
</property>
This sets the maximum heartbeats that can be missed before a node is considered missing. I set this to a high value because I kept having a problem of nodes leaving and coming back a few seconds later. Alternatively, instead of using multicasting I could have fixed the IPs of the machines with running nodes using other properties in the in the config file. I didn't do this but if you are using the same machines over and over it probably would be more reliable.
The other thing I did is:
<property name="collisionSpi">
<bean class="org.gridgain.grid.spi.collision.jobstealing.GridJobStealingCollisionSpi">
<property name="activeJobsThreshold" value="2"/>
<property name="waitJobsThreshold" value="4"/>
<property name="maximumStealingAttempts" value="10"/>
<property name="stealingEnabled" value="true"/>
<property name="messageExpireTime" value="1000"/>
</bean>
</property>
<property name="failoverSpi">
<bean class="org.gridgain.grid.spi.failover.jobstealing.GridJobStealingFailoverSpi">
<property name="maximumFailoverAttempts" value="10"/>
</bean>
</property>
For the first one, the activeJobsThreshold value tells the node how many jobs it can run at the same time. This is a better way of doing throttling than changing the number of threads in the executor service. Also, it does some load balancing and idle nodes can 'steal' work from other nodes to get everything done faster.
There are better ways to do this also. Gridgain can do size the jobs based on the measured performance of each node, apparently, which would improve overall performance, especially if you have fast and slow computers in the grid.
For the future I am going to study the configuration file and compare that to the javadocs to learn all about all the different options, to get this to run even faster.
Related
Due to limitations of integrating with existing application we need to use a separate database connection per chunk and manage the commit boundary one commit at the end of chunk.
We designed to use remote partitioning and process multiple partitions at workers. partition step expected to execute chunks sequentially.
We tried an approach of combining Chink Listener and processor, connection obtained at before chunk listener method as instance variable and using it while processing,
but this connection being replaced right after the first item processed by beforeChunk method. Used Resource less Transaction manager in this case. How ever this transaction manager is not recommended to use for production.
We are expected to process chunks in parallel and not expected this.
We have also observed that when we hold the connection at writer by beforeChunk method is getting closed when write method called.
Second approach by using DataSourceTransactionManager , but not sure how to get the connection from transaction used at chunk level.
Step Configuration as follows:-
<step id="senExtractGeneratePrintRequestWorkerStep" xmlns=http://www.springframework.org/schema/batch>
<tasklet>
<chunk reader="senExtractGeneratePrintRequestWorkerItemReader"
processor="senExtractGeneratePrintRequestWorkerItemProcessor"
writer="senExtractGeneratePrintRequestWorkerMultiItemWriter"
commit-interval="${senExtractGeneratePrintRequestWorkerStep.commit-interval}"
skip-limit="${senExtractGeneratePrintRequestWorkerStep.skip-limit}">
<skippable-exception-classes>
<batch:include class="java.lang.Exception" />
</skippable-exception-classes>
</chunk>
<listeners>
<listener ref="senExtractGeneratePrintRequestWorkerItemProcessor" />
</listeners>
</tasklet>
</step>
<bean id="senExtractGeneratePrintRequestWorkerItemProcessor"
scope="step"
class="com.abc.batch.senextract.worker.SENExtractGeneratePrintRequestItemProcessor"/>
Connection is closed by datasource before write is called. Screen shots of call hierarchy as below.
enter image description here
I have a Spring batch described as like below,
I I change commit-interval from 1 to 10000, will this change improve the performance?
<batch:job id="weeklyPartnerPointAddJob" restartable="true">
<batch:step id="weeklyPartnerPointAddStep" parent="noTransactionStep">
<batch:tasklet task-executor="asyncTaskExecutor" throttle-limit="1">
<batch:chunk reader="selectPartnerListForPointReader" processor="weeklyPartnerPointAccumulateItemProcessor" commit-interval="1" />
</batch:tasklet>
</batch:step>
The weeklyPartnerPointAccumulateItemProcessor processor manipulate the read input and update the record inside the processor. so I didn't create a ItemWriter for update logic.
The noTransactionStep described like following, which doesn't maintain transaction.
<bean id="noTransactionStep" class="org.springframework.batch.core.step.factory.SimpleStepFactoryBean" abstract="true">
<property name="transactionManager" ref="resourcelessTransactionManager" />
<property name="jobRepository" ref="jobRepository" />
<property name="startLimit" value="10" />
<property name="commitInterval" value="3" />
</bean>
Yes, the commit-interval may impact the performance. There is no best value, it depends on the context. Determining the "best" value for your specific use case is an empirical process, ie you need to try different values and find the one that leads to best performance.
A commit-interval of 1 is too small and will lead to a large number of transactions (equal to the number of input items). This could degrade the performance significantly. A value of 10.000 can lead to long running transactions and probably high memory usage (depending on the use case) which can also degrade the performance. In my experience, a value of 100 or 1000 is a good start.
I'm wondering whether it is possible, to configure a job in a way, that I could repeat several chunked-steps until the whole data is processed?
Background is, that I need to work on some real big data and while processing it, there's a risk of unwanted aborts. To prevent restarting from scratch over-and-over again, I'd like to do some partitioning of the data, that could be used to loop over the chunked steps.
Due to the given data, it is unfortunately not possible to make use of the spring-batch restartable-job feature to reach my goal.
My source database consists of several more-or-less loose connected tables, each of them is processed in its own step. So I have something like:
... omitting job-configuration ...
<batch:step="A" next="B">
<batch:tasklet>
<batch:chunk reader="readerA" writer="writerA" commit-interval="1000" />
</batch:tasklet>
</batch:step>
<batch:step="B" next="C">
<batch:tasklet>
<batch:chunk reader="readerB" writer="writerB" commit-interval="5000" />
</batch:tasklet>
</batch:step>
... some more steps with similar set-up...
Each reader has it's own SQL-Statement, to get the necessary data from the source-db and will write the result in another table of the target-db.
Now, my idea would be to, to adapt those SQLs in a way, that the data will be partitioned into some disjoint but consistent(*) parts, so that I could repeat the processing using the chunked steps as before. Maybe only adding some "parent-step" to control whether the loop has to be ended.
(*) By "disjoint but consistent" I mean, that although the data in the different steps is fetched from different tables, there are dependencies. For example, fetching the data to be processed for step B would do a join with table A, choosing only sets which were successfully processed.
Thanx for any advices!
/Andreas
Since there are dependencies between tables, I don't think going parallel is appropriate. Going parallel makes sense when partitions are independent from each other.
Your current setup should allow you to restart your job from where it left off at two levels:
In between steps: If the job fails at step B, step A would not be re-executed
Within each step: If the job fails in the middle of step A, it will restart from the last successfully committed chunk of step A.
You need to make sure to use a persistent job repository and restart the same job instance in case of failure (using the same identifying job parameters as the previous run).
I am new to spring batch.i have a requirement to read and process 500 000 lines from text to csv. My item processor is taking five min to process 100 lines which will result in almost 2 days for processing and writing 500k lines.
How to invoke the item reader and processor concurrently?
You can use "SimpleAsyncTaskExecutor" for parallel processing and use it in your spring application context as follows:
<bean id="taskExecutor"
class="org.springframework.core.task.SimpleAsyncTaskExecutor">
</bean>
And then you can specify this taskExecutor in some specific tasklet as follows:
<tasklet task-executor="taskExecutor">
<chunk reader="deskReader" processor="deskProcessor"
writer="deskWriter" commit-interval="1" />
</tasklet>
Note that you need to define the ItemReader, ItemWriter and ItemProcessor classes as specified here.
Also, the for parallel processing, you can specify the throttle-limit which specifies how many threads how want to run in parallel which is by default 4 if throttle-limit is not being specified.
I have a spring batch job with the following definition :
<batch:step id="step1">
<batch:tasklet task-executor="simpleTaskExecutor">
<batch:chunk reader="itemReader" processor="itemProcessor"
writer="itemWriter" >
</batch:chunk>
</batch:tasklet>
</batch:step>
<bean id="itemReader" class="CustomReader">
</bean>
Custom reader , reads a row from database and pass it to processor for further processing.
My problem is i want to have multiple threads at the same time to run this job at the same time ( each read a row and process) . based on documentation i used taskExecutor but it didn't worked.
note : my scenario doesn't fit with partitioner.
What do you mean by "doesn't" work?
If you want to read and process one entry with each thread, you need to have a "commit-interval" of exactly one. (http://docs.spring.io/spring-batch/reference/html/configureStep.html)
But note: since several threads will call the reader and writer (they are singleton instances) in parallel you have to ensure that both are thread-safe. The simplest thing to do this would be to synchronize the read, resp. the write method of the reader and writer.