In spring batch JdbcCursorItemReader is ResultSet processing is TYPE_FORWARD_ONLY.
In my case, I need to point the cursor to 1 row back, hence I want to set TYPE_SCROLL_SENSITIVE, so that I can go back.
Any idea how to do it in spring batch or some workaround?
<bean id="databaseItemReader"
class="org.springframework.batch.item.database.JdbcCursorItemReader">
<property name="dataSource" ref="dataSource" />
<property name="sql"
value="select * from document udd, field uff where uff.docid = udd.docid AND uff.field_name IN ('address','contractNb','city','locale','login','mobile','name','phone') ORDER BY udd.docid, uff.field_name ASC" />
<property name="rowMapper">
<bean class="com.migration.springbatch.UDocumentResultRowMapper" />
</property>
<property name="verifyCursorPosition" value="false"/>
</bean>
The contract of an ItemReader in Spring Batch is forward only, and the JdbcCursorItemReader is implemented according to this contract (Hence the TYPE_FORWARD_ONLY). So it is not possible to rewind the cursor back once it is opened.
That said, when the underlying resource is transactional (such as a database or a JMS queue) then calling read may return the same logical item on subsequent calls in a rollback scenario.
Please find more details in the documentation here: https://docs.spring.io/spring-batch/4.0.x/reference/html/readersAndWriters.html#itemReader
Related
I have a Spring batch described as like below,
I I change commit-interval from 1 to 10000, will this change improve the performance?
<batch:job id="weeklyPartnerPointAddJob" restartable="true">
<batch:step id="weeklyPartnerPointAddStep" parent="noTransactionStep">
<batch:tasklet task-executor="asyncTaskExecutor" throttle-limit="1">
<batch:chunk reader="selectPartnerListForPointReader" processor="weeklyPartnerPointAccumulateItemProcessor" commit-interval="1" />
</batch:tasklet>
</batch:step>
The weeklyPartnerPointAccumulateItemProcessor processor manipulate the read input and update the record inside the processor. so I didn't create a ItemWriter for update logic.
The noTransactionStep described like following, which doesn't maintain transaction.
<bean id="noTransactionStep" class="org.springframework.batch.core.step.factory.SimpleStepFactoryBean" abstract="true">
<property name="transactionManager" ref="resourcelessTransactionManager" />
<property name="jobRepository" ref="jobRepository" />
<property name="startLimit" value="10" />
<property name="commitInterval" value="3" />
</bean>
Yes, the commit-interval may impact the performance. There is no best value, it depends on the context. Determining the "best" value for your specific use case is an empirical process, ie you need to try different values and find the one that leads to best performance.
A commit-interval of 1 is too small and will lead to a large number of transactions (equal to the number of input items). This could degrade the performance significantly. A value of 10.000 can lead to long running transactions and probably high memory usage (depending on the use case) which can also degrade the performance. In my experience, a value of 100 or 1000 is a good start.
I am new to spring batch.i have a requirement to read and process 500 000 lines from text to csv. My item processor is taking five min to process 100 lines which will result in almost 2 days for processing and writing 500k lines.
How to invoke the item reader and processor concurrently?
You can use "SimpleAsyncTaskExecutor" for parallel processing and use it in your spring application context as follows:
<bean id="taskExecutor"
class="org.springframework.core.task.SimpleAsyncTaskExecutor">
</bean>
And then you can specify this taskExecutor in some specific tasklet as follows:
<tasklet task-executor="taskExecutor">
<chunk reader="deskReader" processor="deskProcessor"
writer="deskWriter" commit-interval="1" />
</tasklet>
Note that you need to define the ItemReader, ItemWriter and ItemProcessor classes as specified here.
Also, the for parallel processing, you can specify the throttle-limit which specifies how many threads how want to run in parallel which is by default 4 if throttle-limit is not being specified.
I have a spring batch job with the following definition :
<batch:step id="step1">
<batch:tasklet task-executor="simpleTaskExecutor">
<batch:chunk reader="itemReader" processor="itemProcessor"
writer="itemWriter" >
</batch:chunk>
</batch:tasklet>
</batch:step>
<bean id="itemReader" class="CustomReader">
</bean>
Custom reader , reads a row from database and pass it to processor for further processing.
My problem is i want to have multiple threads at the same time to run this job at the same time ( each read a row and process) . based on documentation i used taskExecutor but it didn't worked.
note : my scenario doesn't fit with partitioner.
What do you mean by "doesn't" work?
If you want to read and process one entry with each thread, you need to have a "commit-interval" of exactly one. (http://docs.spring.io/spring-batch/reference/html/configureStep.html)
But note: since several threads will call the reader and writer (they are singleton instances) in parallel you have to ensure that both are thread-safe. The simplest thing to do this would be to synchronize the read, resp. the write method of the reader and writer.
I'm using a paging Jdbc reader in a spring batch job. I have 16 rows in a table and am expecting to see all the rows but the configuration below is only returning 10.
<bean id="pagingQuery" class="org.springframework.batch.item.database.support.SqlPagingQueryProviderFactoryBean">
<property name="dataSource" ref="myDataSource"/>
<property name="selectClause" value="select policy_number,val_date,name,sequence,amount,rate,frequency,start_date,end_date,basis"/>
<property name="fromClause" value="from MyTable"/>
<property name="sortKey" value="policy_number"/>
<property name="whereClause" value="load_id=:jobid"/>
</bean>
<bean id="tableReader"
class="org.springframework.batch.item.database.JdbcPagingItemReader"
scope="step">
<property name="dataSource" ref="myDataSource"/>
<property name="queryProvider" ref="pagingQuery"/>
<property name="parameterValues">
<map>
<entry key="jobid" value="1"/>
</map>
</property>
<property name="pageSize" value="10"/>
<property name="fetchSize" value="10"/>
<property name="rowMapper">
<bean class="org.springframework.jdbc.core.ColumnMapRowMapper"/>
</property>
</bean>
Without just raising the value of the paging/fetch size to above 16 is there an alternative way to configure this reader to return all rows?
I had same issue I needed to get all the data from table, Rather than jdbc I was using mybatis paging. The fact is that Since the view I was fetching was so complex. I was asked to hit(query) database once(paging requires multiple query to be executed for each page, where clause was an overhead). Hence I increased pagesize to accommodate all data, I run into memory issue( which is obvious). Since I'm not using UI to display data, paging is not really a requirement. Hence I used JDBCCursorItemReader. which queries the table once and keeps the data in database cache or temp table space( not a database expert, but because of mess up I found result set is cached in temp table space). the data is sent back to you using result set( it depends on fetch size) which you read and create object till commit interval. once the commit interval is reached, data is written. Next it again fetches the data from result set(if result set doesn't have data it will get another chunk from database as per fetch size) and creates object(row mapper) till commit interval reaches and then calls item writer. Hence I did not run out of memory. Hence you can configure commit interval based on your memory limitation. I would recommend keep fetchsize and commit-interval same to get better performance. When I tweaked fetchsize by increasing I got better execution time but not sure How It will impact else where. Here is sample JDBC cursor Item Reader
<bean id="cursorReader" class="org.springframework.batch.item.database.JdbcCursorItemReader">
<property name="dataSource" ref="dataSource" />
<property name="sql" value="SELECT C1,C2,C3,C4,C5 FROM KP_TBL_VW" />
<property name="rowMapper" ref="rowMapperDomain" />
<property name="fetchSize" value="50000"/>
<property name="driverSupportsAbsolute" value="true" />
</bean>
Apologize if my db understanding is wrong
I have implemented my first GridGain application and am not getting the performance improvements I expected. Sadly it is slower. I would like some help in improving my implementation so it can be faster.
The gist of my application is I am doing a brute force optimization with millions of possible parameters that take a fraction of a second for each function evaluation. I have implemented this by dividing up the millions of iterations into a few groups, and each group is executed as one job.
The relevant piece of code is below. the function maxAppliedRange calls function foo for every value in the range x, and returns the maximum, and the result becomes the maximum of all the maximums found by each job.
scalar {
result = grid !*~
(for (x <- (1 to threads).map(i => ((i - 1) * iterations / threads, i * iterations / threads)))
yield () => maxAppliedRange(x, foo), (s: Seq[(Double, Long)]) => s.max)
}
My code can chose between a multi-threaded execution on one machine or use several GridGain nodes using the code above. When I run the gridgain version it starts out like it is going to be faster, but then a few things always happen:
One of the nodes (on a different machine) misses a heartbeat, causing the node on my main computer to give up on that node and to start executing the job a second time.
The node that missed a heartbeat continues doing the same job. Now I have two nodes doing the same thing.
Eventually, all jobs are being executed on my main machine, but since some of the jobs started later, it takes way longer for everything to finish.
Sometimes an exception gets thrown by GridGain because a node timed out and the whole task gets failed.
I get annoyed.
I tried setting it up to have many jobs so if one failed then it wouldn't be as big of a deal, but when I do this I end up with many jobs being executed on each node. That puts a much bigger burden on each machine making it more likely for a node to miss a heartbeat, causing everything to go downhill faster. If I have one job per CPU then if one job fails, a different node has to start over from the beginning. Either way I can't win.
What I think would work best is if I could do two things:
Increase the timeout for heartbeats
Throttle each node so that it only does one job at a time.
If I could do this, I could divide up my task into many jobs. Each node would do one job at a time and no machine would become overburdened to cause it to miss a heartbeat. If a job failed then little work would be lost and recovery would be quick.
Can anyone tell me how to do this? What should I be doing here?
I figured it out.
First, there is an xml configuration file that controls the details of how the grid nodes operate. The default configuration file is in GRIDGAIN_HOME/config/default-spring.xml. I could either edit this or copy it and pass the new file to ggstart.sh when I start the grid node. The two things I needed to add are:
<property name="networkTimeout" value="25000"/>
which sets the timeout for network messages to 25 seconds, and
<property name="executorService">
<bean class="org.gridgain.grid.thread.GridThreadPoolExecutor">
<constructor-arg type="int" value="1"/>
<constructor-arg type="int" value="1"/>
<constructor-arg type="long">
<util:constant static-field="java.lang.Long.MAX_VALUE"/>
</constructor-arg>
<constructor-arg type="java.util.concurrent.BlockingQueue">
<bean class="java.util.concurrent.LinkedBlockingQueue"/>
</constructor-arg>
</bean>
</property>
The first two constructor arguments are for 1 thread to start and a max thread size of 1. The executor service controls the threadpool that executes the gridgain jobs. The default is 100, which is why my application was being overwhelmed and the heartbeats were being timed out.
The other change I had to make to my code is:
scalar.apply("/path/to/gridgain home/config/custom-spring.xml") {
result = grid !*~
(for (x <- (1 to threads).map(i => ((i - 1) * iterations / threads, i * iterations / threads)))
yield () => maxAppliedRange(x, kalmanBruteForceObj.performKalmanIteration), (s: Seq[(Double, Long)]) => s.max)
}
Because without the .apply statement it starts a grid node with all default options, not the configuration file with the above edits, which is what I want.
Now it works exactly as I need it to. I can divide up the task into small pieces and even my weakest and slowest computer can make a contribution to this effort.
Now I have it working correctly. In my situation for my application I am getting about a 50% speed-up over a multithreaded application on one machine, but that is not the best that I can do. More work is to be done.
To use gridgain, it seems that the configuration file is critical to getting everything working. This is where node behavior is set and must match your application's needs.
One thing I needed in my xml configuration file is this:
<property name="discoverySpi">
<bean class="org.gridgain.grid.spi.discovery.multicast.GridMulticastDiscoverySpi">
<property name="maxMissedHeartbeats" value="20"/>
<property name="leaveAttempts" value="10"/>
</bean>
</property>
This sets the maximum heartbeats that can be missed before a node is considered missing. I set this to a high value because I kept having a problem of nodes leaving and coming back a few seconds later. Alternatively, instead of using multicasting I could have fixed the IPs of the machines with running nodes using other properties in the in the config file. I didn't do this but if you are using the same machines over and over it probably would be more reliable.
The other thing I did is:
<property name="collisionSpi">
<bean class="org.gridgain.grid.spi.collision.jobstealing.GridJobStealingCollisionSpi">
<property name="activeJobsThreshold" value="2"/>
<property name="waitJobsThreshold" value="4"/>
<property name="maximumStealingAttempts" value="10"/>
<property name="stealingEnabled" value="true"/>
<property name="messageExpireTime" value="1000"/>
</bean>
</property>
<property name="failoverSpi">
<bean class="org.gridgain.grid.spi.failover.jobstealing.GridJobStealingFailoverSpi">
<property name="maximumFailoverAttempts" value="10"/>
</bean>
</property>
For the first one, the activeJobsThreshold value tells the node how many jobs it can run at the same time. This is a better way of doing throttling than changing the number of threads in the executor service. Also, it does some load balancing and idle nodes can 'steal' work from other nodes to get everything done faster.
There are better ways to do this also. Gridgain can do size the jobs based on the measured performance of each node, apparently, which would improve overall performance, especially if you have fast and slow computers in the grid.
For the future I am going to study the configuration file and compare that to the javadocs to learn all about all the different options, to get this to run even faster.