Spring Batch SQL Command with JobParameters - spring-batch

I am new to spring-batch, here i am getting some data from DB using following reader statements. Here i need to pass value dynamically(thru arguments).
<bean id="ItemReader"
class="org.springframework.batch.item.database.JdbcCursorItemReader">
<property name="dataSource" ref="dataSource" />
<property name="sql">
<value>
<![CDATA[
select * from table where section = #{jobParameters['section']}
]]>
</value>
</property>
<property name="rowMapper">
<bean class="xyzRowMapper" />
</property>
</bean>
JUnit Code:
JobParameters jobParameters = = new JobParametersBuilder()
.addString("section", section);
Can any body help on this?

As explained in ยง5.4 Late Binding of Job and Steps Attributes of official Spring Batch documentation, you need to add scope="step" to your step :
Using a scope of Step is required in order to use late binding since
the bean cannot actually be instantiated until the Step starts, which
allows the attributes to be found. Because it is not part of the
Spring container by default, the scope must be added explicitly,
either by using the batch namespace or by including a bean definition
explicitly for the StepScope (but not both)
Giving this :
<bean id="ItemReader" scope="step" class="org.springframework.batch.item.database.JdbcCursorItemReader">
<property name="dataSource" ref="dataSource" />
<property name="sql">
<value>
<![CDATA[
select * from table where section = #{jobParameters['section']}
]]>
</value>
</property>
<property name="rowMapper">
<bean class="xyzRowMapper" />
</property>
</bean>

Related

Spring Batch issue w/ CompositeItemWriter and ClassifierCompositeItemWriter

I am using spring batch in order to read data from the database (using partitioning) and writing the same to a set of files based upon entry keys - 1,2,3,4.
I have created a CompositeItemWriter which is a composition of two ClassifierCompositeItemWriter(s). Even though I have registered the individual writers as stream, I still get the following exception:
org.springframework.batch.item.WriterNotOpenException: Writer must be open before it can be written to
I even tried registering ItemWriter1 and ItemWriter2 as streams, but, that gives me a different error:
Caused by: java.lang.IllegalStateException: Cannot convert value of type [com.sun.proxy.$Proxy13 implementing org.springframework.batch.item.ItemWriter,java.io.Serializable,org.springframework.aop.scope.ScopedObject,org.springframework.aop.framework.AopInfrastructureBean,org.springframework.aop.SpringProxy,org.springframework.aop.framework.Advised] to required type [org.springframework.batch.item.ItemStream] for property 'streams[0]': no matching editors or conversion strategy found
at org.springframework.beans.TypeConverterDelegate.convertIfNecessary(TypeConverterDelegate.java:264)
at org.springframework.beans.TypeConverterDelegate.convertIfNecessary(TypeConverterDelegate.java:128)
at org.springframework.beans.TypeConverterDelegate.convertToTypedArray(TypeConverterDelegate.java:463)
at org.springframework.beans.TypeConverterDelegate.convertIfNecessary(TypeConverterDelegate.java:195)
at org.springframework.beans.BeanWrapperImpl.convertIfNecessary(BeanWrapperImpl.java:448)
... 74 more
I have even implemented the ItemStream in the writers, but, it does not work yet.
public class WriterA1 implements ItemWriter<List<Object>>, ItemStream {
...
}
The following is the xml configuration:
...
<job id="abcJob" xmlns="http://www.springframework.org/schema/batch"
restartable="true">
<step id="masterStep">
<partition step="slaveStep" partitioner="abcPartitioner">
<handler grid-size="${grid-size}" task-executor="abcTaskExecutor" />
</partition>
</step>
</job>
<step id="slaveStep" xmlns="http://www.springframework.org/schema/batch">
<tasklet transaction-manager="transactionManager">
<chunk reader="abcReader" writer="abcWriter"
processor="abcProcessor" commit-interval="${a}" skip-limit="${b}" retry-limit="${c}" >
<streams>
<!--
<stream ref="ItemWriter1"/>
<stream ref="ItemWriter2"/>
-->
<stream ref="WriterA1"/>
<stream ref="WriterB2"/>
<stream ref="WriterC3"/>
<stream ref="WriterD4"/>
<stream ref="WriterA5"/>
<stream ref="WriterB6"/>
<stream ref="WriterC7"/>
<stream ref="WriterD8"/>
</streams>
</chunk>
<listeners>
...
</listeners>
</tasklet>
</step>
<bean id="abcWriter" class="org.springframework.batch.item.support.CompositeItemWriter" scope="step">
<property name="delegates">
<list>
<ref bean="ItemWriter1" />
<ref bean="ItemWriter2" />
</list>
</property>
</bean>
<bean id="ItemWriter1" class="org.springframework.batch.item.support.ClassifierCompositeItemWriter" scope="step">
<property name="classifier">
<bean
class="org.springframework.classify.BackToBackPatternClassifier">
<property name="routerDelegate">
<bean class="xxx.xxx.xxx.xxx.Classifier1" scope="step"/>
</property>
<property name="matcherMap">
<map>
<entry key="1" value-ref="WriterA1" />
<entry key="2" value-ref="WriterB2" />
<entry key="3" value-ref="WriterC3" />
<entry key="4" value-ref="WriterD4" />
</map>
</property>
</bean>
</property>
</bean>
<bean id="ItemWriter2" class="org.springframework.batch.item.support.ClassifierCompositeItemWriter" scope="step">
<property name="classifier">
<bean
class="org.springframework.classify.BackToBackPatternClassifier">
<property name="routerDelegate">
<bean class="xxx.xxx.xxx.xxx.Classifier2" scope="step"/>
</property>
<property name="matcherMap">
<map>
<entry key="1" value-ref="WriterA5" />
<entry key="2" value-ref="WriterB6" />
<entry key="3" value-ref="WriterC7" />
<entry key="4" value-ref="WriterD8" />
</map>
</property>
</bean>
</property>
</bean>
<bean id="WriterA1" class="xxx.xxx.xxx.xxx.WriterA1" scope="step">
</bean>
<bean id="WriterB2" class="xxx.xxx.xxx.xxx.WriterB2" scope="step">
</bean>
<bean id="WriterC3" class="xxx.xxx.xxx.xxx.WriterC3" scope="step">
</bean>
<bean id="WriterD4" class="xxx.xxx.xxx.xxx.WriterD4" scope="step">
</bean>
<bean id="WriterA5" class="xxx.xxx.xxx.xxx.WriterA5" scope="step">
</bean>
<bean id="WriterB6" class="xxx.xxx.xxx.xxx.WriterB6" scope="step">
</bean>
<bean id="WriterC7" class="xxx.xxx.xxx.xxx.WriterC7" scope="step">
</bean>
<bean id="WriterD8" class="xxx.xxx.xxx.xxx.WriterD8" scope="step">
</bean>
Please advise.
You have three types of writers. From top to bottom:
abcWriter is a CompositeItemWriter. It implements ItemStream by delegating the ItemStream method calls to the delegates (here ItemWriter1 and ItemWriter2), provided they implement ItemStream. Which is not the case. But even if they implemented ItemStream, you shouldn't separately register ItemWriter1 and ItemWriter2 as streams in the step configuration (there's another independent reason in the next bullet point).
ItemWriter1/ItemWriter2 are ClassifierCompositeItemWriters. This class doesn't implement ItemStream, so you must not register them as streams in the step configuration.
WriterA1 are of type BeanIOFlatFileItemWriter and therefore implement ItemStream. Because the ClassifierCompositeItemWriter that wraps them doesn't call their ItemStream methods (unlike CompositeItemWriter), you must register each one of them as streams in the step configuration.
But this is what you claim you have. Yet your scoped-scoped WriterXX beans are being proxied (with interface proxy mode) through singleton beans that don't implement ItemStream or ItemStreamWriter, instead only ItemWriter. Make sure that the classes you have inside the <bean> elements do implement ItemStream. You can also try creating the scoped proxy bean explicitly (using ScopedProxyFactoryBean and setting the interfaces property). Or you can try putting a breakpoint in ScopedProxyFactoryBean::setBeanFactory breaking when targetBeanName is contains the string WriterXX (it will be something like stepScopedTarget.WriterD8) and try to understand why the ItemStream interface is not being proxied.

Spring Batch : Field or property 'stepExecutionContext' cannot be found

I have the following spring batch job configuration. There is a single reader which then passes details to a composite writer which has two specific writers. Both writers share a common parent and need to use the same JobId for the INSERT operations they execute.
<bean id="job" parent="simpleJob">
<property name="steps">
<list>
<bean parent="simpleStep">
<property name="itemReader" ref="policyReader"/>
<property name="itemWriter" ref="stagingCompositeWriter"/>
</bean>
</list>
</property>
</bean>
<bean id="stagingCompositeWriter" class="org.springframework.batch.item.support.CompositeItemWriter">
<property name="delegates">
<list>
<ref bean="stagingLoadWriter"/>
<ref bean="stagingPolicyWriter"/>
</list>
</property>
</bean>
<bean id="abstractStagingWriter" class="a.b.c.AbstractStagingWriter" abstract="true">
<property name="stepExecution" value="#{stepExecutionContext}"/>
<property name="hedgingStagingDataSource" ref="hedgingStagingDataSource"/>
</bean>
<bean id="stagingLoadWriter" class="a.b.c.StagingLoadWriter" parent="abstractStagingWriter"/>
<bean id="stagingPolicyWriter" class="a.b.c.StagingPolicyWriter" parent="abstractStagingWriter"/>
When i run my code i get the following error
Caused by: org.springframework.expression.spel.SpelEvaluationException: EL1008E:(pos 0): Field or property 'stepExecutionContext' cannot be found on object of type 'org.springframework.beans.factory.config.BeanExpressionContext'
at org.springframework.expression.spel.ast.PropertyOrFieldReference.readProperty(PropertyOrFieldReference.java:208)
at org.springframework.expression.spel.ast.PropertyOrFieldReference.getValueInternal(PropertyOrFieldReference.java:72)
at org.springframework.expression.spel.ast.SpelNodeImpl.getValue(SpelNodeImpl.java:93)
at org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:88)
at org.springframework.context.expression.StandardBeanExpressionResolver.evaluate(StandardBeanExpressionResolver.java:139)
I have tried setting the scope="step" in various place but to no avail. Any suggestions?
You can access the stepExecutionContext only within a bean defined in the scope="step".
Change your bean definition to
<bean id="stagingLoadWriter" scope="step" class="a.b.c.StagingLoadWriter" parent="abstractStagingWriter" />
<bean id="stagingPolicyWriter" scope="step" class="a.b.c.StagingPolicyWriter" parent="abstractStagingWriter"/>

how partition reader from database, writer in different files and optimize thread load

EDIT:
I think that there are something wrong with this clause:
I tried to run my first test that runs single thread and take about 35 minutes with this whereCause and the execution is terribly slow. When I just do an select * from table, whitout whereClause the process happens normally.
I trying to use Step Partitioning in a Job with Spring Batch, but I dont realize if is it's
appropriate to my case:
I have read from a database with ~30 million records. In the record, I have a column bank_id and there is about 23 differents banks.
I have to read the value from this column and separate the records from each bank into different txt files.
I want my job parallelize the work in 4 or 8 threads, in a first moment I try to use step partitioning and I split the job in 4 slaves and set the id_bank that I process in a parameter for a query in SqlPagingQueryProviderFactoryBean and I use only 4 different Ids. But the amount of records from one bank_id to another varies widely resulting in a slave finish they job before anothers.
I want that when the slave finish they work, he begin to process another bank_id.
I need a help to do anything like this in spring batch. I use the 2.1 version of spring batch.
here is my files:
<bean id="arquivoWriter"
class="org.springframework.batch.item.file.FlatFileItemWriter"
scope="step">
<property name="encoding" value="ISO-8859-1" />
<property name="lineAggregator">
<bean
class="org.springframework.batch.item.file.transform.FormatterLineAggregator">
<property name="fieldExtractor">
<bean
class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
<property name="names"
value="name_bank, id_bank, etc" />
</bean>
</property>
<property name="format"
value="..." />
</bean>
</property>
<property name="resource"
value="file:./arquivos/#{stepExecutionContext[faixa]}.txt" />
</bean>
<job id="partitionJob" xmlns="http://www.springframework.org/schema/batch">
<step id="masterStep">
<partition step="slave" partitioner="rangePartitioner">
<handler task-executor="taskExecutor" />
</partition>
</step>
</job>
<step id="slave" xmlns="http://www.springframework.org/schema/batch">
<tasklet>
<chunk reader="pagingReader" writer="arquivoWriter"
commit-interval="#{jobParameters['commit.interval']}" />
<listeners>
<listener ref="myChunkListener"></listener>
</listeners>
</tasklet>
</step>
<bean id="rangePartitioner" class="....RangePartitioner" />
<bean id="pagingReader"
class="org.springframework.batch.item.database.JdbcPagingItemReader"
scope="step">
<property name="dataSource" ref="dataSource" />
<property name="fetchSize" value="#{jobParameters['fetch.size']}"></property>
<property name="queryProvider">
<bean
class="org.springframework.batch.item.database.support.SqlPagingQueryProviderFactoryBean">
<property name="dataSource" ref="dataSource" />
<property name="selectClause">
<value>
<![CDATA[
SELECT ...
]]>
</value>
</property>
<property name="fromClause" value="FROM my_table" />
<property name="whereClause" value="where id_bank = :id_op" />
</bean>
</property>
<property name="parameterValues">
<map>
<entry key="id_op" value="#{stepExecutionContext[id_op]}" />
</map>
</property>
<property name="maxItemCount" value="#{jobParameters['max.rows']}"></property>
<property name="rowMapper">
<bean class="....reader.MyRowMapper" />
</property>
</bean>
The range partitioner:
public class RangePartitioner implements Partitioner {
#Autowired
BancoDao bancoDao;
final Map<String, ExecutionContext> result = new HashMap<String, ExecutionContext> ();
#Override
public Map<String, ExecutionContext> partition(int gridSize) {
List<OrgaoPagadorQuantidadeRegistrosTO> lista = bancoDao.findIdsOps();
for (OrgaoPagadorQuantidadeRegistrosTO op:lista){
String name = String.valueOf(op.getIdOrgaoPagador());
ExecutionContext ex = new ExecutionContext();
ex.putLong("id_op", op.getIdBank());
ex.putString ("faixa", name);
result.put("p"+name, ex);
}
return result;
}
}
What you're asking for should work assuming that you have enough work for each of the slaves to work on. For example, if you have 23 banks but one has 20 million records and the others each have 100,000, the slaves not working on the big bank will free up quickly.
Are you creating a StepExecution per bank or per thread? I'd recommend doing it per bank. This would allow threads to pick up work as they finish. Otherwise, you end up being responsible for that load balancing by implementing a Partitioner that does this normalization.

In-memory Job-Explorer definition in Spring batch

I was trying to share My in-memory jobRepository to the jobExplorer. But it throws an error as,
Nested exception is
org.springframework.beans.ConversionNotSupportedException:
Failed to convert property value of type '$Proxy1 implementing
org.springframework.batch.core.repository.JobRepository,org.
springframework.aop.SpringProxy,org.springframework.aop.framework.Advised'
to required type
Even i tried putting '&' sign before jobRepository when passing to jobExplorer for sharing.But attempt end in vain.
I am using Spring Batch 2.2.1
Is the dependency for jobExplorer is only database not in-memory?
Definition is,
<bean id="jobRepository"
class="com.test.repository.BatchRepositoryFactoryBean">
<property name="cache" ref="cache" />
<property name="transactionManager" ref="transactionManager" />
</bean>
<bean id="jobOperator" class="test.batch.LauncherTest.TestBatchOperator">
<property name="jobExplorer" ref="jobExplorer" />
<property name="jobRepository" ref="jobRepository" />
<property name="jobRegistry" ref="jobRegistry" />
<property name="jobLauncher" ref="jobLauncher" />
</bean>
<bean id="jobExplorer" class="test.batch.LauncherTest.TestBatchExplorerFactoryBean">
<property name="repositoryFactory" ref="&jobRepository" />
</bean>
<bean id="transactionManager"
class="org.springframework.batch.support.transaction.ResourcelessTransactionManager" />
<bean id="jobLauncher" class="com.scb.smartbatch.core.BatchLauncher">
<property name="jobRepository" ref="jobRepository" />
</bean>
<!-- To store Batch details -->
<bean id="jobRegistry" class="com.scb.smartbatch.repository.SmartBatchRegistry" />
<bean id="jobRegistryBeanPostProcessor"
class="org.springframework.batch.core.configuration.support.JobRegistryBeanPostProcessor">
<property name="jobRegistry" ref="jobRegistry" />
</bean>
<!--Runtime cache of batch executions -->
<bean id="cache" class="com.scb.cache.TCRuntimeCache" />
thanks for your valuable inputs.
But I used '&' before the job repository reference, which allowed me to use it for my job explorer as a shared resource.
problem solved.
kudos.
Usually you have to wire interface instead of implementation.
Else, probably, you have to add <aop:config proxy-target-class="true"> to create CGLIB-based proxy instead of standard Java-based proxy.
Read Spring official documentation about that

Invoking Stored Procedure using Spring JdbcBatchItemWriter

I would like to execute a Stored Procedure using spring JdbcBatchItemWriter. My current code looks like :
<bean id="xyzWriter" class="org.springframework.batch.item.database.JdbcBatchItemWriter">
......
<property name="sql" value="update abc where x=:paramX" />
......
</bean>
I would like to replace this update sql query with a Stored Proc call. I would like to handle it in the xml file itself. Any help is really appreciated.
Thanks
Did you tried running SP through JdbcBatchItemWriter?
because I also had same requirement and i just tried and it worked for me
<bean id="trackItemWriter" class="org.springframework.batch.item.database.JdbcBatchItemWriter">
<property name="dataSource" ref="mySQLDatasource"/>
<property name="itemPreparedStatementSetter">
<bean class="com.MyDataPreparedStatmentSetter"/>
</property>
<property name="assertUpdates" value="false" />
<property name="sql" value="Call my_Stored_Proc (?,?,?,?)"/>
</bean>
Hope it helps.