Spring batch StoredProcedureItemReader - spring-batch

I get the following error when I run a stored procedure using Spring Batch:
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: L'index 0 du paramètre de sortie n'est pas valide.
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(SQLServerException.java:190)
at com.microsoft.sqlserver.jdbc.SQLServerCallableStatement.getterGetParam(SQLServerCallableStatement.java:354)
at com.microsoft.sqlserver.jdbc.SQLServerCallableStatement.getObject(SQLServerCallableStatement.java:659)
at org.springframework.batch.item.database.StoredProcedureItemReader.openCursor(StoredProcedureItemReader.java:214)
... 18 more
The stored procedure contains a create table command which is responsible for the error.
The stored procedure:
ALTER PROCEDURE [dbo].[TEST]
#orgKeyParam bigint
AS
BEGIN
CREATE TABLE #tmpPatients (
programID bigint NOT NULL)
drop table #tmpPatients;
SELECT last_name from patient;
END
The StoredProcedureItemReader configuration:
<bean id="DBReader"
class="org.springframework.batch.item.database.StoredProcedureItemReader">
<property name="dataSource" ref="dataSource2" />
<property name="procedureName" value="[${sql.RPMDBName}].dbo.Test" />
<property name="fetchSize" value="50" />
<property name="parameters">
<list>
<bean class="org.springframework.jdbc.core.SqlParameter">
<constructor-arg index="0" value="orgKeyParam" />
<constructor-arg index="1">
<util:constant static-field="java.sql.Types.INTEGER" />
</constructor-arg>
</bean>
</list>
</property>
<property name="rowMapper" ref="dataRowMapper" />
<property name="preparedStatementSetter" ref="preparedStatementSetter" />
</bean>
<bean id="preparedStatementSetter"
class="org.springframework.batch.core.resource.ListPreparedStatementSetter" scope="step">
<property name="parameters">
<list>
<value>1</value>
</list>
</property>
</bean>

Use nocount on in sql server stored proc.
Shoul b used before begin statement

Without seeing your stored procedure or the way you've configured your StoredProcedureItemReader, I can't give you an exact fix. However, the error is that your parameter at index 0 is invalid. By default, the StoredProcedureItemReader looks for the cursor that is returned to be the first out parameter. I'm going to assume that either you've configured your reader to look elsewhere or your stored procedure isn't returning a cursor for it's first out parameter.

Related

Spring Batch SQL Command with JobParameters

I am new to spring-batch, here i am getting some data from DB using following reader statements. Here i need to pass value dynamically(thru arguments).
<bean id="ItemReader"
class="org.springframework.batch.item.database.JdbcCursorItemReader">
<property name="dataSource" ref="dataSource" />
<property name="sql">
<value>
<![CDATA[
select * from table where section = #{jobParameters['section']}
]]>
</value>
</property>
<property name="rowMapper">
<bean class="xyzRowMapper" />
</property>
</bean>
JUnit Code:
JobParameters jobParameters = = new JobParametersBuilder()
.addString("section", section);
Can any body help on this?
As explained in §5.4 Late Binding of Job and Steps Attributes of official Spring Batch documentation, you need to add scope="step" to your step :
Using a scope of Step is required in order to use late binding since
the bean cannot actually be instantiated until the Step starts, which
allows the attributes to be found. Because it is not part of the
Spring container by default, the scope must be added explicitly,
either by using the batch namespace or by including a bean definition
explicitly for the StepScope (but not both)
Giving this :
<bean id="ItemReader" scope="step" class="org.springframework.batch.item.database.JdbcCursorItemReader">
<property name="dataSource" ref="dataSource" />
<property name="sql">
<value>
<![CDATA[
select * from table where section = #{jobParameters['section']}
]]>
</value>
</property>
<property name="rowMapper">
<bean class="xyzRowMapper" />
</property>
</bean>

Spring Batch : Field or property 'stepExecutionContext' cannot be found

I have the following spring batch job configuration. There is a single reader which then passes details to a composite writer which has two specific writers. Both writers share a common parent and need to use the same JobId for the INSERT operations they execute.
<bean id="job" parent="simpleJob">
<property name="steps">
<list>
<bean parent="simpleStep">
<property name="itemReader" ref="policyReader"/>
<property name="itemWriter" ref="stagingCompositeWriter"/>
</bean>
</list>
</property>
</bean>
<bean id="stagingCompositeWriter" class="org.springframework.batch.item.support.CompositeItemWriter">
<property name="delegates">
<list>
<ref bean="stagingLoadWriter"/>
<ref bean="stagingPolicyWriter"/>
</list>
</property>
</bean>
<bean id="abstractStagingWriter" class="a.b.c.AbstractStagingWriter" abstract="true">
<property name="stepExecution" value="#{stepExecutionContext}"/>
<property name="hedgingStagingDataSource" ref="hedgingStagingDataSource"/>
</bean>
<bean id="stagingLoadWriter" class="a.b.c.StagingLoadWriter" parent="abstractStagingWriter"/>
<bean id="stagingPolicyWriter" class="a.b.c.StagingPolicyWriter" parent="abstractStagingWriter"/>
When i run my code i get the following error
Caused by: org.springframework.expression.spel.SpelEvaluationException: EL1008E:(pos 0): Field or property 'stepExecutionContext' cannot be found on object of type 'org.springframework.beans.factory.config.BeanExpressionContext'
at org.springframework.expression.spel.ast.PropertyOrFieldReference.readProperty(PropertyOrFieldReference.java:208)
at org.springframework.expression.spel.ast.PropertyOrFieldReference.getValueInternal(PropertyOrFieldReference.java:72)
at org.springframework.expression.spel.ast.SpelNodeImpl.getValue(SpelNodeImpl.java:93)
at org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:88)
at org.springframework.context.expression.StandardBeanExpressionResolver.evaluate(StandardBeanExpressionResolver.java:139)
I have tried setting the scope="step" in various place but to no avail. Any suggestions?
You can access the stepExecutionContext only within a bean defined in the scope="step".
Change your bean definition to
<bean id="stagingLoadWriter" scope="step" class="a.b.c.StagingLoadWriter" parent="abstractStagingWriter" />
<bean id="stagingPolicyWriter" scope="step" class="a.b.c.StagingPolicyWriter" parent="abstractStagingWriter"/>

how partition reader from database, writer in different files and optimize thread load

EDIT:
I think that there are something wrong with this clause:
I tried to run my first test that runs single thread and take about 35 minutes with this whereCause and the execution is terribly slow. When I just do an select * from table, whitout whereClause the process happens normally.
I trying to use Step Partitioning in a Job with Spring Batch, but I dont realize if is it's
appropriate to my case:
I have read from a database with ~30 million records. In the record, I have a column bank_id and there is about 23 differents banks.
I have to read the value from this column and separate the records from each bank into different txt files.
I want my job parallelize the work in 4 or 8 threads, in a first moment I try to use step partitioning and I split the job in 4 slaves and set the id_bank that I process in a parameter for a query in SqlPagingQueryProviderFactoryBean and I use only 4 different Ids. But the amount of records from one bank_id to another varies widely resulting in a slave finish they job before anothers.
I want that when the slave finish they work, he begin to process another bank_id.
I need a help to do anything like this in spring batch. I use the 2.1 version of spring batch.
here is my files:
<bean id="arquivoWriter"
class="org.springframework.batch.item.file.FlatFileItemWriter"
scope="step">
<property name="encoding" value="ISO-8859-1" />
<property name="lineAggregator">
<bean
class="org.springframework.batch.item.file.transform.FormatterLineAggregator">
<property name="fieldExtractor">
<bean
class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor">
<property name="names"
value="name_bank, id_bank, etc" />
</bean>
</property>
<property name="format"
value="..." />
</bean>
</property>
<property name="resource"
value="file:./arquivos/#{stepExecutionContext[faixa]}.txt" />
</bean>
<job id="partitionJob" xmlns="http://www.springframework.org/schema/batch">
<step id="masterStep">
<partition step="slave" partitioner="rangePartitioner">
<handler task-executor="taskExecutor" />
</partition>
</step>
</job>
<step id="slave" xmlns="http://www.springframework.org/schema/batch">
<tasklet>
<chunk reader="pagingReader" writer="arquivoWriter"
commit-interval="#{jobParameters['commit.interval']}" />
<listeners>
<listener ref="myChunkListener"></listener>
</listeners>
</tasklet>
</step>
<bean id="rangePartitioner" class="....RangePartitioner" />
<bean id="pagingReader"
class="org.springframework.batch.item.database.JdbcPagingItemReader"
scope="step">
<property name="dataSource" ref="dataSource" />
<property name="fetchSize" value="#{jobParameters['fetch.size']}"></property>
<property name="queryProvider">
<bean
class="org.springframework.batch.item.database.support.SqlPagingQueryProviderFactoryBean">
<property name="dataSource" ref="dataSource" />
<property name="selectClause">
<value>
<![CDATA[
SELECT ...
]]>
</value>
</property>
<property name="fromClause" value="FROM my_table" />
<property name="whereClause" value="where id_bank = :id_op" />
</bean>
</property>
<property name="parameterValues">
<map>
<entry key="id_op" value="#{stepExecutionContext[id_op]}" />
</map>
</property>
<property name="maxItemCount" value="#{jobParameters['max.rows']}"></property>
<property name="rowMapper">
<bean class="....reader.MyRowMapper" />
</property>
</bean>
The range partitioner:
public class RangePartitioner implements Partitioner {
#Autowired
BancoDao bancoDao;
final Map<String, ExecutionContext> result = new HashMap<String, ExecutionContext> ();
#Override
public Map<String, ExecutionContext> partition(int gridSize) {
List<OrgaoPagadorQuantidadeRegistrosTO> lista = bancoDao.findIdsOps();
for (OrgaoPagadorQuantidadeRegistrosTO op:lista){
String name = String.valueOf(op.getIdOrgaoPagador());
ExecutionContext ex = new ExecutionContext();
ex.putLong("id_op", op.getIdBank());
ex.putString ("faixa", name);
result.put("p"+name, ex);
}
return result;
}
}
What you're asking for should work assuming that you have enough work for each of the slaves to work on. For example, if you have 23 banks but one has 20 million records and the others each have 100,000, the slaves not working on the big bank will free up quickly.
Are you creating a StepExecution per bank or per thread? I'd recommend doing it per bank. This would allow threads to pick up work as they finish. Otherwise, you end up being responsible for that load balancing by implementing a Partitioner that does this normalization.

In-memory Job-Explorer definition in Spring batch

I was trying to share My in-memory jobRepository to the jobExplorer. But it throws an error as,
Nested exception is
org.springframework.beans.ConversionNotSupportedException:
Failed to convert property value of type '$Proxy1 implementing
org.springframework.batch.core.repository.JobRepository,org.
springframework.aop.SpringProxy,org.springframework.aop.framework.Advised'
to required type
Even i tried putting '&' sign before jobRepository when passing to jobExplorer for sharing.But attempt end in vain.
I am using Spring Batch 2.2.1
Is the dependency for jobExplorer is only database not in-memory?
Definition is,
<bean id="jobRepository"
class="com.test.repository.BatchRepositoryFactoryBean">
<property name="cache" ref="cache" />
<property name="transactionManager" ref="transactionManager" />
</bean>
<bean id="jobOperator" class="test.batch.LauncherTest.TestBatchOperator">
<property name="jobExplorer" ref="jobExplorer" />
<property name="jobRepository" ref="jobRepository" />
<property name="jobRegistry" ref="jobRegistry" />
<property name="jobLauncher" ref="jobLauncher" />
</bean>
<bean id="jobExplorer" class="test.batch.LauncherTest.TestBatchExplorerFactoryBean">
<property name="repositoryFactory" ref="&jobRepository" />
</bean>
<bean id="transactionManager"
class="org.springframework.batch.support.transaction.ResourcelessTransactionManager" />
<bean id="jobLauncher" class="com.scb.smartbatch.core.BatchLauncher">
<property name="jobRepository" ref="jobRepository" />
</bean>
<!-- To store Batch details -->
<bean id="jobRegistry" class="com.scb.smartbatch.repository.SmartBatchRegistry" />
<bean id="jobRegistryBeanPostProcessor"
class="org.springframework.batch.core.configuration.support.JobRegistryBeanPostProcessor">
<property name="jobRegistry" ref="jobRegistry" />
</bean>
<!--Runtime cache of batch executions -->
<bean id="cache" class="com.scb.cache.TCRuntimeCache" />
thanks for your valuable inputs.
But I used '&' before the job repository reference, which allowed me to use it for my job explorer as a shared resource.
problem solved.
kudos.
Usually you have to wire interface instead of implementation.
Else, probably, you have to add <aop:config proxy-target-class="true"> to create CGLIB-based proxy instead of standard Java-based proxy.
Read Spring official documentation about that

Invoking Stored Procedure using Spring JdbcBatchItemWriter

I would like to execute a Stored Procedure using spring JdbcBatchItemWriter. My current code looks like :
<bean id="xyzWriter" class="org.springframework.batch.item.database.JdbcBatchItemWriter">
......
<property name="sql" value="update abc where x=:paramX" />
......
</bean>
I would like to replace this update sql query with a Stored Proc call. I would like to handle it in the xml file itself. Any help is really appreciated.
Thanks
Did you tried running SP through JdbcBatchItemWriter?
because I also had same requirement and i just tried and it worked for me
<bean id="trackItemWriter" class="org.springframework.batch.item.database.JdbcBatchItemWriter">
<property name="dataSource" ref="mySQLDatasource"/>
<property name="itemPreparedStatementSetter">
<bean class="com.MyDataPreparedStatmentSetter"/>
</property>
<property name="assertUpdates" value="false" />
<property name="sql" value="Call my_Stored_Proc (?,?,?,?)"/>
</bean>
Hope it helps.