Spring Batch: Getting step context in processor for partitioned step - spring-batch

I'm developing a Spring Batch job that processes multiple input files in parallel using a MultiResourcePartitioner. In the ItemProcessor I need to get the number of records in the current input file. I get the current file name from the step context and read the number of lines in the file:
StepSynchronizationManager.register(stepExecution);
StepContext stepContext = StepSynchronizationManager.getContext();
StepSynchronizationManager.close();
log.trace("stepContext: " + stepContext.getStepExecution().getExecutionContext().entrySet().toString());
...
UrlResource currentFile = new UrlResource(stepContext.getStepExecution().getExecutionContext().getString("fileName"));
This all seems to work but I'm getting exceptions when accessing the step context from the processor threads:
2013-05-15 11:44:35,178 DEBUG [org.springframework.batch.core.step.tasklet.TaskletStep] <taskExecutor-3> - Rollback for RuntimeException: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'scopedTarget.processor': Scope 'step' is not active for the current thread; consider defining a scoped proxy for this bean if you intend to refer to it from a singleton; nested exception is java.lang.IllegalStateException: No context holder available for step scope
2013-05-15 11:44:35,194 DEBUG [org.springframework.jdbc.datasource.DataSourceUtils] <taskExecutor-3> - Returning JDBC Connection to DataSource
2013-05-15 11:44:35,194 DEBUG [org.springframework.batch.repeat.support.RepeatTemplate] <taskExecutor-3> - Handling exception: org.springframework.beans.factory.BeanCreationException, caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'scopedTarget.processor': Scope 'step' is not active for the current thread; consider defining a scoped proxy for this bean if you intend to refer to it from a singleton; nested exception is java.lang.IllegalStateException: No context holder available for step scope
2013-05-15 11:44:35,194 DEBUG [org.springframework.batch.repeat.support.RepeatTemplate] <taskExecutor-3> - Handling fatal exception explicitly (rethrowing first of 1): org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'scopedTarget.processor': Scope 'step' is not active for the current thread; consider defining a scoped proxy for this bean if you intend to refer to it from a singleton; nested exception is java.lang.IllegalStateException: No context holder available for step scope
2013-05-15 11:44:35,210 ERROR [org.springframework.batch.core.step.AbstractStep] <taskExecutor-3> - Encountered an error executing the step
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'scopedTarget.processor': Scope 'step' is not active for the current thread; consider defining a scoped proxy for this bean if you intend to refer to it from a singleton; nested exception is java.lang.IllegalStateException: No context holder available for step scope
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:341)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
at org.springframework.aop.target.SimpleBeanTargetSource.getTarget(SimpleBeanTargetSource.java:33)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:182)
at $Proxy14.process(Unknown Source)
at org.springframework.batch.core.step.item.SimpleChunkProcessor.doProcess(SimpleChunkProcessor.java:125)
at org.springframework.batch.core.step.item.SimpleChunkProcessor.transform(SimpleChunkProcessor.java:291)
at org.springframework.batch.core.step.item.SimpleChunkProcessor.process(SimpleChunkProcessor.java:190)
at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:74)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:386)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130)
at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:264)
at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:76)
at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:367)
at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:214)
at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:143)
at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:250)
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:195)
at org.springframework.batch.core.partition.support.TaskExecutorPartitionHandler$1.call(TaskExecutorPartitionHandler.java:120)
at org.springframework.batch.core.partition.support.TaskExecutorPartitionHandler$1.call(TaskExecutorPartitionHandler.java:118)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.IllegalStateException: No context holder available for step scope
at org.springframework.batch.core.scope.StepScope.getContext(StepScope.java:197)
at org.springframework.batch.core.scope.StepScope.get(StepScope.java:139)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:327)
... 24 more
Is there a way to get the current input file name from the processor of a partitioned job?
Here is the relevant config:
<batch:job id="acct">
<batch:step id="init" parent="abstractStep" next="processStep">
<batch:tasklet ref="fileDeleteTasklet" />
</batch:step>
<batch:step id="processStep">
<batch:partition step="processInput" partitioner="partitioner">
<batch:handler grid-size="2" task-executor="taskExecutor" />
</batch:partition>
</batch:step>
</batch:job>
<batch:step id="processInput">
<batch:tasklet>
<batch:chunk reader="reader" processor="processor" writer="writer" commit-interval="3">
<batch:streams>
<batch:stream ref="reader"/>
<batch:stream ref="flatFileItemWriter"/>
</batch:streams>
</batch:chunk>
</batch:tasklet>
</batch:step>
<bean id="partitioner" class="org.springframework.batch.core.partition.support.MultiResourcePartitioner" scope="step">
<property name="keyName" value="fileName"/>
<property name="resources" value="file:#{jobParameters['inputFilePattern']}"/>
</bean>
<bean id="taskExecutor" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
<property name="corePoolSize" value="5"/>
<property name="maxPoolSize" value="5"/>
</bean>
<bean id="reader" class="org.springframework.batch.item.file.FlatFileItemReader" scope="step">
<property name="resource" value="#{stepExecutionContext[fileName]}" />
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer">
<bean class="org.springframework.batch.item.file.transform.FixedLengthTokenizer">
<property name="names" value="recordType,reserved,aCode,bCode,acctNumber,idType,cCode" />
<property name="columns" value="1,2,3-6,7-9,10-15,16,17-19" />
<property name="strict" value="false" />
</bean>
</property>
<property name="fieldSetMapper">
<bean class="com.example.batch.acct.AcctInputFieldSetMapper" />
</property>
</bean>
</property>
</bean>
<bean id="processor" class="com.example.batch.acct.AcctProcessor" scope="step">
<property name="soapClient" ref="soapClient" />
<property name="maxNumIds" value="${batch.soap.numberOfIds}" />
<aop:scoped-proxy />
</bean>

Answered on the Spring Batch forum:
The ItemProcessor interface does not expose a way to access the
StepContext, so you need to inject it yourself:
<bean id="itemProcessor" class="com.myApp.MyItemProcessor" scope="step">
<property name="stepContext" value="#{stepExecutionContext}"/>
</bean>
Assuming your processor class has an exposed property named
stepContext.
-- Michael Minella, Spring Batch Lead

<bean id="itemProcessor" class="com.myApp.MyItemProcessor" scope="step">
<property name="stepContext" value="#{stepExecutionContext}"/>
</bean>
In this case when stepContex is a type of org.springframework.batch.core.scope.context.StepContext, how suggests name of this property, I am getting exceptions:
org.springframework.beans.factory.BeanCreationException: Error
creating bean with name
'scopedTarget.itemProcessor' defined in
class path resource [META-INF/jobs/Job.xml]:
Initialization of bean failed; nested exception is
org.springframework.beans.ConversionNotSupportedException: Failed to
convert property value of type 'java.util.Collections$UnmodifiableMap'
to required type
'org.springframework.batch.core.scope.context.StepContext' for
property 'stepContext'; nested exception is
java.lang.IllegalStateException: Cannot convert value of type
'java.util.Collections$UnmodifiableMap' to required type
'org.springframework.batch.core.scope.context.StepContext' for
property 'stepContext': no matching editors or conversion strategy
found
I needed to access jobExecutionId inside ItemProcessor, so I have changed this configuration to:
<bean id="itemProcessor" class="com.myApp.MyItemProcessor" scope="step">
<property name="stepExecution" value="#{stepExecution}"/>
</bean>
Property "stepExecution" is a type org.springframework.batch.core.StepExecution

Related

CrafterCMS: How to use crafter engine properties in an application-context bean?

I am connecting to an external database with a class that extends JdbcTemplate. My problem is that I can't use the globalProperties of the Groovy API because of the Jdbc.
I added these properties I needed in the server-config.properties:
studio.db.driverClassName
studio.db.url
studio.db.username
studio.db.password
I am trying to access them in my application-context.xml with this:
<bean id="jdbc" class="com.dbJdbcTemplate">
<constructor-arg ref="datasource"/>
</bean>
<bean id="datasource" class="org.apache.commons.dbcp.BasicDataSource">
<property name="url" value="${studio.db.url}"/>
<property name="driverClassName" value="${studio.db.driverClassName}"/>
<property name="username" value="${studio.db.username}"/>
<property name="password" value="${studio.db.password}"/>
</bean>
I receive this error:
Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not
get JDBC Connection; nested exception
org.apache.commons.dbcp.SQLNestedException: Cannot load JDBC driver class '${studio.db.driverClassName}'
How do I access the properties from my bean correctly?
Add a <bean class="org.springframework.context.support.PropertySourcesPlaceholderConfigurer" parent="crafter.properties"/> in your site application-context.xml, like is shown in here https://docs.craftercms.org/en/3.0/site-administrators/engine/engine-site-configuration.html#id3. That lines gives you access to Engine's global properties.

postgres: Error querying database

Project run for some time after the database connection is not found, the error message is as follows:
2017-05-02 10:02:17,224 ERROR [main] (line:com.unis.license.agent.management.aop.ManagementServicesAop.afterThrowing(ManagementServicesAop.java:47)) - BeforeMethod:com.unis.license.agent.management.service.impl.LicenseServiceImpl.loadLicenseInfo Params: []
2017-05-02 10:02:17,225 ERROR [main] (line:com.unis.license.agent.management.aop.ManagementServicesAop.afterThrowing(ManagementServicesAop.java:50)) - methodException:com.unis.license.agent.management.service.impl.LicenseServiceImpl.loadLicenseInfo Exception:org.mybatis.spring.MyBatisSystemException: nested exception is org.apache.ibatis.exceptions.PersistenceException:
Error querying database. Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (FATAL: the database system is starting up)
The error may exist in licenseAgent/orm/LicenseInfo.xml
The error may involve com.unis.license.agent.management.dao.ILicenseInfoOper.getAll
The error occurred while executing a query
Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (FATAL: the database system is starting up)
2017-05-02 10:02:17,226 ERROR [main] (line:org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:318)) - Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.unis.license.agent.management.filter.Initialization#0' defined in class path resource [spring-context.xml]: Invocation of init method failed; nested exception is org.springframework.transaction.NoTransactionException: No transaction aspect-managed TransactionStatus in scope
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1553)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:475)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:304)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:300)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:195)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:700)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:760)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:381)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:293)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:106)
at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:799)
at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:446)
at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:791)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:296)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1347)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:743)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:492)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:117)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:99)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:60)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:154)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
My DataSource configuration information is as follows:
<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource">
<property name="driverClassName" value="org.postgresql.Driver"></property>
<property name="url" value="jdbc:postgresql://127.0.0.1:5432/ucsm"></property>
<property name="username" value="uqdm"></property>
<property name="password" value="unis123"></property>
<property name="maxActive" value="100"></property>
<property name="maxIdle" value="30"></property>
<property name="maxWait" value="500"></property>
<property name="defaultAutoCommit" value="true"></property>
</bean>
<bean id="transactionManager"
class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="dataSource" />
</bean>
<tx:annotation-driven transaction-manager="transactionManager" />
<bean id="sqlSessionFactory" class="org.mybatis.spring.SqlSessionFactoryBean">
<property name="configLocation" value="classpath:licenseAgent/mybatiscfg.xml"></property>
<property name="dataSource" ref="dataSource" />
</bean>
<bean class="org.mybatis.spring.mapper.MapperScannerConfigurer">
<property name="basePackage" value="com.unis.license.agent.management.dao" />
</bean>
This is the key part of your stack trace:
FATAL: the database system is starting up
It means that the PostgreSQL server is starting or restarting and not ready to accept connections yet.
You should check the PostgreSQL server log to find the cause of this unexpected start.
Chances are that it is a restart caused by a crashing PostgreSQL server process. Such crashes can be caused (in rough order of likelihood) by buggy server extensions, buggy hardware or PostgreSQL bugs.

Create new output file using FlatFileItemWriter in spring-batch

I have a simple spring batch job - read a file line by line, do something with the input string, and write some output. Output file contains every line of input plus some processing status for that line (success/failure.) The reads a file from: <dir>/<inputFolder>/<inputFileName> and writes processed output to <dir>/<outputFolder>/<inputFileName> All these values are passed as jobParameters
File Reader is like so:
<bean id="itemReader" class="org.springframework.batch.item.file.FlatFileItemReader" scope="step">
<property name="resource" value="file:#{jobParameters['cwd']}/#{jobParameters['inputFolder']}/#{jobParameters['inputFile']}" />
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer">
<bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="delimiter" value="," />
</bean>
</property>
<property name="fieldSetMapper" >
<bean class="org.springframework.batch.item.file.mapping.PassThroughFieldSetMapper" />
</property>
</bean>
</property>
</bean>
Item writer is like so:
<bean id="itemWriter" class="org.springframework.batch.item.file.FlatFileItemWriter" scope="step" >
<property name="resource" value="#{jobParameters['cwd']}/#{jobParameters['outputFolder']}/#{jobParameters['inputFile']}" />
<property name="lineAggregator">
<bean class="org.springframework.batch.item.file.transform.PassThroughLineAggregator" />
</property>
</bean>
When I run this batch job, the reader reads the file properly, processor does its job but FileNotFound exception is thrown by the itemWriter
2014/06/27 18-02-31,168:OUT:ERROR[Encountered an error executing the step]
org.springframework.batch.item.ItemStreamException: Could not convert resource to file: [class path resource [S:/temp/seller-optin-batch/output/sellersToOptin_test.txt]]
at org.springframework.batch.item.file.FlatFileItemWriter.getOutputState(FlatFileItemWriter.java:374)
at org.springframework.batch.item.file.FlatFileItemWriter.open(FlatFileItemWriter.java:314)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...
Caused by: java.io.FileNotFoundException: class path resource [S:/temp/seller-optin-batch/output/sellersToOptin_test.txt] cannot be resolved to URL because it does not exist
at org.springframework.core.io.ClassPathResource.getURL(ClassPathResource.java:179)
at org.springframework.core.io.AbstractFileResolvingResource.getFile(AbstractFileResolvingResource.java:48)
at org.springframework.batch.item.file.FlatFileItemWriter.getOutputState(FlatFileItemWriter.java:371)
... 58 more
2014/06/27 18-02-31,168:ERR:ERROR[Encountered an error executing the step]
[org.springframework.batch.item.file.FlatFileItemWriter.getOutputState threw org.springframework.batch.item.ItemStreamException: Could not convert resource to file: [class path resource [S:/temp/seller-optin-batch/output/sellersToOptin_test.txt]]]
Batch Execution Failed!
Everytime the batch job runs, the output file does not exist yet. The itemWriter has to create it. Is it not possible using FlatFileItemWriter ?
Adding 'file://' prefix solved my problem. Thanks #LucaBassoRicci.

spring-data embadded-database configuration

I try to configure springdata with embedded-database. Context file inspired from http://cooldevstuff.wordpress.com/2012/09/20/in-memory-database-using-spring-3-2/
My context file:
<jdbc:embedded-database id="embeddedDataSource">
<jdbc:script location="classpath:schemaFile.sql" />
<jdbc:script location="classpath:dataFile.sql" />
</jdbc:embedded-database>
<jpa:repositories base-package="aa.bb.repository" />
<bean class="org.springframework.orm.hibernate4.HibernateExceptionTranslator" />
<bean id="entityManagerFactory">
<property name="dataSource" ref="embeddedDataSource" />
<property name="persistenceUnitName" value="SamplePU"></property>
</bean>
but I get error:
No unique bean of type [javax.persistence.EntityManagerFactory] is defined: expected single bean but found 0.
what I missed?
The configuration is just invalid. Your bean entityManagerFactory does not declare any class attribute.

No bean named 'springbatch.readerDataSource' is defined

While executing my job I am getting below exception
Cannot resolve reference to bean 'springbatch.readerDataSource' while setting bean property 'dataSource'; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'springbatch.readerDataSource' is defined
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:329)
Note -- I am not creating seperate reader file .Just using JdbcCursorItemReader.
My configuration file
<bean id="itemReader"
class="org.springframework.batch.item.database.JdbcCursorItemReader" scope="step">
<property name="dataSource" ref="springbatch.batchDataSource"/>
<property name="sql"
value=
"select Cust_Id from Customer "/>
<property name="rowMapper">
<bean class="com.insurance.premiumrecalculation.batch.CustDto" />
</property>
</bean>
<bean id="policy.premium.recalculation.PremiumRecalculationWriter"
class="com.insurance.premiumrecalculation.batch.PremiumRecalculationProcessWriter" scope="step"/>
<batch:job id="policy.job.premiumRecalculation"
job-repository="springbatch.jobRepository" parent="springbatch.job.baseJob">
<batch:step id="policy.step.premiumrecalculation" parent="springbatch.step.baseStep">
<batch:tasklet allow-start-if-complete="false" transaction-manager="powTransactionManager">
<batch:chunk commit-interval="10"
reader="itemReader"
writer="policy.premium.recalculation.PremiumRecalculationWriter"/>
</batch:tasklet>
</batch:step>
</batch:job>
Thanks in advance
The source of this error is the following line:
<property name="dataSource" ref="springbatch.batchDataSource"/>
Which means that you need a bean definition with id "springbatch.batchDataSource" configured as a data source for your environment which seems to not exist. You may use the following as a template; do not forget to provide your database driver and connection information.
<bean id="springbatch.batchDataSource" class="org.apache.commons.dbcp.BasicDataSource">
<property name="driverClassName" value="org.hsqldb.jdbcDriver" />
<property name="url" value="jdbc:hsqldb:mem:testdb" />
<property name="username" value="sa" />
<property name="password" value="" />
</bean>