Spring Batch Using CompositeItemWriter and CompositeItemProcessor - spring-batch

Using Spring Batch, I have to write in two different table, but using the same ItemReader.
I can't figure out how to use one ItemReader and a CompositeItemWriter.
Here's the JobConfiguration :
public class JobConfiguration {
#Autowired
#SuppressWarnings("squid:S3305")
private ItemReaderSurveillance itemReaderSurveillance;
#Autowired
#SuppressWarnings("squid:S3305")
private ItemWriterSurveillance itemWriterSurveillance;
#Autowired
#SuppressWarnings("squid:S3305")
private ItemWriterSurveillanceEcheance itemWriterSurveillanceEcheance;
#Autowired
#SuppressWarnings("squid:S3305")
private CompositeItemProcessorSurveillance compositeItemProcessor;
#Bean(name = "importSurveillanceJob")
public Job job(JobBuilderFactory jobs) {
return jobs.get("importSurveillanceStep")
.listener(jobListener)
.start(stepTaskletCreationRepertoireReport())
.next(stepTaskletCreationRepertoireArchive())
.next(stepSurveillanceReadProcessWrite())
.next(stepZipFile())
.build();
}
#Bean
protected Step stepSurveillanceReadProcessWrite() {
return stepBuilderFactory.get("stepSurveillanceReadProcessWrite")
.<SurveillanceLineFile, CompositeResultSurveillance>chunk(Integer.valueOf(commitInterval))
.reader(itemReaderSurveillance)
.processor(compositeItemProcessor)
.writer(compositeItemWriter())
.faultTolerant()
.retryLimit(0)
.build();
}
#Bean
public CompositeItemWriter<CompositeResultSurveillance> compositeItemWriter(){
CompositeItemWriter compositeItemWriter = new CompositeItemWriter();
compositeItemWriter.setDelegates(Arrays.asList(itemWriterSurveillance, itemWriterSurveillanceEcheance));
return compositeItemWriter;
}
}
Item Writers :
#Slf4j
#StepScope
#Component
public class ItemWriterSurveillance implements ItemWriter<FoaSurveillance>, StepExecutionListener {
String fileName;
JobExecution mJobExecution;
StepExecution stepExecution;
#Autowired
private FoaSurveillanceDao foaSurveillanceDao;
#Override
public void write(List<? extends FoaSurveillance> foaSurveillances) {
ExecutionContext executionContext = stepExecution.getExecutionContext();
// Process data
}
#Override
public void beforeStep(StepExecution stepExecution) {
mJobExecution = stepExecution.getJobExecution();
this.stepExecution = stepExecution;
}
#Override
public ExitStatus afterStep(StepExecution stepExecution) {
return ExitStatus.COMPLETED;
}
}
#Slf4j
#StepScope
#Component
public class ItemWriterSurveillanceEcheance implements ItemWriter<FoaSurveillanceEcheance>, StepExecutionListener {
String fileName;
JobExecution mJobExecution;
StepExecution stepExecution;
#Autowired
private FoaSurveillanceEcheanceDao foaSurveillanceEcheanceDao;
#Override
public void write(List<? extends FoaSurveillanceEcheance> foaSurveillanceEcheances) {
ExecutionContext executionContext = stepExecution.getExecutionContext();
// Process data
}
#Override
public void beforeStep(StepExecution stepExecution) {
mJobExecution = stepExecution.getJobExecution();
this.stepExecution = stepExecution;
}
#Override
public ExitStatus afterStep(StepExecution stepExecution) {
return ExitStatus.COMPLETED;
}
}
Compose the two itemProcessor :
#Slf4j
#Component
public class CompositeItemProcessorSurveillance implements ItemProcessor<SurveillanceLineFile, CompositeResultSurveillance>, StepExecutionListener {
private StepExecution stepExecution;
#Autowired
ItemProcessorSurveillance itemProcessorSurveillance;
#Autowired
ItemProcessorSurveillanceEcheance itemProcessorSurveillanceEcheance;
#Override
public CompositeResultSurveillance process(SurveillanceLineFile surveillanceLineFile) throws Exception {
CompositeResultSurveillance compositeResultSurveillance = new CompositeResultSurveillance();
compositeResultSurveillance.setFoaSurveillance(itemProcessorSurveillance.process(surveillanceLineFile));
compositeResultSurveillance.setFoaSurveillanceEcheance(itemProcessorSurveillanceEcheance.process(surveillanceLineFile));
return compositeResultSurveillance;
}
#Override
public void beforeStep(StepExecution stepExecution) {
this.stepExecution = stepExecution;
}
#Override
public ExitStatus afterStep(StepExecution stepExecution) {
return null;
}
}
The ItemProcessorSurveillance :
#Slf4j
#Component
public class ItemProcessorSurveillance implements ItemProcessor<SurveillanceLineFile, FoaSurveillance>, StepExecutionListener {
String fileName;
private StepExecution stepExecution;
#Override
public void beforeStep(StepExecution stepExecution) {
this.stepExecution = stepExecution;
}
#Override
public ExitStatus afterStep(StepExecution stepExecution) {
return ExitStatus.COMPLETED;
}
#Override
public FoaSurveillance process(SurveillanceLineFile surveillanceLineFile) throws Exception {
ExecutionContext executionContext = stepExecution.getExecutionContext();
// Process Data
}
And the CompositeResult returned by the Processor :
#Getter
#Setter
public class CompositeResultSurveillance {
private FoaSurveillance foaSurveillance;
private FoaSurveillanceEcheance foaSurveillanceEcheance;
}
For now I've got a NPE on the ItemProcessorSurveillance because stepExecution is null on the process method.
I can't figure out what's wrong. Any help ?

This is because your ItemProcessorSurveillance implements two interfaces: ItemProcessor and StepExecutionListener but is only registered as an ItemProcessor in the step. It should also be registered as a listener so that beforeStep is called when appropriate to set the stepExecution field.

Related

Configuration for Mvc testing

guys. I have spring MVC project and I want to test CoursesController, but can not find out how to do it.
Do I need to make separated configuration class for tests?
Before springMvc I used separated configuration class for test with embedded database.
I'd appreciate all the help I can get.
CoursesController class:
#Controller
#RequestMapping("/courses")
public class CoursesController {
private final CourseService courseService;
#Autowired
public CoursesController(CourseService courseService) {
this.courseService = courseService;
}
#GetMapping()
public String index(Model model, #RequestParam("page") Optional<Integer> page,
#RequestParam("size") Optional<Integer> size) throws ServiceException {
int currentPage = page.orElse(1);
int pageSize = size.orElse(10);
Page<Course> coursePage = courseService.findPaginated(PageRequest.of(currentPage - 1, pageSize));
model.addAttribute("coursePage", coursePage);
int totalPages = coursePage.getTotalPages();
if (totalPages > 0) {
List<Integer> pageNumbers = IntStream.rangeClosed(1, totalPages).boxed().collect(Collectors.toList());
model.addAttribute("pageNumbers", pageNumbers);
}
return "courses/index";
}
}
Configuration class:
#Configuration
#ComponentScan("com.university")
#PropertySource("classpath:/application.properties")
#EnableWebMvc
public class Config implements WebMvcConfigurer {
#Autowired
private Environment env;
private final ApplicationContext applicationContext;
#Autowired
public Config(ApplicationContext applicationContext) {
this.applicationContext = applicationContext;
}
#Bean
public HikariDataSource dataSource() {
return (HikariDataSource) DataSourceBuilder.create().type(HikariDataSource.class)
.url(env.getProperty("spring.datasource.url"))
.driverClassName(env.getProperty("spring.datasource.driverClassName"))
.username(env.getProperty("spring.datasource.username"))
.password(env.getProperty("spring.datasource.password")).build();
}
#Bean
public JdbcTemplate jdbcTemplate() {
return new JdbcTemplate(dataSource());
}
#Bean
public SpringResourceTemplateResolver templateResolver() {
SpringResourceTemplateResolver templateResolver = new SpringResourceTemplateResolver();
templateResolver.setApplicationContext(applicationContext);
templateResolver.setPrefix("/WEB-INF/views/");
templateResolver.setSuffix(".html");
return templateResolver;
}
#Bean
public SpringTemplateEngine templateEngine() {
SpringTemplateEngine templateEngine = new SpringTemplateEngine();
templateEngine.setTemplateResolver(templateResolver());
templateEngine.setEnableSpringELCompiler(true);
return templateEngine;
}
#Override
public void configureViewResolvers(ViewResolverRegistry registry) {
ThymeleafViewResolver resolver = new ThymeleafViewResolver();
resolver.setTemplateEngine(templateEngine());
registry.viewResolver(resolver);
}
#Bean
public SessionLocaleResolver localeResolver() {
SessionLocaleResolver localeResolver = new SessionLocaleResolver();
localeResolver.setDefaultLocale(Locale.ENGLISH);
return localeResolver;
}
}
I did separate configuration class for tests with H2 database.

How to write junit test for graceful shutdown in spring boot

I'm new to code coverage. I have started to write test cases for my spring boot application.
The below highlighted part in red, I'm unable to cover. Could you please suggest how to test these?
Here is my code.
#SpringBootApplication
public class ImsApplication extends SpringBootServletInitializer implements WebApplicationInitializer {
#Override
protected SpringApplicationBuilder configure(SpringApplicationBuilder builder) {
return builder.sources(ImsApplication.class);
}
public static void contextDestroyed(ConfigurableApplicationContext ctx) {
int exitCode = SpringApplication.exit(ctx, new ExitCodeGenerator() {
#Override
public int getExitCode() {
return 0;
}
});
System.exit(exitCode);
}
public static void main(String[] args) {
ConfigurableApplicationContext ctx = SpringApplication.run(ImsApplication.class, args);
contextDestroyed(ctx);
}
public ImsApplication() {
super();
setRegisterErrorPageFilter(false);
}
#PostConstruct
public void init() {
TimeZone.setDefault(TimeZone.getTimeZone("IST"));
}
}

Issue retrieving a ExecutionContext from a SkipListener

I am trying to retrieve a spring batch ExecutionContext from a SkipListener.
Here is what I have attempted (I rely on annotations instead of interfaces in order to implement my listeners):
import com.xxxx.domain.UserAccount;
import lombok.extern.slf4j.Slf4j;
import org.springframework.batch.core.StepExecution;
import org.springframework.batch.core.annotation.BeforeStep;
import org.springframework.batch.core.annotation.OnSkipInWrite;
import org.springframework.mail.MailSendException;
import org.springframework.stereotype.Component;
#Slf4j
#Component
public class MailSkipListener {
private StepExecution stepExecution;
#BeforeStep
public void saveStepExecution(StepExecution stepExecution) {
this.stepExecution = stepExecution;
}
#OnSkipInWrite
public void logSkippedEmail(UserAccount userAccount, Throwable t) {
if (t instanceof MailSendException) {
MailSendException e = (MailSendException) t;
log.warn("FailedMessages: " + e.getFailedMessages());
}
}
}
However, the logSkippedEmail method is never executed when a MailSendException is raised. When I remove the saveStepExecution method, the logSkippedEmail is again executed in case of a MailSendException.
I register my MailSkipListener as follows:
#Bean
public Step messagesDigestMailingStep(EntityManagerFactory entityManagerFactory) {
return stepBuilderFactory
.get("messagesDigestMailingStep")
.<UserAccount, UserAccount>chunk(5)
...
.writer(itemWriter)
.listener(mailSkipListener)//Here
.build();
}
What I am trying to achieve here is retrieving an ExecutionContext from my SkipListener. How can this be achieved? It seems there's no way to autowire theExecutionContext.
This is quite an old question, but I just struggled with this too.
I ended up registering the skiplistener twice in order for it to work, once as a StepExecutionListener and another as a SkipListener.
It sucks, but it seems to work:
#Bean
public Step messagesDigestMailingStep(EntityManagerFactory entityManagerFactory) {
return stepBuilderFactory
.get("messagesDigestMailingStep")
.<UserAccount, UserAccount>chunk(5)
...
.writer(itemWriter)
.listener((StepExecutionListener) mailSkipListener) // <--- 1
.listener((SkipListener) mailSkipListener) // <--- 2
.build();
}
I know this is an old Question, but I had to deal with this myself, and have put together the following implementation, wherein I made the SkipListener also implement the StepExecutionListener, and adding the same class as both the SkipListener and StepExecutionListener.
#Component
public class PersonImportListener implements SkipListener<Person, Person>, StepExecutionListener {
private final Logger logger = LoggerFactory.getLogger(this.getClass());
private StepExecution stepExecution;
#Override
public void beforeStep(StepExecution stepExecution) {
this.stepExecution = stepExecution;
}
#Override
public ExitStatus afterStep(StepExecution stepExecution) {
return ExitStatus.COMPLETED;
}
#Override
public void onSkipInRead(Throwable throwable) {
logger.warn("Line skipped on read", throwable);
}
#Override
public void onSkipInWrite(Person person, Throwable throwable) {
logger.warn("Bean skipped on write", throwable);
logger.warn("Execution Context" + stepExecution);
}
#Override
public void onSkipInProcess(Person person, Throwable throwable) {
logger.warn("Bean skipped on process", throwable);
}
}
And use this class as a listener for StepExecutionListener and also SkipListener.
#Bean
public Step step1(JdbcBatchItemWriter<Person> writer) {
PersonImportListener listener = new PersonImportListener();
return stepBuilderFactory.get("step1")
.<Person, Person> chunk(10)
.reader(reader())
.faultTolerant()
.skipLimit(10)
.skip(DataIntegrityViolationException.class)
.listener((StepExecutionListener) listener)
.listener((SkipListener) listener)
.processor(processor())
.writer(writer)
.build();
}
You can implement StepExecutionListener on your MailSkipListener to save the context in your stepExecution during the beforeStep() method :
public class MailSkipListener implements StepExecutionListener {
#Override
public void beforeStep(StepExecution stepExecution) {
this.stepExecution = stepExecution;
}

Save LocaTime via SpringData to MongoDB

I try to save a LocalTime (joda) field to the MongoDB with SpringData using spring-boot-starter-parent (org.springframework.boot 1.2.3.RELEASE) and get a StackOverflowError.
The StackOverflowError is in BeanWrapper in the method
public <S> S getProperty(PersistentProperty<?> property, Class<? extends S> type)
Stacktrace:
http-nio-8080-exec-2#5509 daemon, prio=5, in group 'main', status: 'RUNNING'
at org.springframework.data.mapping.model.BeanWrapper.getProperty(BeanWrapper.java:120)
at org.springframework.data.mapping.model.BeanWrapper.getProperty(BeanWrapper.java:100)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter$3.doWithPersistentProperty(MappingMongoConverter.java:419)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter$3.doWithPersistentProperty(MappingMongoConverter.java:412)
at org.springframework.data.mapping.model.BasicPersistentEntity.doWithProperties(BasicPersistentEntity.java:307)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.writeInternal(MappingMongoConverter.java:412)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.writePropertyInternal(MappingMongoConverter.java:511)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter$3.doWithPersistentProperty(MappingMongoConverter.java:424)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter$3.doWithPersistentProperty(MappingMongoConverter.java:412)
at org.springframework.data.mapping.model.BasicPersistentEntity.doWithProperties(BasicPersistentEntity.java:307)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.writeInternal(MappingMongoConverter.java:412)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.writePropertyInternal(MappingMongoConverter.java:511)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter$3.doWithPersistentProperty(MappingMongoConverter.java:424)
at org.springframework.data.mongodb.core.convert.MappingMongoConverter$3.doWithPersistentProperty(MappingMongoConverter.java:412)
at org.springframework.data.mapping.model.BasicPersistentEntity.doWithProperties(BasicPersistentEntity.java:307)...
Adding these two Converters to the CustomConversions fix the problem.
#Configuration
public class MongoConfiguration extends AbstractMongoConfiguration {
#Override
protected String getDatabaseName() {
return "databasename";
}
#Override
public Mongo mongo() throws Exception {
return new MongoClient("localhost");
}
#Override
public CustomConversions customConversions() {
List<Converter<?, ?>> converters = new ArrayList<>();
converters.add(new LocalTimeToStringConverter());
converters.add(new StringToLocalTimeConverter());
return new CustomConversions(converters);
}
}
public class LocalTimeToStringConverter implements Converter<LocalTime, String> {
#Override
public String convert(LocalTime localTime) {
return localTime.toString();
}
}
public class StringToLocalTimeConverter implements Converter<String, LocalTime> {
#Override
public LocalTime convert(String s) {
return LocalTime.parse(s);
}
}

Not able to access stepExecutionContext value in writer

I am setting the stepExecutionContext value in my partitioner and trying to get it in Writer.
But i could not able to access it.
The writer is step scoped.
Could any one help me how to get the step execution context values in writer?
Thanks
Shankar
you can implement StepExecutionListener in writer to get stepExecution in writer
public class ExampleWriter implements ItemWriter<T>,StepExecutionListener {
private JobExecution jobExecution;
#Override
public void write(List<? extends T> items) {
String executionContextValue=jobExecution.getExecutionContext().get("KEY");
System.out.println("ExecutionContextValue is:"+executionContextValue);
}
#Override
public ExitStatus afterStep(StepExecution stepExecution) {
return null;
}
#Override
public void beforeStep(StepExecution stepExecution) {
this.jobExecution= stepExecution.getJobExecution();
}
}
If needed register this class as a listener in xml configuration
<listener>ExampleWriter</listener>