Current flow:
1.BatchItemReader implements ItemReader<List<SingleJsonRowInput>>
2.BatchItemProcessor implements ItemProcessor<List<SingleJsonRowInput>>
3.BatchItemWriter implements ItemWriter<List<String>>
The input is a text file with each row represent a Json file. currently the program runs well with a single file, i would like to implement MultiResourceItemReader but since my reader doesn't imlement this ResourceAwareItemReaderItemStream - it cannot be applied to MultiResourceItemReader. i tried:
1. Implementing ResourceAwareItemReaderItemStream
2. Changing my reader to be FlatFileItemReader as advised here:
Spring Batch: How to setup a FlatFileItemReader to read a json file?
but failed to do so.
Reader:
public class BatchItemReader implements ItemReader<List<SingleJsonRowInput>>{
private int count = 0;
private FileManager fileManager;
private Gson gson = new Gson();
public List<SingleJsonRowInput> read() {
return readLine();
}
public BatchItemReader(FileManager fileManager) {
this.fileManager = fileManager;
}
private List<SingleJsonRowInput> readLine() {
List<String> result = fileManager.readTextJsonFile("C:\\Users\\orenl\\Desktop\\small.json");
List<SingleJsonRowInput> singles = new LinkedList<>();
SingleJsonRowInput singleJsonRowInput = null;
for (String line : result) {
System.out.println("#### Reading line: " + line);
singleJsonRowInput = gson.fromJson(line, SingleJsonRowInput.class);
singles.add(singleJsonRowInput);
}
if (count > 5) {
return null;
}
count++;
return singles;
}
}
MultiResourceItemReader:
#Bean
public MultiResourceItemReader<SingleJsonRowInput> multiResourceItemReader(){
Resource resources[]=new Resource[]{new FileSystemResource("small.json")};
MultiResourceItemReader<SingleJsonRowInput> multiResourceItemReader=new MultiResourceItemReader<>();
multiResourceItemReader.setResources(resources);
multiResourceItemReader.setDelegate(new FlatFileItemReader<>());
return multiResourceItemReader;
}
Related
I am trying to read files from AWS s3 using spring batch but the file name becomes null in stepExecutionContext. Same code was working when i read the files from the windows mount but when we migrate the code and reading it from S3 it is becoming null.
#Bean
#JobScope
public CustomMultiResourcePartitioner partitioner() {
CustomMultiResourcePartitioner partitioner = new CustomMultiResourcePartitioner();
Set<String > filesToProcess= fileRepository.findAllFilesByFileState("NEW");
List<Resource> resourceList = new ArrayList<>();
for(String file:filesToProcess) {
Resource resource = getS3Resource(file);
resourceList.add(resource);
log.info("resourceList Size"+resourceList.size());
}
if(resourceList.size()>0 && resourceList.toArray()!=null) {
resources = resourceList.stream().toArray(Resource[]::new);
ExecutionContext executionContext = new ExecutionContext();
executionContext.put("FILE_NAME",filesToProcess);
}
else
{
resources = new Resource[0];
}
partitioner.setResources(resources);
return partitioner;
}
#Bean
#StepScope
public FlatFileItemReader<RosterInput> itemReader(#Value("#{stepExecutionContext[fileName]}") String filename) throws UnexpectedInputException, ParseException {
FlatFileItemReader<RosterInput> reader = new FlatFileItemReader<RosterInput>();
DelimitedLineTokenizer tokenizer = new DelimitedLineTokenizer();
tokenizer.setStrict(false);
Resource resource =getS3Resource(filename);
}
I was using ByteArrayResource in my getS3Resource() method it was causing the file name to null. After modifying the code to use below the problem is solved.
public class FileNameAwareByteArrayResource extends ByteArrayResource {
private String fileName;
public FileNameAwareByteArrayResource(String fileName, byte[] byteArray) {
super(byteArray);
this.fileName = fileName;
}
#Override
public String getFilename() {
return fileName;
}
}
I've been building a Spring integration Service Email using Java DSL.
This service must have a recovery policy in order to retry sending the emails but I'm not getting success.
A brief story: The application recieve a Payload and Header and try to send to email server. It tries 3 times and in case of failure, it creates a new file with Header and Body of message.
How could I get the original Message(Header and Payload) and put the information pair in a json file, in case of failure to send the email?
Thanks.
This is my beans and the service:
/**
* #################
* MESSAGE ENDPOINTS
* #################
*/
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata poller() {
return Pollers
.fixedRate(NumberUtils.createLong(QUEUE_RATE))
.maxMessagesPerPoll(NumberUtils.createLong(QUEUE_CAPACITY))
.errorHandler(e -> LOG.error("Exception : " + e.getMessage()))
.get();
}
#Bean
public MessageChannel recoveryChannel() {
return MessageChannels.direct().get();
}
#MessagingGateway
public static interface MailService {
#Gateway(requestChannel = "mail.input")
void sendMail(String body, #Headers Map<String,String> headers);
}
#Bean
public RetryPolicy retryPolicy() {
final Map<Class<? extends Throwable>, Boolean> map =
new HashMap<Class<? extends Throwable>, Boolean>() {
{
put(MailSendException.class,true);
put(RuntimeException.class, true);
}
private static final long serialVersionUID = -1L;
};
final RetryPolicy ret = new SimpleRetryPolicy(3, map, true);
return ret;
}
#Bean
public RetryTemplate retryTemplate() {
final RetryTemplate ret = new RetryTemplate();
ret.setRetryPolicy(retryPolicy());
ret.setThrowLastExceptionOnExhausted(false);
return ret;
}
#Bean
public Advice retryAdvice() {
final RequestHandlerRetryAdvice advice = new RequestHandlerRetryAdvice();
advice.setRetryTemplate(retryTemplate());
RecoveryCallback<Object> recoveryCallBack = new ErrorMessageSendingRecoverer(recoveryChannel());
advice.setRecoveryCallback(recoveryCallBack);
return advice;
}
private MailSendingMessageHandlerSpec mailOutboundAdapter(){
MailSendingMessageHandlerSpec msmhs =
Mail.outboundAdapter(emailServerHost())
.port(serverPort())
.credentials(MAIL_USER_NAME, MAIL_PASSWORD)
.protocol(emailProtocol())
.javaMailProperties(p -> p
.put("mail.debug", "true")
.put("mail.smtp.ssl.enable",enableSSL())
.put("mail.smtp.connectiontimeout", 5000)
.put("mail.smtp.timeout", 5000));
return msmhs;
}
#Bean
public FileWritingMessageHandler fileOutboundAdapter(){
FileWritingMessageHandler fwmhs = Files
.outboundAdapter(new File("logs/errors/"))
.autoCreateDirectory(true)
.get();
return fwmhs;
}
/**
* ################
* FLOWS
* ################
*/
#Bean
public IntegrationFlow smtp(){
return IntegrationFlows.from("mail.input")
.channel(MessageChannels.queue())
.handle(this.mailOutboundAdapter(),
e -> e.id("smtpOut")
.advice(retryAdvice())
)
.get();
}
#Bean
public IntegrationFlow errorFlow(){
return IntegrationFlows.from(recoveryChannel())
.transform(Transformers.toJson())
.enrichHeaders(c -> c.header(FileHeaders.FILENAME, "emailErrors"))
.handle(this.fileOutboundAdapter())
.get();
}
}
The error message has a payload MessagingException. It has two properties cause and failedMessage.
The failed message is the message at the point of failure, with headers and payload.
I have a weird issue with my FlatFileItemWriter callbacks.
I have a custom ItemWriter implementing both FlatFileFooterCallback and FlatFileHeaderCallback. Consequently, I set header and footer callbacks in my FlatFileItemWriter like this :
ItemWriter Bean
#Bean
#StepScope
public ItemWriter<CityItem> writer(FlatFileItemWriter<CityProcessed> flatWriter, #Value("#{jobExecutionContext[inputFile]}") String inputFile) {
CityItemWriter itemWriter = new CityItemWriter();
flatWriter.setHeaderCallback(itemWriter);
flatWriter.setFooterCallback(itemWriter);
itemWriter.setDelegate(flatWriter);
itemWriter.setInputFileName(inputFile);
return itemWriter;
}
FlatFileItemWriter Bean
#Bean
#StepScope
public FlatFileItemWriter<CityProcessed> flatFileWriterArchive(#Value("#{jobExecutionContext[outputFileArchive]}") String outputFile) {
FlatFileItemWriter<CityProcessed> flatWriter = new FlatFileItemWriter<CityProcessed>();
FileSystemResource isr;
isr = new FileSystemResource(new File(outputFile));
flatWriter.setResource(isr);
DelimitedLineAggregator<CityProcessed> aggregator = new DelimitedLineAggregator<CityProcessed>();
aggregator.setDelimiter(";");
BeanWrapperFieldExtractor<CityProcessed> beanWrapper = new BeanWrapperFieldExtractor<CityProcessed>();
beanWrapper.setNames(new String[]{
"country", "name", "population", "popUnder25", "pop25To50", "pop50to75", "popMoreThan75"
});
aggregator.setFieldExtractor(beanWrapper);
flatWriter.setLineAggregator(aggregator);
flatWriter.setEncoding("ISO-8859-1");
return flatWriter;
}
Step Bean
#Bean
public Step stepImport(StepBuilderFactory stepBuilderFactory, ItemReader<CityFile> reader, ItemWriter<CityItem> writer, ItemProcessor<CityFile, CityItem> processor,
#Qualifier("flatFileWriterArchive") FlatFileItemWriter<CityProcessed> flatFileWriterArchive, ExecutionContextPromotionListener executionContextListener) {
return stepBuilderFactory.get("stepImport").<CityFile, CityItem> chunk(10).reader(reader(null)).processor(processor).writer(writer).stream(flatFileWriterArchive)
.listener(executionContextListener).build();
}
I have the classic content in my writeFooter, writeHeader and write methods.
ItemWriter code
public class CityItemWriter implements ItemWriter<CityItem>, FlatFileFooterCallback, FlatFileHeaderCallback, ItemStream {
private FlatFileItemWriter<CityProcessed> writer;
private static int totalUnknown = 0;
private static int totalSup10000 = 0;
private static int totalInf10000 = 0;
private String inputFileName = "-";
public void setDelegate(FlatFileItemWriter<CityProcessed> delegate) {
writer = delegate;
}
public void setInputFileName(String name) {
inputFileName = name;
}
private Predicate<String> isNullValue() {
return p -> p == null;
}
#Override
public void write(List<? extends CityItem> cities) throws Exception {
List<CityProcessed> citiesCSV = new ArrayList<>();
for (CityItem item : cities) {
String populationAsString = "";
String less25AsString = "";
String more25AsString = "";
/*
* Some processing to get total Unknown/Sup 10000/Inf 10000
* and other data
*/
// Write in CSV file
CityProcessed cre = new CityProcessed();
cre.setCountry(item.getCountry());
cre.setName(item.getName());
cre.setPopulation(populationAsString);
cre.setLess25(less25AsString);
cre.setMore25(more25AsString);
citiesCSV.add(cre);
}
writer.write(citiesCSV);
}
#Override
public void writeFooter(Writer fileWriter) throws IOException {
String newLine = "\r\n";
String totalUnknown= "Subtotal:;Unknown;" + String.valueOf(nbUnknown) + newLine;
String totalSup10000 = ";Sum Sup 10000;" + String.valueOf(nbSup10000) + newLine;
String totalInf10000 = ";Sum Inf 10000;" + String.valueOf(nbInf10000) + newLine;
String total = "Total:;;" + String.valueOf(nbSup10000 + nbInf10000 + nbUnknown) + newLine;
fileWriter.write(newLine);
fileWriter.write(totalUnknown);
fileWriter.write(totalSup10000);
fileWriter.write(totalInf10000);
fileWriter.write(total );
}
#Override
public void writeHeader(Writer fileWriter) throws IOException {
String newLine = "\r\n";
String firstLine= "FILE PROCESSED ON: ;" + new SimpleDateFormat("MM/dd/yyyy").format(new Date()) + newLine;
String secondLine= "Filename: ;" + inputFileName + newLine;
String colNames= "Country;Name;Population...;...having less than 25;...having more than 25";
fileWriter.write(firstLine);
fileWriter.write(secondLine);
fileWriter.write(newLine);
fileWriter.write(colNames);
}
#Override
public void close() throws ItemStreamException {
writer.close();
}
#Override
public void open(ExecutionContext context) throws ItemStreamException {
writer.open(context);
}
#Override
public void update(ExecutionContext context) throws ItemStreamException {
writer.update(context);
}
}
When I run my batch, I only have the data for each city (write method part) and the footer lines. If I comment the whole content of write method and footer callback, I still don't have the header lines. I tried to add a System.out.println() text in my header callback, it looks like it's never called.
Here is an example of the CSV file produced by my batch :
France;Paris;2240621;Unknown;Unknown
France;Toulouse;439553;Unknown;Unknown
Spain;Barcelona;1620943;Unknown;Unknown
Spain;Madrid;3207247;Unknown;Unknown
[...]
Subtotal:;Unknown;2
;Sum Sup 10000;81
;Sum Inf 10000;17
Total:;;100
What is weird is that my header used to work before, when I added both footer and header callbacks. I didn't change them, and I don't see what I've done in my code to "broke" my header callback... And of course, I have no save of my first code. Because I see only now that my header has disappeared (I checked my few last files, and it looks like my header is missing for some time but I didn't see it), I can't just remove my modifications to see when/why it happens.
Do you have any idea to solve this problem ?
Thanks
When using Java config as you are, it's best to return the most specific type possible (the opposite of what you're normally told to do in java programming). In this case, your writer is returning ItemWriter, but is step scoped. Because of this a proxy is created that can only see the type that your java config returns which in this case is ItemWriter and does not expose the methods on the ItemStream interface. If you return CityItemWriter, I'd expect things to work.
I wrote an annotator that extracts all CD tagged tokens and the code looks like this
public class WeightAnnotator extends JCasAnnotator_ImplBase {
private Logger logger = Logger.getLogger(getClass().getName());
public static List<Token> weightTokens = new ArrayList<Token>();
public static final String PARAM_STRING = "stringParam";
#ConfigurationParameter(name = PARAM_STRING)
private String stringParam;
#Override
public void initialize(UimaContext context) throws ResourceInitializationException {
super.initialize(context);
}
#Override
public void process(JCas jCas) throws AnalysisEngineProcessException {
logger.info("Starting processing.");
for (Sentence sentence : JCasUtil.select(jCas, Sentence.class)) {
List<Token> tokens = JCasUtil.selectCovered(jCas, Token.class, sentence);
for (Token token : tokens) {
int begin = token.getBegin();
int end = token.getEnd();
if (token.getPos().equals( PARAM_STRING)) {
WeightAnnotation ann = new WeightAnnotation(jCas, begin, end);
ann.addToIndexes();
System.out.println("Token: " + token.getCoveredText());
}
}
}
}
}
but when I am trying to create an iterator on it in a pipeline, the iterator is returning null. Here is how my pipeline looks.
AnalysisEngineDescription weightDescriptor = AnalysisEngineFactory.createEngineDescription(WeightAnnotator.class,
WeightAnnotator.PARAM_STRING, "CD"
);
AggregateBuilder builder = new AggregateBuilder();
builder.add(sentenceDetectorDescription);
builder.add(tokenDescriptor);
builder.add(posDescriptor);
builder.add(weightDescriptor);
builder.add(writer);
for (JCas jcas : SimplePipeline.iteratePipeline(reader, builder.createAggregateDescription())) {
Iterator iterator1 = JCasUtil.iterator(jcas, WeightAnnotation.class);
while (iterator1.hasNext()) {
WeightAnnotation weights = (WeightAnnotation) iterator1.next();
logger.info("Token: " + weights.getCoveredText());
}
}
I generated WeightAnnotator and WeightAnnotator_Type using JCasGen. I debugged the entire code but I don't understand where I am getting this wrong. Any ideas on how to improve this are appreciated.
I am using spring data mongo-db 1.4.1.RELEASE.
My entity 'Event' has a getter method which is calculated based on other properties:
public int getStatus() {
return (getMainEventId() == null) ? (elapseTimeInMin() < MINIMUM_TIME ? CANDIDATE :
VALID) : POINTER;
}
I wanted the property 'status' to be persisted only through the getter ,so I wrote converters:
#WritingConverter
public class EventWriteConverter implements Converter<Event ,BasicDBObject > {
static final Logger logger = LoggerFactory.getLogger(EventWriteConverter.class.getCanonicalName());
public BasicDBObject convert(Event event) {
logger.info("converting " +event );
if (event.getMainEventId() != null)
return new BasicDBObject("mainEventId", event.getMainEventId() );
BasicDBObject doc = new BasicDBObject("status",event.getStatus()).
append("updated_date",new Date()).
append("start",event.getS0()).
append("end",event.getS1()).
append("location",event.getLocation()).
;
BasicDBList list = new BasicDBList();
doc.append("access_points",event.getHotPoints());
return doc;
}
#ReadingConverter
public class EventReadConverter implements Converter<BasicDBObject, Event> {
#Inject
HotPointRepositry hotRepositry;
static final Logger logger = LoggerFactory.getLogger(EventReadConverter.class.getCanonicalName());
public Event convert(BasicDBObject doc) {
logger.info(" converting ");
Event event = new Event();
event.setId(doc.getObjectId("_id"));
event.setS0(doc.getDate("start"));
event.setS1(doc.getDate("end"));
BasicDBList dblist = (BasicDBList) doc.get("hot_points");
if (dblist != null) {
for (Object obj : dblist) {
ObjectId hotspotId = ((BasicDBObject) obj).getObjectId("_id");
event.addHot(hotRepositry.findOne(hotId));
}
}
dblist = (BasicDBList) doc.get("devices");
if (dblist != null) {
for (Object obj : dblist)
event.addDevice(obj.toString());
}
event.setMainEventId(doc.getObjectId("mainEventId"));
return event;
}
}
My test mongo configuration is
#Profile("test")
#Configuration
#EnableMongoRepositories(basePackages = "com.echo.spring.data.mongo")
#ComponentScan(basePackages = "com.echo.spring.data.mongo" )
public class MongoDbTestConfig extends AbstractMongoConfiguration {
static final Logger logger = LoggerFactory.getLogger(MongoDbTestConfig.class.getCanonicalName());
#Override
protected String getDatabaseName() {
return "echo";
}
#Override
public Mongo mongo() {
return new Fongo("echo-test").getMongo();
}
#Override
protected String getMappingBasePackage() {
return "com.echo.spring.data.mongo";
}
#Bean
#Override
public CustomConversions customConversions() {
logger.info("loading custom converters");
List<Converter<?, ?>> converterList = new ArrayList<Converter<?, ?>>();
converterList.add(new EventReadConverter());
converterList.add(new EventWriteConverter());
CustomConversions cus = new CustomConversions(converterList);
return new CustomConversions(converterList);
}
}
And my test (using fongo) is
ActiveProfiles("test")
#RunWith(SpringJUnit4ClassRunner.class)
#ContextConfiguration(classes = MongoDbTestConfig.class )
public class SampleMongoApplicationTests {
#Test
#ShouldMatchDataSet(location = "/MongoJsonData/events.json")
public void shouldSaveEvent() throws IOException {
URL url = Resources.getResource("MongoJsonData/events.json");
List<String> lines = Resources.readLines(url,Charsets.UTF_8);
for (String line : lines) {
Event event = objectMapper.readValue(line.getBytes(),Event.class);
eventRepository.save(event);
}
}
I can see the converters are loaded when the configuration customConversions() is called
I added logging and breakpoints in the convert methods but they do not seems to be
called when I run or debug, though they are loaded .
What am I doing wrong ?
I had a similar situation, I followed Spring -Mongodb storing/retrieving enums as int not string
and I need both the converter AND converterFactory wired to get it working.