I want to do some validation on line I process and if validation fail, I would like to send email that we can't process that line.
How can I achieved that? I'm using FlatFileItemReader and processor after it.
I'm using FlatFileItemReader and processor after it
That's how I would do it too. In your processor, you can implement the validation logic and throw an exception for invalid items. This is one of the typical use cases for an item processor BTW. The Validating input section of the reference documentation shows an example.
You can then configure the exception to be skippable and use a SkipListener to store invalid items (either in memory or to a file or a db table, etc). Finally, you would use an additional tasklet step to send an email with those invalid items.
HTH.
I have problem after adding faultTolerant, skipLimit an skip into the step config. In this setup I'm unable to use build()
return stepBuilderFactory.get("job")
.allowStartIfComplete(true)
.<RegisterUserRequest, RegisterUserRequest>chunk(2000)
.reader(registerUserFromFileReader)
.processor(registerUserProcessor)
.faultTolerant()
.skipLimit(50000)
.skip(RegisterUserLineException.class)
.listener(registerUserSkipListener)
.writer(userRegistration)
.listener(registerUserFinalizer)
.build() //doesn't work
Related
I am planning to use S3Streaming message source to process the import file (xml)received in S3. I am not sure how to transform MessageSource to job launch request as job parameter doesn't support parameters other than primitive type, please throw some light on how to proceed on this
- Thanks
If you mean you want to pass the InputStream payload to an ItemReader, no, that can't be done with a JobLauncher.
Instead, the ItemReader itself needs to open the input stream (perhaps using a Spring Integration SftpRemoteFileTemplate).
Just pass the information (filename, server, credentials, etc) in the JobParameters and close the SI input stream payload without reading.
I am using a FIX protocol to communicate with one of our counterparties. I have used Camel with Spring to build my communication routes.
I have a requirement where in my counterparty is expecting an ACK for every request it sends to me.
For example:
TradeCaptureRequestAck in response to TradeCaptureRequest
AllocationReportAck in response to AllocationReport
Confirmation_Ack in response to Confirmation
They are expecting a response irrespective of what happens at our end (even if something fails or exception occurs).
One way I know we can intercept the incoming message via MessageFactory. We can create a custom messagefactory and inject it in while creating QuickFixJComponent bean.
Problem with this approach is at factory level I will just be able to get the message type like TradeCaptureReport, AllocationReport etc. but not the content because factory only creates (and returns) the appropriate Message object. Actual work of populating this message object with incoming message data happens in Session class I guess (not sure about this).
Can someone please tell me if there is a way I can get or intercept the request message as soon as it reaches the route so that I can send the appropriate ACK to counterparty?
Thanks to NiFi How to use InvokeHTTP Processor with SOAP, I've been able to make a SOAP call. Strategy used was to use the GenerateFlowFile processor, and insert my content into custom content. The call needed a username and password, as well as another variable or two, and expression language neatly populated it.
Good.
Now, what I really need is two SOAP calls, where the first SOAP call returns a sequence number that I provide to the second call to get the specific data file I'm returning. I've set up a ProcessorGroup for the first SOAP interaction, which results in a flow attribute being set for the sequence number.
The rub: GenerateFlowFile just works on a timer. I've not been able to fathom a way where I can use the result from the first SOAP flow to then trigger an appropriately built FlowFile for the 2nd call.
Thoughts? Things I've puzzled over but haven't yet wrestled a solution forth include RouteOnAttribute, RouteOnContent, MergeContent, Wait, ...
You can use ReplaceText to accept an incoming flowfile with the correct sequenceNumber attribute and populate the flowfile content with the new SOAP body you need. The Replacement Value property supports Expression Language, so you could provide a value like:
Search Value: (?s)(^.*$)
Replacement Value: <xml><sequenceNumber>${sequence_number}</sequenceNumber></xml>
If you only need to replace part of the content or maintain some of the existing content text, you can use regex matching groups and backreferences to identify those.
The output from the ReplaceText processor would then be routed to a second InvokeHTTP processor to perform the second SOAP call.
Just to add on top of what #Andy had said. The response that you receive which contains the sequenceNumber will be in XML, right? So you could use EvaluateXPath processor to parse and get the sequence number and then use the approach that Andy had mentioned. i.e. Use ReplaceText processor to generate the SOAP Request body which would be sent to the second InvokeHTTP
So the overall flow would look like:
GenerateFlowfile -> InvokeHTTP -> EvaluateXPath -> ReplaceText -> InvokeHTTP -> (YOUR_LOGIC)
I've developed a method that does the following steps, in this order:
1) Get a report's metadata via /gdc/md//obj/
2) From that, get the report definition and use that as payload for a call to /gdc/xtab2/executor3
3) Use the result from that call as payload for a call to /gdc/exporter/executor
4) Perform a GET on the returned URI to download the generated CSV
So this all works fine, but the problem is that I often get back a blank CSV or an incomplete CSV. My workaround has been to put a sleep() in between getting the URI back and actually calling a GET on the URI. However, as our data grows, I have to keep increasing the delay on this, and even then it is no guarantee that I got complete data. Is there a way to make sure that the report has finished exporting data to the file before calling the URI?
The problem is that export runs as asynchronous task - result on the URL returned in payload of POST to /gdc/exporter/executor (in form of /gdc/exporter/result/{project-id}/{result-id}) is available after exporter task finishes its job.
If the task has not been done yet, GET to /gdc/exporter/result/{project-id}/{result-id} should return status code 202 which means "we are still exporting, please wait".
So you should periodically poll on the result URL until it returns status 200 which will contain a payload (or 40x/50x if something wrong happened).
We have an Apache Camel application providing SOAP service. The "initial route" starts from a Apache CFX -provided endpoint.
We need a simple mechanism to prevent the messages from being handled "too fast" (and don't have massive scalability needs).
Thus we ended up trying Throttler. Now, the problem is that after adding throttle to our route, something goes wrong.
The initial route, somewhat cleaned:
from("cxf:bean:sapEndpoint").routeId(SOAP_ENDPOINT)
.throttle(1)
.onException(Exception.class)
.to("direct:emailFaultNotification").handled(false)
.end()
.transacted(joinJpaTx)
.to(xsltRemoveEmptyElements) // Cleaning done with XSLT endpoint
.to("direct:inboundWorkOrderXml"); // Forward to actual processing
// direct:inboundWorkOrderXml contains various validation, persistance & response
Error in our log:
2013-02-18 16:50:16,257 [tp1636587648-50] ERROR DefaultErrorHandler - Failed delivery for exchangeId: ID-...-4. Exhausted after delivery attempt: 1 caught: javax.xml.transform.TransformerException: javax.xml.transform.TransformerException: com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Content is not allowed in prolog.
javax.xml.transform.TransformerException: javax.xml.transform.TransformerException: com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Content is not allowed in prolog.
at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:735)[:1.6.0_37]
at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:336)[:1.6.0_37]
at org.apache.camel.builder.xml.XsltBuilder.process(XsltBuilder.java:98)[camel-core-2.7.0.jar:2.7.0]
at org.apache.camel.impl.ProcessorEndpoint.onExchange(ProcessorEndpoint.java:102)[camel-core-2.7.0.jar:2.7.0]
at org.apache.camel.impl.ProcessorEndpoint$1.process(ProcessorEndpoint.java:72)[camel-core-2.7.0.jar:2.7.0]
...
I suppose that throttler doesn't work straight the way I supposed.
It seems that with throttling enabled, the XSLT endpoint receives empty or invalid XML? Without throttle definition everything works fine. With short try, the message body still seems to contain XML string?
Some ideas?
When using Camel error handling for redelivery then mind about streaming payloads. See about stream-caching at: http://camel.apache.org/stream-caching.html
There is also a tip in the top of the Camel CXF documentation page at: http://camel.apache.org/cxf about this
Finally the solution was more simple than I thought. Instead of using throttle in the route starting from "cxf:bean:sapEndpoint", I added throttle to route handling "direct:inboundWorkOrderXml".
Don't know the exact reason, could be somehow related to that some parts of throttle functionality might vary related on the from-endpoint of the route. (So problem with cxf-endpoint not encountered with direct-endpoint)