Scenario 1
The skip listener interface is as below:
public interface SkipListener<T,S> extends StepListener {
void onSkipInRead(Throwable t);
void onSkipInProcess(T item, Throwable t);
void onSkipInWrite(S item, Throwable t);
}
This interface is best used to log the skipped item and the error.
Is is possible to get the number of the skipped item in the input. For e.g. if the 10th item in the input is getting skipped, I should be able to log "Item number 10 was skipped!" through above listener.
I need this since I have input as a file where the rows are not having any identifying key. So just by logging out the item, it would not be possible to pin point the item itself in the file.
What if instead of file, the input is a database table ? Is it possible to get the position number of the skipped item there as well ?
Scenario 2
My bean has three properties one, two and three (all strings) where the input is read from a file through appropriate row mapper and then a database table gets loaded with the data after some processing.
Below is a code block from processor:
if(two.charAt(4) == '_')
{ // do some processing }
Clearly if field two is coming empty from the file above block will throw "string index out of bound exception" and will get skipped.
So, inside skip listener, what I want is the information about the column which threw error.
Here since field named two gave error, the information I would like to log in skip listener would be like "Property one threw error "string index out of bound exception" in line number 10" or if possible, even more specific "property one is empty in line number 10" which makes more sense to business who does not know java jargons.
Hope I made my doubts clear.
Thanks for reading!
Scenario 1
to get line number for skipped item from reading a file you can:
for onSkipInRead - implement own/wrap the reader to act on exception
onSkipInProcess - implement own linemapper which writes line number to item or writes current line number in step context
onsKipInWrite - same as for onSkipInProcess
to get line number for skipped item from reading a table you can:
for onSkipInRead - implement own/wrap the reader to act on exception, but i'm not sure if it's possible to the get the line number here, might only be the current chunk-start-number
onSkipInProcess - implement own rowmapper which writes row number to item or writes current row number in step context
onsKipInWrite - same as for onSkipInProcess
Scenario 2
see scenario 1 and add a try/catch block inside your processor, so you throw your own exception e.g. with information on property position, or alter the step context in a similar way
Related
We are receiving fix messages from WebICE exchange in a text file and our application is reading and parsing them line by line using QuickFixJ. We noticed that in some messages the repeating group fields are not being parsed and upon validating with data dictionary getting error.
quickfix.FieldException: Out of order repeating group members, field=326
For example in the sample file data-test.csv the first 2 rows parsed successfully but third one fails with the above error message.
Upon investigation I found , in first 2 rows tag 326 comes after tag 9133 but in the third row it comes before that and hence fails in validation. If I adjust data dictionary as per the third one it succeeds but ofcourse the first one starts failing.
This is happening only for few messages for most of the other fix messages are getting validated and parsed quite fine. This is part of the migration project from existing C# application using QuickFix/N to our scala application using QuickFix/J. And its been working fine at the source end (with QuickFIx/N). Is there any difference in both the libraries QuickFIx/J and QuickFIx/N in terms of dealing with group fields ?
To help recreate the issue , I have shared the data file having 3 fix messages as explained above.
Data file : data-test.csv
Data dictionary : ICE-FIX42.xml
Here is the test code snippet
val dd: DataDictionary = new DataDictionary("ICE-FIX42.xml")
val mfile = new File("data-test.csv")
for (line <- Source.fromFile(mfile).getLines) {
val message = new quickfix.Message(line,dd)
dd.setCheckUnorderedGroupFields(true)
dd.validate(message)
val noOfunderlyings= message.getInt(711)
println("Number of Underlyings "+noOfunderlyings)
for(i <- 1 to noOfunderlyings ) {
val FixGroup: Group = message.getGroup(i, 711)
println("UnderlyingSecurityID : " + FixGroup.getString(311))
}
}
Request to fellow SO users , If you can help me with this.
Many Thanks
You should use setCheckUnorderedGroupFields(false) to disable the validation of the ordering in repeating groups. However, this is only a workaround.
I would suggest to approach your counterparty about this because especially in repeating groups the field order is required to follow the message definition, i.e. the order in the data dictionary.
FIX TagValue encoding spec
Field sequence within a repeating group
...
Fields within repeating groups must be specified in the order that the fields are specified in the message definition.
When we are processing EOS(end of service) of an employee then system automatically generate the zero line 0.00 as Encashment quantity for rendom employee somehow.My client don't want this zero line in this form becuase when they are trying to post EOS to general ledger its showing the error as amount can't be post without value.So how to code in x++ to remove the zero amount line from my EOS form if there is any 0.00 encashment amount.
enter image description here
In Above screenshot system has calculated the encashment quantity as zero so when we processed EOS. Kindly requesting give some suggestions to handle this scenario by X++ code.
If you are using a datasource to get the value then there is a property called "Insert if empty" on the datasource (In this case the datasource with the EncashmentQuantity field) which is automatically set as "yes" but you can change it to "no" and the value should not be shown if it is 0. I am not sure this is what you are talking about but it might be.
I hope to understood the question.
"My client don't want this zero line in this form becuase...."
I think you have this way, If you want to filter the record without 0 Value by code, you can use filter in DataSource rightClick override executequery method and insert code looklike follow:
public void executeQuery()
{
this.query().dataSourceTable(tableNum(**YourTablename**)).addRange(fieldNum(**YourTableName**,**YoutFilterField**)).value(strFmt("%1", '>0'));
super();
}
ForExample :
this.query().dataSourceTable(tableNum(LedgerJournalTrans)).addRange(fieldNum(LedgerJournalTrans,AmountCurCredit)).value(strFmt("%1", '>0')); // DISCARD all record lower than 0
or
this.query().dataSourceTable(tableNum(LedgerJournalTrans)).addRange(fieldNum(LedgerJournalTrans,AmountCurCredit)).value(SysQuery::valueNot(0));
I share this useful links :
Expressions in query ranges
SimilarQuestion
StandardGuide
Good work!
I am trying to create some joblets in Talend that will speed up some processes.
I have an input from a MSSQLInput, the results are then sorted and filtered a little. Then I have a tMemorizeRows and a tJavaFlex, the purpose of this is to memorize the rows in a column to preform a count. The count is based on a customer ID, once the the id changes the count starts back to 1 and the proccess begine again and continues to the end. I have refactored this as a joblet but it does not work, the error is:
ID_tMemorizeRows_1 cannot be resolved to a variable
I have a tJavaFlex which starts with
int counte = 1;
The Main code is
if(ID_tMemorizeRows_1[0].equals(ID_tMemorizeRows_1[1]))
{
counte = counte + 1;
}
else
{
counte = 1;
}
context.Enqnum = counte;
The Enqnum variable and is created correctly and added into a tMaps component.
Does anyone know why this is happening, one person told me it is because when you move something to a joblet it gets a new/different name so it has to be specifically called in the Java, if this is the case how do I find the name out?
Thank you
Rich
I do have a resolution. I have tried to add images however my reputation is not high enough.
When using joblets we know that Talend essentially recycles the code used in the joblet by inserting it into the code for the main job.
This is the joblet I have created, i know it works because I have refactored it to a joblet instead of building it from sctatch. What its doing is simply memorises row 0 and row 1 in an ordered data set, the java performs a count and the tMap appends the result to the job (as Mentioned above).
(I will try it inser image in my question, I do not have enough reputation point to insert it into a question).
When the job is run it runs fine. But problems occur when I want to reuse the same joblet in another part of the job. What Talend does is it assigns names within the source code to each component depending on the name of the joblet.
For example, if the Joblet was called ThisJob, then tMemorizeRows_1 would be called ThisJob_1_tMemorizeRows_1.
The row within the component (in this example ReferenceID) would renamed as:
ReferenceID_ThisJob_1_tMemorizeRows_1.
But when you add a second joblet to your job it gives it a new name, eg ThisJob_2. This name will be different depending on how much you have been altering your job before you add the second joblet. Therefore the number within the name will depend on this activity.
If you add the joblet into your job immediately then the joblet would be called ThisJob_2, if you have added 5 other components before you add it in then the joblet is likely to be called ThisJob_6 etc. (I'm not 100% sure how talend renames components)
When you add a joblet, You can see the name of the joblet on the joblet component, this then reverts back the the original joblet name when you create any links/joins to other components.
Its also important that each component within the code is assigned to a variable called currentComponent.
Resolution
What I did was used the Java code to split the name using the code below. This way I can get the current name of the of the joblet and use this name in my Java.
String string = currentComponent;
String[] parts = string.split("_");
String part1 = parts[0];
String part2 = parts[1];
String joblet = part1+'_'+part2;
String newrow = "ReferenceID_"+joblet+"_tMemorizeRows_1";
I hope this makes sense.
Thanks
I have almost completed my work with Spring Batch, it's working but then I have problems to handle the errors. I'll make a simple example:
I read one flat file, that I (later) map with 3 variables:
ID CODE NAME
AAA3333333Alex
AAA3333333Mark
BBB4444444Paul
I want the reader to read the flat file with a control break (I don't know if it's the right term in english, in italian it's something like "key break"): I read the elements with the same ID and CODE and only when the key changes return them to the reader:
while ((line = (Person) peek()) != null) { //while there are elements to read
if (line.getId().equals(prevElement.getId()) && line.getCode().equals(prevElement.getCode())
//do something
}
This works fine: when the ID or CODE changes I return the elements to the writer.
To make this work I had to set the commit-interval to 1 from the application-context. The thing is that in the worst case, if the elements are different for each line, I commit every single element and it becomes all very very slow.
So I said: let's put an outer control. Instead of returning the elements to the writer each time the key changes, I put them in a list, and then I return the list every 200 key changes (like a...handmade commit-interval):
while (controlBreakCount < 200 &&) {
while (!exit && (line = (Person) peek()) != null) { //while there are elements to read
if (line.getId().equals(prevElement.getId()) && line.getCode().equals(prevElement.getCode())
//do something
else { //if the key changes
//there is a controlBreakCount++; to increase the count
//add the elements to a list
}
}
}
return the list
and this works too (the real code has more controls, but this was to explain in a simple way).
The problem comes here: how to handle the errors with the listener in this case. With the outer while I have put (the one with the controlBreakCount), if even one of the 200 elements has an error all the elements currently in the list go to the listener and so it's very difficult to recognize the element with the error.
I guess my solution is not the best way to handle the "control break", but I can't find really much about this (and I'm not very pro with Spring Batch)...may I have some help?
Thank you very much :)
The 'do something' in your code is suspect... :/ Which operations do you do on the element just read? In reader you should only aggregate items and pass them to processor or writer.
IMHO you have to set a commit-interval greater than 1 to make process faster and try to group your data with same ID+CODE key in your own custom reader and THEN pass to writer in this way:
class PersonList {
String id;
String code;
List<Person> persons = new ArrayList<Person>();
}
when you find a keybreak (ID+CODE differs from previous) you have to create a new PersonList object and until ID+CODE are the same of previous line add to current PersonList.person list.
If your problem is about manage single line error with my solution you you can manage in your reader and, if you want, skip items with same ID+CODE in your reader as well.
Your reader must change its signature to MyPersonItemReader<PersonList> and your writer will write PersonList objects, but you have under your control the object (with ID+CODE).
I hope I had understood correctly your problem.
When fields are nested, there is a problem.
foreach (Word.Field field in this.Application.ActiveDocument.Fields)
{
field.Update();
text = field.Result.Text;
}
The above code does not work.
The process starts, but winds up in an endless loop or some other process that hangs the system.
Thinking about it, I can surmise that when you update a field, it might have an effect on the fields collection - thus, the loop fails.
Does anyone have any ideas on implementing this?
P.S. I know there is a Document.UpdateFields() method to update ALL fields. However, there are reasons why I cannot use this and need to only update specific field types.
My apologies! I was going to give an example of a nested field but was trying to test some more before sending anyone (Jack) on a goose-chase.
I waited and waited and waited, and after a good 2 or 3 minutes, it finished. After the last field, it crashed with this message:
Object has been deleted.
The error was generated from the following line inside the loop:
string text = field.Code.Text;
The template is being tested on mergefields that are not being found because I am testing without database connectivity. It would be odd, but explainable, that it goes through all the fields and then, at the end of the day, the very OUTER IF field's result is "Error! Reference source not found." But I still don't get why this could happen.
Nor do I understand why looping takes 3 minutes while a call to document.Fields.Update() will do the same thing in about 1 second and NOT result in the error described above.
Again, my apologies. I never considered updating inside a loop would be vastly slower that a call to doc.fields.update().