QuickFIX/J not reading all the repeating groups in FIX message - scala

We are receiving fix messages from WebICE exchange in a text file and our application is reading and parsing them line by line using QuickFixJ. We noticed that in some messages the repeating group fields are not being parsed and upon validating with data dictionary getting error.
quickfix.FieldException: Out of order repeating group members, field=326
For example in the sample file data-test.csv the first 2 rows parsed successfully but third one fails with the above error message.
Upon investigation I found , in first 2 rows tag 326 comes after tag 9133 but in the third row it comes before that and hence fails in validation. If I adjust data dictionary as per the third one it succeeds but ofcourse the first one starts failing.
This is happening only for few messages for most of the other fix messages are getting validated and parsed quite fine. This is part of the migration project from existing C# application using QuickFix/N to our scala application using QuickFix/J. And its been working fine at the source end (with QuickFIx/N). Is there any difference in both the libraries QuickFIx/J and QuickFIx/N in terms of dealing with group fields ?
To help recreate the issue , I have shared the data file having 3 fix messages as explained above.
Data file : data-test.csv
Data dictionary : ICE-FIX42.xml
Here is the test code snippet
val dd: DataDictionary = new DataDictionary("ICE-FIX42.xml")
val mfile = new File("data-test.csv")
for (line <- Source.fromFile(mfile).getLines) {
val message = new quickfix.Message(line,dd)
dd.setCheckUnorderedGroupFields(true)
dd.validate(message)
val noOfunderlyings= message.getInt(711)
println("Number of Underlyings "+noOfunderlyings)
for(i <- 1 to noOfunderlyings ) {
val FixGroup: Group = message.getGroup(i, 711)
println("UnderlyingSecurityID : " + FixGroup.getString(311))
}
}
Request to fellow SO users , If you can help me with this.
Many Thanks

You should use setCheckUnorderedGroupFields(false) to disable the validation of the ordering in repeating groups. However, this is only a workaround.
I would suggest to approach your counterparty about this because especially in repeating groups the field order is required to follow the message definition, i.e. the order in the data dictionary.
FIX TagValue encoding spec
Field sequence within a repeating group
...
Fields within repeating groups must be specified in the order that the fields are specified in the message definition.

Related

How to add Currency and SettlType to InstrmtMDReqGrp in MarketDataRequest <V>

I'm implementing a FIX program as initiator that connect to market and send & receive fix messages. In one step I need send MarketDataRequest message to market, but something happened that the message is rejected by market.
I have tried add Fields Currency.FIELD and SettlType.FIELD, but in final message in the final message they do not appear.
I need send message like:
8=FIXT.1.1 9=222 35=V 34=1013 49=**** 52=20190117-17:23:48.670 56=**** 128=FGW 262=MD1 263=0 264=5 265=1 266=Y 146=1 55=GGAL 48=GGAL-0002-C-CT-ARS 167=GO 207=XMEV 15=ARS 63=2 267=9 269=0 269=1 269=2 269=4 269=5 269=7 269=8 269=B 269=3 10=215
But I can not put the tag 15 and 63 before the tag 267.
My code now:
InstrmtMDReqGrp instrmtMDReqGrp = new InstrmtMDReqGrp();
NoRelatedSym noRelatedSym = new NoRelatedSym(1); //146
instrmtMDReqGrp.set(noRelatedSym);
SettlType settlType = new SettlType(); // 63
settlType.setValue("2");
Currency currency = new Currency(); // 15
currency.setValue("ARS");
instrmtMDReqGrp.setField(Currency.FIELD, currency);
instrmtMDReqGrp.setField(SettlType.FIELD, settlType);
In the final message these fields do not appear.
I'm using quickfixj with quickfixj-spring-boot-starter 2.2.1.
You are forgetting to call addGroup on the message, like so:
fixMessage.addGroup(instrmtMDReqGrp);
A small documentation page exists for repeating groups in QuickFIX/J, which demonstrates this:
Creating Messages With Repeating Groups
Here is an example of a message that distributes market data. When the
message is created the required field with the number of repeating
groups is set to zero. This is because QuickFIX will automatically set
this field for you when you add groups. This way there is never an
inconsistancy between the number of entries in the field and in the
message.
package quickfix;
quickfix.fix42.MarketDataSnapshotFullRefresh message =
new quickfix.fix42.MarketDataSnapshotFullRefresh(new Symbol("QF"));
quickfix.fix42.MarketDataSnapshotFullRefresh.NoMDEntries group =
new quickfix.fix42.MarketDataSnapshotFullRefresh.NoMDEntries();
group.set(new MDEntryType('0'));
group.set(new MDEntryPx(12.32));
group.set(new MDEntrySize(100));
group.set(new OrderID("ORDERID"));
message.addGroup(group);
group.set(new MDEntryType('1'));
group.set(new MDEntryPx(12.32));
group.set(new MDEntrySize(100));
group.set(new OrderID("ORDERID"));
message.addGroup(group);
What's reject fix message you get back? It usually gives the tag number that contains the error plus some error description.
Usually the structure of fix messages is defined by a data dictionary provided by the Acceptor. If this data dictionary differs from the one built in the quickfixj library, you will need to recompile quickfixj core library with this data dictionary so messages are parsed correctly but also tags are placed in the correct order when creating new messages.

mirth connect Database Reader automatic column mapping

Please could somebody confirm the following..
I am using Mirth Connect 3.5.08232.
My Source Connector is a Database Reader.
Say, I am using a query that returns multiple rows, and return the result (via JavaScript), as documentation suggests, so that Mirth would treat each row as a separate message. I also use a couple of mappers as source transformers, and save the mapped fields in my channel map (which ends up to contain only those fields that I define in transformers)
In the destination, and specifically, in destination response transformer (or destination body, if it is a JavaScript writer), how do I access the source fields?
the only way I found by trial and error is
var rawMsg = connectorMessage.getRawData();
var xmlMsg = new XML(rawMsg);
logger.info(xmlMsg.some_field); // ignore the root element of rawMsg
Is this the right way to do this? I thought that maybe the fields that were nicely automatically detected would be put in some kind of a map, like sourceMap - but that doesn't seem to be the case, right?
Thank you
If you are using Mapper steps in your transformer to extract the data and put it into a variable map (like the channel map), then you can use any of the following methods to retrieve it from a subsequent JavaScript context (including a JavaScript Writer, and your response transformer):
var value = channelMap.get('key');
var value = $c('key');
var value = $('key');
Look at the Variable Maps section of the User Guide for more information.
So to recap, say you're selecting a column "mycolumn" with a Database Reader. The XML sent to the channel will be something like this:
<result>
<mycolumn>value</mycolumn>
</result>
Then you can choose to extract pieces of that message into specific variables for later use. The transformer allows you to easily drag-and-drop pieces of the sample inbound message.
Finally in your JavaScript Writer (or in any subsequent filter, transformer, or response transformer), just drag the value into the field you want:
And the corresponding JavaScript code will automatically be inserted:
One last note, if you are selecting a lot of variables and don't want to make Mapper steps for each one individually, you can use a JavaScript Step to iterate through the message and extract each column into a separate map variable:
for each (child in msg.children()) {
channelMap.put(child.localName(), child.toString());
}
Or, you can just reference the columns directly from within the JavaScript Writer:
var msg = new XML(connectorMessage.getEncodedData());
var column1 = msg.column1.toString();
var column2 = msg.column2.toString();
...

Quickfix 58=Conditionally Required Field Missing

If I try to replace or cancel an order I get a message
58=Conditionally Required Field Missing
and the next message contains
58=Invalid MsgType
Here are the logs:
Replacing an order (tgFZctx200U61 is my side. FG is an exchange.):
20170203-15:44:04.225 : 8=FIX.4.49=15135=G34=349=tgFZctx200U6152=20170203-15:44:04.22556=FG1=U6111=270071221=138=240=241=2700744=11640054=155=RTS-3.1760=20170203-18:44:04.20510=028
20170203-15:44:04.225 : 8=FIX.4.49=23235=849=FG56=tgFZctx200U6134=352=20170203-15:43:56.98137=572984433198=F:572984433526=$01$11=270071241=2700717=exec-201702031001027616150=E39=E55=RTS-3.17461=FXXXXX54=138=140=2151=114=06=060=19700101-00:00:00.00010=213
20170203-15:44:04.275 : 8=FIX.4.49=11535=j34=449=tgFZctx200U6152=20170203-15:44:04.27556=FG45=358=Conditionally Required Field Missing372=8380=510=065
20170203-15:44:04.275 : 8=FIX.4.49=33335=849=FG56=tgFZctx200U6134=452=20170203-15:43:56.98237=572984753198=F:572984753526=$01$11=270071241=27007453=1448=tgFZctx200U61447=C452=317=3355471052150=539=01=FZ00U6155=RTS-3.1754=138=240=244=116400.00000336=9291151=214=06=060=20170203-15:43:56.98920008=-922337203685372211120018=[51000-3355471052-0]10=100
20170203-15:44:04.285 : 8=FIX.4.49=10335=349=FG56=tgFZctx200U6134=552=20170203-15:43:57.03345=4371=372373=1158=Invalid MsgType372=810=164
cancelling an order:
20170203-15:26:19.178 : 8=FIX.4.49=15435=F34=349=tgFZctx200U6152=20170203-15:26:19.17856=FG11=270061237=57286383038=141=2700644=116470.0000054=155=RTS-3.1760=20170203-18:26:19.17810=013
20170203-15:26:19.188 : 8=FIX.4.49=20735=849=FG56=tgFZctx200U6134=352=20170203-15:26:11.92437=572863830198=F:572863830526=$01$11=270061241=2700617=exec-201702031001027615150=639=655=RTS-3.17461=FXXXXX54=138=140=2151=114=06=010=239
20170203-15:26:19.418 : 8=FIX.4.49=11535=j34=449=tgFZctx200U6152=20170203-15:26:19.41856=FG45=358=Conditionally Required Field Missing372=8380=510=070
20170203-15:26:19.418 : 8=FIX.4.49=33335=849=FG56=tgFZctx200U6134=452=20170203-15:26:11.92437=572863830198=F:572863830526=$01$11=270061241=27006453=1448=tgFZctx200U61447=C452=317=3354681208150=439=41=FZ00U6155=RTS-3.1754=138=140=244=116470.00000336=9291151=014=06=060=20170203-15:26:11.93120008=-922337203685267353520018=[51000-3354681208-0]10=080
20170203-15:26:19.418 : 8=FIX.4.49=10335=349=FG56=tgFZctx200U6134=552=20170203-15:26:12.16445=4371=372373=1158=Invalid MsgType372=810=161
Best regards, Mikhail
"Conditionally Required Field Missing" means you are trying to extract an optional field that isn't present. (It's not required by the DD, but the user's logic expects it to be there, hence "conditionally required".)
The first 35=j message says:
45=3 - sequence number of message where these happened
58=Conditionally Required Field Missing
372=8 - type of message where this happened
380=5 - same code as explained in 58
Unfortunately, the message doesn't say which field is the problem, but basically, you're doing this (forgive my pseudocode):
var x = msg.getSomeOptionalField()
but you need to do this:
var x = null;
if (msg.checkIfSomeOptionalFieldIsPresent())
x = msg.getSomeOptionalField();
In order to parse your own FIX messages use FIXimate.
58 is a text field. The text after 58 in this case is the error message. The tag value pair 372=83 means: The message referred to (i.e. the missing tag) is tag 83.
Tag 83 is the sequence number of message within report series. FIXimate says that 83 is "Used to carry reporting sequence number of the fill as represented on the Trade Report Side."
This is your FIX engine sending an error back to the exchange. You can tell by looking at the SenderCompID and TargetCompID for each message.
You send a message:
20170203 15:44:04.225:8=FIX.4.49=15135=G34=349=tgFZctx200U6152=20170203-15:44:04.22556=FG
You get an Execution Report (35=8, probably acknowledging order cancellation/replace):
20170203-15:44:04.225 : 8=FIX.4.49=23235=8 49=FG 56=tgFZctx200U61lo9
You send an Business Reject (35=j):
20170203-15:44:04.275 : 8=FIX.4.49=115 35=j 34=4 49=tgFZctx200U61 52=20170203-15:44:04.275 56=FG 45=358=Conditionally Required Field Missing372=8380=510=065
What this last message coming in from the exchange is, is hard to tell without further analysis, but it most likely the execution report for the replaced order. It seems to have been sent 1 ms after the original execution report.
Your FIX engine expects certain data to be present within the messages. The expectation is set in your data dictionary, an xml file which should be provided by your counterparty. Sometimes (like now) there are errors in this file and you have to open it up, find the message in question (in this case the original execution report), and tell your data dictionary not to expect tag 83.
That should clear things up. Let me know if it doesn't work.

Scala Gatling keep inserting user until end of feed file is reached

I'm trying to inject users to a scenario in such a way that it will keep inserting user until every single entry of the feed file is used since the feed file contains log in information. I would like all the users in the feed file to log in. Right now all I could think of is two possible approaches.
Here I insert the number of rows in the feedfile at once.
scenario("Verified_Login")
.exec(LoginScenario.scn)
.inject(atOnceUsers(number_of_entries_in_feedfile))
Here I insert a very high time duration, for example, 100 seconds and then make the feedfile circular.
scenario("Verified_Login")
.exec(LoginScenario.scn)
.inject(atOnceUsers(1),constantUsersPerSec(1) during(100 seconds)
The problem with the first approach is I have to find the number of entries in the feed file which can be tedious as there could be thousands there. The problem with the second is that entries could and probably will be repeated. So is there a way to keep injecting users till feed file runs out of entries?
According to this source, from last year, Stéphane Landelle - who is the leading contributor of gatling, says that you must provide enough data for a simulation to complete using this method.
The post I linked from Stéphane does suggest to simply read the length of the file and use that to drive the amount of users, as you have already mentioned in your question.
I suggest you read the post as it will give you an alternate method to achieving what you want. Seems to be as close as you will ever get unless things have changed.
Here is their code.
val systemsIdentifier = jdbcFeeder(databaseUrl, databaseUser, databasePassword, sql_systemsIdentifier)
val count = new AtomicInteger(systemsIdentifier.records.size).asLongAs(_ => count.getAndIncrement < systemsIdentifier.records.size)
val comScn = scenario("My scenario")
.repeat(systemsIdentifier.records.size / count) {
feed(systemsIdentifier)
.exec(performActionsChain)
}
setUp(comScn.inject(rampUsers(count) over (60 seconds))).protocols(httpConf)

Spark: RDD.saveAsTextFile when using a pair of (K,Collection[V])

I have a dataset of employees and their leave-records. Every record (of type EmployeeRecord) contains EmpID (of type String) and other fields. I read the records from a file and then transform into PairRDDFunctions:
val empRecords = sc.textFile(args(0))
....
val empsGroupedByEmpID = this.groupRecordsByEmpID(empRecords)
At this point, 'empsGroupedByEmpID' is of type RDD[String,Iterable[EmployeeRecord]]. I transform this into PairRDDFunctions:
val empsAsPairRDD = new PairRDDFunctions[String,Iterable[EmployeeRecord]](empsGroupedByEmpID)
Then, I go for processing the records as per the logic of the application. Finally, I get an RDD of type [Iterable[EmployeeRecord]]
val finalRecords: RDD[Iterable[EmployeeRecord]] = <result of a few computations and transformation>
When I try to write the contents of this RDD to a text file using the available API thus:
finalRecords.saveAsTextFile("./path/to/save")
the I find that in the file every record begins with an ArrayBuffer(...). What I need is a file with one EmployeeRecord in each line. Is that not possible? Am I missing something?
I have spotted the missing API. It is well...flatMap! :-)
By using flatMap with identity, I can get rid of the Iterator and 'unpack' the contents, like so:
finalRecords.flatMap(identity).saveAsTextFile("./path/to/file")
That solves the problem I have been having.
I also have found this post suggesting the same thing. I wish I saw it a bit earlier.