I have a FIX message as a string, and I am creating a message object from this message using QuickFix4J, so that I can send it to the other party.
The DataDictionary that I am using is given to me by the other party.
But when I resolve it with the DD and create the message, many fields especially the ones that are repeating are getting truncated. Basically when a group of field repeats, the final message has only ONE instance of the repeating fields.
This is my original message:
8=FIXT.1.1|9=1288|35=X|34=1163|49=XX|52=20190410-10:27:43|56=XXXXXXXXXXXXXXXXXXXXX|131=XXXXXXXXXXXXXX_2019410_155743|146=1|55=[X/X]|48=58013XXX5|22=1|6360=XXXXXXXX XXXXXXXX|454=1|455=XX58013XXX54|456=4|20200=2|20201=1|20202=6|20203=99.06300000|20204=1111.00|20205=3|20206=XXX2|20201=2|20202=6|20203=0.14400000|20204=2222.00|20205=3|20206=XXX2|460=3|1227=XXXX|29703=XXXX XXXX|167=XXXX|541=20350410|225=20170410|223=0.03500000|106=XXXXXXXXX XXXX XXXXXX XXXX XXX XXXX XXXXX|107=XXX 3.500 3/1/27 X26|873=20170309|54=2|38=188000|64=20190412|15=XXX|126=20190410-10:32:43|60=20190410-10:27:43|663=1|699=9128286X1|761=1|29715=XX9128286X18|29716=4|29717=XXX|29718=0.02625000|29719=20290215|423=6|453=7|448=XXXXXXX1|447=X|452=11|802=1|523=XXXXXXX XXXXXX XX XXX|803=9|448=XXXX|447=X|452=13|448=XXX XXXXXXXXXX 5|447=X|452=13|448=XXXXXX33|447=X|452=17|802=1|523=0355|803=17|448=XXXX|447=X|452=17|448=XXXXXX XXXXXX XXXXXXXXXX (XXX) XXX|447=X|452=17|448=XXXXXXX|447=X|452=13|58=XXXXXX5 (XXXXXXXXXX, XXXXXXX XXXXXX XX XXX) XXXXXXXX XXX XX $100,000 XXX 3.500 03/01/27 X26, XXXXXXXXX XXX 2.625 02/29, XXX XX 2 XXXX XXXXX-XXXXXXX , XXXX XXXX.|5625=2|20117=10155743|5961=XXXXXX|5626=3|20012=1 3|5215=X|5627=XXXXXXXXX|5630=XXXXXXX, XXXX|20120=X2X-XXX-XXXX|21031=X|21032=X|20013=0.3|29724=60|22203=XXXX|29741=000X|29742=1000|10=144|
And this is the message after creating the message object:
8=FIXT.1.1|9=262|35=X|34=656|49=XX|52=20190410-10:27:45.566|56=XXXXXXXXXXXXXXXXXXXXX|131=XXXXXXXXXXXXXX_2019410_155743|146=1|55=[X/X]|48=58013XXX5|22=1|6360=XXXXXXXX XXXXXXXX|454=1|455=XX58013XXX54|456=4|20200=2|20201=2|20202=6|20203=99.06300000|20204=1111.00|20205=3|20206=XXX2|10=111|
This is the code with which I am creating the message:
rawMessage = new Message(newmessage, dataDictionary, false);
Session.sendToTarget(rawMessage, sessionID)
Is there a way I can send the message as is without quickfix4j trying to resolve it and hence truncating fields. Unfortunately I can't share the DD.
I see two issues:
First issue: Your approach is a mistake
rawMessage = new Message(newmessage, dataDictionary, false);
Session.sendToTarget(rawMessage, sessionID)
You can't do that! The sequence numbers, timestamps, etc are not valid anymore!
Are you creating a naive message replayer? (Why do people keep trying to do that?) It's not going to work! FIX messages flows have state that you can't just blindly replay.
If you're creating a test harness, it needs to be smarter than this. Stop what you're doing and reconsider your approach.
Second issue: your DD probably has errors
Truncated repeating groups always means a DataDictionary problem. The DD is not matching the message being parsed. One of the following is surely happening:
the message has put a field in the group that is not in the DD's definition for that group
the message's group has fields out of order (compared to the DD's order)
When parsing a group, the engine will end the group as soon as it sees a field that it does not expect.
The DataDictionary that I am using is given to me by the other party.
Don't trust it! Counterparties make errors with their own docs, because they don't actually use them, or because their internal version are newer than what they've published.
Start parsing your message manually against the DD that they gave you, and I bet you'll find the error.
Thank you Grant for your inputs, finally we figured out that the error was that repeating groups were coming out of order:
datadictionary.setCheckUnorderedGroupFields(false);
solved it.
Related
I am building a mailout capability and it is working OK as far as it goes. However, I want to distinguish between various potential (high level) outcomes in order to determine what happens to each message after the current send attempt.
This must be a common requirement so I seem to be missing something pretty obvious, but I can't find anything that addresses it, either here or via Google or on PHPMailer site or .. . Possibly because there are so many questions about specific errors that I just can't find anything useful in all the other results.
At very high level:
Attempt send, and assess resulting error/result. Identify whether this message has been sent, must be retried later, or failed permanently.
- success -> update message status as 'SENT: OK'
- sent, but some issues (e.g. one recipient failed, others processed OK)-> 'SENT: some error'
- failed, due to temporary problem (e.g. connection problem, attachment open) -> 'TRY LATER'
- failed, due to message-specific problem that we should NOT try to resend-> 'FAILED: some error'
As I was unable to find an existing resource with e.g. a table of errors, I spent some time working through the phpmailerException code to try to build one myself, but it's not simple because a) they don't appear to have been designed in terms of this kind of grouping logic, b) it is not easy to uniquely identify a particular error: PHPMailer provides human-friendly messages, which are different in different languages, rather than an identifiable code - given that my solution will need to work across different language installations that's a problem!
Obviously SMTP itself provides a range of errorcodes which I could potentially use for this purpose, but how do I access these via PHPMailer? (This would work for me as I only use SMTP at this point - however, this would NOT work if other message transport like sendmail was used, so I would prefer a PHPMailer solution)
If you want individual result codes for individual address, you really need to send each message separately. If you do get errors on some recipients, they will be listed in the ErrorInfo property - look in the smtpSend function to see how the error string is assembled. I agree that it's not especially easy to parse that info out. The error messages in PHPMailer are generally more for the developer than the end user, so the translations are not that significant. You can get slightly more information about errors if you enable exceptions rather than relying only on return values.
Hi I need help to understand, if 35=X message should contain Symbol/SecID within the repeating group.
The FIX Specification indicates that under the repeating group both 55 and 48/22 are optional.
I received a message from my client without a symbol tag, please help me undersatnd if that was a bad formed message
20150923-15:06:14.976 : 8=FIXT.1.19=33635=X34=19153349=SENDER52=20150923-15:06:14.63756=RECEIVER268=8279=0269=1270=99.609375271=289279=0269=1270=99.6171875271=241279=0269=1270=99.625271=154279=0269=1270=99.6328125271=139279=0269=0270=99.6015625271=268279=0269=0270=99.59375271=244279=0269=0270=99.5859375271=171279=0269=0270=99.578125271=21610=198
You are advised to treat the default FIX message and field definitions as a set of suggested definitions.
In practice, no commercial FIX counterparty uses these definitions as-is. Every counterparty I've connected to makes modifications, adding or removing fields from messages or groups, creating new fields, or sometimes adding entirely new messages. No counterparty supports every message and field.
When connecting to a counterparty, do not assume anything. Your counterparty should provide documentation on how they expect their interface to be used, and which messages and fields they will send and which they expect to receive from you.
You need to read their specs and modify your FIXnn.xml DataDictionary file to match what they will be sending you.
If their spec says they will send you Symbol and/or SecurityID in a 35=X message, you need to make sure your DD file matches that.
This page might be helpful to you. (It's technically for the C# QuickFIX/n, but the DD file is the same for all QF versions.)
http://quickfixn.org/tutorial/custom-fields-groups-and-messages.html
I am noticing that I get an email from my server each time an order is placed. It looks like the customer confirmation emails are not sending.
This is part of the error message:
A message that you sent contained one or more recipient addresses that were
incorrectly constructed:
=?utf-8?B?R3Vlc3Q=?= <>: missing or malformed local part
This address has been ignored. The other addresses in the message were
syntactically valid and have been passed on for an attempt at delivery.
------ This is a copy of your message, including all the headers. ------
To: =?utf-8?B?R3Vlc3Q=?= <>
Subject: =?utf-8?B?SW50ZWxsaWdlbnQgV29ya3Nob3A6IE5ldyBPcmRlciAjIDEwMDAwMDAzMA==?=
It looks like it’s only when the user checks out without registering
=?utf-8?B?R3Vlc3Q=?= <>: missing or malformed local part
The error message is quite obvious: There should be an email address between <>, but there is none. The local part is the part of the address before the #, and as there is nothing at all here, there is no local part. Thus your server is complaining.
You have to fix whatever application is trying to send mails to <> to get rid of the error.
The email address is not written in proper format, please use <>
I really don't Understand what you are trying to do. But This is what I found out playing around with this error.
$to = "'Name of Person' <email#example.com>";
$headers = "From: ".$_POST['theirName']."<".$_POST['theirEmail'].">\r\n";
$subject = $_POST['subject'];
$messageBody = $_POST['message']."\r\n ---\n This Message was sent from ".HOME." contact form.";
mail($to,$subject,$messageBody,$headers);
Notice that around the 'Name of Person' has single quotes. If you are trying to send emails with an additional name, title or anything else in front of the emails. With the single quotes will let PHP know that it is not an email but a string, and it will add the string but be ignored as an email, not giving you the error anymore.
The header where it states from does not need to have the single quotes.
=?utf-8?B?R3Vlc3Q=?= <-This line I don't know what it supposed to be doing.
But it's not an Email on the $to/$recipient and for that reason it giving the error. Once again, must have only emails: "email1#example.com, email2#example.com" other than that: 'String' with single quotes no error will produce...
When people email each other, they generally include the original email in their reply to a sender, adding a little more information each time to the email. Each email client seems to have a different way of adding the original email to a reply.
I need to parse email arriving at our mail server and try and extract the new part of the message, and I'm wondering if there is a sensible way to strip this appended (or prepended) information (the "original message") and just get the new information in a mail body? I believe sadly, that there is no encoding, the original email is simply added to the new message, but I thought I'd check with the experts?
thanks.
No, there is no simple, straightforward algorithm to separate quoted or forwarded text from new content. Quoting and forwarding are poorly standardized and different conventions have existed at different times.
Having said that, e.g. Google's Gmail succeeds fairly well in practice. With enough samples, you can clearly come up with reasonable heuristics.
Good indicators for quoted material are forwarded (pseudo-) headers and indented text, perhaps with a quote indicator along the left margin before the quoted text. You occasionally see outdents as well.
Traditionally, on Usenet in the early 1990s, people would use different, unique quoting styles.
: ~ | This seems to be the original.
: ~ This is the first reply.
: This is the second reply.
This is the third reply, quoting the
previous three messages in sequence.
Around 1995, both clients and standardization initiatives by and large converged on "wedge" quotes;
> >> This seems to be the original.
> > This is the first reply.
> This is the second reply.
This is the third reply, quoting the
previous three messages in sequence.
Then along came Microsoft and ruined it all. I suppose that top quoting makes sense in some corporate settings where you quickly need to collect all the background from a thread to a new participant, but even for that purpose it's a horrible abomination.
This is the third reply, quoting the
previous three messages in sequence.
---- Begin forwarded message ----
From: Him [smtp:bogus]
To: His Friend
Subject: VS: Re: Same as on this message
Date: nothing machine-readable
This is the second reply.
---- Alkuperäinen viesti ----
Lähettäjä: His Friend [smtp:poppycock]
Saaja: Some Guy
Aihe: Re: Same as on this message
Päivämäärä: olisiko eilen ehkä
This is the first reply.
----- Original message ----
From: Somebody Else [smtp:mindless]
To: Some Guy
Subject: Same as on this message
Date: like, the day before
This seems to be the original.
I have a very special problem.
If we create a mail in Outlook, we add a UserProperty which contains a DataBase-ID of our System, so we can Link the mail to the representing DataBase-Item. On the service which reads the mails in each Mailbox and imports them automatically I can read this property by using ExtendedPropertyDefinitions. So far everything is fine...
If the User now forwards the message in Outlook, Olk copies the UserProperty to the new message. And now my problems beginn. Now my Service thinks the new message is also linked to our database and updates DB-Entry with the new Body and new Subject.
So does anyone now how to find out if a message is a forwarded one or how to tell Outlook not to copy the userproperty to the forwarded (new) message?
thx. Jay
What we thought about, but isnt working for our case
- a second userproperty containing a simple tag linke "fromSystem". Cause this would be copied too.
- a second userproperty containing a hashsum calculated from subject and Body. Cause both could be changed by the user. We just create the message, add all properties and Display it. from this Point on we no longer have control what is Happening to the mail until the Service handles it.
Your service consuming EWS should check the ConversationIndex and only update the database if it's 22 bytes long (original source message). Forward emails and reply emails keep appending 5 bytes (10 chars) to the ConversationIndex extending it beyond 22 bytes.
Sample ConversationIndexes
Original: 01CDD15D80E51C1D4522172840ACA96287DA28A15D97
Reply: 01CDD15D80E51C1D4522172840ACA96287DA28A15D970000018630
Forward: 01CDD15D80E51C1D4522172840ACA96287DA28A15D970000018630000000FC30
ConversationIndex represents the sequential ordering of the ConversationTopic (essentially GUID + timestamp). See Working with Conversations on MSDN. ConversationIndex is explicitly defined on MSDN here.
if (message.ConversationIndex.Length == 22)
{
// update DB body, subject, etc.
}
Also make sure you load the EmailMessageSchema.ConversationIndex before trying to access its value.