I'd like to use syslog-ng to do nested filtering of messages from sendmail. I think its called nested filtering. I don't know much about the intricacies of pattern matching and filtering in syslog-ng, I apologize.
I need to find a message that matches an IP address, then save the QID in that same message. Using that QID, I want to then capture all messages from the 'sm-mta' process that are for that specific QID. These messages don't always include that IP address, which makes this problem seem tricky to me.
Example:
Mar 28 22:58:22 mail-server.com sm-mta[18687]: 02SMwLVG018687: from=<someone#somewhere.com>, size=1581, class=0, nrcpts=7, msgid=<user#ip>, proto=ESMTP, daemon=MTA, tls_verify=NONE, auth=NONE, relay=source-server.com [3.3.3.3]
Look for relay=, and then match on 3.3.3.3. Save the 02SMwLVG018687 QID, then capture and send all messages that have that QID to a specific destination.
Can this be handled? Maybe I need post processing outside of syslog-ng?
Related
I am implementing a client to connect to a server which as far as I can tell uses a hybrid of FIX4.2 and FIX4.4.
The server sends group 453 (NoPartyIDs) with fields in a non-standard order when some events occur.
According to the specification document, the first field should be PartyID (448). With certain messages, the first field in the group is PartyIDSource (447) and the message is rejected. PartyIDSource is the second field in the group as per the specification.
I get the following error:
<event> Message 140 Rejected: Group 453's first entry does not start with delimiter 448 (Field=453)
From the documentation and trial and error, I cannot find a way through this issue. Amongst a few guesses, I have tried adding field 447 as the first (non-required) field in the group definition in the data-dictionary. I have also set ValidateFieldsOutOfOrder to N in the config.
Is there something I can do to not reject and process the message?
Relevant documentation:
Groups are a little more nuanced than other parts of the Data Dictionary.
A group is defined within a message, with the group tag. The first child element of the group tag is the group-counter tag, followed by the other fields in the group in the order in which they should appear in the message.
ValidateFieldsOutOfOrder is not relevant here, so you can take that out.
If I understand you correctly, you're saying that
sometimes 447 comes before 448
but at other times 448 comes before 447
If this is true, then unfortunately your counterparty is being really stupid. Per the FIX spec, the order of fields in repeating groups is supposed to be in a consistent order. (And also, the first field of the each group-sequence is always required to be present.) If they're flip-flopping fields, they're violating FIX.
If the order was consistent, you would just edit your DD to change the order, and it sounds like you tried that. But if your counterparty is flip-flopping, then your DD will always be wrong part of the time.
I don't have a good answer for you. QF/n is not designed to handle all the ways that counterparties do FIX wrong (nor should it be).
Your counterparty's implementation is sloppy. Try contacting their support and seeing if they'll fix it?
Using ConsumeIMAP to read emails from an Inbox and trying to select only emails that have
- attachment to download
- sent "From" xyz#yahoo.com
- send "To" abc#gmail.com
- Have "Daily" in their subject
- at 8 am EST
Please let me know if it can be set in any component. I tried to use EvaluateJsonPath, ExtractEmailHeaders and RouteonAttribute but no luck yet.
It sounds like you have been exploring the correct path. You should be able to achieve this using a flow consisting of:
ConsumeIMAP >> ExtractEmailHeaders >> RouteOnAttribute
ConsumeIMAP will download messages from the email server and create a single FlowFile for each message, storing the email message raw bytes in the FlowFile contents.
ExtractEmailHeaders attempts to parse a FlowFile's contents as email (must be RFC-2822 compliant), extract email headers, and write each header field to a FlowFile attribute, including:
email.headers.from.*
email.headers.to.*
email.headers.subject
email.headers.sent_date
Note that ExtractEmailHeaders is not doing any filtering, just populating FlowFile attributes based on the FlowFile content, thus making the FlowFiles more easily routable downstream in the flow. Start just by creating a flow that has these two processors and verify that the output of the ExtractEmailHeaders processor meets these expectations. If not, its possible the email messages are malformed or not RFC-2822 compliant.
After you have successfully sent email FlowFiles through ExtractEmailHeaders, you can do the filtering using one or more RouteOnAttribute processors using the NiFi Expression Language to define your match conditions, e.g.:
${email.headers.subject.contains("Daily")}
If you have verified that your flow is working correctly through ExtractEmailHeaders, but the filtering in RouteOnAttribute is not working as expected, make sure your attribute expressions and assumptions about email header values (e.g., capitalization, datetime format) are correct. Consult the Apache NiFi Expression Language Guide and if you have specific questions relating to the expression language itself, search here or post another question on that specifically.
I hope this helps!
I am using Elasticsearch to search an email inbox. I want to be able to search subject lines as well as message bodies. Simple enough. But if I just do a simple search that ends up matching a subject line, all the email messages in that thread will be returned (because they all have the same subject line). How can I use Elasticsearch to collapse all those messages into one response, while still having it return individual messages that have matches in the message bodies? I want results that behave similar to gmail search results, in other words. Is this possible without having to do two individual searches (one over subjects, one over message bodies), and then combining the results? I'd like to avoid that if possible.
Each message includes a thread identifier, which is unique per message thread.
I'm pretty new to the FIX protocol and was hoping someone could help clarify some terms.
In particular could someone explain (perhaps with an example) the flow of NewOrderSingle, ExecutionReport, CancelReplaceRequest and how the fields ClOrdID, OrdID, OrigClOrdID are used within those messages?
A quick note about usages of fields. My experience is that many who implement FIX do it slightly differently. So be aware that though I am trying to explain correct usage you may find that there are differences between implementations. When I connect to a new broker I get a FIX specification which details exactly how they use the protocol. I have to be very careful to make sure where they have deviated from other implementations.
That said I will give you a rundown of what you have asked for.
There are more complicated orders but NewOrderSingle is the one most used. It allows you to create a trade for any asset. You will need to create a new order using this object / msg type. Then you will send it through your session using the method sendToTarget(). You can modify the message after this point through the toApp() method, assuming your application implements the quickfix.Application interface.
The broker (or whoever you are connected to) will send you a reply in the form of and Execution report. Using quickfix that reply will enter your application through the fromApp() callback. From there the best thing to do is to implement your app inheriting from the MessageCracker class (or implement it elsewhere) using the crack method from MessageCracker it will then call back a relevant onMessage() method call. You will need to implement a number of these onMessage() methods (it depends on specifically what you are doing as to which methods you will need), the main one being onMessage(ExecutionReport msg, SessionID session). This method will be called by message cracker when you receive and Execution report from the broker. This is the standard reply to a new order.
From there you handle the reply as required.
Some orders do not get filled immediately like Limit orders. They can be changed. For that you will need the CancelReplaceRequest. Your broker will give you details of how to do this specifically for them (again there are differences and not everyone does it the same). You will have to have done a NewOrderSingle first and then you will use this MsgType to update it.
ClOrdID is an ID that the client uses to identify the order. It is sent with the NewOrderSingle and returned in the ExecutionReport. The OrdID tag is in the ExecutionReport message, it is the ID that the broker will use to identify the order. OrgClOrdID is usually used to identify the original order in when you do and update (using CancelReplaceRequest), it is supposed to contain the ClOrdID of the original order. Some brokers want the original order only, others want the ClOrdID of the last update, so the first OrigClOrdID or will be the ClOrdID of the NewOrderSingle, then if there are subsequent updates to the same order then they will be the ClOrderID from the last CancelReplaceRequest. Some brokers want the last OrderID and not ClOrderID. Note that the CancelReplaceRequest will require a ClOrdID as well.
I have a questing regarding MSMQ...
I designed an async arhitecture like this:
CLient - > WCF Service (hosted in WinService) -> MSMQ
so basically the WCF service takes the requests, processes them, adds them to an INPUT queue and returns a GUID. The same WCF service (through a listener) takes first message from queue (does some stuff...) and then it puts it back into another queue (OUTPUT).
The problem is how can I retrieve the result from the OUTPUT queue when a client requests it... because MSMQ does not allow random access to it's messages and the only solution would be to iterate through all messages and push them back in until I find the exact one I need. I do not want to use DB for this OUTPUT queue, because of some limitations imposed by the client...
You can look in your Output-Queue for your message by using
var mq = new MessageQueue(outputQueueName);
mq.PeekById(yourId);
Receiving by Id:
mq.ReceiveById(yourId);
A queue is inherently a "first-in-first-out" kind of data structure, while what you want is a "random access" data structure. It's just not designed for what you're trying to achieve here, so there isn't any "clean" way of doing this. Even if there was a way, it would be a hack.
If you elaborate on the limitations imposed by the client perhaps there might be other alternatives. Why don't you want to use a DB? Can you use a local SQLite DB, perhaps, or even an in-memory one?
Edit: If you have a client dictating implementation details to their own detriment then there are really only three ways you can go:
Work around them. In this case, that could involve using a SQLite DB - it's just a file and the client probably wouldn't even think of it as a "database".
Probe deeper and find out just what the underlying issue is, ie. why don't they want to use a DB? What are their real concerns and underlying assumptions?
Accept a poor solution and explain to the client that this is due to their own restriction. This is never nice and never easy, so it's really a last resort.
You may could use CorrelationId and set it when you send the message. Then, to receive the same message you can pick the specific message with ReceiveByCorrelationId as follow:
message = queue.ReceiveByCorrelationId(correlationId);
Moreover, CorrelationId is a string with the following format:
Guid()\\Number