Instead of using the quickfix engine and its ways of send/receive fix messages, if i were to implement fix protocol over dds how would you recommend i go over it?
I am new to dds. I understand that i might have to configure topics but what would the topics be for fix?
You should consider using TargetCompID (tag 49) and/or other header tags (DeliverToCompID tag 128) to contrive the topic. Either the values of those tags are the topic, or you use them to look up the topic on which to publish (i.e. lookup in a routing table / in-memory map). This is the common approach for building a buyside/sellside hub.
Alternatively, you can use Symbol (tag 55), Account (tag 1), ExDest (tag 207) or any other tags in a similar manner. It all depends on the use case.
FIX over RV is pretty common - there are entire large-scale applications in production which use FIX to receive data and RV to route to other "outbound" FIX sessions. You're on the right track.
Related
I still cannot wrap my head around how is kafka producers / consumers and schema registry intended to reuse KafkaProperties. Or is it not intended to reuse same structures?
so for schema registry, I have to configure for example following properties:
spring.kafka.basic.auth.credentials.source
spring.kafka.basic.auth.user.info
spring.kafka.producer.properties.schema.registry.url
spring.kafka.consumer.properties.schema.registry.url
but if I do so, and call org.springframework.boot.autoconfigure.kafka.KafkaProperties#buildConsumerProperties
and proceed with build hashmap I will get for example warning:
The configuration 'schema.registry.url' was supplied but isn't a known config.
I saw recommendation to set schema registry urls as such, I also saw setting basic.auth… like this. I really cannot get it working. I mean the app is working, I just get several pages of these warnings. I'd like to know how to configure the app correctly, as it was intended by design, so that I can share 1 configuration for confluent kafka and schema-registry configuration
Sure I can get separate configuration of properties "to be added" for schema registry, or bend it somehow, so that I can build separate set of properties for both, but it just don't feel right, this clearly isn't how it was (I hope) designed. What is the correct procedure here? Maybe it's hidden somewhere in depth of autoconfiguration, but if its there I cannot find it.
I answered a similar question on Gitter earlier today; it comes up a lot:
https://gitter.im/spring-projects/spring-kafka?at=60d490459cf3171730f0d2d4
It's rather unfortunate that Kafka supports configuring (de)serializers via the producer/consumer configs, but then emits these annoying (ignorable) log messages.
If it bothers you, I suggest you open a JIRA against the kafka-clients.
We are currently working on a FIX connection, whereby data that should only be validated can be marked. It has been decided to mark this data with a specific TargetSubID. But that implies a new session.
Let's say we send the messages to the session FIX.4.4:S->T. If we then get a message that should only be validated with TargetSubID V, this implies the session FIX.4.4:S->T/V. If this Session is not configured, we get the error
Unknown session: FIX.4.4:S->T/V
and if we explicitly configure this session next to the other, there is the error
quickfix.Session – [FIX/Session] Disconnecting: Encountered END_OF_STREAM
what, as bhageera says, is that you log in with the same credentials.
(...) the counterparty I was connecting to allows only 1 connection
per user/password (i.e. session with those credentials) at a time.
I'm not a FIX expert, but I'm wondering if the TargetSubID is not just being misused here. If not, I would like to know how to do that. We develop the FIX client with camel-quickfix.
It depends a lot on what you system is like and what you want to achieve in the end.
Usually the dimensions to assess are:
maximising the flexibility
minimising the amount of additional logic required to support the testing
minimising the risk of bad things happening on accidental connection from a test to a prod environment (may happen, despite what you might think).
Speaking for myself, I would not use tags potentially involved in the sesson/routing behavior for testing unless all I need is routing features and my system reliably behaves the way I expect (probably not your case).
Instead I would consider one of these:
pick something from a user defined range (5000-9999)
use one of symbology tags (say Symbol(55)) corrupted in some reversible way (say "TEST_VOD.L" in the tag 55 instead of "VOD.L")
A tag from a custom range would give a lot of flexibility, a corrupted symbology tag would make sure a test order would bounce if sent to prod by accident.
For either solution you may potentially need a tag-based routing and transformation layer. Both are done in couple of hours in generic form if you are using something Java-based (I'd look towards javax.scripting / Nashorn).
It's up to the counterparties - sometimes Sender/TargetSubID are considered part of the unique connection, sometimes they distinguish messages on one connection.
Does your library have a configuration option to exclude the sub IDs from the connection lookups? e.g. in QuickFix you can set the SessionQualifier.
With ZeroMQ PUB/SUB model, is it possible for a subscriber to filter based on contents of more than just the first frame?
For example, if we have a multi-frame message that contains three frames1) data type,2) instrument, and then 3) the actual data,is it possible to subscribe to a specific data type, instrument pair?
(All of the examples I've seen only shows filtering based off of the first message of a multipart message).
Different modes of ZeroMQ PUB/SUB filtering
Initial ZeroMQ model used a subscriber-side subscription-based filtering.
That was easier for publisher, as it need not handle subscriber-specific handling and simply fed all data to all SUB-s ( yes, at the cost of the vasted network traffic and causing SUB-side workload to process all incoming BLOBs, all the way up from the PHY-media, even if it did not subscribe to the particular topic. Yes, pretty expensive in low-latency designs )
PUB-side filtering was initially proposed for later implementation
still, this mode does not allow your idea to fly just on using the PUB/SUB Scaleable Formal Communication Pattern.
ZeroMQ protocol design strives to inspire users, how to achieve just-enough designed distributed systems' behaviour.
As understood from your 1-2-3 intention, a custom logic shall be just enough to achieve the desired processing.
So, how to approach the solution?
Do not hesitate to setup several messaging / control relays in parallel in your application domain-specific solution, it works much better for any tailor-made solutions and a way safer than to try to "bend" a library's primitive archetype ( in this case the trivial PUB/SUB topic-based filtering ) to make something a bit different, than for what an original use-case was designed.
The more true, if your domain-specific use is FOREX / Equity trading, where latency is your worst enemy, so a successful approach has to minimise stream-decoding and alternative branching as much as possible.
nanoseconds do matter
If one reads into details about multiframe composition ( a sender-side BLOB assembly ), there is no latency-wise advantage, as the whole BLOB gets onto wire only after it has been completed - se there is no advantage for your SUB-side in case your idea was to handle initial frame content for signalling, as the whole BLOB arrives "together"
I'm very much at the beginning of using / understanding EventStore or get-event-store as it may be known here.
I've consumed the documentation regarding clients, projections and subscriptions and feel ready to start using on some internal projects.
One thing I can't quite get past - is there a guide / set of recommendations to describe the difference between event metadata and data ? I'm aware of the notional differences; Event data is 'Core' to the domain, Meta data for describing, but it is becoming quite philisophical.
I wonder if there are hard rules regarding implementation (querying etc).
Any guidance at all gratefully received!
Shamelessly copying (and paraphrasing) parts from Szymon Kulec's blog post "Enriching your events with important metadata" (emphases mine):
But what information can be useful to store in the metadata, which info is worth to store despite the fact that it was not captured in
the creation of the model?
1. Audit data
who? – simply store the user id of the action invoker
when? – the timestamp of the action and the event(s)
why? – the serialized intent/action of the actor
2. Event versioning
The event sourcing deals with the effect of the actions. An action
executed on a state results in an action according to the current
implementation. Wait. The current implementation? Yes, the
implementation of your aggregate can change and it will either because
of bug fixing or introducing new features. Wouldn’t it be nice if
the version, like a commit id (SHA1 for gitters) or a semantic version
could be stored with the event as well? Imagine that you published a
broken version and your business sold 100 tickets before fixing a bug.
It’d be nice to be able which events were created on the basis of the
broken implementation. Having this knowledge you can easily compensate
transactions performed by the broken implementation.
3. Document implementation details
It’s quite common to introduce canary releases, feature toggling and
A/B tests for users. With automated deployment and small code
enhancement all of the mentioned approaches are feasible to have on a
project board. If you consider the toggles or different implementation
coexisting in the very same moment, storing the version only may be
not enough. How about adding information which features were applied
for the action? Just create a simple set of features enabled, or map
feature-status and add it to the event as well. Having this and the
command, it’s easy to repeat the process. Additionally, it’s easy to
result in your A/B experiments. Just run the scan for events with A
enabled and another for the B ones.
4. Optimized combination of 2. and 3.
If you think that this is too much, create a lookup for sets of
versions x features. It’s not that big and is repeatable across many
users, hence you can easily optimize storing the set elsewhere, under
a reference key. You can serialize this map and calculate SHA1, put
the values in a map (a table will do as well) and use identifiers to
put them in the event. There’s plenty of options to shift the load
either to the query (lookups) or to the storage (store everything as
named metadata).
Summing up
If you create an event sourced architecture, consider adding the
temporal dimension (version) and a bit of configuration to the
metadata. Once you have it, it’s much easier to reason about the
sources of your events and introduce tooling like compensation.
There’s no such thing like too much data, is there?
I will share my experiences with you which may help. I have been playing with akka-persistence, akka-persistence-eventstore and eventstore. akka-persistence stores it's event wrapper, a PersistentRepr, in binary format. I wanted this data in JSON so that I could:
use projections
make these events easily available to any other technologies
You can implement your own serialization for akka-persistence-eventstore to do this, but it still ended up just storing the wrapper which had my event embedded in a payload attribute. The other attributes were all akka-persistence specific. The author of akka-persistence-eventstore gave me some good advice, get the serializer to store the payload as the Data, and the rest as MetaData. That way my event is now just the business data, and the metadata aids the technology that put it there in the first place. My projections now don't need to parse out the metadata to get at the payload.
I consider to buy this connector:
MICEX FIX/FAST Market Data Adaptor http://www.b2bits.com/trading_solutions/market-data-solutions/micex-fixfast.html
But I don't like propriety software by some reasons and would prefer to replace this connector with QuickFix + DIY code.
100 usec perfomance difference is not critical for me, but I do care about features.
In particular MICEX uses FIX+Fast and referenced connector automatically decodes fast: "Hides FAST functionality from user, automatically applies FAST decoding."
The question is how to do the same with quickfix? Is it good idea at all? How easy would be to implement referenced connector using quickfix?
Have you looked at http://code.google.com/p/quickfast/ I have used it and is mostly works but it is not the best library.
I don't believe QuickFIX supports FAST. FAST is a complex compression specification for FIX messages, and to implement FAST on top of QuickFIX or any FIX engine in a performing way can be tricky.
You want to choose a FAST engine that can generate template-decoding source code, in other words, it reads the template XML file from the exchange and spills out code to parse each template. Done this way it's automatic, easy and crucial for speed as the code generated avoids recursion calls otherwise necessary to parse repeating groups.
Take a look on CoralFIX which is an intuitive FIX engine with support for FAST decoding.
Disclaimer: I am one of the developers of CoralFIX.