If I write an order routing system based on QuickfixJ, can I just start submitting my trades to an exchange? Or do I need to register myself with the exchange or get permission or something like that?
I am not able to understand how QuickfixJ, the order routing system, the actual trading engine and the exchange fits together. Any online architecture diagram would be very helpful for how these components fit together.
FIX is just a transmission protocol. By itself, it's pretty dumb. QuickFIX (any language port) is just an engine that does all the boring dirty work of managing a FIX connection.
The FIX specification includes a list of messages and fields. In reality, you can treat these as suggestions that, in practice, no commercial FIX counterparty uses as-is. Every counterparty I've connected to makes modifications to those messages and fields, sometimes adding entirely new messages. No counterparty supports every message and field.
When connecting to a counterparty, do not assume anything. Your counterparty should provide documentation on how they expect their interface to be used, and which messages and fields they will send and which they expect to receive from you.
Their docs should tell you which message to send them to request market data and any special fields/options you must use.
Their docs will tell you how to submit a trade.
Their docs will tell you how to do anything that they support, and which messages/fields you will receive in return.
Do not try to send any message type to your counterparty unless their docs say they support it.
If you are writing the ORS side... then you have no docs. If you haven't written a FIX client before, you probably shouldn't be writing a FIX server without some assistance from someone who has. At the least, you should try to get ahold of some other systems' FIX interface docs to get an idea of how to go about it. (Unfortunately, such firms usually only give them to client-developers.)
Related
I'm writing an application in which users will be able to send money to each other. I've built out most of it, but now comes the most important part, managing transactions as they occur.
What I'd like to do upon a successful transaction is to send an update to the recipient. Right now, my thinking is to do this via SSE or WebSockets. For this particular app, it doesn't appear that I need bi-directional communication, since the response would only be sent to the recipient's instance which should be listening for a response from the server.
I might be answering my own question here, but I also wanted to factor in scale. If my app grows to a million users, for instance, which technology would best be able to handle the number of transactions being processed without failure?
I'm also a little unsure as to how to implement this for the case where there is a multiplicity of users, but I only want a particular user to receive the update.
Any advise would be greatly appreciated.
Thanks!
I am trying to create a network architecture which has a single server and multiple clients. The clients can connect and disconnect at any time so they need to announce their existence or shut-down to the server. The server must be able to send data to any particular client.
What is the best scalability protocols/architecture to use for this?
Currently I use a REQ/REP so that clients can 'login' and 'logout', and a SURVEY socket so that the server can send data to all clients. The message sent has an ID for the particular client it wants to process the message.
Is this good, or is there something better?
Sounds more like you need publisher subscriber. With both 0MQ and nanomsg you don't need to do anything in particular to manage connection / disconnection, the library does that for you.
However if you want more sophisticated message management (such as caching outgoing messages just in case another client chooses to connect) then you will have to manage that yourself. You might use a single push pull from the clients for them to announce their presence (they'd send a message saying who they were), followed by more push pulls from the server to each of the clients to send the messages from the cache that you have. Fiddly, but still a whole lot easier than programming up with raw sockets.
Using req rep can be problematic - if either end crashes or unexpectedly disconnects the other can be left in a stalled, unrecoverable state.
Sorry, in real world, there is no "One Size Fits All"
There are many further aspects, that influence architecture - The Bigger Picture.
While both Martin SUSTRIK's cool kids -- ZeroMQ & nanomsg -- have made a gread help in providing excellend bases + LEGO-type building blocks of Scaleable Formal Communication Patterns, they are only the beginning and saying that REQ/REP or SURVEY Behavioural Primitives ( great innovation, nevertheless still a building block ) are The Architecture would get upset almost all architects and evangelists.
The original question is important, however you already have gotten a first proposal to get it administratively closed as some people with "wider wingspan" feel your question is "too wide" or "opinion"-oriented instead an MCVE code-example driven ( ... yes, StackOverflow life is sometimes fast and cruel ).
So, without any further details available, my recommendation would be to check recent versions of PUB/SUB ( which can and do filter on PUB-side ( not on the SUB as was the design in the early ZeroMQ versions, once already xmited / delivered zillions of bytes round the world to just realise on the globally distributed peers level that no-one has yet SUB-ed to receive anything of that ) instead of the mentioned SURVEY.
Without any context it is nonsense to seriously judge, the less to improve what you try to design and implement.
I would do a poor service I were trying to do so.
The best next step?
What I can do for you right now is to direct you to see a bigger picture on this subject >>> with more arguments, a simple signalling-plane / messaging-plane illustration and a direct link to a must-read book from Pieter HINTJENS.
I am using Tigase(XMPP) server. I want to block every incoming message from a particular JID. At the moment i am blocking a particular JID by dropping it's packet in Message.java inside
/tigase/xmpp/impl
package. Is it the right way to do this, if not please guide me.
Thanks
An advantage of blocking messages in Message plugin is that the performance penalty for this filtering is reduced to minimal. However there are quite a few disadvantages of doing it this way:
You modify Tigase's code which makes you version update painful and time consuming
It does not allow you to filter out any other packets (such as presence or iq)
Even if you block messages in Message plugin this message may still be processed by other plugins which intercept messages (such as offline message, message archiver, etc...)
Now, what is the best way to implement such a filtering it depends on what you really want to do and why do you want to do it. Have you heard of privacy lists? Please take a look at it. Tigase fully implements privacy lists, why you do not want to use them? Why you cannot use them?
Usually such a filtering you speak about is done in the Tigase filter called preprocessor. Please take a look at the privacy lists plugin or domain filter for a code example.
I'm a real Erlang newbie (started 1 week ago), and I'm trying to learn this language by creating a small but efficient chat server. (When I say efficient I mean I have 5 servers used to stress test this with hundreds of thousands connected client - A million would be great !)
I have find some tutorials doing so, the only thing is, that every tutorial i found, are IRC like. If one user send a message, all user except sender will receive it.
I would like to change that a bit, and use one-to-one discussion.
What would be the most effective client pool for searching a connected user ?
I thought about registering the process, because it seems to do everything I need, but I really don't think this is the better way to do it. (Or most pretty way to do it anyway).
Does anyone would have any suggestions doing this ?
EDIT :
Every connected client is affected to an ID.
When the user is connected, it first send a login command to give it's id.
When an user wants to send a message to another one the message looks like this
[ID-NUMBER][Message] %% ID-NUMBER IS A FIXED LENGTH
When I ask for "the most effective client pool", I'm actually looking for the fastest way to retrieve/add/delete one client on the connected client list which could potentially be large (hundred of thousands -- maybe millions)
EDIT 2 :
For answering some questions :
I'm using Raw Socket (Using telnet right now to communicate with server) - will probably move to ssl later...
It is my own protocol
Every Client is a spawned Pid
Every Client's Pid is linked to it's own monitor (mostly for debugging reason - The client if disconnected should reconnect by it's own starting auth from scratch)
I have read a couple a book before starting coding, So I do not master yet every aspect of Erlang but I'm not unaware of it, I will read more about it when needed I guess.
What I'm really looking for is the best way to store and search thoses PIDs to send message directly from process to process.
Should I write my own search Client function using lists ?
or should I use ets ?
Or even use register/2 unregister/1 and whereis/1 to maintain my client list, using it's unique id as atom, it seems to be the simplest way to do so, I really don't know if it is efficient, but I'm pretty sure this is the ugly solution ;-) ?
I'm doing something similar to your chat program using gproc as a pubsub (similar to the demo on that page). Each client registers as it's id. To find a particular client, you do a lookup on that client id. To subscribe to a client, you add a property to that process of the client id being subscribed to. To publish, you call gproc:send(ClientId,Message). This covers your use case, the more general room based chat as well, and can handle distributed masterless registry of processes.
I haven't tested to see if it scales to millions, but it uses ets to do the storage and gproc is rock solid code by Ulf Wiger. I wouldn't count on being able to write a better implementation.
I'm also kind of new to Erlang (a couple of months), so I hope this can put you in the correct path :)
First of all, since you're a "newbie", you should know about these sites:
Erlang Official Documentation:
Most common modules are in the stdlib application, so start from there.
Alternative Documentation:
There's a real time search engine, so it is really good when searching
for specific modules.
Erlang Programming Rules:
For you to enter in the mindset of erlang programming.
Learn You Some Erlang Book:
A must read for everyone starting with Erlang. It's really comprehensive
and fun to read!
Trapexit.org:
Forum and cookbooks, to search for common problems faced by programmers.
Well, thinking about a non persistent database, I would suggest the sets or gb_sets modules (documentation here).
If you want persistence, you should try dets (see documentation above), but I can't state anything about efficiency, so you should research this topic a bit further.
In the book Learn You Some Erlang there is a chapter on data structures that says that sets are better for read intensive systems, while gb_sets is more appropriate for a balanced usage.
Now, Messaging systems are what everyone wants to do when they come to Erlang because the two naturally blend. However, there are a number of things to look into before one continues. Messaging basically involves the following things: User Registration, User Authentication, Sessions Management,Logging, Message Switching/routing e.t.c. Now, to do all or most of these, one needs to have a Database, certainly IN-MEMORY, thats leads me to either Mnesia or ETS Tables. Since you are new to Erlang, i suppose you have not yet really mastered working with these. At one moment, you will need to maintain Who is communicating with who, Who is available for Chat e.t.c. Hence you might need to look up things and write things some where.Another thing is you have not told us the Client. Is it going to be a Web Client (HTTP), is it an entirely new protocol you are implementing over raw Sockets ? Which ever way, you will need to master something called: Concurrency in Erlang. If a user connects and is assigned an ID, if your design is A process Per User, then you will have to save the Pids of these Processes or register them against some criteria, yet again monitor them if they die e.t.c. Which brings me to OTP and Supervision trees. There is quite alot, however, tell us more about the Client and Server interaction, the Network Communication you need e.t.c. Or is it just a simple Erlang RPC project you are doing for your own revision ?
EDIT Use ETS Tables, or use Mnesia RAM tables. Do not think of registering these Pids or Storing them in a list, Array or set. Look at this solution which was given to this question
I was recently approached by my management with an interesting problem - where I am pretty sure I am telling my bosses the correct information but I really want to make sure I am telling them the correct stuff.
I am being asked to develop some software that has this function:
An application at one location is constantly processing real-time data every second and only generates data if the underlying data has changed in any way.
On the event that the data has changed send the results to another box over a network
Maintains a persistent connection between the both machines, altering the remote box if for some reason the network connection went down
From what I understand, I imagine that I need to do some reading on doing some sort of TCP/IP socket-level stuff. That way if the connection is dropped the remote location will be aware that the data it has received may be stale.
However management seems to be very convinced that this can be accomplished using SOAP. I was under the impression that SOAP is more or less a way for a client to initiate a procedure from a server and get some results via the HTTP protocol. Am I wrong in assuming this? I haven't been able to find much information on how SOAP might be able to solve a problem like this.
I feel like a lot of people around my office are using SOAP as a buzzword and that has generated a bit of confusion over what SOAP actually is - and is capable of.
Any thoughts on how to accomplish this task would be appreciated!
I think SOAP is the wrong tool. SOAP is a spec for exchanging structured data. For your problem, the simplest thing would be to write a program to just transfer data and figure out if the other end is alive. Sockets are a good way to go. There are lots of socket programming tutorials on the net. Pick your language, and ask Mr. Google. Write a couple of demo programs to teach yourself how it works. Ask if you have more specific questions.
For the problem, you'll need a sender and a receiver. The sender sends data when it gets it, the receiver waits for data and hands it off when it arrives. Get that working first. Next, add in heartbeats; a message that says "I'm alive", sent periodically. Get that working next. You'll need to be determine the exact behavior you want -- should both sides send heartbeats to the other end, the maximum time you are willing to wait for a heartbeat, and what action you take should heartbeats stop arriving. The network connection can drop, the other end can crash, the other end can hang, and perhaps there are other conditions you should think about (e.g., what if the real time data is nonsense?). Figure out how to handle each condition, and code up the error handling. Test it out, and serve with a side of documentation.
SOAP certainly won't tell you when the data source goes down, though you could use "heartbeats" to add that.
Probably you are right and they are just repeating a buzz word, and don't actually know much about what SOAP is or does or have any real argument for why it ought to be used here.