Order Placing System
A user places an order and the corresponding amount is kept on hold for the order and an order is created. This order is then pushed to some queue to be used by trade matching system. User gets back a reference order id for the order placed in return to the API call.
Trade Matching System
The system feeds on data from the queue generated by order placing system and looks for possible match and if possible to execute, executes them and push to another queue.
User Notification System
The system fetches data from the executed queue and broadcasts it to the user it belonged to. User can also fetch status of the order from the reference id which was shared on first API call
These two systems are right now communicating indirectly via a queue. Now the requirement is, in order placing system, when a user places order, along with order id, we also need to return execution status (i.e. Whether it got executed or not, if yes, rate and fee charged etc).
What should be mode of communication between order placing and trade matching system to make it possible to return execution details in first api call itself ?
Challenges
Matching System being single threaded, we cannot merge it with order engine
Polling and waiting for execution from execution queue, will probably make our order placing API slow
Right now our order placing system and matching system are separate.
Just looking for possible solution and opinion. Please let me know if something is unclear.
Related
With respect to FIX 4.2 or greater:
Q1.a. How are incoming and outgoing sequence #’s correlated/linked? Is there a buyer specific FIX tag a buyer can embed/use explicitly for tracking upon submitting a buy order that is also included in subsequent incoming status message sequences from the broker?
Q1.b. If not, then how does a buyer manage/track individually several IOC buy orders that are submitted in quick succession or concurrently of securities which may or may not be identical, at different price levels, where units or shares are “filled” at varying rates?
Q1.a. How are incoming and outgoing sequence #’s correlated/linked?
They are not linked (i.e. they are independant). Any FIX application/engine (such as the QuickFIX family) maintains two sequence numbers per session, one for incoming and one for outgoing. See also this answer on Stack Overflow which pretty much tells you the same.
When using an engine like any of the QuickFIX family (QuickFIX, QuickFIX/J, QuickFIX/N), these will be managed for you and apart from some configuration vis-a-vis your counterparty you should not bother about managing these.
Q1.a. Is there a buyer specific FIX tag a buyer can embed/use explicitly for tracking upon submitting a buy order that is also included in subsequent incoming status message sequences from the broker?
These tags are already present in e.g. the FIX Order Single message (D) - ClOrdId:
Unique identifier for Order as assigned by the buy-side (institution, broker, intermediary etc.) [...]. Uniqueness must be guaranteed within a single trading day. Firms, particularly those which electronically submit multi-day orders, trade globally or throughout market close periods, should ensure uniqueness across days, for example by embedding a date within the ClOrdID field.
This field is mandatory when creating a new order using FIX Order Single, and is used the refer to the order in subsequent messaging (e.g. Execution Report, or Status messages).
Note that the ClOrdId changes when an order is changed using an Order Cancel/Replace Request <G>, i.e. you assign a new ClOrdId to the order when changing or canceling it.
I have a very complicated job process and it's not 100% clear to me where to handle what.
I don't want to have code, it just the question who is responsible for what.
Given is the following:
There is a root directory "C:\server"
Inside are two directories "ftp" and "backup"
Imagine the following process:
An external customer sends a file into the ftp directory.
An importer application get's the file and now the fun starts.
A job aggregate have to be created for this file.
The command "CreateJob(string file)" is fired.
?. The file have to be moved from ftp to backup. Inside the CommandHandler or inside the Aggregate or on JobCreated event?
StartJob(Guid jobId) get's called. A third folder have to be created "in-progress", File have to be copied from backup to in-progress. Who does it?
So it's unclear for me where Filesystem things have to be handled if the Aggregate can not work correctly without the correct filesystem.
Because my first approach was to do that inside an Infrastructure layer/lib which listen to the events from the job layer. But it seems not 100% correct?!
And top of this, what is with replaying?
You can't replay things/files that were moved, you have to somehow simulate that a customer sends the file to the ftp folder...
Thankful for answers
The file have to be moved from ftp to backup. Inside the CommandHandler or inside the Aggregate or on JobCreated event?
In situations like this, I move the file to the destination folder in the Application service that sends the command to the Aggregate (or that calls a command-like method on the Aggregate, it's the same) before the command is sent to the Aggregate. In this way, if there are some problems with the file-system (not enough permissions or space is not available etc) the command is not sent. These kind of problems should not reach our Aggregate. We most protect it from the infrastructure. In fact we should keep the Aggregate isolated from anything else; it must contain only pure business logic that is used to decide what events get generated.
Because my first approach was to do that inside an Infrastructure layer/lib which listen to the events from the job layer. But it seems not 100% correct?!
Indeed, this seems like over engineering to me. You must KISS.
StartJob(Guid jobId) get's called. A third folder have to be created "in-progress", File have to be copied from backup to in-progress. Who does it?
Whoever's calling the StartJob could do the moving, before the StartJob gets called. Again, keep the Aggregate pure. In this case it depends on your framework/domain details.
And top of this, what is with replaying? You can't replay things/files that where moved, you have to somehow simulate that a customer sends the file to the ftp folder...
The events are loaded from the event store and replayed in two situations:
Before every command gets sent to the Aggregate, the Aggregate Repository loads all the events from the event store then it applies every one of them to the Aggregate, probably calling some applyThisEvent(TheEvent) method on the Aggregate. So, this methods should be with no side effects (pure) otherwise you change the outside world again and again at every command execution and you don't want that.
The read-models (the projections, the query-models) that present data to the user listen to those events and update the database tables that hold the data that the users see. The events are sent to those read-models after they are generated and every time the read-models are being recreated. When you invent a new read-model, you must pass it all the events that were previous generated by the aggregates in order to build the correct/complete state on them. If your read-model's event listeners have side effects what do you think happens when you replay those long past events? The outside world is modified again and again and you don't want that! The read-models only interpret the events, they don't generate other events and they don't change the outside world.
There is a special third case when events reach another type of model, a Saga. A Saga must receive an event only once! This is the case that you thought to use in Because my first approach was to do that inside an Infrastructure layer/lib which listen to the events from the job layer. You could do this in your case but is not KISS.
I have a very complicated job process and it's not 100% clear to me where to handle what. I don't want to have code, it just the question who is responsible for what.
The usual answer is that the domain model -- aka the "aggregate" makes decisions, and saves them. Observing those decisions, some event handler induces side effects.
And top of this, what is with replaying? You can't replay things/files that where moved, you have to somehow simulate that a customer sends the file to the ftp folder...
You replay the events to the aggregate, so that it is restored to the state where it made the last decision. That's a separate concern from replaying the side effects -- which is part of the motivation for handling the side effects elsewhere.
Where possible, of course, you prefer to have the side effects be idempotent, so that a duplicated message doesn't create a problem. But notice that from the point of view of the model, it doesn't actually matter whether the side effect succeeds or not.
Suppose I have an aggregate containing some data and when it reaches a certain state, I'd like to take all that state and pass it to some outside service. For argument and simplicity's sake, lets just say it is an aggregate that has a list and when all items in that list are checked off, I'd like to send the entire state to some outside service. Now when I'm handling the command for checking off the last item in the list, I'll know that I'm at the end but it doesn't seem correct to send it to the outside system from the processing of the command. So given this scenario what is the recommended approach if the outside system requires all of the state of the aggregate. Should the outside system build its own copy of the data based on the aggregate events or is there some better approach?
Should the outside system build its own copy of the data based on the aggregate events.
Probably not -- it's almost never a good idea to share the responsibility of rehydrating an aggregate from its history. The service that owns the object should be responsible for rehydration.
First key idea to understand is when in the flow the call to the outside service should happen.
First, the domain model processes the command arguments, computing the update to the event history, including the ChecklistCompleted event.
The application takes that history, and saves it to the book of record
The transaction completes successfully.
At this point, the application knows that the operation was successful, but the caller doesn't. So the usual answer is to be thinking of an asynchronous operation that will do the rest of the work.
Possibility one: the application takes the history that it just saved, and uses that history to create schedule a task to rehydrate a read-only copy of the aggregate state, and then send that state to the external service.
Possibility two: you ditch the copy of the history that you have now, and fire off an asynchronous task that has enough information to load its own copy of the history from the book of record.
There are at least three ways that you might do this. First, you could have the command schedule the task as before.
Second, you could have a event handler listening for ChecklistCompleted events in the book of record, and have that handler schedule the task.
Third, you could read the ChecklistCompleted event from the book of record, and publish a representation of that event to a shared bus, and let the handler in the external service call you back for a copy of the state.
I was under the impression that one bounded context should not reach out to get state from another bounded context but rather keep local copies of the data it needed.
From my experience, the key idea is that the services shouldn't block each other -- or more specifically, a call to service B should not block when service A is unavailable. Responding to events is fundamentally non blocking; does it really matter that we respond to an asynchronously delivered event by making an asynchronous blocking call?
What this buys you, however, is independent evolution of the two services - A broadcasts an event, B reacts to the event by calling A and asking for a representation of the aggregate that B understands, A -- being backwards compatible -- delivers the requested representation.
Compare this with requiring a new release of B every time the rehydration logic in A changes.
Udi Dahan raised a challenging idea - the notion that each piece of data belongs to a singe technical authority. "Raw business data" should not be replicated between services.
A service is the technical authority for a specific business capability.
Any piece of data or rule must be owned by only one service.
So in Udi's approach, you'd start to investigate why B has any responsibility for data owned by A, and from there determine how to align that responsibility and the data into a single service. (Part of the trick: the physical view of a service can span process boundaries; in other words, a process may be composed from components that belong to more than one service).
Jeppe Cramon series on microservices is nicely sourced, and touches on many of the points above.
You should never externalise your state. Reporting on that state is a function of the read side, as it produces reports and you'll need that data to call the service. The structure of your state is plastic, and you shouldn't have an external service that relies up that structure otherwise you'll have to update both in lockstep which is a bad thing.
There is a blog that puts forward a strong argument that the process manager is the correct place to put this type of feature (calling an external service), because that's the appropriate place for orchestrating events.
I found time as the best value as event version.
I can merge perfectly independent events of different event sources on different servers whenever needed without being worry about read side event order synchronization. I know which event (from server 1) had happened before the other (from server 2) without the need for global sequential event id generator which makes all read sides to depend on it.
As long as the time is a globally ever sequential event version , different teams in companies can act as distributed event sources or event readers And everyone can always relay on the contract.
The world's simplest notification from a write side to subscribed read sides followed by a query pulling the recent changes from the underlying write side can simplify everything.
Are there any side effects I'm not aware of ?
Time is indeed increasing and you get a deterministic number, however event versioning is not only serves the purpose of preventing conflicts. We always say that when we commit a new event to the event store, we send the new event version there as well and it must match the expected version on the event store side, which must be the previous version plus exactly one. If there will be a thousand or three millions of ticks between two events - I do not really care, this does not give me the information I need. And if I have missed one event on the go is critical to know. So I would not use anything else than incremental counter, with events versioned per aggregate/stream.
I have a scala client that talk to the CallFire API. I can't find anything in the documentation about having a phone number be immediately available (accept phone calls) after placing an order from the API. Here is the specific line I use: https://github.com/oGLOWo/callfire-scala-client/blob/master/src/main/scala/com/oglowo/callfire/Client.scala#L166
I need these numbers to be available when my customers purchase them. Are there any parameters that I don't know about or something that I'm doing wrong that is causing the numbers to not pick up for several minutes?
Number purchases can take several minutes to fulfil as the order is processed by the upstream number provider, which can vary according to the region and number type. As such, this is necessarily an asynchronous process.
My suggestion would be that after you create the number order, each time that it is necessary to know the status of the number you purchased, you can invoke the GetNumber operation to get status information for that number.
The most relevant field for your purposes would be the "Status" field, which indicates where in the number fulfilment process that number is. Once the status has transitioned to "Active", your number should be fully available.
Additionally, you can look at the "CallFeature" and "TextFeature" fields, in the NumberConfiguraton section of the Number resource, to see whether the number has confirmed call or text service yet, respectively.
Alternatively, you can also invoke the GetNumberOrder operation to get the status of your order. This will give you information on the status of the number order itself, but in my opinion is less useful for your purposes than querying the number status directly.
It is also worth mentioning that there are cases where the number is technically being serviced, but CallFire's number inventory hasn't yet been updated to indicate this. This can be pushed along by creating inbound traffic to the number on each of the features. That is, you might have a number which "activates" the numbers you purchase more rapidly by sending a call or text to them. This is due to the slight delay between the number being configured upstream, and CallFire's systems being notified of that fact. By sending traffic to the number, you more rapidly give CallFire's systems feedback that the number is enabled. This can save you up to a couple of minutes, if time is of the essence.
Your question has prompted me to create a feature request for CallFire internally, to add an event type to CreateSubscription for when number orders transition between statuses. This way, you could avoid having to poll for number/order statuses repeatedly, and instead we would notify your server by HTTP POST when the number order transitions to finished.