I have a service A that creates an email and sends it to a customer. The customer will receive the email and will, eventually, click on the link in the body to trigger service B.
How can I correlate two different and completely isolated services that are part of the same business process with sleuth?
Should I leave the span "opened" or is there a way to "embed" the trace id somehow on the email?
You can use asynchronous communication (http://cloud.spring.io/spring-cloud-sleuth/spring-cloud-sleuth.html#_asynchronous_communication) for example via a trace representation of the ExecutorService called the TraceableExecutorService (http://cloud.spring.io/spring-cloud-sleuth/spring-cloud-sleuth.html#_executor_executorservice_and_scheduledexecutorservice). You emit a completable future that will process the data in a separate thread. At some point you block and then you can retrieve the data. The trace representation of the ExecutorService will take care of passing of tracing data.
UPDATE:
If however, these are completely two separate processes then I'd close the span and create a completely separate span the moment someone clicks on the link. You should never leave spans explicitly open. What will bind the 2 processes will be the trace id. Zipkin doesn't yet support these long living tasks in the best possible way from the point of view of the UI but there's some work in progress going on to improve it (via so-called linked spans)
Related
When designing an application's back-end you will often need to abstract the systems that do things from the systems that actually do them.
There are elements of this in the CQRS and PubSub design patterns.
By way of example:
A new user submits a registration form
Your application receives that data and pushes out a message saying “hey i have some new user data, please do something with this”
A listener / handler / service grabs the data and processes it
(please let me know if that makes no sense)
In my applications I would usually:
Fire a new Event that a Listener is set up to process Event::fire('user.new', $data)
Create a new Command with the data, which is bound to a CommandHandler new NewUserCommand($data)
Call a method in a Service and pass in the data UserService::newUser($data)
While these are nearly exactly the same, I am just wondering - how do you go about deciding which one to use when you are creating the architecture of your applications?
Fire a new Event that a Listener is set up to process
Event::fire('user.new', $data)
Event pattern implies that there could be many handlers, subscribing to the same event and those handlers are disconnected form the sender. Also event handlers usually do not return information to the sender (because there can be actually many handlers and there is a confusion about whose information to return).
So, this is not your case.
Create a new Command with the data, which is bound to a CommandHandler
new NewUserCommand($data)
Commands are an extended way to perform some operation. They can be dispatched, pipelined, queued etc. If you don't need all that capabilities, why to complicate things?
Call a method in a Service and pass in the data
UserService::newUser($data)
Well, this is the most suitable thing for your case, isn't it?
While these are nearly exactly the same, I
am just wondering - how do you go about deciding which one to use when
you are creating the architecture of your applications?
Easy. From many solutions choose only those, which:
metaphorically suitable (do not use events, where your logic does not look like an event)
the simplest (do not go too deep into the depths of programming theories and methods. Always choose solution, that lowers your project development complexity)
When to use command over event?
Command: when I have some single isolated action with few dependencies which must be called from different application parts. The closest analogue is some editor command, which is accessible both from toolbar and menu.
Event: when I have several (at least in perspective) dependent actions, which may be called before/after some other action is executed. For example, if you have a number of services, you can use events to perform cache invalidation for them. Service, that changes a particular object emits "IChangedObject" event. Other services subscribe to such events and respond to them invalidating their cache.
In the old days, when each request to a web application was handled by one thread, it was fairly easy to understand the logs. One could, for example, use a servlet filter to name the thread that was handling a request with some sort of request id. This request id then could be output in the logs. In this world, a simple grep was all it took to collect the log lines for a given request.
In my current position, I'm building web applications with Scala (we're using Scalatra but that isn't specifically relevant to my question). Each request creates a scala.concurrent.Future and is then parked until that future has completed. The important bit here is that the thread that actually handles the business logic is different from the thread that handled the request which is different (I think) from the thread that completes the request and so the context of that request is lost during processing. The business logic can log all it likes but it is hard to associate that logging with the specific request it relates to.
Now from the standpoint of supporting my web services in production, the old approach was great and I'd like to come up with something similar for my asynchronous services. I've been trying to come up with a way to do it but have come up empty. That is, I haven't come up with anything nearly as light weight as the old, name-the-thread model. Does the Stack Overflow crowd have any suggestions?
Thanks
As you have written, assign an id to each request, and pass that to the business logic function. You can also do this with implicit parameter, so your code won't be cluttered.
This should be possible with MDC logging available with SLF4j which uses Thread local storage to store the context of the each request.
Also you will have to create a MDC Context Propagating execution context, to move the context across threads.
This post describes it well:
http://code.hootsuite.com/logging-contextual-info-in-an-asynchronous-scala-application/
I am attempting to learn and apply the CQRS design approach (pattern and architecture) to a new project but seem to be missing a key piece.
My client application executes a query and retrieves a list of light-weight, read-only DTOs from the read model. The user selects an item and clicks a button to initiate some action. The action is performed by creating and sending the corresponding command object to the write model (where the command handler carries out the action, updates the data store, etc.) At some point, however, I need to update the UI to reflect changes to the state of the application resulting from the action.
How does the UI know when it is time to refresh the original list?
Additional Info
I have noticed that most articles/blogs discussing CQRS use MVC client apps in their examples. I am working on a Silverlight client right now and am beginning to wonder if the pattern simply doesn't work in that case.
Follow-Up Question
After thinking more about Bartlomiej's response and subsequent discussion, I am wondering about error handling in CQRS. Given that commands are basically fire-and-forget asynchronous operations, how do we report an error condition to the UI?
I see 'refreshing the UI' to take one of two forms:
The operation succeeds, data has changed and the UI should be updated to reflect these changes
The operation fails, data has not changed but the user should be notified of the failure and potential corrective actions.
Even with a Post-Redirect-Get pattern in an MVC, you can't really Redirect until you know the outcome of the operation. None of the examples I've seen thus far address these real-world concerns.
I've been struggling with similar issues for a WPF client. The re-query trigger for any data is dependent on the data your updating, commands tend to fall into categories:
The command is a true fire and forget method, it informs the back-end of a state change but this change does not need to be reflected in the UI, or the change simply isn't important to the UI.
The command will alter the result of a single query
The command will alter the result of multiple queries, usually (in my domain at least) in a cascading fashion, that is, changing the state of a single "high level" piece of data will likely affect many "low level" caches.
My first trigger is the page load, very few items are exempt from this as most pages must assume data has been updated since it was last visited. Though some systems may be able to escape with only updating financial and other critical data in this way.
For short commands I also update data when 'success' is returned from a command. Though this is mostly laziness as IMHO all CQRS commands should be fired asynchronously. It's still an option I couldn't live without but one you may have to if your implementation expects high latency between command and query.
One pattern I'm starting to make use of is the mediator (most MVVM frameworks come with one). When I fire a command, I also fire a message to the mediator specifying which command was launched. Each Cache (A view model property Retriever<T>) listens for commands which affect it and then updates appropriately. I try to minimise the number of messages while still minimising the number of caches that update unnecessary from a single message so I'll (hopefully) eventually end up with a shortlist of update reasons, with each 'reason' updating a list of caches.
Another approach is simple honesty, I find that by exposing graphically how the system updates itself makes users more willing to be patient with it. On firing a command show some UI indicating you're waiting for the successful response, on error you could offer to retry / show the error, on success you start the update of the relevant fields. Baring in mind that this command could have been fired from another terminal (of which you have no knowledge) so data will need to timeout eventually to avoid missing state changes invoked by other machines also.
Noting the irony that the only efficient method of updating cache's and values on a client is to un-separate the commands and queries again, be it through hardcoding or something like a hashmap.
My two cents.
I think MVVM actually fits into CQRS quite well. The ViewModel simply becomes an observable ReadModel.
1 - You initialize your ViewModel state via a query on the ReadModel.
2 - Changes on your ViewModel are automatically reflected on any Views that are bound to it.
3 - Certain changes on your ViewModel trigger a command to propegate to a message queue, an object responsible for sending those commands to the server takes those messages off the queue and sends them to the WriteModel.
4 - Clients should be well formed, meaning the ViewModel should have performed appropriate validation before it ever triggered the command. Once the command has been triggered, any event notifications can be published onto an event bus for the client to communicate changes to other ViewModels or components in the system interested in those changes. These events should carry the relevant information necessary. Typically, this means that other view models usually don't have to re-query the read model as a result of the change unless they are dependent on other data that needs to be retrieved.
5 - There is an object that connects to the message bus on the server for real-time push notifications when other clients make changes that this client is interested in knowing about, falling back to long-polling if necessary. It propagates those to the internal message bus that ties the components on the client together.
6 - The last part to handle is the fact that clients can be occasionally connected, which should be the only reason a command fails (they don't have internet access at the moment), which is when the client should be notified of problems.
In my ASP.NET MVC 3 I use 2 techniques depending on use case:
already well-known Post-Redirect-Get pattern which fits nicely with CQRS. Your MVC action that triggers the command returns a redirection to action that performs a query.
in some cases, like real-time updates of other clients, I rely on domain events/messages. I create an event handler that uses singlarR to push changes to all connected and interested clients.
There are two major ways you can take as far as I know :
1) design your UI , so that the user does not see its changes right away. Like for instance a message to tell him his action is a success, and offering him different choices to continue his work. this should buy you enough time to have updated your readmodel.
2) more complex, but you might keep the information you have send to the server and shows them in the interface.
The most important I guess, educate your user if you can so that they know why the data is not here... yet!
I am thinking about it only now, but these are for sync command handling, not async, in async things go really harder on the brain...the client interface becomes an event eater too..
In my Service Broker design, I need to make an asynchronous calls and needed some work to get done in background (Inside SQL Server only, like updating tables).
There are certain points to be taken under consideration based on the requirement :
It's kind of one-way data push. Just place a message into the SB queue and forget. No acknowledgement required.
Only one database involved in the design. There is no need for multiple databases.
Message will be placed to the SB queue using a Stored Proc ( This SP will be called by an application).
By observing above points, it seems that requirement doesn't suits for creating 2 different SB services as only one service would suffice. I designed the scenario with having only one SB Service, and while creating a conversation dialog, I assigned same service name to the 'From' & 'To' clauses. Program pushes data to the SB queue and activator will activate associated Store Procedure.. It works just fine.
BEGIN DIALOG CONVERSATION #RecordConversationHandle
FROM SERVICE **UpdateQueueStatus**
TO SERVICE '**UpdateQueueStatus**'
WITH ENCRYPTION = OFF;
Please help me by any suggestion on the above proposed design.. ? Any suggestions/issues or anything which demands attention to improve the design for better performance & scalability would be much appreciated.
Service broker is designed for dialogs, not monolog conversations. Don't design something new (There is tons of good reasons why they are always dialogs)
You can create sending service (Service1), witch is used for sending messages and receives "End Dialog" messages and ends dialog. The other (Service2) witch receives messages and does some processing with them + ends dialog when work is done.
The main reason of two services in a dialog and dialog-oriented conversations is the ability to disable queue. The initiator's queue may be enabled while at the same time, for some purpose or reasons, the target's queue may be disabled. In this case, sedning messages runs without the "disabled queue" error and messages will wait in the transmition queue until the target queue become enabled again.
That is why a contract may contain just one message type, and a queue may be created without specifying any contract. It's the initiator's queue.
There is a caveat: BEGIN CONVERSATION TIMER. It puts the standard message https://schemas.microsoft.com/SQL/ServiceBroker/DialogTimer into a local queue the specified conversation dialog belongs to.
One use case when a dialog on the same service may be usefull, is a recovery process. However, in this case, there should be a specific message type received in a higher priority than ordinary messages. The activation procedure first receives a recovery message, tries to recover, rollback if unsuccessfull, then receives an ordinary messages and commits receiving messages of both the types or just rollback if unsuccessfull again.
I have an application in Asp.net MVC where at some point I would like to display a modal dialog to the user that would display process execution progress indicator.
The process behind the scenes does a lot of database data processing (based on existing data it generates lots of resulting records that get written back to database as well). Process may take anything from a brief moment to a very long time (depending on existing data).
Application will initiate this process asynchronously (via Ajax request) and display progress in the same manner.
The problem
I've read a bit about Async controllers where one can asynchronously start a process and will informed about the end of it but there's no progress indication and I'm not really sure how browser timeouts are handled. As far as the client goes an async request is the same as synchronous one. Client will therefore wait for response (as I understand it). the main difference being that server will execute something in async manner so it won't block other incoming requests. What I should actually do is:
make a request that would start the process and respond to the client taht process has started.
client would them periodically poll the server for process progress status getting immediate response back with percentage value (probably as JSON)
when progress would be 100% it would mean that it ended so client would know to make a request for results.
I'm not convinced that async controllers work this way...
The thing is that I'm not really sure I understand async controllers hence am not sure which approach should I use approach this problem as just described? I see two possibilities myself:
Asp.net MVC Async controllers if they can work this way
Windows Service app that processes data on request and reports its progress - this service would be started by writing a particular record to DB using a normal controller action; that would start it and then service would be writing its progress status to DB so my Asp.net MVC app would be able to read it on client process polling requests.
I haven't used Asynch controllers myself in a project. However here's a link to someone who has.
asynchronous-processing-in-asp-net-mvc-with-ajax-progress-bar
I have personally used Number 2 in a large production project.
Number 2 was a Service App running on a separate server using OpenSSH to communicate between the two servers. We'd poll for progress periodically to update the progress bar to the clients UI via AJAX.
Additionally by separating your web server from your long running process you are separating your concerns. You web server is not interested in writing files to disk, handling IO, etc and so shouldn't be burdended with such.
If your long running process has to be killed or fails then this wont affect your web server handling requests, and processing transactions.
Another suggestion would be for an extremely long running process is not to burden the client with waiting, give them an option to come back later to see the progress. I.e. send them an e-mail when its done.
Or actually show them something interesting, in our case we had a signed Java Applet show exactly what their process is doing at that exact moment.