Whats a clean way to make a Observable/Flowable for a global api like say
public void log(String s)
All components use this log method that currently has listeners and the various listeners log to various kinds of logging. I want to make it an observable so that multiple subscribers can can subscribe to this observable and log to various files/db.
I am hoping that I won't have to rework the legacy classes (do have access to code of the library).
Currently I add the string to a queue like data structure which is my observable and use subscribers to achieve what I want.
A more generic question would be , how do you create an observable for an infinite source (for me the log method is the source) and what data structure would be recommended for hold the data of this infinite source that can become a good observable.
Any pointers/recommendations/snippets will be greatly appreciated.
[edit]
As suggested in the response comment, using PublishSubject solved my problem.
When I need to subscribe I access the subject object and call the subscribe on it.
static PublishSubject<LogEvent> subject = PublishSubject.create();
public static void log (LogEvent evt) {
subject.onNext(evt);
}
Related
I'm trying to write a Vert.x Web handler that could be used to hide any processing latencies from an API to prevent figuring out existence of accounts as well as other information from an API. I would like to be able to just write something like:
router
.post("/uri")
.handler(new LatencyNormalizer())
.handler(new UriHandler());
In other words, make it as easy to use as possible and as easy to integrate into existing code bases as possible. Looking at the docs for Router and RoutingContext, I see only the following method as a possible candidate for implementing this:
https://vertx.io/docs/apidocs/io/vertx/ext/web/RoutingContext.html#addHeadersEndHandler-io.vertx.core.Handler-
I could then write code like:
public void handle(RoutingContext ctx) {
long start = System.nanoTime();
ctx.addHeadersEndHandler(v -> {
public void handle(RoutingContext ctx) {
long end = System.nanoTime();
Thread.sleep(...);
});
ctx.next();
}
Of course, this doesn't work, since sleep here blocks the thread. It looks like the handlers in the addHeaderEndHandlers list maintained internally by the RoutingContext are called synchronously, so there is no way to use e.g. vertx.SetTimer() inside the addHeaderEndHandler.
In other words, does Vert.x offer any interface that allows creating a handler which is called asynchronously before writing out to the wire (and with nothing written until the async call finishes)? This is for example how Netty works under the hood, which Vert.x leverages. I know I could implement this LatencyNormalizer as a base class for my other handlers, but it would not be as easy to integrate in existing code in that case.
We are using the Axon Framework to implement the Saga Pattern in Java. Axon uses two tables (ASSOCIATION_VALUE_ENTRY and SAGA_ENTRY) to store all the necessary information after each step of the saga. And at the end of the process (if it is correct, or, in case of error, all the compensations have been executed), it deletes the registers.
If for any reason, after an error, the compensations cannot be executed, we are able to resume the execution at the point where it failed, based on the stored information. Until here, everything is ok.
The issue came when we wanted to improve the resilience of the process and we checked what happened if the service died during the execution of a saga. According to the above, we expected the information of the execution to be persisted in the tables, but they were empty: the information only appeared when the process couldn't continue due to an error in a compensation (and no final delete action was executed).
Analyzing the source code of the Axon's JpaSagaStore class implementation, the interactions with the database (insert, update and delete) are persisted with a flush instead of a commit. The global commit is managed in the AbstractUnitOfWork class (as far as we understand). And here is where we have the doubts:
According to the literature, the flush writes in the database but the register is in a READ_UNCOMMITED state. The only way to see them in the database would be activating the READ_UNCOMMITED isolation level, with the problematic of the 'dirty reads', right? There would be any additional consideration/issue to have into account?
Does Axon have an alternative in order to ensure the persistence of the saga registers? Mainly if we couldn't activate the READ_UNCOMMITED mode (due to internal policies).
EDIT:
Summarizing it a lot, all starts with this method
public void startSaga(SagaWorkflow sagaWorkflow, Serializable sagaInput) {
StartSagaEvt startSagaEvt = StartSagaEvt.builder().sagaWorkflow(sagaWorkflow).sagaInput(sagaInput).build();
eventBus.publish(GenericEventMessage.asEventMessage(startSagaEvt));
}
Where:
eventBus is the Axon's internal one
sagaInput is simply a Serializable with some input values
SagaWorkflow is a Serializable that models the whole saga flow, whose main attribute is a LinkedList of nodes (the different steps of the saga, each one can have a different logic)
StartSagaEvt is just the POJO that models the event sent to the bus
After this, Axon performs all its 'magic' and finally arrives to the internal code:
AnnotatedSagaRepository.doCreateInstance --> AnnotatedSagaRepository.storeSaga --> [...] --> JpaSagaStore.insertSaga
public void insertSaga(Class<?> sagaType, String sagaIdentifier, Object saga, Set<AssociationValue> associationValues) {
EntityManager entityManager = entityManagerProvider.getEntityManager();
AbstractSagaEntry<?> entry = createSagaEntry(saga, sagaIdentifier, serializer);
entityManager.persist(entry);
for (AssociationValue associationValue : associationValues) {
storeAssociationValue(entityManager, sagaType, sagaIdentifier, associationValue);
}
if (logger.isDebugEnabled()) {
logger.debug("Storing saga id {} as {}", sagaIdentifier, serializedSagaAsString(entry));
}
if (useExplicitFlush) {
entityManager.flush();
}
}
The same applies for the update and delete phases. As far as I know, all the handle of the commit/rollback is performed in the class AbstractUnitOfWork, that intervenes just at the end of the complete saga flow.
This leads me to the following considerations/questions:
what sense has to keep the transaction open during the whole process instead of committing after each step? If for any reason the process fails, goes down, the database is not accessible,... all the saved information is lost.
There must be a design reason for this behavior, but I'm not able to see it. Or maybe there is a configuration to change it (hopefully, although I doubt it).
Thanks in advance for any comment!
EDIT 2
Effectively, we are using it as a kind of state machine, where the saga flow is a sequence of steps, each one with an action and a compensation, and we jump from one to another until reach an "END" status.
#Saga
class GenericSaga {
private EventBus eventBus;
private CustomCommandGateway commandGateway;
[...]
#StartSaga
#SagaEventHandler(associationProperty = "sagaId")
public void startStep(StartSagaEvt startSagaEvt) {
// Initializes de GenericSaga and associate several properties with SagaLifecycle.associateWith(key, value);
[...]
// Transit to the next (first) step
eventBus.publish(GenericEventMessage.asEventMessage(new StepSagaEvt(startSagaEvt)));
}
#SagaEventHandler(associationProperty = "sagaId")
public void nextStep(StepSagaEvt stepSagaEvt) {
// Identifies what is the next step in the defined flow, considering if it should be executed sequentially or concurrently, or if it is the end of the flow and then call the SagaLifecycle.end()
[...]
// Also checks if it has to execute the compensation logic of the step
[...]
// Execute
Serializable actionOutput = commandGateway.sendAndWaitEx(stepAction.getActionInput());
}
#SagaEventHandler(associationProperty = "sagaId")
public void resumeSaga(ResumeSagaEvt resumeSagaEvt) {
// Recover information from the execution that we want to resume
[...]
// Transit to the next step
eventBus.publish(GenericEventMessage.asEventMessage(new StepSagaEvt(resumeSagaEvt)));
}
}
As you can see, we don't have an endSaga annotation, and maybe that's the problem. But in our current situation we have kicked forward, and be have defined our custom implementation of the JpaSagaStore, in order to force a local transaction in the insertSaga and updateSaga methods.
Based on my understanding, I think you are somehow misusing the Saga component from Axon Framework. I assume from your question that you are trying to build a form of a 'state machine' using your own SagaWorkflow object. If that is the case, I have to say this is not how Axon intends the usage of Sagas.
To add to that, let me give you a pseudo-sample of what a Saga should look like.
#Saga
class SagaWorkflow {
private transient CommandGateway commandGateway;
#StartSaga
#SagaEventHandler(associationProperty = "yourProperty")
public void on(SagaInputEvent event) {
// validate, associate with another property and fire a command
SagaLifecycle.associateWith("associationPropertyKey", "associationPropertyValue");
commandGateway.send(new GivenCommand());
}
#SagaEventHandler(associationProperty = "associationPropertyValue")
public void on(AnotherEvent event) {
// validate and fire a command or finish the saga
SagaLifecycle.end();
}
#EndSaga
#SagaEventHandler(associationProperty = "anyProperty")
public void on(FinishSagaEvent event) {
// check if you need to fire extra commands to tell others it's finished or just do it silently
}
}
#Saga Annotation will make sure Axon Framework handles the whole Saga process for you, storing (serializing) it to the database when each (Saga)EventHandler is executed
#SagaEventHandler will make sure the 'Event Handling method' reacts to a given Event, only if it contains the associationProperty as part of the Event (to understand it better, I will share our docs link)
#EndSaga will tell Axon Framework to finalize the Saga after the execution of the method (finalizing means deleting it from the database)
SagaLifecycle provides several 'utilities' methods to interact with the Saga's lifecycle and associations
In the example, I made the CommandGateway transient because the Saga is serialized and stored on the database. You would not Axon to serializer any external component, like the gateway, as well
Of course, there is more to it.
You can check Axon's docs for that. But I hope this gives you enough material and ideas to use Sagas within Axon Framework better!
KR
I have some code in a class that takes FileSystemWatcher events and flattens them into an event in my domain:
(Please note, the *AsObservable methods are extensions from elsewhere in my project, they do what they say 🙂.)
watcher = new FileSystemWatcher(ConfigurationFilePath);
ChangeObservable = Observable
.Merge(
watcher.ChangedAsObservable().Select((args) =>
{
return new ConfigurationChangedArgs
{
Type = ConfigurationChangeType.Edited,
};
}),
watcher.DeletedAsObservable().Select((args) =>
{
return new ConfigurationChangedArgs
{
Type = ConfigurationChangeType.Deleted,
};
}),
watcher.RenamedAsObservable().Select((args) =>
{
return new ConfigurationChangedArgs
{
Type = ConfigurationChangeType.Renamed,
};
})
);
ChangeObservable.Subscribe((args) =>
{
Changed.Invoke(this, args);
});
Something that I'm trying to wrap my head around as I'm learning are best practices around naming, ownership and cleanup of the IObservable and IDisposable returned by code like this.
So, some specific questions:
Is it okay to leak IObservables from a class that creates them? For example, is the property I'm assigning this chain to okay to be public?
Does the property name ChangeObservable align with what most people would consider best practice when using the .net reactive extensions?
Do I need to call Dispose on any of my subscriptions to this chain, or is it safe enough to leave everything up to garbage collection when the containing class goes out of scope? Keep in mind, I'm observing events from watcher, so there's some shared lifecycle there.
Is it okay to take an observable and wire them into an event on my own class (Changed in the example above), or is the idea to stay out of the native .net event system and leak my IObservable?
Other tips and advice always appreciated! 😀
Is it okay to leak IObservables from a class that creates them? For
example, is the property I'm assigning this chain to okay to be
public?
Yes.
Does the property name ChangeObservable align with what most
people would consider best practice when using the .net reactive
extensions?
Subjective question. Maybe FileChanges? The fact that it's an observable is clear from the type.
Do I need to call Dispose on any of my subscriptions to
this chain, or is it safe enough to leave everything up to garbage
collection when the containing class goes out of scope?
The ChangeObservable.Subscribe at the end could live forever, preventing the object from being garbage collected if the event is subscribed to, though that could also be your intention. Operator subscriptions are generally fine. I can't see the code for your ChangedAsObservable like functions. If they don't include a Subscribe or an event subscription, they're probably fine as well.
Keep in mind,
I'm observing events from watcher, so there's some shared lifecycle
there.
Since FileWatcher implements IDisposable, you should probably use Observable.Using around it so you can combine the lifecycles.
Is it okay to take an observable and wire them into an event on
my own class (Changed in the example above), or is the idea to stay
out of the native .net event system and leak my IObservable?
I would prefer to stay in Rx. The problem with event subscriptions is that they generally live forever. You lose the ability to control subscription lifecycle. They're also feel so much more primitive. But again, that's a bit subjective.
We are using Azure Service Fabric and are using actors to model specific devices, using the id of the device as the ActorId. Service Fabric will instantiate a new actor instance when we request an actor for a given id if it is not already instantiated, but I cannot seem to find an api that allows me to query if a specific device id already has an instantiated actor.
I understand that there might be some distributed/timing issues in obtaining the point-in-time truth but for our specific purpose, we do not need a hard realtime answer to this but can settle for a best guess. We would just like to, in theory, contact the current primary for the specific partition resolved by the ActorId and get back whether or not the device has an instantiated actor.
Ideally it is a fast/performant call, essentially faster than e.g. instantiating the actor and calling a method to understand if it has been initialized correctly and is not just an "empty" actor.
You can use the ActorServiceProxy to iterate through the information for a specific partition but that does not seem to be a very performant way of obtaining the information.
Anyone with insights into this?
The only official way you can check if the actor has been activated in any Service Partition previously is using the ActorServiceProxy query, like described here:
IActorService actorServiceProxy = ActorServiceProxy.Create(
new Uri("fabric:/MyApp/MyService"), partitionKey);
ContinuationToken continuationToken = null;
do
{
PagedResult<ActorInformation> page = await actorServiceProxy.GetActorsAsync(continuationToken, cancellationToken);
var actor = page.Items.FirstOrDefault(x => x.ActorId == idToFind);
continuationToken = page.ContinuationToken;
}
while (continuationToken != null);
By the nature of SF Actors, they are virtual, that means they always exist, even though you didn't activated then previously, so it make a bit harder to do this check.
As you said, it is not performant to query all actors, so, the other workarounds you could try is:
Store the IDs in a Reliable Dictionary elsewhere, every time an Actor is activated you raise an event and insert the ActorIDs in the Dictionary if not there yet.
You can use the OnActivateAsync() actor event to notify it's creation, or
You can use the custom actor factory in the ActorService to register actor activation
You can store the dictionary in another actor, or another StatefulService
Create a property in the actor that is set by the actor itself when it is activated.
The OnActivateAsync() check if this property has been set before
If not set yet, you set a new value and store in a variable (a non persisted value) to say the actor is new
Whenever you interact with actor you set this to indicate it is not new anymore
The next activation, the property will be already set, and nothing should happen.
Create a custom IActorStateProvider to do the same as mentioned in the option 2, instead of handle it in the actor it will handle a level underneath it. Honestly I think it is a bit of work, would only be handy if you have to do the same for many actor types, the option 1 and 2 would be much easier.
Do as Peter Bons Suggested, store the ActorID outside the ActorService, like in a DB, I would only suggest this option if you have to check this from outside the cluster.
.
The following snipped can help you if you want to manage these events outside the actor.
private static void Main()
{
try
{
ActorRuntime.RegisterActorAsync<NetCoreActorService>(
(context, actorType) => new ActorService(context, actorType,
new Func<ActorService, ActorId, ActorBase>((actorService, actorId) =>
{
RegisterActor(actorId);//The custom method to register the actor if new
return (ActorBase)Activator.CreateInstance(actorType.ImplementationType, actorService, actorId);
})
)).GetAwaiter().GetResult();
Thread.Sleep(Timeout.Infinite);
}
catch (Exception e)
{
ActorEventSource.Current.ActorHostInitializationFailed(e.ToString());
throw;
}
}
private static void RegisterActor(ActorId actorId)
{
//Here you will put the logic to register elsewhere the actor creation
}
Alternatively, you could create a stateful DeviceActorStatusActor which would be notified (called) by DeviceActor as soon as it's created. (Share the ActorId for correlation.)
Depending on your needs you can also register multiple Actors with the same status-tracking actor.
You'll have great performance and near real-time information.
I'm connecting to an object that asyncronously loads a collection of objects into an IEnumerable. At the time I connect, the IEnumerable may have items already in it's collection, and may add items during the lifetime of the application that I need to be notified of as they occur. As an example, it could be a bank account containing a list of bank transactions.
The challenge is this. I want to combine the processing of the initial values in the IEnumerable with any new additions. They are currently two processes. I would like to eliminate the use of NotifyCollectionChanged entirely.
I can modify the backend holding the IEnumerable. It does not need to remain as an IEnumerable if a solution to this question exists otherwise.
I would suggest that the object should not expose a IEnumerable as that is for "cold observable values", where in your case you need something which can get additional items in future also.
The best way to model this would be to use ReplaySubject<T> instead of IEnumerable. Below is an example that demonstrate the situation similar of yours:
//Function to generate the subject with future values
public static ReplaySubject<int> GetSubject()
{
var r = new ReplaySubject<int>();
r.OnNext(1); r.OnNext(2); r.OnNext(3);
//Task to generate future values
Task.Factory.StartNew(() =>
{
while (true)
{
Thread.Sleep(3000);
r.OnNext(DateTime.Now.Second);
}
});
return r;
}
Consuming code:
var sub = GetSubject();
sub.Subscribe(Console.WriteLine);
Every time anyone subscribes to sub they will get all the values that have been published in the subject till now and as well as new values that this subject generates in future
You can use Defer/Replay Operator