How can I create topic and subscribe on it multiple independent subscribers with different subscriptions for each without specifying subscription names. If the subscriber disconnect, the corresponding subscription should be automatic removed. This case can be realised with rabbitmq server for logging purposes, for example. https://www.rabbitmq.com/tutorials/tutorial-three-dotnet.html.
In the .NET client, when we supply no parameters to queueDeclare() we create a non-durable, exclusive, autodelete queue with a generated name.
If it is impossible, how can I wrap .net client for realising this case? Thanks.
As you mentioned in your comment, you can create new subscription with unique GUID as subscription name when the client connect (or app start). And specifying SubscriptionDescription.AutoDeleteOnIdle property to set the TimeSpan idle interval after which the subscription is automatically deleted.
var namespaceManager = NamespaceManager.CreateFromConnectionString(connectionString);
var subscriptionname = Guid.NewGuid().ToString();
if (!namespaceManager.SubscriptionExists(topicname, subscriptionname))
{
SqlFilter updatedMessagesFilter =
new SqlFilter("mypro = 'test'");
namespaceManager.CreateSubscription(new SubscriptionDescription(topicname, subscriptionname) { AutoDeleteOnIdle = TimeSpan.FromMinutes(5) },
updatedMessagesFilter);
}
When client disconnect, you can delete the subscription manually.
if (namespaceManager.SubscriptionExists(topicname, subscriptionname))
{
namespaceManager.DeleteSubscription(topicname, subscriptionname);
}
Note: to guarantee 100% delete subscription, you can retain information about client and subscriptionname (unique GUID) in an external storage, and every time when a client connect/reconnect, you can detect if a record exists in external storage that indicates a subscription (used by this client before) is still not be deleted for current client, if the record exists, you can delete that subscription before you create a new subscription.
Related
I am currently testing the scaling of my application and I ran into something I did not expect.
The application is running on a 5 node cluster, it has multiple services/actortypes and is using a shared process model.
For some component it uses actor events as a best effort pubsub system (There are fallbacks in place so if a notification is dropped there is no issue).
The problem arises when the number of actors grows (aka subscription topics). The actorservice is partitioned to 100 partitions at the moment.
The number of topics at that point is around 160.000 where each topic is subscribed 1-5 times (nodes where it is needed) with an average of 2.5 subscriptions (Roughly 400k subscriptions).
At that point communications in the cluster start breaking down, new subscriptions are not created, unsubscribes are timing out.
But it is also affecting other services, internal calls to a diagnostics service are timing out (asking each of the 5 replicas), this is probably due to the resolving of partitions/replica endpoints as the outside calls to the webpage are fine (these endpoints use the same technology/codestack).
The eventviewer is full with warnings and errors like:
EventName: ReplicatorFaulted Category: Health EventInstanceId {c4b35124-4997-4de2-9e58-2359665f2fe7} PartitionId {a8b49c25-8a5f-442e-8284-9ebccc7be746} ReplicaId 132580461505725813 FaultType: Transient, Reason: Cancelling update epoch on secondary while waiting for dispatch queues to drain will result in an invalid state, ErrorCode: -2147017731
10.3.0.9:20034-10.3.0.13:62297 send failed at state Connected: 0x80072745
Error While Receiving Connect Reply : CannotConnect , Message : 4ba737e2-4733-4af9-82ab-73f2afd2793b:382722511 from Service 15a5fb45-3ed0-4aba-a54f-212587823cde-132580461224314284-8c2b070b-dbb7-4b78-9698-96e4f7fdcbfc
I've tried scaling the application but without this subscribe model active and I easily reach a workload twice as large without any issues.
So there are a couple of questions
Are there limits known/advised for actor events?
Would increasing the partition count or/and node count help here?
Is the communication interference logical? Why are other service endpoints having issues as well?
After time spent with the support ticket we found some info. So I will post my findings here in case it helps someone.
The actor events use a resubscription model to make sure they are still connected to the actor. Default this is done every 20 seconds. This meant a lot of resources were being used and eventually the whole system overloaded with loads of idle threads waiting to resubscribe.
You can decrease the load by setting resubscriptionInterval to a higher value when subscribing. The drawback is that it will also mean the client will potentially miss events in the mean time (if a partition is moved).
To counteract the delay in resubscribing it is possible to hook into the lower level service fabric events. The following psuedo code was offered to me in the support call.
Register for endpoint change notifications for the actor service
fabricClient.ServiceManager.ServiceNotificationFilterMatched += (o, e) =>
{
var notification = ((FabricClient.ServiceManagementClient.ServiceNotificationEventArgs)e).Notification;
/*
* Add additional logic for optimizations
* - check if the endpoint is not empty
* - If multiple listeners are registered, check if the endpoint change notification is for the desired endpoint
* Please note, all the endpoints are sent in the notification. User code should have the logic to cache the endpoint seen during susbcription call and compare with the newer one
*/
List<long> keys;
if (resubscriptions.TryGetValue(notification.PartitionId, out keys))
{
foreach (var key in keys)
{
// 1. Unsubscribe the previous subscription by calling ActorProxy.UnsubscribeAsync()
// 2. Resubscribe by calling ActorProxy.SubscribeAsync()
}
}
};
await fabricClient.ServiceManager.RegisterServiceNotificationFilterAsync(new ServiceNotificationFilterDescription(new Uri("<service name>"), true, true));
Change the resubscription interval to a value which fits your need.
Cache the partition id to actor id mapping. This cache will be used to resubscribe when the replica’s primary endpoint changes(ref #1)
await actor.SubscribeAsync(handler, TimeSpan.FromHours(2) /*Tune the value according to the need*/);
ResolvedServicePartition rsp;
((ActorProxy)actor).ActorServicePartitionClientV2.TryGetLastResolvedServicePartition(out rsp);
var keys = resubscriptions.GetOrAdd(rsp.Info.Id, key => new List<long>());
keys.Add(communicationId);
The above approach ensures the below
The subscriptions are resubscribed at regular intervals
If the primary endpoint changes in between, actorproxy resubscribes from the service notification callback
This ends the psuedo code form the support call.
Answering my original questions:
Are there limits known/advised for actor events?
No hard limits, only resource usage.
Would increasing the partition count or/and node count help here? Partition count not. node count maybe, only if that means there are less subscribing entities on a node because of it.
Is the communication interference logical? Why are other service endpoints having issues as well?
Yes, resource contention is the reason.
I am trying to allocate additional permissions to a private clustered MSMQ queue.
I found How to set permissions on MSMQ Cluster queues? which is the same problem but I can't get it to work.
The key from that article is to set the environment variable:
$env:_CLUSTER_NETWORK_NAME_ = 'myclusterMSMQ'
But what should myclusterMSMQ contain?
In my case I have a cluster MSMQCluster with two nodes MSMQNodeA and MSMQNodeB. The queue is in a clustered resource QueueResource.
The logical to me is to set it to QueueResource, but when I did that, the permission appeared on a queue with the same name on MSMQNodeA... this is where I am running my powershell and it is the resource owner - the queue shouldn't be here but during script testing a queue with that name ended up here too.
My code looks like this:
$queueUser= "domain\svc-account"
$queueName=".\private$\log.queue"
$env:_CLUSTER_NETWORK_NAME_ = "$queueResource"
$queue = New-Object System.Messaging.MessageQueue $queueName
$queue.SetPermissions($queueuser, [System.Messaging.MessageQueueAccessRights]::WriteMessage)
I tried a variation that retrieved the queue using GetPrivateQueuesByMachine($queueResource)
but that gave me the error:
System.Messaging.MessageQueueException (0x80004005): The specified
format name does not support the requested operation. For example, a
direct queue format name cannot be deleted
I am sure that this last method was returning the queue from the QueueResource because when I listed all queues, I didn't get any of the other garbage queues that are on MSMQNodeA.
How do I set additional queue permissions on a private clustered queue with Powershell?
It looks like it wasn't working in part because the service account had lost MSMQ admin permissions so it didn't matter what value I used when testing.
The following is the correct syntax:
$env:_CLUSTER_NETWORK_NAME_ = $QueueResource
Is there a way to find the list of ServerSession associated with the given userId?
public boolean associate(final String userId, final ServerSession session)
{
if (session == null)
throw new NullPointerException();
LocalLocation location = new LocalLocation(userId, session);
boolean wasAssociated = isAssociated(userId);
boolean added = associate(userId, location);
if (added)
{
session.addListener(location);
if (_logger.isDebugEnabled())
_logger.debug("Associated session {} to user {}", session, userId);
if (!wasAssociated)
{
if (_logger.isDebugEnabled())
_logger.debug("Broadcasting association addition for user {}", userId);
// Let everyone in the cluster know that this session is here
_session.getChannel(SETI_ALL_CHANNEL).publish(new SetiPresence(true, userId));
}
}
return added;
}
If the user is already associated, seti doesn't publish on SETI_ALL_CHANNEL to notify other comets.
Detail Explanation-
I have implemented cometd oort cluster to push the database change notification to the browser.
Two nodes are considered as master/slave. Master node only receives the notification from the database.
Users are connected through the slave node.
When the user first handshake with the slave node, seti publishes an event user presence added.
Both the nodes in the cluster know about the users in the cloud.
When, user refreshes the browser a new(second) handshake is initiated. The userId(loginUserId) remains the same. Seti doesn’t publishes it to the cluster which is correct by design.
After certain time, the session through the first handshake is removed due to inactivity. The presence removed event is fired by seti, which is also the expected by design.
Slave nodes only know the user is connected through second handshake, however the master node doesn’t know that the user is present in the cloud.
When new event arrives from database, the master node doesn’t see the user in the cloud and thus no events are transferred to the browser. Moreover, the master and slave nodes are connected at this point of time. The users connected to the cluster is not synchronized between two nodes.
I was thinking to seti/disassociate and then seti/associate the session for the same user.
We can get a list of sessions connected-
seti.getOort().getBayeuxServer().getSessions();
// clientId - previous client session id need to be identified
ServerSession serverSession=seti.getOort().getBayeuxServer().getSession(clientId);
Through client Id, Server Session can be retrieved and the disassociation can be done for the user.
seti.disassociate(loginUserId, serverSession);
boolean association=seti.associate(loginUserId, serverSession);//new handshake- serverSession
If you have two sessions connected to the slave node, and one of them goes away, then Seti will not broadcast a presence removed event, because that is only fired when all associations for a userId on a node are gone.
This is exactly to avoid that one node still has an association, but other nodes think the user is gone from the cloud.
If you have evidence that this is happening, then it's a bug, but this specific behavior is tested in the CometD test suite.
I am trying to create an observable that meets the following requirements:
1) When the first client subscribes then the observable needs to connect to some backend service, and push out an initial value
2) When successive clients subscribe then the observable should push out a new value
3) When the final client disposes then the observable should disconnect from the backend system.
4) The backend service also calls OnNext regularly with other messages
So far I have something like below. I can't work out how I can react to each subscribe call but only call the disposer on the final dispose.
var o = Observable.Create((IObserver<IModelBrokerEvent> observer) =>
{
observer.OnNext(newValue);
_backendThingy.Subscribe(observer.OnNext);
return Disposable.Create(() =>
{
_backendThingy.Unsubscribe(observer.OnNext);
});
}
_observable = Observable.Defer(() => o).Publish().RefCount();
There are several ways to do things similar to what you are talking about, but without exact semantics I am limited to proving a few generic solutions...
Replay Subject
The simplest is as such:
var source = ...;
var n = 1
var shared = source.Replay(n).RefCount();
The Replay operator ensures that each new subscription receives the latest n values from the source Observable. However, it does not re-invoke any subscription logic to the source Observable to achieve this. In effect, assuming the source stream has emitted values already, subsequent subscriptions will receive the previous n values synchronously upon subscription. RefCount does what you might think it should do: Connect the Replay upon the first subscription, and dispose of the connection upon the last unsubscribe.
Bidirection Communication via Proxy
Replay solves the most common use case, in that the source stream is capable of keeping itself up-to-date relatively well. However, if the source stream is updated only periodically, and new subscriptions should constitute an immediate update, then you may want to get more fancy:
var service = ...;
var source = service.Publish().RefCount();
var forceUpdate = Observable.Defer(() => Observable.Start(service.PingForUpdate));
var shared = Observable.Merge(source, forceUpdate);
Where the subscription to the server constitutes a new connection, and the PingForUpdate method indicates to the service that it's consumer would like to have a new value ASAP (which then forces the service to output said value.
Merging Periodic Updates with Initial Latest Value
Using the bidirectional communication method denotes that all consumers of this service will receive the latest value from the source upon any new subscription. However, it may be possible that we only want the latest value for the new subscriber, and all other consumers should receive values on the periodic basis.
For this, we can change the code a bit.
var service = ...;
var source = service.Publish().RefCount();
var latest = Observable.Defer(() => service.GetLatestAsObservable());
var shared = Observable.Merge(source, forceUpdate);
Where the subscription to the server constitutes a new connection, and the GetLatestAsObservable method simply retrieves the latest value from the service asynchronously via an Observable.
However, with this method you must also choose how to handle race conditions, such that you request the latest value, but a newer value is yielded before the request for the latest value is returned.
I am creating a Windows Service in C# that processes messages from a queue. I want to give ops the flexibility of partitioning the service in production according to properties of the message. For example, they should be able to say that one instance processes web orders from Customer A, another batch orders from Customer A, a third web or batch orders from Customer B, and so on.
My current solution is to assign separate queues to each customer\source combination. The process that puts orders into the queues has to make the right decision. My Windows Service can be configured to pull messages from one or more queues. It's messy, but it works.
No, but you can PEEK into the queue and decide if you really want to consume the message.
Use GetMessageEnumerator2() like this:
MessageEnumerator en = q.GetMessageEnumerator2();
while (en.MoveNext())
{
if (en.Current.Label == label)
{
string body = ((XmlDocument)en.Current.Body).OuterXml;
en.RemoveCurrent();
return body;
}
}