Strange one. We have a multi-threaded app which pulls messages off a MSMQ Queue and then subsequently performs actions based on the messages. All of this is done using DTC.
Sometimes, for some reason I can't describe, we get message read errors when pulling Messages off the queue.
The code that is being used in the app:
Message[] allMessagesOnQueue = this.messageQueue.GetAllMessages();
foreach (Message currentMessage in allMessagesOnQueue)
{
if ((currentMessage.Body is IAMessageIDealWith))
{
// do something;
}
}
When the currentMessage.Body is accessed, at times it throws an exception:
System.InvalidOperationException: Property Body was not retrieved when receiving the message. Ensure that the PropertyFilter is set correctly.
Now - this only happens some of the time - and it appears as though the MessageReadPropertyFilter on the queue has the Body property set to false.
As to how it gets like this is a bit of a mystery. The Body property is one of the defaults and we absolutley never explicitly set it to false.
Has anyone else seen this kind of behaivour or has some idea why this value is getting set to be false?
As alluded to earlier, you could explicitly set the boolean values on the System.Messaging.MessagePropertyFilter object that is accessible on your messageQueue object via the MessageReadPropertyFilter property.
If you want all data to be extracted from a message when received or peaked, use:
this.messageQueue.MessageReadPropertyFilter.SetAll(); // add this line
Message[] allMessagesOnQueue = this.messageQueue.GetAllMessages();
// ...
That may hurt performance of reading many messages, so if you want just a few additional properties, create a new MessagePropertyFilter with custom flags:
// Specify to retrieve selected properties.
MessagePropertyFilter filter= new MessagePropertyFilter();
filter.ClearAll();
filter.Body = true;
filter.Priority = true;
this.messageQueue.MessageReadPropertyFilter = filter;
Message[] allMessagesOnQueue = this.messageQueue.GetAllMessages();
// ...
You can also set it back to default using:
this.messageQueue.MessageReadPropertyFilter.SetDefaults();
More info here: http://msdn.microsoft.com/en-us/library/system.messaging.messagequeue.messagereadpropertyfilter.aspx
I have seen it as well, and have tried initializing it with the properties I'm accessing explicitly set, and not setting them anywhere else. I periodically get the same error you are getting, my app is multi-threaded as well, what I ended up doing is trapping that error and reconnecting to MSMQ when I get it.
Sometimes, for some reason I can't describe, we get message read errors when pulling Messages off the queue.
Are you using the same MessageQueue instance from more than one thread, without locking? In that case, you will encounter spurious changes in MessageReadPropertyFilter - at least I did, when I tried.
Why? Because
Only the GetAllMessages method is thread safe.
What can you do? Either
wrap a lock (_messageQueue) around all access to your messageQueue OR
create multiple MessageQueue instances, one per thread
Related
I need to log trace events during boot so I configure an AutoLogger with all the required providers. But when my service/process starts I want to switch to real-time mode so that the file doesn't explode.
I'm using TraceEvent and I can't figure out how to do this move correctly and atomically.
The first thing I tried:
const int timeToWait = 5000;
using (var tes = new TraceEventSession("TEMPSESSIONNAME", #"c:\temp\TEMPSESSIONNAME.etl") { StopOnDispose = false })
{
tes.EnableProvider(ProviderExtensions.ProviderName<MicrosoftWindowsKernelProcess>());
Thread.Sleep(timeToWait);
}
using (var tes = new TraceEventSession("TEMPSESSIONNAME", TraceEventSessionOptions.Attach))
{
Thread.Sleep(timeToWait);
tes.SetFileName(null);
Thread.Sleep(timeToWait);
Console.WriteLine("Done");
}
Here I wanted to make that I can transfer the session to real-time mode. But instead, the file I got contained events from a 15s period instead of just 10s.
The same happens if I use new TraceEventSession("TEMPSESSIONNAME", #"c:\temp\TEMPSESSIONNAME.etl", TraceEventSessionOptions.Create) instead.
It seems that the following will cause the file to stop being written to:
using (var tes = new TraceEventSession("TEMPSESSIONNAME"))
{
tes.EnableProvider(ProviderExtensions.ProviderName<MicrosoftWindowsKernelProcess>());
Thread.Sleep(timeToWait);
}
But here I must reenable all the providers and according to the documentation "if the session already existed it is closed and reopened (thus orphans are cleaned up on next use)". I don't understand the last part about orphans. Obviously some events might occur in the time between closing, opening and subscribing on the events. Does this mean I will lose these events or will I get the later?
I also found the following in the documentation of the library:
In real time mode, events are buffered and there is at least a second or so delay (typically 3 sec) between the firing of the event and the reception by the session (to allow events to be delivered in efficient clumps of many events)
Does this make the above code alright (well, unless the improbable happens and for some reason my thread is delayed for more than a second between creating the real-time session and starting processing the events)?
I could close the session and create a new different one but then I think I'd miss some events. Or I could open a new session and then close the file-based one but then I might get duplicate events.
I couldn't find online any examples of moving from a file-based trace to a real-time trace.
I managed to contact the author of TraceEvent and this is the answer I got:
Re the exception of the 'auto-closing and restarting' feature, it is really questions about the OS (TraceEvent simply calls the underlying OS API). Just FYI, the deal about orphans is that it is EASY for your process to exit but leave a session going. This MAY be what you want, but often it is not, and so to make the common case 'just work' if you do Create (which is the default), it will close a session if it already existed (since you asked for a new one).
Experimentation of course is the touchstone of 'truth' but I would frankly expecting unusual combinations to just work is generally NOT true.
My recommendation is to keep it simple. You need to open a new session and close the original one. Yes, you will end up with duplicates, but you CAN filter them out (after all they are IDENTICAL timestamps).
The other possibility is use SetFileName in its intended way (from one file to another). This certainly solves your problem of file size growth, and often is a good way to deal with other scenarios (after all you can start up you processing and start deleting files even as new files are being generated).
In Tibco, sometimes warnings are printed to the console, e.g.:
2014-06-25 18:13:22 RV: TIB/Rendezvous Error Not Handled by Process:
{ADV_CLASS="WARN" ADV_SOURCE="RVCM"
ADV_NAME="REGISTRATION.NOT_CERTIFIED.cm.test.subject"
subject="cm.test.subject" sender="cm.sender.cmname"}
I am using the .NET wrappers for Tibco. It appears as if these errors are not actually making it into the .NET, and they can't be caught with try/catch so they can be handled appropriately.
Is there any way to handle this error in .NET? Perhaps some method of registering a handler to handle errors such as this? Or alternatively, is there a method to redirect these warnings into a sink other than the console, e.g. a log file?
The solution is to add a "catch all" handler.
The current subject that I was listening to was:
private readonly string _subjectDeliveryConfirm = "_RV.INFO.RVCM.DELIVERY.CONFIRM.>";
To add a catch all, add another listener on:
private readonly string _subjectDeliveryGlobal = ">";
When you add a new listener, remember to use a separate Listener concrete class per listener, or else Tibco will mysteriously stop working after the first message (see the demo code for how to create multiple listeners).
_confirmListener1 = new Listener(Queue.Default, _netTransport, _subjectDeliveryConfirm, null);
_confirmListener1.MessageReceived += new MessageReceivedEventHandler(OnCertifiedMessageConfirmed);
// Subscribe to error messages, in particular error messages related to remote listener processes
// disappearing.
_confirmListener2 = new Listener(Queue.Default, _netTransport, _subjectDeliveryGlobal, null);
_confirmListener2.MessageReceived += new MessageReceivedEventHandler(OnTibcoCatchAll);
There is lots of sample C# code in the \src\ directory for the Tibco install that illustrates the techniques shown above.
I have a windows service, running workflows. The workflows are XAMLs loaded from database (users can define their own workflows using a rehosted designer). It is configured with one instance of the SQLWorkflowInstanceStore, to persist workflows when becoming idle. (It's basically derived from the example code in \ControllingWorkflowApplications from Microsoft's WCF/WF samples).
But sometimes I get an error like below:
System.Runtime.DurableInstancing.InstanceOwnerException: The execution of an InstancePersistenceCommand was interrupted because the instance owner registration for owner ID 'a426269a-be53-44e1-8580-4d0c396842e8' has become invalid. This error indicates that the in-memory copy of all instances locked by this owner have become stale and should be discarded, along with the InstanceHandles. Typically, this error is best handled by restarting the host.
I've been trying to find the cause, but it is hard to reproduce in development, on production servers however, I get it once in a while. One hint I found : when I look at the LockOwnersTable, I find the LockOnwersTable lockexpiration is set to 01/01/2000 0:0:0 and it's not getting updated anymore, while under normal circumstances the should be updated every x seconds according to the Host Lock Renewal period...
So , why whould SQLWorkflowInstanceStore stop renewing this LockExpiration and how can I detect the cause of it?
This happens because there are procedures running in the background and trying to extend the lock of the instance store every 30 seconds, and it seems that once the connection fail connecting to the SQL service it will mark this instance store as invalid.
you can see the same behaviour if you delete the instance store record from [LockOwnersTable] table.
The proposed solution is when this exception fires, you need to free the old instance store and initialize a new one
public class WorkflowInstanceStore : IWorkflowInstanceStore, IDisposable
{
public WorkflowInstanceStore(string connectionString)
{
_instanceStore = new SqlWorkflowInstanceStore(connectionString);
InstanceHandle handle = _instanceStore.CreateInstanceHandle();
InstanceView view = _instanceStore.Execute(handle,
new CreateWorkflowOwnerCommand(), TimeSpan.FromSeconds(30));
handle.Free();
_instanceStore.DefaultInstanceOwner = view.InstanceOwner;
}
public InstanceStore Store
{
get { return _instanceStore; }
}
public void Dispose()
{
if (null != _instanceStore)
{
var deleteOwner = new DeleteWorkflowOwnerCommand();
InstanceHandle handle = _instanceStore.CreateInstanceHandle();
_instanceStore.Execute(handle, deleteOwner, TimeSpan.FromSeconds(10));
handle.Free();
}
}
private InstanceStore _instanceStore;
}
you can find the best practices to create instance store handle in this link
Workflow Instance Store Best practices
This is an old thread but I just stumbled on the same issue.
Damir's Corner suggests to check if the instance handle is still valid before calling the instance store. I hereby quote the whole post:
Certain aspects of Workflow Foundation are still poorly documented; the persistence framework being one of them. The following snippet is typically used for setting up the instance store:
var instanceStore = new SqlWorkflowInstanceStore(connectionString);
instanceStore.HostLockRenewalPeriod = TimeSpan.FromSeconds(30);
var instanceHandle = instanceStore.CreateInstanceHandle();
var view = instanceStore.Execute(instanceHandle,
new CreateWorkflowOwnerCommand(), TimeSpan.FromSeconds(10));
instanceStore.DefaultInstanceOwner = view.InstanceOwner;
It's difficult to find a detailed explanation of what all of this
does; and to be honest, usually it's not necessary. At least not,
until you start encountering problems, such as InstanceOwnerException:
The execution of an InstancePersistenceCommand was interrupted because
the instance owner registration for owner ID
'9938cd6d-a9cb-49ad-a492-7c087dcc93af' has become invalid. This error
indicates that the in-memory copy of all instances locked by this
owner have become stale and should be discarded, along with the
InstanceHandles. Typically, this error is best handled by restarting
the host.
The error is closely related to the HostLockRenewalPeriod property
which defines how long obtained instance handle is valid without being
renewed. If you try monitoring the database while an instance store
with a valid instance handle is instantiated, you will notice
[System.Activities.DurableInstancing].[ExtendLock] being called
periodically. This stored procedure is responsible for renewing the
handle. If for some reason it fails to be called within the specified
HostLockRenewalPeriod, the above mentioned exception will be thrown
when attempting to persist a workflow. A typical reason for this would
be temporarily inaccessible database due to maintenance or networking
problems. It's not something that happens often, but it's bound to
happen if you have a long living instance store, e.g. in a constantly
running workflow host, such as a Windows service.
Fortunately it's not all that difficult to fix the problem, once you
know the cause of it. Before using the instance store you should
always check, if the handle is still valid; and renew it, if it's not:
if (!instanceHandle.IsValid)
{
instanceHandle = instanceStore.CreateInstanceHandle();
var view = instanceStore.Execute(instanceHandle,
new CreateWorkflowOwnerCommand(), TimeSpan.FromSeconds(10));
instanceStore.DefaultInstanceOwner = view.InstanceOwner;
}
It's definitely less invasive than the restart of the host, suggested
by the error message.
you have to be sure about expiration of owner user
here how I am used to handle this issue
public SqlWorkflowInstanceStore SetupSqlpersistenceStore()
{
SqlWorkflowInstanceStore sqlWFInstanceStore = new SqlWorkflowInstanceStore(ConfigurationManager.ConnectionStrings["DB_WWFConnectionString"].ConnectionString);
sqlWFInstanceStore.InstanceCompletionAction = InstanceCompletionAction.DeleteAll;
InstanceHandle handle = sqlWFInstanceStore.CreateInstanceHandle();
InstanceView view = sqlWFInstanceStore.Execute(handle, new CreateWorkflowOwnerCommand(), TimeSpan.FromSeconds(30));
handle.Free();
sqlWFInstanceStore.DefaultInstanceOwner = view.InstanceOwner;
return sqlWFInstanceStore;
}
and here how you can use this method
wfApp.InstanceStore = SetupSqlpersistenceStore();
wish this help
We have a HTTP end-point that takes a long time to run and can also be called concurrently by users. As part of this request, we update the model inside a synchronized block so that other (possibly concurrent) requests pick up that change.
E.g.
MyModel m = null;
synchronized (lockObject) {
m = MyModel.findById(id);
if (m.status == PENDING) {
m.status = ACTIVE;
} else {
//render a response back to user that the operation is not allowed
}
m.save(); //Is not expected to be called unless we set m.status = ACTIVE
}
//Long running operation continues here. It can involve further changes to instance "m"
The reason for the synchronized block is to ensure that even concurrent requests get to pick up the latest status. However, the underlying JPA does not commit my changes (m.save()) until the request is complete. Since this is a long-running request, I do not want to wait until the request is complete and still want to ensure that other callers are notified of the change in status. I tried to call "m.em().flush(); JPA.em().getTransaction().commit();" after m.save(), but that makes the transaction unavailable for the subsequent action as part of the same request. Can I just given "JPA.em().getTransaction().begin();" and let Play handle the transaction from then on? If not, what is the best way to handle this use-case?
UPDATE:
Based on the response, I modified my code as follows:
MyModel m = null;
synchronized (lockObject) {
m = MyModel.findById(id);
if (m.status == PENDING) {
m.status = ACTIVE;
} else {
//render a response back to user that the operation is not allowed
}
m.save(); //Is not expected to be called unless we set m.status = ACTIVE
}
new MyModelUpdateJob(m.id).now();
And in my job, I have the following line:
doJob() {
MyModel m = MyModel.findById(id);
print m.status; //This still prints the old status as-if m.save() had no effect...
}
What am I missing?
Put your update code in a job an call
new MyModelUpdateJob(id).now().get();
thus the update will be done in another transaction that is commited at the end of the job
ouch, as soon as you add more play servers, you will be in trouble. You may want to play with optimistic locking in your example or and I advise against it pessimistic locking....ick.
HOWEVER, looking at your code, maybe read the article Building on Quicksand. I am not sure you need a synchronized block in that case at all...try to go after being idempotent.
In your case if
1. user 1 and user 2 both call that method and it is pending, then it goes to active(Idempotent)
If user 1 or user 2 wins, well that would be like you had the synchronization block anyways.
I am sure however you have a more complex scenario not shown here, BUT READ that article Building on Quicksand as it really changes the traditional way of thinking and is how google and amazon and very large scale systems operate.
Another option for distributed transactions across play servers is zookeeper which the big large nosql guys use BUT only as a last resort ;) ;)
later,
Dean
ScopedDBConnection's constructor gets a connection from pool(if can't it will create a new one) and save it as a private member variable.Its get method returns a pointer of DBClientBase,I thinks client code don't need to delete this pointer because the done method will return it back to the pool. Here is my code,am I right.
ScopedDbConnection con(...);
DBClientBase* session = con.get();
//do something using session
...
//
con.done();// ignore session because done will return it back to connection pool
You can find a number of good ScopedDbConnection examples in the MongoDB github. Here's a file that shows some basic usage of that class:
https://github.com/mongodb/mongo/blob/master/src/mongo/client/model.cpp
Check out lines 24-46 (Model::load).