I have a Stateful service with 1000 partitions and 1 replica.
This service in the RunAsync method have an infinte while cycle where I call a Reliable Queue to get messages.
If there are no messages I wait 5 seconds, then retry.
I used to do exactly that with Azure Storage Queue with success.
But with Service Fabric I'm getting thousands of FabricNotReadableExceptions, the Service become unstable and I'm not able to update it or delete it, I need to cancel the entire cluster.
I tried to update it and after 18 hours it was still stuck, so there is something terribly wrong in what I'm doing.
This is the method code:
public async Task<QueueObject> DeQueueAsync(string queueName)
{
var q = await StateManager.GetOrAddAsync<IReliableQueue<string>>(queueName);
using (var tx = StateManager.CreateTransaction())
{
try
{
var dequeued = await q.TryDequeueAsync(tx);
if (dequeued.HasValue)
{
await tx.CommitAsync();
var result = dequeued.Value;
return JSON.Deserialize<QueueObject>(result);
}
else
{
return null;
}
}
catch (Exception e)
{
ServiceEventSource.Current.ServiceMessage(this, $"!!ERROR!!: {e.Message} - Partition: {Partition.PartitionInfo.Id}");
return null;
}
}}
This is the RunAsync
protected override async Task RunAsync(CancellationToken cancellationToken)
{
while (true)
{
var message = await DeQueueAsync("MyQueue");
if (message != null)
{
//process, takes around 500ms
}
else
{
Thread.Sleep(5000);
}
}
}
I also changed Thread.Sleep(5000) with Task.Delay and was having thousands of "A task was canceled" errors.
What I'm missing here?
It's the cycle too fast and SF cannot update the other replicas in time?
Should I remove all the replicas leaving just one?
Should I use the new ConcurrentQueue instead?
I have the problem in production and in local with 50 or 1000 partitions, doesn't matter.
I'm stuck and confused.
Thanks
You need to honor the cancellationToken that is passed in to your RunAsync implementation. Service Fabric will cancel the token when it wants to stop your service for any reason - including upgrades - and it will wait indefinitely for RunAsync to return after cancelling the token. This could explain why you couldn't upgrade your application.
I would suggest checking cancellationToken.IsCancelled inside your loop, and breaking out if it has been cancelled.
FabricNotReadableException can happen for a variety of reasons - the answer to this question has a comprehensive explanation, but the takeaway is
You can consider FabricNotReadableException retriable. If you see it, just try the call again and eventually it will resolve into either NotPrimary or Granted.
Related
I'm on the way to evaluate Dart for a German company by porting various Java programs to Dart and compare and analyze the results. In the browser Dart wins hands down. For server software performance seemed to be a serious isssue (see this question of me) but that got mostly defused.
Now I'm in the area of porting some "simple" command-line tools where I did not expect any serious problems at all but there is at least one. Some of the tools do make HTTP requests to collect some data and the stand-alone Dart virtual machine only supports them in an asynchronous fashion. Looking through all I could find it does not seem to be possible to use any asynchronous call in a mostly synchronous software.
I understand that I could restructure the available synchronous software into an asynchronous one. But this would transform a well-designed piece of software into something less readable and more difficult to debug and maintain. For some software pieces this just does not make sense.
My question: Is there an (overlooked by me) way to embed an asynchronous call into a synchronously called method?
I imagine that it would not be to difficult to provide a system call, usable only from within the main thread, which just transfers the execution to the whole list of queued asynchronous function calls (without having to end the main thread first) and as soon as the last one got executed returns and continues the main thread.
Something which might look like this:
var synchFunction() {
var result;
asyncFunction().then(() { result = ...; });
resync(); // the system call to move to and wait out all async execution
return result;
}
Having such a method would simplify the lib APIs as well. Most "sync" calls could be removed because the re-synchronisation call would do the job. It seems to be such a logical idea that I still think it somehow exists and I have missed it. Or is there a serious reason why that would not work?
After thinking about the received answer from lm (see below) for two days I still do not understand why the encapsulation of an asynchronous Dart call into a synchronous one should not be possible. It is done in the "normal" synchronous programing world all the time. Usually you can wait for a resynchronization by either getting a "Done" from the asynchronous routine or if something fails continue after a timeout.
With that in mind my first proposal could be enhanced like that:
var synchFunction() {
var result;
asyncFunction()
.then(() { result = ...; })
.whenComplete(() { continueResync() }); // the "Done" message
resync(timeout); // waiting with a timeout as maximum limit
// Either we arrive here with the [result] filled in or a with a [TimeoutException].
return result;
}
The resync() does the same that would normally happen after ending the main method of an isolate, it starts executing the queued asynchronous functions (or waits for events to make them executable). As soon as it encounters a continueResync() call a flag is set which stops this asynchronous execution and resync() returns to the main thread. If no continueResync() call is encountered during the given timeout period it too aborts the asynchronous execution and leaves resync() with a TimeoutException.
For some groups of software which benefit from straight synchronous programing (not the client software and not the server software) such a feature would solve lots of problems for the programer who has to deal with asynchrounous-only libraries.
I believe that I have also found a solution for the main argument in lm's argumentation below. Therefore my question still stands with respect to this "enhanced" solution which I proposed: Is there anything which really makes it impossible to implement that in Dart?
The only time that you can wrap an async method in a synchronous one is when you don't need to get a return value.
For example if you want to disable the save button, save results to the server asynchronously and re-enable the save button when the job is done you can write it like this:
Future<bool> save() async {
// save changes async here
return true;
}
void saveClicked() {
saveButton.enabled = false;
save()
.then((success) => window.alert(success ? 'Saved' : 'Failed'))
.catchError((e) => window.alert(e))
.whenComplete(() { saveButton.enabled = true; });
}
Note that the saveClicked method is fully synchronous, but executes the save method asynchronously.
Note that if you make saveClicked async, not only do you have to call it using the async pattern, but the entire method body will run asynchronously so the save button will not be disabled when the function returns.
For completeness the async version of saveClicked looks like this:
Future<Null> saveClicked() async {
saveButton.enabled = false;
try {
bool success = await save();
window.alert(success ? 'Saved' : 'Failed');
}
catch (e) {
window.alert(e);
}
finally {
saveButton.enabled = true;
}
}
Yes, this is way late, but I think this is a cool feature new people should know about.
There is a way, but the Dart docs warn against it (and it's somehow "experimental", although the implications aren't really discussed).
The waitFor command.
You basically pass in an asynchronous function that returns a Future, an optional timeout parameter, and the waitFor function will return the result.
For example:
final int number = waitFor<int>(someAsyncThatReturnsInt);
The resync function cannot be implemented in Dart's current execution model.
Asynchronous execution is contagious. A synchronous function must return before any other asynchronous events can execute, so there is no way to synchronously wait for asynchronous execution.
Execution in Dart is single-threaded and event based. There is no way for the resync function to block without it also blocking all other execution in the same isolate, so the pending async operations will never happen.
To block the synchronous execution, and continue executing something else, you need to preserve the entire call stack up to that point, and reinstate it later when the synchronous operations have completed. If you have that functionality, then there are probably better ways to do things than Future and Stream :)
Also, waiting for "all async execution" isn't well-defined in an event based system. There might be a broadcast Stream emitting events coming in from the network, a periodic timer, or a receive port getting data from another isolate, or some other source of events that you can't wait for because they come from outside the isolate, or event the process. When the current isolate shuts down, it might send a final shut-down message to another isolate, so effectively the "async execution" isn't over until the isolate dies.
Using the async/await syntax, you won't get synchronous operation, but it will be easier to code the similar asynchronous operation:
function() async {
var result = await asyncFunction();
return result;
}
It won't wait for async operations that aren't reflected in the Future returned by asyncFunction, but that's the job of asyncFunction to not complete until its operations are complete.
Dart is inherently async. Trying to avoid asynchronity won't work out.
There are sync versions of some API calls for example in dart:io and in some situations it might seem simpler to use them instead but because there aren't sync versions for all methods/functions you can't avoid async entirely.
With the recent introduction of the async/await feature programming async become much simpler and the code looks almost like sync code (but it isn't).
If a call went async it stays async. As far as I know there is nothing you can do about it.
import 'package:synchronized_lite/synchronized_lite.dart';
import 'dart:async';
// Using Lock as a mixin to further mimic Java-style synchronized blocks
class SomeActivity with Lock {
bool _started = false;
Future<bool> start() async {
// It's correct to return a Future returned by synchronized()
return synchronized(() async {
if(_started)
return false;
// perform the start operation
await Future.delayed(Duration(seconds: 1));
print("Started");
_started = true;
return true;
});
}
Future<void> stop() async {
// It's also correct to await a synchronized() call before returning
// It's incorrect to neither await a synchronized() call nor return its Future.
await synchronized(() async {
if(!_started)
return;
// perform the stop operation`enter code here`
await Future.delayed(Duration(seconds: 1));
print("Stopped");
_started = false;
});
}
}
// Prints:
// Started
// Stopped
main() async {
var a = SomeActivity();
print("Hello");
a.start();
a.start();
a.stop();
await a.stop();
}
/*Since the Await statement can be used in only asynchronous methods. Then we do two methods.I thinking first we call the async method and then we constantly query the null result for the non-async method. Then we get a synchronized model. In this way, we will wait for the answer in the non-async method. Such a method comes to my mind. But as far as I can see, there is no escape from the async working model in flutter dart language. Need to get used to it.It may be unprofessional, but I wanted to share the solution that came to my mind. hope it helps.
Stock resultStockQueryByBarcodeAsync;
bool waitStockQueryByBarcodeAsyncCompleted = false;
Stock WaitStockQueryByBarcodeAsync(String barcode, int timeOut) {
CallStockQueryByBarcodeAsync(barcode);
var startTime = new DateTime.now();
while (!waitStockQueryByBarcodeAsyncCompleted) {
Duration difference = DateTime.now().difference(startTime);
if (difference.inMilliseconds > timeOut) {
throw TimeoutException("Timeout Exceeded");
}
//we must scope time. Because it can be enter endless loop.
}
return resultStockQueryByBarcodeAsync;
}
void CallStockQueryByBarcodeAsync(String barcode) async {
waitStockQueryByBarcodeAsyncCompleted = false;
resultStockQueryByBarcodeAsync = null;
var stock = await StockQueryByBarcodeAsync(barcode);/*your target async method*/
waitStockQueryByBarcodeAsyncCompleted = true;
resultStockQueryByBarcodeAsync = stock;
}
In my case, I had to initialize the database connection from constructor. I am pretty new in Flutter and I don't know what are the best practices right now. But, here is what I did.
class Storage {
late Database database;
Storage() {
getConnection().then((value) => database = value);
}
Future<Database> getConnection() async {
return await openDatabase('ims.db');
}
}
All I have done, is used the callback method to assign the value when the value is available.
Here's a solution based on staggering the start of the async function with start times at least 1 second apart, when calls come in almost simultaneously.
Steps:
Use the lastKnownTime to calculate the delta, where the initial value is 0
Once the delta is not some huge number, you know it's a duplicate call.
class StartConversationState extends State<StartConversationStatefulWidget> {
#override
Widget build(BuildContext context) {
_delayPush(); // this is the call that gets triggered multiple times
}
int lastKnownTime = 0;
int delayMillis = 3000;
_delayPush() async {
delayMillis += 1500;
await new Future.delayed(Duration(milliseconds: delayMillis));
int millisSinceEpoch = new DateTime.now().millisecondsSinceEpoch;
int delta = millisSinceEpoch - lastKnownTime;
// if delta is less than 10 seconds, means it was a subsequent interval
if (delta < 10000) {
print('_delayPush() , SKIPPING DUPLICATE CALL');
return;
}
// here is the logic you don't want to duplicate
// eg, insert DB record and navigate to next screen
}
How can I process a list of delayed jobs in Vertx (actually
hundreds of HTTP GET requests, to limited API that bans fast requesting hosts)? now, I am using this code and it gets blocked because Vertx starts all requests at once. It is desirable to process each request with a 5-second delay between each request.
public void getInstrumnetDailyInfo(Instrument instrument,
Handler<AsyncResult<OptionInstrument>> handler) {
webClient
.get("/Loader")
.addQueryParam("i", instrument.getId())
.timeout(30000)
.send(
ar -> {
if (ar.succeeded()) {
String html = ar.result().bodyAsString();
Integer thatData = processHTML(html);
instrument.setThatData(thatData);
handler.handle(Future.succeededFuture(instrument));
} else {
// error
handler.handle(Future.failedFuture("error " +ar.cause()));
}
});
}
public void start(){
List<Instrument> instruments = loadInstrumentsList();
instruments.forEach(
instrument -> {
webClient.getInstrumnetDailyInfo(instrument,
async -> {
if(async.succeeded()){
instrumentMap.put(instrument.getId(), instrument);
}else {
log.warn("getInstrumnetDailyInfo: ", async.cause());
}
});
});
}
You can consider using a timer to fire events (rather than all at startup).
There are two variants in Vertx,
.setTimer() that fires a specific event after a delay
vertx.setTimer(interval, new Handler<T>() {});
and
2. .setPeriodic() that fires every time a specified period of time has passed.
vertx.setPeriodic(interval, new Handler<Long>() {});
setPeriodic seems to be what you are looking for.
You can get more info from the documentation
For more sophisticated Vertx scheduling use-cases, you can have a look at Chime or other schedulers or this module
You could use any out of the box rate limiter function and adapt it for async use.
An example with the RateLimiter from Guava:
// Make permits available at a rate of one every 5 seconds
private RateLimiter limiter = RateLimiter.create(1 / 5.0);
// A vert.x future that completes when it obtains a throttle permit
public Future<Double> throttle() {
return vertx.executeBlocking(p -> p.complete(limiter.acquire()), true);
}
Then...
throttle()
.compose(d -> {
System.out.printf("Waited %.2f before running job\n", d);
return runJob(); // runJob returns a Future result
});
I'm on the way to evaluate Dart for a German company by porting various Java programs to Dart and compare and analyze the results. In the browser Dart wins hands down. For server software performance seemed to be a serious isssue (see this question of me) but that got mostly defused.
Now I'm in the area of porting some "simple" command-line tools where I did not expect any serious problems at all but there is at least one. Some of the tools do make HTTP requests to collect some data and the stand-alone Dart virtual machine only supports them in an asynchronous fashion. Looking through all I could find it does not seem to be possible to use any asynchronous call in a mostly synchronous software.
I understand that I could restructure the available synchronous software into an asynchronous one. But this would transform a well-designed piece of software into something less readable and more difficult to debug and maintain. For some software pieces this just does not make sense.
My question: Is there an (overlooked by me) way to embed an asynchronous call into a synchronously called method?
I imagine that it would not be to difficult to provide a system call, usable only from within the main thread, which just transfers the execution to the whole list of queued asynchronous function calls (without having to end the main thread first) and as soon as the last one got executed returns and continues the main thread.
Something which might look like this:
var synchFunction() {
var result;
asyncFunction().then(() { result = ...; });
resync(); // the system call to move to and wait out all async execution
return result;
}
Having such a method would simplify the lib APIs as well. Most "sync" calls could be removed because the re-synchronisation call would do the job. It seems to be such a logical idea that I still think it somehow exists and I have missed it. Or is there a serious reason why that would not work?
After thinking about the received answer from lm (see below) for two days I still do not understand why the encapsulation of an asynchronous Dart call into a synchronous one should not be possible. It is done in the "normal" synchronous programing world all the time. Usually you can wait for a resynchronization by either getting a "Done" from the asynchronous routine or if something fails continue after a timeout.
With that in mind my first proposal could be enhanced like that:
var synchFunction() {
var result;
asyncFunction()
.then(() { result = ...; })
.whenComplete(() { continueResync() }); // the "Done" message
resync(timeout); // waiting with a timeout as maximum limit
// Either we arrive here with the [result] filled in or a with a [TimeoutException].
return result;
}
The resync() does the same that would normally happen after ending the main method of an isolate, it starts executing the queued asynchronous functions (or waits for events to make them executable). As soon as it encounters a continueResync() call a flag is set which stops this asynchronous execution and resync() returns to the main thread. If no continueResync() call is encountered during the given timeout period it too aborts the asynchronous execution and leaves resync() with a TimeoutException.
For some groups of software which benefit from straight synchronous programing (not the client software and not the server software) such a feature would solve lots of problems for the programer who has to deal with asynchrounous-only libraries.
I believe that I have also found a solution for the main argument in lm's argumentation below. Therefore my question still stands with respect to this "enhanced" solution which I proposed: Is there anything which really makes it impossible to implement that in Dart?
The only time that you can wrap an async method in a synchronous one is when you don't need to get a return value.
For example if you want to disable the save button, save results to the server asynchronously and re-enable the save button when the job is done you can write it like this:
Future<bool> save() async {
// save changes async here
return true;
}
void saveClicked() {
saveButton.enabled = false;
save()
.then((success) => window.alert(success ? 'Saved' : 'Failed'))
.catchError((e) => window.alert(e))
.whenComplete(() { saveButton.enabled = true; });
}
Note that the saveClicked method is fully synchronous, but executes the save method asynchronously.
Note that if you make saveClicked async, not only do you have to call it using the async pattern, but the entire method body will run asynchronously so the save button will not be disabled when the function returns.
For completeness the async version of saveClicked looks like this:
Future<Null> saveClicked() async {
saveButton.enabled = false;
try {
bool success = await save();
window.alert(success ? 'Saved' : 'Failed');
}
catch (e) {
window.alert(e);
}
finally {
saveButton.enabled = true;
}
}
Yes, this is way late, but I think this is a cool feature new people should know about.
There is a way, but the Dart docs warn against it (and it's somehow "experimental", although the implications aren't really discussed).
The waitFor command.
You basically pass in an asynchronous function that returns a Future, an optional timeout parameter, and the waitFor function will return the result.
For example:
final int number = waitFor<int>(someAsyncThatReturnsInt);
The resync function cannot be implemented in Dart's current execution model.
Asynchronous execution is contagious. A synchronous function must return before any other asynchronous events can execute, so there is no way to synchronously wait for asynchronous execution.
Execution in Dart is single-threaded and event based. There is no way for the resync function to block without it also blocking all other execution in the same isolate, so the pending async operations will never happen.
To block the synchronous execution, and continue executing something else, you need to preserve the entire call stack up to that point, and reinstate it later when the synchronous operations have completed. If you have that functionality, then there are probably better ways to do things than Future and Stream :)
Also, waiting for "all async execution" isn't well-defined in an event based system. There might be a broadcast Stream emitting events coming in from the network, a periodic timer, or a receive port getting data from another isolate, or some other source of events that you can't wait for because they come from outside the isolate, or event the process. When the current isolate shuts down, it might send a final shut-down message to another isolate, so effectively the "async execution" isn't over until the isolate dies.
Using the async/await syntax, you won't get synchronous operation, but it will be easier to code the similar asynchronous operation:
function() async {
var result = await asyncFunction();
return result;
}
It won't wait for async operations that aren't reflected in the Future returned by asyncFunction, but that's the job of asyncFunction to not complete until its operations are complete.
Dart is inherently async. Trying to avoid asynchronity won't work out.
There are sync versions of some API calls for example in dart:io and in some situations it might seem simpler to use them instead but because there aren't sync versions for all methods/functions you can't avoid async entirely.
With the recent introduction of the async/await feature programming async become much simpler and the code looks almost like sync code (but it isn't).
If a call went async it stays async. As far as I know there is nothing you can do about it.
import 'package:synchronized_lite/synchronized_lite.dart';
import 'dart:async';
// Using Lock as a mixin to further mimic Java-style synchronized blocks
class SomeActivity with Lock {
bool _started = false;
Future<bool> start() async {
// It's correct to return a Future returned by synchronized()
return synchronized(() async {
if(_started)
return false;
// perform the start operation
await Future.delayed(Duration(seconds: 1));
print("Started");
_started = true;
return true;
});
}
Future<void> stop() async {
// It's also correct to await a synchronized() call before returning
// It's incorrect to neither await a synchronized() call nor return its Future.
await synchronized(() async {
if(!_started)
return;
// perform the stop operation`enter code here`
await Future.delayed(Duration(seconds: 1));
print("Stopped");
_started = false;
});
}
}
// Prints:
// Started
// Stopped
main() async {
var a = SomeActivity();
print("Hello");
a.start();
a.start();
a.stop();
await a.stop();
}
/*Since the Await statement can be used in only asynchronous methods. Then we do two methods.I thinking first we call the async method and then we constantly query the null result for the non-async method. Then we get a synchronized model. In this way, we will wait for the answer in the non-async method. Such a method comes to my mind. But as far as I can see, there is no escape from the async working model in flutter dart language. Need to get used to it.It may be unprofessional, but I wanted to share the solution that came to my mind. hope it helps.
Stock resultStockQueryByBarcodeAsync;
bool waitStockQueryByBarcodeAsyncCompleted = false;
Stock WaitStockQueryByBarcodeAsync(String barcode, int timeOut) {
CallStockQueryByBarcodeAsync(barcode);
var startTime = new DateTime.now();
while (!waitStockQueryByBarcodeAsyncCompleted) {
Duration difference = DateTime.now().difference(startTime);
if (difference.inMilliseconds > timeOut) {
throw TimeoutException("Timeout Exceeded");
}
//we must scope time. Because it can be enter endless loop.
}
return resultStockQueryByBarcodeAsync;
}
void CallStockQueryByBarcodeAsync(String barcode) async {
waitStockQueryByBarcodeAsyncCompleted = false;
resultStockQueryByBarcodeAsync = null;
var stock = await StockQueryByBarcodeAsync(barcode);/*your target async method*/
waitStockQueryByBarcodeAsyncCompleted = true;
resultStockQueryByBarcodeAsync = stock;
}
In my case, I had to initialize the database connection from constructor. I am pretty new in Flutter and I don't know what are the best practices right now. But, here is what I did.
class Storage {
late Database database;
Storage() {
getConnection().then((value) => database = value);
}
Future<Database> getConnection() async {
return await openDatabase('ims.db');
}
}
All I have done, is used the callback method to assign the value when the value is available.
Here's a solution based on staggering the start of the async function with start times at least 1 second apart, when calls come in almost simultaneously.
Steps:
Use the lastKnownTime to calculate the delta, where the initial value is 0
Once the delta is not some huge number, you know it's a duplicate call.
class StartConversationState extends State<StartConversationStatefulWidget> {
#override
Widget build(BuildContext context) {
_delayPush(); // this is the call that gets triggered multiple times
}
int lastKnownTime = 0;
int delayMillis = 3000;
_delayPush() async {
delayMillis += 1500;
await new Future.delayed(Duration(milliseconds: delayMillis));
int millisSinceEpoch = new DateTime.now().millisecondsSinceEpoch;
int delta = millisSinceEpoch - lastKnownTime;
// if delta is less than 10 seconds, means it was a subsequent interval
if (delta < 10000) {
print('_delayPush() , SKIPPING DUPLICATE CALL');
return;
}
// here is the logic you don't want to duplicate
// eg, insert DB record and navigate to next screen
}
No endpoint found for the service '{serviceB}' partition '{guid}' that matches the specified TargetReplicaSelector : 'RandomSecondaryReplica'
This is an error that has not always showed up, but it does sometimes.
I'm calling a stateful service B from another stateful service A, with service remoting, asking for a random secondary replica, to access state written to the primary.
I can see in Explorer that the partition is there and shows OK, and it has a primary and two ActiveSecondaries.
The service B has following:
protected override IEnumerable<ServiceReplicaListener> CreateServiceReplicaListeners()
{
return new[] { new ServiceReplicaListener(context =>
this.CreateServiceRemotingListener(context), listenOnSecondary: true) };
}
I get all the partitions by this:
return Enumerable.Range(0, PartitionConstants.Partitions).Select(x =>
ServiceProxy.Create<IServiceB>(
ServiceBUri,
new ServicePartitionKey(x),
TargetReplicaSelector.RandomSecondaryReplica));
And the overall settings must be OK since sometimes it does work. And I know the primary is responding because I have saved state there.
So, what could cause this error when I can actually see the partition there, with the secondary replicas?
Update1 : Restarting the calling service made connection work. But they started together, and well after both had been running and working, the problem persisted, until I restarted. Howcome?
Update2 : This happens when whole cluster is started. At startup, Service A primaries calls Service B primaries for some registration. A polls B to know that it has initiated its internal state before doing this.
Then when this is complete, Service A goes on to check if its internal state needs update, and if so, it will call Service B again to retrieve state. Since it will not do any writing to B state, it calls secondary replicas. And here is when endpoint is not found.
When I restart Service A, endpoints are found.
Could it be that primaries are working and OK, but the secondaries are not yet OK?
How can I ascertain this? Is there some service fabric class that I can access to know whether the secondary will be found if I call for it?
Using a service primer found here, solved this issue. Seems like not all partition replicas was ready when being called.
Basically, what it does is counting all replicas of all partitions via FabricClient, until expected count is found.
Here is code:
public async Task WaitForStatefulService(Uri serviceInstanceUri, CancellationToken token)
{
StatefulServiceDescription description =
await this.Client.ServiceManager.GetServiceDescriptionAsync(serviceInstanceUri) as StatefulServiceDescription;
int targetTotalReplicas = description.TargetReplicaSetSize;
if (description.PartitionSchemeDescription is UniformInt64RangePartitionSchemeDescription)
{
targetTotalReplicas *= ((UniformInt64RangePartitionSchemeDescription)description.PartitionSchemeDescription).PartitionCount;
}
ServicePartitionList partitions = await this.Client.QueryManager.GetPartitionListAsync(serviceInstanceUri);
int replicaTotal = 0;
while (replicaTotal < targetTotalReplicas && !token.IsCancellationRequested)
{
await Task.Delay(this.interval);
//ServiceEventSource.Current.ServiceMessage(this, "CountyService waiting for National Service to come up.");
replicaTotal = 0;
foreach (Partition partition in partitions)
{
ServiceReplicaList replicaList = await this.Client.QueryManager.GetReplicaListAsync(partition.PartitionInformation.Id);
replicaTotal += replicaList.Count(x => x.ReplicaStatus == System.Fabric.Query.ServiceReplicaStatus.Ready);
}
}
}
I am using rabbitmq and I want to make sure that if I have a connection problem in the client, the messages that I posted won't be lost. I simulate it with eclipse: I do system.exit the program of fetching after 100 messages. I posted 1000 messages. The second run I don't limit the number of messages and it returns me 840 messages with 3 times. Can you help me?
the code of the producer is:
public void run() {
String json =SimpleQueueServiceSample.getFromList();
while (!(json.equals(""))){
json =SimpleQueueServiceSample.getFromList();
try {
c.basicPublish("", "test",
MessageProperties.PERSISTENT_TEXT_PLAIN, json.getBytes());
} catch (IOException e) {
e.printStackTrace();
}
}
try {
c.waitForConfirmsOrDie();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
the code of the consumber is:
QueueingConsumer consumer = new QueueingConsumer(channel);
channel.basicConsume(QUEUE_NAME, true, consumer);
while (true) {
System.out.println(count++);
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
String message = new String(delivery.getBody());
System.out.println(" [x] Received '" + message + "'");
}
So the challenge for your scenario is how you're handling the acknowledgements.
channel.basicConsume(QUEUE_NAME, true, consumer);
Is the problem. The second parameter of true is the auto-acknowledge field.
To fix that, use:
channel.basicConsume(QUEUE_NAME, false, consumer);
while (true) {
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
//...
channel.basicAck(delivery.getEnvelope().getDeliveryTag(), false);
}
It looks like you're using RabbitMQ's tutorials, and your code snippet is from part one. If you look at part two, they start talking about acknowledgements and setting up quality of service to provide round-robin dispatch.
It's worth pointing out that the basicConsume() and nextDelivery() combination rely upon a hidden queue that lives within the consumer. So when you call basicConsume() several messages are pulled down to the client to local storage.
The benefit at that approach is that it avoids additional network overhead from calling for each individual message. The problem is that it can put more messages within your local consumer than you wish and you may lose messages if the consumer drops before processing all of the messages in the local hidden queue.
If you truly want your consumers only working on one message a time so that nothing is lost, you probably want to look at the basicGet() method instead of the basicConsume().