Im using kue to schedule a job, but its only is executed the first time.
Here is my code:
Queue.clear(function(error, response) {
var job = Queue
.createJob("sendStatus")
.priority('normal')
.removeOnComplete(true);
Queue.every('20 seconds', job);
Queue.process("sendStatus", telegram.checkStatus);
});
I forgot to call done() on checkStatus
Related
I want to use gulp-imagemin to minify images. The relevant part of my gulpfile.js looks like this:
const gulp = require('gulp');
// a couple more require('')s
function minifyImages(cb) {
import('gulp-imagemin')
.then(module => {
const imagemin = module.default;
gulp.src('src/img/**/*')
.pipe(imagemin())
.pipe(gulp.dest('img'));
cb();
})
.catch(err => {
console.log(err);
cb();
});
}
function buildCSS(cb) { /* ... */ }
exports.build = gulp.series(buildCSS, minifyImages);
The reason I'm using a dynamic import here is because I think I have to - gulp-imagemin doesn't support the require('') syntax, and when I say import imagemin from 'gulp-imagemin I get an error saying "Cannot use import statement outside a module".
I would expect the build task to only finish after minifyImages has finished. After all, I'm calling cb() only at the very end, at a point where the promise should be resolved.
However, build seems to finish early, while minifyImages is still running. This is the output I get:
[21:54:47] Finished 'buildCSS' after 6.82 ms
[21:54:47] Starting 'minifyImages'...
[21:54:47] Finished 'minifyImages' after 273 ms
[21:54:47] Finished 'build' after 282 ms
<one minute later>
[21:55:49] gulp-imagemin: Minified 46 images (saved 5.91 MB - 22.8%)
How can I make sure the task doesn't finish early, and all tasks are run in sequence?
Let me know if there's something wrong with my assumptions; I'm somewhat new to gulp and importing.
Streams are always asynchronous, so if the cb() callback is called just after creating the gulp stream as in your then handler, it's only obvious that the stream by that time has not finished yet (in fact, it hasn't even started).
The simplest solution to call a callback when the gulp.dest stream has finished is using stream.pipeline, i.e.:
function minifyImages(cb) {
const { pipeline } = require('stream');
return import('gulp-imagemin')
.then(module => {
const imagemin = module.default;
pipeline(
gulp.src('src/img/**/*'),
imagemin(),
gulp.dest('img'),
cb
);
})
.catch(cb);
}
Or similarly, with an async function.
async function minifyImages(cb) {
const { pipeline } = require('stream');
const { default: imagemin } = await import('gulp-imagemin');
return pipeline(
gulp.src('src/img/**/*'),
imagemin(),
gulp.dest('img'),
cb
);
}
Another approach I have seen is to split the task in two sequential sub-tasks: the first sub-task imports the plugin module and stores it in a variable, and the second sub-task uses the plugin already loaded by the previous sub-task to create and return the gulp stream in the usual way.
Then the two sub-tasks can be combined with gulp.series.
I’m new to proto.actor/actor programming and I’m wondering is this possible to achieve this behavior:
Actor A is asking actor B via async command – he should await for response to achieve request/response model but using tasks.
Actor B is using HTTP request so it would be some async IO operation so I don’t want it to be blocked for other actors in this time, so when 10 actors will ask him in the same time each request will be queued but while first request is waiting for process second should get a chance to proceed. Once firs request will be finished it should have priority in queue and get response to actor A.
How to get this flow?
For example I have 3 clients that ask service for some data, service call is taking 5 seconds and most of this time service is spending in IO. With current implementation we have 15 second in total for all requests but I would like it to take ~5-6 second
public static class ProtoTest
{
public static PID Service;
public static async Task Start()
{
var context = new RootContext();
var props = Props.FromProducer(() => new ClientActor());
var serviceProps = Props.FromProducer(() => new ServiceActor());
Service = context.Spawn(serviceProps);
var jobs = new List<Task>();
for (int i = 0; i < 3; i++)
{
string actorName = $"Actor_{i}";
jobs.Add(Task.Run(() =>
{
var client = context.SpawnNamed(props, actorName);
context.Send(client, new Command());
}));
}
Console.ReadLine();
}
}
public class ClientActor : IActor
{
public virtual async Task ReceiveAsync(IContext context)
{
if (context.Message is Command)
{
Console.WriteLine($"{DateTime.Now.ToLongTimeString()} START processing by {context.Self.Id}");
var result = await context.RequestAsync<string>(ProtoTest.Service, new Query());
Console.WriteLine($"{DateTime.Now.ToLongTimeString()} End processing by {context.Self.Id}");
}
return;
}
}
public class ServiceActor : IActor
{
public async virtual Task ReceiveAsync(IContext context)
{
if (context.Message is Query)
{
// this operation is taking long time so actor could handle others in this time
await Task.Delay(5000);
context.Respond("result");
}
return;
}
}
One of the core principles of an actor is that it does not perform multiple operations in parallel. If I understand your problem correctly, what you can do instead is to create a new actor for each operation that you want to run in parallel (actors are cheap so creating many is not an issue). So if actor A needs to send N commands to be processed asynchronously and receive each result as they come in, it could spawn N actors, B1,B2...Bn (one for each command) and send a Request to each of them. The B actors await the result and then Respond back to the A actor. Each response would then be sent as a message to actor A's mailbox and be processed sequentially in the order they complete.
I have a Stateful service with 1000 partitions and 1 replica.
This service in the RunAsync method have an infinte while cycle where I call a Reliable Queue to get messages.
If there are no messages I wait 5 seconds, then retry.
I used to do exactly that with Azure Storage Queue with success.
But with Service Fabric I'm getting thousands of FabricNotReadableExceptions, the Service become unstable and I'm not able to update it or delete it, I need to cancel the entire cluster.
I tried to update it and after 18 hours it was still stuck, so there is something terribly wrong in what I'm doing.
This is the method code:
public async Task<QueueObject> DeQueueAsync(string queueName)
{
var q = await StateManager.GetOrAddAsync<IReliableQueue<string>>(queueName);
using (var tx = StateManager.CreateTransaction())
{
try
{
var dequeued = await q.TryDequeueAsync(tx);
if (dequeued.HasValue)
{
await tx.CommitAsync();
var result = dequeued.Value;
return JSON.Deserialize<QueueObject>(result);
}
else
{
return null;
}
}
catch (Exception e)
{
ServiceEventSource.Current.ServiceMessage(this, $"!!ERROR!!: {e.Message} - Partition: {Partition.PartitionInfo.Id}");
return null;
}
}}
This is the RunAsync
protected override async Task RunAsync(CancellationToken cancellationToken)
{
while (true)
{
var message = await DeQueueAsync("MyQueue");
if (message != null)
{
//process, takes around 500ms
}
else
{
Thread.Sleep(5000);
}
}
}
I also changed Thread.Sleep(5000) with Task.Delay and was having thousands of "A task was canceled" errors.
What I'm missing here?
It's the cycle too fast and SF cannot update the other replicas in time?
Should I remove all the replicas leaving just one?
Should I use the new ConcurrentQueue instead?
I have the problem in production and in local with 50 or 1000 partitions, doesn't matter.
I'm stuck and confused.
Thanks
You need to honor the cancellationToken that is passed in to your RunAsync implementation. Service Fabric will cancel the token when it wants to stop your service for any reason - including upgrades - and it will wait indefinitely for RunAsync to return after cancelling the token. This could explain why you couldn't upgrade your application.
I would suggest checking cancellationToken.IsCancelled inside your loop, and breaking out if it has been cancelled.
FabricNotReadableException can happen for a variety of reasons - the answer to this question has a comprehensive explanation, but the takeaway is
You can consider FabricNotReadableException retriable. If you see it, just try the call again and eventually it will resolve into either NotPrimary or Granted.
I want to run periodic tasks in with a restriction that at most only one execution of a method is running at any given time.
I was experimenting with Rx, but I am not sure how to impose at most once concurrency restriction.
var timer = Observable.Interval(TimeSpan.FromMilliseconds(100));
timer.Subscribe(tick => DoSomething());
Additionally, if a task is still running, I want the subsequent schedule to elapse. i.e I don't want the tasks to queue up and cause problems.
I have 2 such tasks to execute periodically. The tasks being executed is currently synchronous. But, I could make them async if there is a necessity.
You are on the right track, you can use Select + Concat to flatten out the observable and limit the number of inflight requests (Note: if your task takes longer than the interval time, then they will start to stack up since they can't execute fast enough):
var source = Observable.Interval(TimeSpan.FromMilliseconds(100))
//I assume you are doing async work since you want to limit concurrency
.Select(_ => Observable.FromAsync(() => DoSomethingAsync()))
//This is equivalent to calling Merge(1)
.Concat();
source.Subscribe(/*Handle the result of each operation*/);
You should have tested your code as is because this is exactly what Rx imposes already.
Try this as a test:
void Main()
{
var timer = Observable.Interval(TimeSpan.FromMilliseconds(100));
using (timer.Do(x => Console.WriteLine("!")).Subscribe(tick => DoSomething()))
{
Console.ReadLine();
}
}
private void DoSomething()
{
Console.Write("<");
Console.Write(DateTime.Now.ToString("HH:mm:ss.fff"));
Thread.Sleep(1000);
Console.WriteLine(">");
}
When you run this you'll get this kind of output:
!
<16:54:57.111>
!
<16:54:58.112>
!
<16:54:59.113>
!
<16:55:00.113>
!
<16:55:01.114>
!
<16:55:02.115>
!
<16:55:03.116>
!
<16:55:04.117>
!
<16:55:05.118>
!
<16:55:06.119
It is already ensuring that there's no overlap.
Below are two implementations of a PeriodicSequentialExecution method, that creates an observable by executing an asynchronous method in a periodic fashion, enforcing a no-overlapping-execution policy. The interval between subsequent executions can be extended to prevent overlapping, in which case the period is time-shifted accordingly.
The first implementation is purely functional, while the second implementation is mostly imperative. Both implementations are functionally identical. The first one can be supplied with a custom IScheduler. The second one may be slightly more efficient.
The functional implementation:
/// <summary>
/// Creates an observable sequence containing the results of an asynchronous
/// action that is invoked periodically and sequentially (without overlapping).
/// </summary>
public static IObservable<T> PeriodicSequentialExecution<T>(
Func<CancellationToken, Task<T>> action,
TimeSpan dueTime, TimeSpan period,
CancellationToken cancellationToken = default,
IScheduler scheduler = null)
{
// Arguments validation omitted
scheduler ??= DefaultScheduler.Instance;
return Delay(dueTime) // Initial delay
.Concat(Observable.Using(() => CancellationTokenSource.CreateLinkedTokenSource(
cancellationToken), linkedCTS =>
// Execution loop
Observable.Publish( // Start a hot delay timer before each operation
Delay(period), hotTimer => Observable
.StartAsync(() => action(linkedCTS.Token)) // Start the operation
.Concat(hotTimer) // Await the delay timer
)
.Repeat()
.Finally(() => linkedCTS.Cancel()) // Unsubscription: cancel the operation
));
IObservable<T> Delay(TimeSpan delay)
=> Observable
.Timer(delay, scheduler)
.IgnoreElements()
.Select(_ => default(T))
.TakeUntil(Observable.Create<Unit>(o => cancellationToken.Register(() =>
o.OnError(new OperationCanceledException(cancellationToken)))));
}
The imperative implementation:
public static IObservable<T> PeriodicSequentialExecution2<T>(
Func<CancellationToken, Task<T>> action,
TimeSpan dueTime, TimeSpan period,
CancellationToken cancellationToken = default)
{
// Arguments validation omitted
return Observable.Create<T>(async (observer, ct) =>
{
using (var linkedCTS = CancellationTokenSource.CreateLinkedTokenSource(
ct, cancellationToken))
{
try
{
await Task.Delay(dueTime, linkedCTS.Token);
while (true)
{
var delayTask = Task.Delay(period, linkedCTS.Token);
var result = await action(linkedCTS.Token);
observer.OnNext(result);
await delayTask;
}
}
catch (Exception ex) { observer.OnError(ex); }
}
});
}
The cancellationToken parameter can be used for the graceful termination of the resulting observable sequence. This means that the sequence waits for the currently running operation to complete before terminating. If you prefer it to terminate instantaneously, potentially leaving work running unobserved in a fire-and-forget fashion, you can simply dispose the subscription to the observable sequence as always. Canceling the cancellationToken results to the observable sequence completing in a faulted state (OperationCanceledException).
Here is a factory function that does exactly what you are asking for.
public static IObservable<Unit> Periodic(TimeSpan timeSpan)
{
return Observable.Return(Unit.Default).Concat(Observable.Return(Unit.Default).Delay(timeSpan).Repeat());
}
Here is an example usage
Periodic(TimeSpan.FromSeconds(1))
.Subscribe(x =>
{
Console.WriteLine(DateTime.Now.ToString("mm:ss:fff"));
Thread.Sleep(500);
});
If you run this, each console print will be roughly 1.5 seconds apart.
Note, If you don't want the first tick to run immediately, you could instead use this factory, which won't send the first Unit until after the timespan.
public static IObservable<Unit> DelayedPeriodic(TimeSpan timeSpan)
{
return Observable.Return(Unit.Default).Delay(timeSpan).Repeat();
}
We have implemented recurring tasks using Quartz scheduler in our app. The user may schedule a recurring task starting at any time, even starting in the past. So for example, I can schedule a task to run monthly, starting on the 1st July, even though today is 17th July.
I would expect Quartz to run the first job immediately if it is in the past, and any subsequent jobs I throw at it. However, today I encountered a case when the task didn't get triggered instantly. The task was scheduled for 15th July, today is 17th July. Nothing happened. After I restarted the server and the code to schedule all the tasks in the DB ran, it did get triggered. Why would that happen ?
Code for scheduling the task below. Note that to make it recurring, we just reschedule it with the same code for another date we calculate (but that part of the code doesn't matter for the issue at hand).
Edit: Only the first job gets triggered, any subsequent jobs are not. If I try to use startNow() instead of startAt(Date), it still doesn't work, makes no difference.
JobDetail job = JobBuilder.newJob(ScheduledAppJob.class)
.withIdentity(stringId)
.build();
Trigger trigger = TriggerBuilder.newTrigger()
.withIdentity(stringId)
.startAt(date)
.build();
try
{
scheduler.scheduleJob(job, trigger);
dateFormat = new SimpleDateFormat("dd MMM yyyy, HH:mm:ss");
String recurringTime = dateFormat.format(date);
logger.info("Scheduling recurring job for " + recurringTime);
}
catch (SchedulerException se)
{
se.printStackTrace();
}
quartz.properties file, located under src/main (tried even in WEB-INF and WEB-INF/classes like suggested in the tutorial, but made no difference); even tried with 20 threadCount, still no difference:
org.quartz.scheduler.instanceName = AppScheduler
org.quartz.threadPool.threadCount = 3
org.quartz.jobStore.class = org.quartz.simpl.RAMJobStore
Seems to be working now. Haven't ran into any more problems. Could've been a config issue, as I have moved the config file in /src/main/resources.
Also try turning logging on in order to help with the debug:
log4j.logger.com.gargoylesoftware.htmlunit=DEBUG
We also added a JobTriggerListener to help with the logs:
private static class JobTriggerListener implements TriggerListener
{
private String name;
public JobTriggerListener(String name)
{
this.name = name;
}
public String getName()
{
return name;
}
public void triggerComplete(Trigger trigger, JobExecutionContext context,
Trigger.CompletedExecutionInstruction triggerInstructionCode)
{
}
public void triggerFired(Trigger trigger, JobExecutionContext context)
{
}
public void triggerMisfired(Trigger trigger)
{
logger.warn("Trigger misfired for trigger: " + trigger.getKey());
try
{
logger.info("Available threads: " + scheduler.getCurrentlyExecutingJobs());
}
catch (SchedulerException ex)
{
logger.error("Could not get currently executing jobs.", ex);
}
}
public boolean vetoJobExecution(Trigger trigger, JobExecutionContext context)
{
return false;
}
}
Check the updated config file:
#============================================================================
# Configure Main Scheduler Properties
#============================================================================
org.quartz.scheduler.skipUpdateCheck = true
org.quartz.scheduler.instanceName = MyAppScheduler
org.quartz.scheduler.instanceId = AUTO
#============================================================================
# Configure ThreadPool
#============================================================================
org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount = 25
org.quartz.threadPool.threadPriority = 9
#============================================================================
# Configure JobStore
#============================================================================
org.quartz.jobStore.misfireThreshold = 60000
org.quartz.jobStore.class = org.quartz.simpl.RAMJobStore