Using Bull queue. I have a queue that is rate limited. I would like failed jobs to be retried as soon as possible in the queue, considering the rate limits and backoff strategy. This does not seem to be the case. It seems that the failed job is pushed on, and will be processed behind other existing jobs in the queue.
A contrived example:
const Bull = require('bull');
const queue = new Bull('test-queue', {
limiter: {
max: 1,
duration: 1000
}
});
queue.process(async ({ data, id }) => {
let {fail = false} = data;
console.log(`processing job ${id}`);
if (fail) {
throw new Error('Queue test error');
}
return `${id} completed`;
});
queue.add({fail: true});
for(let i = 0; i < 10; i++) {
queue.add({});
}
This always results in jobs after the first being processed, then the failed job is retried. According to bull, this should not be happening. Is my implementation wrong?
Related
I have an Observable stream (representing a network connection of data values) which I'm Replaying and RefCounting. The underlying source is disconnecting as expected when the subscriber count hits zero, but Replay(1) returns a value from this source to the next subscriber, which I consider to be stale. I've fixed this with a custom implementation of ReplaySubject, which forgets the last value when the last observer disconnects, but it's a leaky abstraction and I'm wondering whether there's a more idomatic way to solve this problem?
Example failing test:
[Test]
public async Task Replay_DoesNotReplay_AfterRefCountDisconnection()
{
int i = 0;
var replay = Observable.Defer(
() =>
{
i++;
return new[] {i, i}.ToObservable();
})
.Replay(1)
.RefCount();
var first = await replay.Take(1).ToTask();
var second = await replay.Take(1).ToTask();
Assert.AreEqual(1, first);
Assert.AreEqual(2, second);
}
I would like to set up an Rx subscription that can respond to an event right away, and then ignore subsequent events that happen within a specified "cooldown" period.
The out of the box Throttle/Buffer methods respond only once the timeout has elapsed, which is not quite what I need.
Here is some code that sets up the scenario, and uses a Throttle (which isn't the solution I want):
class Program
{
static Stopwatch sw = new Stopwatch();
static void Main(string[] args)
{
var subject = new Subject<int>();
var timeout = TimeSpan.FromMilliseconds(500);
subject
.Throttle(timeout)
.Subscribe(DoStuff);
var factory = new TaskFactory();
sw.Start();
factory.StartNew(() =>
{
Console.WriteLine("Batch 1 (no delay)");
subject.OnNext(1);
});
factory.StartNewDelayed(1000, () =>
{
Console.WriteLine("Batch 2 (1s delay)");
subject.OnNext(2);
});
factory.StartNewDelayed(1300, () =>
{
Console.WriteLine("Batch 3 (1.3s delay)");
subject.OnNext(3);
});
factory.StartNewDelayed(1600, () =>
{
Console.WriteLine("Batch 4 (1.6s delay)");
subject.OnNext(4);
});
Console.ReadKey();
sw.Stop();
}
private static void DoStuff(int i)
{
Console.WriteLine("Handling {0} at {1}ms", i, sw.ElapsedMilliseconds);
}
}
The output of running this right now is:
Batch 1 (no delay)
Handling 1 at 508ms
Batch 2 (1s delay)
Batch 3 (1.3s delay)
Batch 4 (1.6s delay)
Handling 4 at 2114ms
Note that batch 2 isn't handled (which is fine!) because we wait for 500ms to elapse between requests due to the nature of throttle. Batch 3 is also not handled, (which is less alright because it happened more than 500ms from batch 2) due to its proximity to Batch 4.
What I'm looking for is something more like this:
Batch 1 (no delay)
Handling 1 at ~0ms
Batch 2 (1s delay)
Handling 2 at ~1000s
Batch 3 (1.3s delay)
Batch 4 (1.6s delay)
Handling 4 at ~1600s
Note that batch 3 wouldn't be handled in this scenario (which is fine!) because it occurs within 500ms of Batch 2.
EDIT:
Here is the implementation for the "StartNewDelayed" extension method that I use:
/// <summary>Creates a Task that will complete after the specified delay.</summary>
/// <param name="factory">The TaskFactory.</param>
/// <param name="millisecondsDelay">The delay after which the Task should transition to RanToCompletion.</param>
/// <returns>A Task that will be completed after the specified duration.</returns>
public static Task StartNewDelayed(
this TaskFactory factory, int millisecondsDelay)
{
return StartNewDelayed(factory, millisecondsDelay, CancellationToken.None);
}
/// <summary>Creates a Task that will complete after the specified delay.</summary>
/// <param name="factory">The TaskFactory.</param>
/// <param name="millisecondsDelay">The delay after which the Task should transition to RanToCompletion.</param>
/// <param name="cancellationToken">The cancellation token that can be used to cancel the timed task.</param>
/// <returns>A Task that will be completed after the specified duration and that's cancelable with the specified token.</returns>
public static Task StartNewDelayed(this TaskFactory factory, int millisecondsDelay, CancellationToken cancellationToken)
{
// Validate arguments
if (factory == null) throw new ArgumentNullException("factory");
if (millisecondsDelay < 0) throw new ArgumentOutOfRangeException("millisecondsDelay");
// Create the timed task
var tcs = new TaskCompletionSource<object>(factory.CreationOptions);
var ctr = default(CancellationTokenRegistration);
// Create the timer but don't start it yet. If we start it now,
// it might fire before ctr has been set to the right registration.
var timer = new Timer(self =>
{
// Clean up both the cancellation token and the timer, and try to transition to completed
ctr.Dispose();
((Timer)self).Dispose();
tcs.TrySetResult(null);
});
// Register with the cancellation token.
if (cancellationToken.CanBeCanceled)
{
// When cancellation occurs, cancel the timer and try to transition to cancelled.
// There could be a race, but it's benign.
ctr = cancellationToken.Register(() =>
{
timer.Dispose();
tcs.TrySetCanceled();
});
}
if (millisecondsDelay > 0)
{
// Start the timer and hand back the task...
timer.Change(millisecondsDelay, Timeout.Infinite);
}
else
{
// Just complete the task, and keep execution on the current thread.
ctr.Dispose();
tcs.TrySetResult(null);
timer.Dispose();
}
return tcs.Task;
}
Here's my approach. It's similar to others that have gone before, but it doesn't suffer the over-zealous window production problem.
The desired function works a lot like Observable.Throttle but emits qualifying events as soon as they arrive rather than delaying for the duration of the throttle or sample period. For a given duration after a qualifying event, subsequent events are suppressed.
Given as a testable extension method:
public static class ObservableExtensions
{
public static IObservable<T> SampleFirst<T>(
this IObservable<T> source,
TimeSpan sampleDuration,
IScheduler scheduler = null)
{
scheduler = scheduler ?? Scheduler.Default;
return source.Publish(ps =>
ps.Window(() => ps.Delay(sampleDuration,scheduler))
.SelectMany(x => x.Take(1)));
}
}
The idea is to use the overload of Window that creates non-overlapping windows using a windowClosingSelector that uses the source time-shifted back by the sampleDuration. Each window will therefore: (a) be closed by the first element in it and (b) remain open until a new element is permitted. We then simply select the first element from each window.
Rx 1.x Version
The Publish extension method used above is not available in Rx 1.x. Here is an alternative:
public static class ObservableExtensions
{
public static IObservable<T> SampleFirst<T>(
this IObservable<T> source,
TimeSpan sampleDuration,
IScheduler scheduler = null)
{
scheduler = scheduler ?? Scheduler.Default;
var sourcePub = source.Publish().RefCount();
return sourcePub.Window(() => sourcePub.Delay(sampleDuration,scheduler))
.SelectMany(x => x.Take(1));
}
}
The solution I found after a lot of trial and error was to replace the throttled subscription with the following:
subject
.Window(() => { return Observable.Interval(timeout); })
.SelectMany(x => x.Take(1))
.Subscribe(i => DoStuff(i));
Edited to incorporate Paul's clean-up.
Awesome solution Andrew! We can take this a step further though and clean up the inner Subscribe:
subject
.Window(() => { return Observable.Interval(timeout); })
.SelectMany(x => x.Take(1))
.Subscribe(DoStuff);
The initial answer I posted has a flaw: namely that the Window method, when used with an Observable.Interval to denote the end of the window, sets up an infinite series of 500ms windows. What I really need is a window that starts when the first result is pumped into the subject, and ends after the 500ms.
My sample data masked this problem because the data broke down nicely into the windows that were already going to be created. (i.e. 0-500ms, 501-1000ms, 1001-1500ms, etc.)
Consider instead this timing:
factory.StartNewDelayed(300,() =>
{
Console.WriteLine("Batch 1 (300ms delay)");
subject.OnNext(1);
});
factory.StartNewDelayed(700, () =>
{
Console.WriteLine("Batch 2 (700ms delay)");
subject.OnNext(2);
});
factory.StartNewDelayed(1300, () =>
{
Console.WriteLine("Batch 3 (1.3s delay)");
subject.OnNext(3);
});
factory.StartNewDelayed(1600, () =>
{
Console.WriteLine("Batch 4 (1.6s delay)");
subject.OnNext(4);
});
What I get is:
Batch 1 (300ms delay)
Handling 1 at 356ms
Batch 2 (700ms delay)
Handling 2 at 750ms
Batch 3 (1.3s delay)
Handling 3 at 1346ms
Batch 4 (1.6s delay)
Handling 4 at 1644ms
This is because the windows begin at 0ms, 500ms, 1000ms, and 1500ms and so each Subject.OnNext fits nicely into its own window.
What I want is:
Batch 1 (300ms delay)
Handling 1 at ~300ms
Batch 2 (700ms delay)
Batch 3 (1.3s delay)
Handling 3 at ~1300ms
Batch 4 (1.6s delay)
After a lot of struggling and an hour banging on it with a co-worker, we arrived at a better solution using pure Rx and a single local variable:
bool isCoolingDown = false;
subject
.Where(_ => !isCoolingDown)
.Subscribe(
i =>
{
DoStuff(i);
isCoolingDown = true;
Observable
.Interval(cooldownInterval)
.Take(1)
.Subscribe(_ => isCoolingDown = false);
});
Our assumption is that calls to the subscription method are synchronized. If they are not, then a simple lock could be introduced.
Use .Scan() !
This is what I use for Throttling when I need the first hit (after a certain period) immediately, but delay (and group/ignore) any subsequent hits.
Basically works like Throttle, but fires immediately if the previous onNext was >= interval ago, otherwise, schedule it at exactly interval from the previous hit. And of course, if within the 'cooling down' period multiple hits come, the additional ones are ignored, just like Throttle does.
The difference with your use case is that if you get an event at 0 ms and 100 ms, they will both be handled (at 0ms and 500ms), which might be what you actually want (otherwise, the accumulator is easy to adapt to ignore ANY hit closer than interval to the previous one).
public static IObservable<T> QuickThrottle<T>(this IObservable<T> src, TimeSpan interval, IScheduler scheduler)
{
return src
.Scan(new ValueAndDueTime<T>(), (prev, id) => AccumulateForQuickThrottle(prev, id, interval, scheduler))
.Where(vd => !vd.Ignore)
.SelectMany(sc => Observable.Timer(sc.DueTime, scheduler).Select(_ => sc.Value));
}
private static ValueAndDueTime<T> AccumulateForQuickThrottle<T>(ValueAndDueTime<T> prev, T value, TimeSpan interval, IScheduler s)
{
var now = s.Now;
// Ignore this completely if there is already a future item scheduled
// but do keep the dueTime for accumulation!
if (prev.DueTime > now) return new ValueAndDueTime<T> { DueTime = prev.DueTime, Ignore = true };
// Schedule this item at at least interval from the previous
var min = prev.DueTime + interval;
var nextTime = (now < min) ? min : now;
return new ValueAndDueTime<T> { DueTime = nextTime, Value = value };
}
private class ValueAndDueTime<T>
{
public DateTimeOffset DueTime;
public T Value;
public bool Ignore;
}
I got another one for your. This one doesn't use Repeat() nor Interval() so it might be what you are after:
subject
.Window(() => Observable.Timer(TimeSpan.FromMilliseconds(500)))
.SelectMany(x => x.Take(1));
Well the most obvious thing will be to use Repeat() here. However, as far as I know Repeat() might introduce problems so that notifications disappear in between the moment when the stream stops and we subscribe again. In practice this has never been a problem for me.
subject
.Take(1)
.Concat(Observable.Empty<long>().Delay(TimeSpan.FromMilliseconds(500)))
.Repeat();
Remember to replace with the actual type of your source.
UPDATE:
Updated query to use Concat instead of Merge
I have stumbled upon this question while trying to re-implement my own solution to the same or similar problem using .Window
Take a look, it seems to be the same as this one and solved quite elegantly:
https://stackoverflow.com/a/3224723/58463
It's an old post, but no answer could really fill my needs, so I'm giving my own solution :
public static IObservable<T> ThrottleOrImmediate<T>(this IObservable<T> source, TimeSpan delay, IScheduler scheduler)
{
return Observable.Create<T>((obs, token) =>
{
// Next item cannot be send before that time
DateTime nextItemTime = default;
return Task.FromResult(source.Subscribe(async item =>
{
var currentTime = DateTime.Now;
// If we already reach the next item time
if (currentTime - nextItemTime >= TimeSpan.Zero)
{
// Following item will be send only after the set delay
nextItemTime = currentTime + delay;
// send current item with scheduler
scheduler.Schedule(() => obs.OnNext(item));
}
// There is still time before we can send an item
else
{
// we schedule the time for the following item
nextItemTime = currentTime + delay;
try
{
await Task.Delay(delay, token);
}
catch (TaskCanceledException)
{
return;
}
// If next item schedule was change by another item then we stop here
if (nextItemTime > currentTime + delay)
return;
else
{
// Set next possible time for an item and send item with scheduler
nextItemTime = currentTime + delay;
scheduler.Schedule(() => obs.OnNext(item));
}
}
}));
});
}
First item is immediately sent, then following items are throttled. Then if a following item is sent after the delayed time, it's immediately sent too.
I have an API limit of 10 calls per second (however thousands per day), however, when I run this function (Called each Style ID of object, > 10 per second):
getStyleByID(styleID: number): void {
this._EdmundsAPIService.getStyleByID(styleID).subscribe(
style => {this.style.push(style); },
error => this.errorMessage = <any>error);
}
from this function (only 1 call, used onInit):
getStylesWithoutYear(): void {
this._EdmundsAPIService.getStylesWithoutYear(this.makeNiceName, this.modelNiceName, this.modelCategory)
.subscribe(
styles => { this.styles = styles;
this.styles.years.forEach(year =>
year.styles.forEach(style =>
this.getStyleByID(style.id)));
console.log(this.styles); },
error => this.errorMessage = <any>error);
}
It makes > 10 calls a second. How can I throttle or slow down these calls in order to prevent from getting a 403 error?
I have a pretty neat solution where you combine two observables with the .zip() operator:
An observable emitting the requests.
Another observable emitting a value every .1 second.
You end up with one observable emitting requests every .1 second (= 10 requests per second).
Here's the code (JSBin):
// Stream of style ids you need to request (this will be throttled).
const styleIdsObs = new Rx.Subject<number>();
// Getting a style means pushing a new styleId to the stream of style ids.
const getStyleByID = (id) => styleIdsObs.next(id);
// This second observable will act as the "throttler".
// It emits one value every .1 second, so 10 values per second.
const intervalObs = Rx.Observable.interval(100);
Rx.Observable
// Combine the 2 observables. The obs now emits a styleId every .1s.
.zip(styleIdsObs, intervalObs, (styleId, i) => styleId)
// Get the style, i.e. run the request.
.mergeMap(styleId => this._EdmundsAPIService.getStyleByID(styleId))
// Use the style.
.subscribe(style => {
console.log(style);
this.style.push(style);
});
// Launch of bunch of requests at once, they'll be throttled automatically.
for (let i=0; i<20; i++) {
getStyleByID(i);
}
Hopefully you'll be able to translate my code to your own use case. Let me know if you have any questions.
UPDATE: Thanks to Adam, there's also a JSBin showing how to throttle the requests if they don't come in consistently (see convo in the comments). It uses the concatMap() operator instead of the zip() operator.
You could use a timed Observable that triggers every n milliseconds. I didn't adapt your code but this one shows how it would work:
someMethod() {
// flatten your styles into an array:
let stylesArray = ["style1", "style2", "style3"];
// create a scheduled Observable that triggers each second
let source = Observable.timer(1000,1000);
// use a counter to track when all styles are processed
let counter = 0;
let subscription = source.subscribe( x => {
if (counter < stylesArray.length) {
// call your API here
counter++;
} else {
subscription.complete();
}
});
}
Find here a plunk that shows it in action
While I didn't test this code, I would do try something along these lines.
Basically I create a variable that keeps track of when the next request is allowed to be made. If that time has not passed, and a new request comes in, it will use setTimeout to allow that function to run at the appropriate time interval. If the delayUntil value is in the past, then the request can run immediately, and also push back the timer by 100 ms from the current time.
delayUntil = Date.now();
getStylesWithoutYear(): void {
this.delayRequest(() => {
this._EdmundsAPIService.getStylesWithoutYear(this.makeNiceName, this.modelNiceName, this.modelCategory)
.subscribe(
styles => { this.styles = styles;
this.styles.years.forEach(year =>
year.styles.forEach(style =>
this.getStyleByID(style.id)));
console.log(this.styles); },
error => this.errorMessage = <any>error);
};
}
delayRequest(delayedFunction) {
if (this.delayUntil > Date.now()) {
setTimeout(delayedFunction, this.delayUntil - Date.now());
this.delayUntil += 100;
} else {
delayedFunction();
this.delayUntil = Date.now() + 100;
}
}
i have a simple finagle thrift server:
import com.twitter.finagle.Thrift
import scala.concurrent.Future
import com.twitter.util.{ Await, Future }
object Main{
def main(args: Array[String]) {
var count = 0
val myserver = Thrift.serveIface("0.0.0.0:9090", new RealTimeDatabasePageImpressions[com.twitter.util.Future] {
def saveOrUpdate(pageImpression: PageImpressions):
com.twitter.util.Future[Boolean] = {
count += 1
println(count)
com.twitter.util.Future.value(true)
}
}
Await.ready(myserver)
}
}
This server works but i have one big problem: i wrote a thrift nodejs client with a for loop. It executes 10.000 thrift request. But it's not asynchronous. It executes 500 request and stops. After a while, 2 or 3 seconds, 300 more requests will executed. Now the question: Why this happen? Is something wrong with my server or client? I use only the apache thrift generated nodejs code. No wrapper. The function executed 10.000 times. I think the nodejs isn't the problem:
function callFunc(i){
console.log("started executing: " + i);
var connection = thrift.createConnection("IP", 9090, {
transport: transport,
protocol: protocol
});
connection.on('error', function (err) {
console.log(err);
});
// Create a Calculator client with the connection
var client = thrift.createClient(Realtime_pageImpressions, connection);
var rand = Math.random() * (20000 - 1);
var trackId = trackIds[Math.round(Math.random() * 10)];
var values = new PageImpressions({
trackId: trackId,
day: 4,
hour: 4,
minute: 13,
pageId: 'blabla',
uniqueImpressions: Math.random() * (13000 - 1),
sumImpressions: Math.random() * (1000450 - 1)
});
client.saveOrUpdate(values, function (error, message) {
if (message) {
console.log("Successful, got Message: " + message);
} else {
console.log("Error with Message: " + error);
}
});
return true;
}
for(var i = 0; i < 10000; i++){
callFunc(i);
}
Your var count is unsynchronized. This is a very big problem, but, probably, not related to your performance issue.
You are also blocking finagle thread, which is also a big problem, but does not matter in your mock case, because there is no wait time.
Think about it this way. Let's say, you have one cpu (you probably have several, but there are other things going on the machine as well), and you are asking it to execute 10000 operations all at the same time.
How can this work? It will have to execute one of the requests, save the context, the stack, flush all caches, switch to the next request, execute that one ...
500 requests in 2 seconds is 4 milliseconds per request. Does not sound that bad, does it?
Also, have you turned your GC (on both server and client)? If requests are processed in bursts followed by long pauses, that's probably a sign of full GC kicking in
I have a node-apn nodejs script running as a daemon on AmazonWS. The daemon runs fine and the script stays up and comes back when it goes down but I believe I am having a synchronous execution and exiting issue with node.js. When I release the process with process.exit(); even though all console.logs output saying they have sent my messages, they never are received on the phone. I decided to remove the exit and let the process "hang" after execution and all messages were sent successfully. This led me to do the following implementation using an ASYNC function, but the same result seems to be happening. Can anyone provide insight to this? There are no errors being thrown from APN or anywhere else.
function closeDB()
{
connection.end(function(err) {
if (err) {
console.log("ERROR: " + util.inspect(err, false, 5));
process.exit(1);
}
console.log("APNS-PUSH: COMPLETED.");
});
setTimeout(function(){process.exit();}, 50);
} // End of closeDB()
function apnsError(err, notification)
{
console.log(err);
console.log(notification);
closeDB();
}
function async(arg, callback)
{
apnsConnection.sendNotification(arg);
console.log(arg);
setTimeout(function() { callback(1); }, 100);
}
/**
* Our MySQL query callback.
*/
function queryCB(err, results)
{
//error in our all, report and exit
if (err) {
console.log("ERROR: " + util.inspect(err, false, 5));
closeDB();
}
if(results.length == 0)
{
closeDB();
}
var notes = [];
var count = 0;
try {
for( var i = 0; i < results.length; i++ ) {
var myDevice = new apns.Device(results[i]['udid']);
var note = new apns.Notification();
note.expiry = Math.floor(Date.now() / 1000) + 3600; // Expires 1 hour from now.
note.badge = results[i]["notification_count"];
note.sound = "ping.aiff";
note.alert = results[i]["message"];
note.device = myDevice;
connection.query('UPDATE `tbl_notifications` SET `sent`=1 WHERE `id`=' + results[i]["id"] , function(err, results) {
if(err)
{
console.log("ERROR: " + util.inspect(err, false, 5));
}
});
notes.push(note);
}
} catch( err ) {
console.log('error: ' + err)
}
console.log(notes.length);
notes.forEach(function(nNode) {
async(nNode, function(result) {
count++;
if(count == notes.length) {
closeDB();
}
})
});
} // End of queryCB()
I had the same problem where killing the process also killed the open socket connections and didn't allow the notifications to be sent. The solution I came up with isn't an an ideal solution but it will work in your situation as well. I looked into the node-apn code and found that the Connection object inherited from EventEmitter so you can monitor events on the object like so:
var apnsConnection = new apn.Connection(options)
apnsConnection.sendNotification(notification)
apnsConnection.on('transmitted', function(){
console.log("Transmitted")
callback()
})
apnsConnection.on('error', function(){
console.log("Error")
callback()
})
This is monitoring the socket that the notification is sent through so I don't know how accurate it is at determining when a notification has successfully been passed off to Apple's APNS servers but it has worked pretty well for me.
The reason you are seeing this problem is that when you use #pushNotification it buffers the notification inside the module and handles sending it asynchronously.
Listening for "transmitted" is valid and this is emitted when the notification has been written to the socket. However, if your objective is to close the socket after all notifications have been sent then the easiest way to accomplish this is using the connectionTimeout property when creating your connection.
Simply set connectionTimeout to something around 1000 (milliseconds) and assuming you have no other connections open then the process will exit automatically. Or you can set an event listener on the timeout event and call process.exit() from there.