I use RequestQueue like so:
const requestQueue = await RequestQueue.open();
requestQueue.timeoutSecs = 60;
await requestQueue.addRequest(...);
but when running scraper I still see the default timeout:
WARN CheerioCrawler: Reclaiming failed request back to the list or queue. request timed out after 30 seconds.
If you do not specify higher level timeout in crawler settings then its a bug, please add issue at github for https://crawlee.dev/api/core/class/RequestQueue#timeoutSecs
Related
When multiple devs use my API, multiple concurrent requests are being sent to Mongoose.
When the concurrency is high, the connection just "dies" and refuses to fulfil any new request, no matter how long I wait (hours!).
I just want to state everything is working fine on regular use. Heavy use leads the connection to crash.
My MongooseModule initialization:
MongooseModule.forRoot(DatabasesService.MONGO_FULL_URL, {
useNewUrlParser: true,
useUnifiedTopology: true,
useFindAndModify: false,
autoEncryption: {
keyVaultNamespace: DatabasesService.keyVaultNamespace,
kmsProviders: DatabasesService.kmsProviders,
extraOptions: {
mongocryptdSpawnArgs: ['--pidfilepath', '/tmp/mongocryptd.pid']
}
} as any
})
Module that imports the feature:
#Module({
imports: [MongooseModule.forFeature([{ name: 'modelName', schema: ModelNameSchema }])],
providers: [ModelNameService],
controllers: [...],
exports: [...]
})
Service:
#Injectable()
export class ModelNameService {
constructor(
#InjectModel('modelName') private modelName: Model<IModelName>
) {}
async findAll(): Promise<IModelName[]> {
const result: IModelName[] = await this.modelName.find().exec();
if (!result) throw new BadRequestException(`No result was found.`);
return result;
}
}
I've tried loadtesting using different utils, the easiest was:
ab -c 200 -n 300 -H "Authorization: Bearer $TOKEN" -m GET -b 0 https://example.com/getModelName
Any new request after the connection hangs gets stuck at ModelNameService.findAll() first line (the request to mongo).
On mongodb logs with verbosity of "-vvvvv" I can see few suspicious lines:
User Assertion: Unauthorized: command endSessions requires authentication src/mongo/db/commands.cpp
Cancelling outstanding I/O operations on connection to 127.0.0.1:33134
And I've also found that it doesn't exceed 12 open connections at the same time. It always waits to close one before opening a new one.
Other key points:
Mongoose doesn't return any value or notifies about any error. It just hangs without notifying anything.
Terminus health check able to ping the DB and returns a healthy status.
NestJS API still works - I'm able to send new requests and receive a response. Just requests that are related to the faulty connection hang.
When I inject connection and check its readyState it returns connected.
Restarting the API fixes it immediately.
MongoDB itself keeps working as normal.
Increasing Mongoose poolSize is able to handle more requests at the same time but will still crash on a larger amount of requests.
My main question here is how do I handle this case? Currently, I've added another health check to try and send a query to the problematic connection every half a minute, and k8s restarts the pod if it determines a failure. This works but it is not optimal.
I have some dns records I'm trying to import with pulumi, and they fail somewhat bluntly with this error:
Diagnostics:
gcp:dns:RecordSet (root/my.domain./NS):
error: Preview failed: refreshing urn:pulumi:root::root::gcp:dns/recordSet:RecordSet::root/my.domain./NS: Error when reading or editing DNS Record Set "my.domain.": Get "https://www.googleapis.com/dns/v1beta2/projects/root-280012/managedZones/root/rrsets?alt=json&name=my.domain.&prettyPrint=false&type=NS": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
pulumi:pulumi:Stack (root-root):
error: preview failed
I'm just getting started with pulumi, so I have no real sense of whether this is a GCP-specific problem or more general with pulumi, so apologies if this is in the wrong place.
Is this just a case of increasing a timeout limit? Is this a problem with the cli? Why would this particular request timeout? (It times out every attempt)
Appreciate any advice!
I solved this by using customTimeouts: https://www.pulumi.com/docs/intro/concepts/programming-model/#customtimeouts
When creating your resource, you can pass an options object as the last parameter. In that, you can add your customTimeouts configuration e.g. (for typescript):
{
customTimeouts: {
create: '5m',
delete: '5m',
update: '5m'
}
})
I've got an ECS/Fargate task that runs every five minutes. Is there a way to tell it to not run if the prior instance is still working? At the moment I'm just passing it a cron expression, and there's nothing in the cron/rate aws doc about blocking subsequent runs.
Conseptually I'm looking for something similar to Spring's #Scheduled(fixedDelay=xxx) where it'll run every five minutes after it finishes.
EDIT - I've created the task using cloudformation, not the cli
This solution works if you are using Cloudwatch Logging for your ECS application
- Have your script emit a 'task completed' or 'script successfully completed running' message so you can track it later on.
Using the describeLogStreams function, first retrieve the latest log stream. This will be the stream that was created for the task which ran 5 minutes ago in your case.
Once you have the name of the stream, check the last few logged events (text printed in the stream) to see if it's the expected task completed event that your stream should have printed. Use the getLogEvents function for this.
If it isn't, don't launch the next task and invoke a wait or handle as needed
Schedule your script to run every 5 minutes as you would normally.
API links to aws-sdk docs are below. This script is written in JS and uses the AWS-SDK (https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS.html) but you can use boto3 for python or a different lib for other languages
API ref for describeLogStreams
API ref for getLogEvents
const logGroupName = 'logGroupName';
this.cloudwatchlogs = new AwsSdk.CloudWatchLogs({
apiVersion: '2014-03-28', region: 'us-east-1'
});
// Get the latest log stream for your task's log group.
// Limit results to 1 to get only one stream back.
var descLogStreamsParams = {
logGroupName: logGroupName,
descending: true,
limit: 1,
orderBy: 'LastEventTime'
};
this.cloudwatchlogs.describeLogStreams(descLogStreamsParams, (err, data) => {
// Log Stream for the previous task run..
const latestLogStreamName = data.logStreams[0].logStreamName;
// Call getLogEvents to read from this log stream now..
const getEventsParams = {
logGroupName: logGroupName,
logStreamName: latestLogStreamName,
};
this.cloudwatchlogs.getLogEvents(params, (err, data) => {
const latestParsedMessage = JSON.parse(data.events[0].message);
// Loop over this to get last n messages
// ...
});
});
If you are launching the task with the CLI, the run-task command will return you the task-arn.
You can then use this to check the status of that task:
aws ecs describe-tasks --cluster MYCLUSTER --tasks TASK-ARN --query 'tasks[0].lastStatus'
It will return RUNNING if it's still running, STOPPED if stopped, etc.
Note that Fargate is very aggressive about harvesting stopped tasks. If that command returns null, you can consider it STOPPED.
I am using ejabberd server version 18.12.1 with stream management enabled. When the user disconnects from the internet, its presence remains online so I decided to use mod_ping to kill the connection after a timeout using mod ping
I used the following config in ejabberd.yml file :
mod_ping:
send_pings: true
ping_ack_timeout: 32
timeout_action: kill
considering the default value of ping_interval : 60.
Ping does not seem to be working with this configuration. Am I missing any other configuration ? should the client enable something to make this working ? is there any ping log that I can check?
Note: using the modules page of the web admin of ejabberd server, the config value of the ping_ack_timeout of mod_ping seems to be different from the one in the ejabberd.yml file, why is that?
[{ping_interval,60},
{ping_ack_timeout,32000},
{send_pings,true},
{timeout_action,kill}]
Note: using the modules page of the web admin of ejabberd server, the config value of the ping_ack_timeout of mod_ping seems to be different from the one in the ejabberd.yml file, why is that?
That is expected: you set the human-configurable option in seconds, and later the internal time value is expressed in milliseconds (the time unit used by erlang).
Am I missing any other configuration ? should the client enable something to make this working ? is there any ping log that I can check?
That should be enough. Try with other clients, just to check if that affects in any way. I've installed ejabberd 18.12, configured like this:
loglevel: 5
...
mod_ping:
send_pings: true
ping_interval: 10
ping_ack_timeout: 15
timeout_action: kill
Then I start ejabberd and login with Tkaber client (but I think any client is good for testing ping). Every ten seconds, the client receives this query:
<iq to='user1#localhost/tka1'
from='user1#localhost'
type='get'
id='rr-1552642185584-13814872912241253802-5xOvCCobbU2TCC/RT4GaqD6M8bo=-55238004'>
<ping xmlns='urn:xmpp:ping'/>
</iq>
And at the same time, the ejabberd log file shows several messages, starting with this one:
10:29:30.585 [debug] route:
#iq{id = <<"rr-1552642185584-13814872912241253802-5xOvCCobbU2TCC/RT4GaqD6M8bo=-55238004">>,
type = get,lang = <<>>,
from = #jid{user = <<"user1">>,server = <<"localhost">>,resource = <<>>,
luser = <<"user1">>,lserver = <<"localhost">>,
lresource = <<>>},
to = #jid{user = <<"user1">>,server = <<"localhost">>,
resource = <<"tka1">>,luser = <<"user1">>,
lserver = <<"localhost">>,lresource = <<"tka1">>},
sub_els = [#ping{}],
meta = #{}}
I'm testing an akka http service on AWS ECS. Each instance is added to a load balancer which regularly makes requests to a health check route. Since this is a test environment I can control for no other traffic going to the server. I notice the debug log indicating that the "default dispatcher" number is consistently increasing:
[DEBUG] [01/03/2017 22:33:03.007] [default-akka.actor.default-dispatcher-41200] [akka://default/system/IO-TCP/selectors/$a/0] New connection accepted
[DEBUG] [01/03/2017 22:33:29.142] [default-akka.actor.default-dispatcher-41196] [akka://default/system/IO-TCP/selectors/$a/0] New connection accepted
[DEBUG] [01/03/2017 22:33:33.035] [default-akka.actor.default-dispatcher-41204] [akka://default/system/IO-TCP/selectors/$a/0] New connection accepted
[DEBUG] [01/03/2017 22:33:59.174] [default-akka.actor.default-dispatcher-41187] [akka://default/system/IO-TCP/selectors/$a/0] New connection accepted
[DEBUG] [01/03/2017 22:34:03.066] [default-akka.actor.default-dispatcher-41186] [akka://default/system/IO-TCP/selectors/$a/0] New connection accepted
[DEBUG] [01/03/2017 22:34:29.204] [default-akka.actor.default-dispatcher-41179] [akka://default/system/IO-TCP/selectors/$a/0] New connection accepted
[DEBUG] [01/03/2017 22:34:33.097] [default-akka.actor.default-dispatcher-41210] [akka://default/system/IO-TCP/selectors/$a/0] New connection accepted
This trend is never reversed and will get up into the tens of thousands pretty soon. Is this normal behavior or indicative of an issue?
Edit: I've updated the log snippet to show that the dispatcher thread number goes way beyond what I would expect.
Edit #2: Here is the health check route code:
class HealthCheckRoutes()(implicit executionContext: ExecutionContext)
extends LogHelper {
val routes = pathPrefix("health-check") {
pathEndOrSingleSlash {
complete(OK -> "Ok")
}
}
}
Probably, yes. I think that's the thread name.
If you do a thread dump on the server, does it have a great many open threads?
It looks like your server is leaking a thread per connection.
(It will probably be much easier to debug and diagnose this on your development machine, rather than on the EC2 VM. Try to reproduce it locally.)
For you Question, check this comment:
Akka http server dispatcher number constantly increasing
About dispatcher:
It is no problem to use default dispatcher for operations like health check.
Threads are controlled by the dispatcher you specified, or default-dispatcher if not specified.
default-dispatcher is setting as following, which means the thread pool size is between 8 to 64 or equal to (number of processors * 3).
default-dispatcher {
type = "Dispatcher"
executor = "default-executor"
default-executor {
fallback = "fork-join-executor"
}
fork-join-executor {
# Min number of threads to cap factor-based parallelism number to
parallelism-min = 8
# The parallelism factor is used to determine thread pool size using the
# following formula: ceil(available processors * factor). Resulting size
# is then bounded by the parallelism-min and parallelism-max values.
parallelism-factor = 3.0
# Max number of threads to cap factor-based parallelism number to
parallelism-max = 64
# Setting to "FIFO" to use queue like peeking mode which "poll" or "LIFO" to use stack
# like peeking mode which "pop".
task-peeking-mode = "FIFO"
}
Dispathcer Document:
http://doc.akka.io/docs/akka/2.4.16/scala/dispatchers.html
Configuration reference:
http://doc.akka.io/docs/akka/2.4.16/general/configuration.html#akka-actor
BTW for operations take a long time and blocks other operations, here is how to specify a custom dispatcher in Akka HTTP for them:
http://doc.akka.io/docs/akka-http/current/scala/http/handling-blocking-operations-in-akka-http-routes.html
According to this akka-http github issue there doesn't seem to be a problem: https://github.com/akka/akka-http/issues/722