i have a problem with my rundeck, in automatisation process i need to perform some long task with Rundeck such as Database Refresh from PRODUCTION to TEST Database.
This case cause problems sometimes.
I've no timeout set on rundeck job, i use script that launch action on a distant server trough SSH.
If a step took long time (Drop database, import database with datapump), Rundeck generate an error
Failed: NonZeroResultCode: Remote command failed with exit status -1
after a certain period of time.
It's look like a timeout, but i don't know where this timeout occured and i can't find it.
But the DROP or the IMPDB still continue his job and if you wait for it, the action has successfully completed but rundeck marked the step as failed, i put some workaround such as error handler to by pass but it's not very clean.
Do you have any idea were that timeout is set ?
In my opininon, after a certain period of time, if rundeck did not receive an answer from the distant server, it cut off and mark as failed with the error code Failed: NonZeroResultCode: Remote command failed with exit status -1
You can increase the ssh timeout globally at the framework.properties file.
Stop the Rundeck server, then add the following lines on the /etc/rundeck/framework.properties file:
framework.ssh-connection-timeout=0
framework.ssh-command-timeout=0
Then start the Rundeck service.
The timeout values are in milliseconds, 0 is indefinite.
More info here.
Related
I need to perform a load test against a pgbouncer. All the clients perform an SSH Tunnel before enstablishing a database connection to the database (through the pgbouncer). It's something like this:
sshpass -p 'MY_PSW' ssh -o StrictHostKeyChecking=no -N -L LOCAL_PORT:127.0.0.1:63666 PGBOUNCER_USER#PGBOUNCER_ADDRESS -p PORT >/dev/null 2>&1 &
My Jmeter project has three thread Groups at the moment:
SetUp Thread Group: In which I make a connection to a different database to select a random username and schema
Query Thread Group: In which I perform the JDBC connection using the previous user (which became a property using props.put("schema", vars.get("schema_1")); into the BeanShell Assertion) and the queries
TearDown Thread Group: In which I close the ssh Tunnel.
Now every first time I run the test from the GUI, the data select (JDBC request) into the Query Thread Group gives me an error:
Cannot create PoolableConnectionFactory (FATAL: "trust" authentication failed)
After that, if I run the test again, everything works. I checked the content of the variables and properties with a Debug sampler and everything is correct.
The main problem starts when I run the test without the GUI. It always fails because of that error.
I actually don't like the fact that I have to enstablish the SSH tunnel running the command with an OS Process Sampler, but I can't find any better solution. The SSH tunnel is a part of the test, I don't need that for the master/slave configuration of JMeter.
I would appreciate a lot for a solution or a suggestion to make this work. Thanks.
If you're using the command in the OS Process Sampler it's being run in the background therefore my expectation is that the OS Process Sampler returns the SampleResult immediately and the tunnel is not up yet.
Then when the "Query Thread Group" starts as per JMeter Test Elements execution order JDBC Connection Configuration tries to establish the connection using local port which is not fully established. The fact that the issue is reproducible in non-GUI mode might be the confirmation for my guess as JMeter works much faster in non-GUI as it doesn't need to waste time and resources for GUI refreshing and propagating sample results to listeners.
My expectation is that if you add i.e. Flow Control Action sampler to the setUp Thread Group and configure it to "sleep" for a couple of seconds it should resolve your issue. If it doesn't - try increasing JMeter logging verbosity for the JDBC Test Elements by adding the next line to log4j2.xml file
<Logger name="org.apache.jmeter.protocol.jdbc" level="debug" />
and compare the entries for "successful" and "failed" executions in the jmeter.log file.
Ok, I managed to find the solution. Thank to Dmitri T Answer, I could track the problem, which was about the property not being properly set.
I was using the BeanShell Assertion to set the property after the result of the JDBC request in the setUp Thread Group. Apparently, the BeanShell Assertion is executed at the end of the entire run, so the property where being set at the end. During the first execution, the property is empty.
I used the Beanshell Sampler instead and now it works.
Rundeck job returns "curl: (52) Empty reply from server" if the jobs takes 2 minutes or more.
Also part of the error "SSH command execution error: NonZeroResultCode: Remote command failed with exit status 52"
We have a rundeck job (curl cmd) that executes on a Linux 7 server that for the majority of the time returns a "success" for the completion of the job.
However sometimes it will return "failed" if the job runs 2 minutes or more.
A value for "5m" has been added to the job "Timeout" on the configuration.
The job actually is successful but rundeck still returns the "failed" status.
When comparing the successful against failed logs I notice some extra steps in the failed log which pushes it past the 2 minutes time.
Any ideas why the failed status as mentioned above?
Thanks,
Ken
It's a problem with your curl command against the web server, usually happens when you call using HTTP against an HTTPS based service. take a look at this. Another possibility is a network problem.
I'm working with ADF and Azure Managed Postgres. I've had a reoccurring issue with look-ups and query-sourced copy activities timing out after about 35 seconds.
Failure happened on 'Source' side. 'Type=Npgsql.NpgsqlException,Message=Exception while reading from stream,Source=Npgsql,''Type=System.IO.IOException,Message=Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.,Source=System,''Type=System.Net.Sockets.SocketException,Message=A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond,Source=System,'
So the error says it's a Npgsql Exception, so I took a look at their documentation and modified the connection string to take Timeout = 60 and CommandTimeout = 60 as well (Internal Timeout will default to CommandTimeout).
And the queries still timeout at ~35 seconds. Could this be a socket issue with the Azure Managed Instance causing the timeout and it's just propagating down to npgsql?
Any help would be appreciated!
I just want to add some precision because I had the same problem (and thanks #DeliciousMalware and #Leon_Yue):
There is a default timeout of 30s for requests with a postgres connection
There is no way to change this timeout from the lookup activity directly.
The only option that does something is to add Timeout=600;CommandTimeout=0; to your connection string in your linked service (if you use a key vault for exemple) or add the options in the linked service additionnal parameters like in #DeliciousMalware screenshot.
Timeout is to establish the connection, and CommandTimeout is the timeout for the command itself (in second, 0 means infinity)
The library behind the connection is npgsql, and the others parameters and details that are usable are there: https://www.npgsql.org/doc/connection-string-parameters.html
I had a hard time to find what the parameters of the connection string are and what they mean, and which one exists, so I was really happy to find this doc. I didn't found a lot of doc on postgres in azure, so I though this list of param would be of some use for others.
I added the 2 parameters suggested by Leon and that resolved the issue I had.
Here is a screenshot of the parameters being added to the linked service:
Here is a screenshot of the error and completed run:
Here is a screenshot of the error and completed run:
When halting service in Process Admin console - using Halt service button - I get following message after a while:
Halting of the Service timed out. Most probably, the Process Server failed to halt the Service within short time period.
From what I can see, it happens only when said service has few (say in tenths)steps, when it has more (hundreds) steps, it works well.
Can somebody see cause of this and tell me what to do? Thanks.
If we fail to halt a hung or long running (infinite loop) service within a short duration, process admin console does not allow to halt it.
This is the default behavior of how halt will work. Only way to restart the server to reinstate the BPM env.
I'd build an application in VB.net which contains some background procedures to be done every day. The exe was scheduled in SQL Server 2008 R2.
I'm getting the following error from the system. Whereas when the same scheduler is run manually it runs successfully. There are no multiple processes running during the scheduler process. Can you suggest why it happens?
Error:
Connection Timeout Expired.
The timeout period elapsed during the post-login phase.
The connection could have timed out while waiting for server to complete the login process and respond; Or it could have timed out while attempting to create multiple active connections.
The duration spent while attempting to connect to this server was -
[Pre-Login] initialization=2; handshake=5; [Login] initialization=0; authentication=0; [Post-Login] complete=14001;
Thanks in advance.