Environment
Spring Boot 1.5.13.RELEASE
Spring Cloud Edgware.SR3
Compiled with Java version "1.8.0_172-ea",Java(TM) SE Runtime Environment (build 1.8.0_172-ea-b03) and source level 1.8
Runtime JRE: in docker with openjdk:10.0.1-jre-slim
Question
I have a ribbon client called serviceA and associated
serviceA.ribbon.ConnectTimeout=5000
serviceA.ribbon.ReadTimeout=15000
hystrix.command.serviceA.execution.isolation.thread.timeoutInMilliseconds = 20000
I have not (knowingly) got spring-retry on the classpath. I execute ./mvnw dependency:list | grep -i retry and get no results.
At runtime I get these warnings:
The Hystrix timeout of 20000ms for the command serviceA is set lower than the combination of the Ribbon read and connect timeout, 40000ms.
I'm not sure where these numbers come from given that I thought I'd set them to 15 and 5 seconds respectively. Why is this figure double?
Actually, ribbon timeout includes all same server retry and next server retry.
ribbonTimeout = (ribbon.ConnectTimeout + ribbon.ReadTimeout) * (ribbon.MaxAutoRetries + 1) * (ribbon.MaxAutoRetriesNextServer + 1);
// ...
if(hystrixTimeout < ribbonTimeout) {
LOGGER.warn("The Hystrix timeout of " + hystrixTimeout + "ms for the command " + commandKey +
" is set lower than the combination of the Ribbon read and connect timeout, " + ribbonTimeout + "ms.");
}
In your configuration:
ribbon.connectionTimeout is 5000
ribbon.readTimeout is 15000
ribbon.maxAutoRetries is 0 (default)
ribbon.maxAutoRetriesNextServer is 1 (default)
So the hystrixTimeout should be:
(5000 + 15000) * (1 + 0) * (1 + 1) // -> 40000 ms
If you choose to not configure Hystrix timeout, the default Hystrix timeout will be 40000ms.
19.13 Zuul Timeouts in Spring Cloud Document
Related
In Spring Integration, I want to disable a poller by setting the autoStartup=false on the InboundChannelAdapter. But with the following setup, none of my pollers are firing on either my Tomcat instance 1 nor Tomcat instance 2. I have two Tomcat instances with the same code deployed. I want the pollers to be disabled on one of the instances since I do not want the same job polling on the two Tomcat instances concurrently.
Here is the InboundChannelAdapter:
#Bean
#InboundChannelAdapter(value = "irsDataPrepJobInputWeekdayChannel", poller = #Poller(cron="${batch.job.schedule.cron.weekdays.irsDataPrepJobRunner}", maxMessagesPerPoll="1" ), autoStartup = "${batch.job.schedule.cron.weekdays.irsDataPrepJobRunner.autoStartup}")
public MessageSource<JobLaunchRequest> pollIrsDataPrepWeekdayJob() {
return () -> new GenericMessage<>(requestIrsDataPrepWeekdayJob());
}
The property files are as follows. Property file for Tomcat instance 1:
# I wish for this job to run on Tomcat instance 1
batch.job.schedule.cron.riStateAgencyTransmissionJobRunner=0 50 14 * * *
# since autoStartup defaults to true, I do not provide:
#batch.job.schedule.cron.riStateAgencyTransmissionJobRunner.autoStartup=true
# I do NOT wish for this job to run on Tomcat instance 1
batch.job.schedule.cron.weekdays.irsDataPrepJobRunner.autoStartup=false
# need to supply as poller has a cron placeholder
batch.job.schedule.cron.weekdays.irsDataPrepJobRunner=0 0/7 * * * 1-5
Property file for Tomcat instance 2:
# I wish for this job to run on Tomcat instance 2
batch.job.schedule.cron.weekdays.irsDataPrepJobRunner=0 0/7 * * * 1-5
# since autoStartup defaults to true, I do not provide:
#batch.job.schedule.cron.weekdays.irsDataPrepJobRunner.autoStartup=true
# I do NOT wish for this job to run on Tomcat instance 2
batch.job.schedule.cron.riStateAgencyTransmissionJobRunner.autoStartup=false
# need to supply as poller has a cron placeholder
batch.job.schedule.cron.riStateAgencyTransmissionJobRunner=0 50 14 * * *
The properties files are passed as a VM option, e.g. "-Druntime.scheduler=dev1". I cannot disable the poller on one of the JVMs using "-" as the cron expression -- something similar to the ask here: Poller annotation with cron expression should support a special disable character
My goal of being able to call the job manually from either Tomcat instance 1 or Tomcat instance 2 is working. My problem with the setup mentioned above, is that none of the pollers are firing as per their cron expression.
Consider to investigate a leader election pattern: https://docs.spring.io/spring-integration/docs/current/reference/html/messaging-endpoints.html#leadership-event-handling.
This way you have those endpoints in non-started state by default and in the same role. The election is going to chose a leader and start only this one.
Of course there has to be some shared external service to control leadership.
I am running airlfow local executor with a postgres database and I am getting a: (psycopg2.OperationalError) FATAL: remaining connection slots are reserved
My configs:
sql_alchemy_pool_size = 5
sql_alchemy_pool_recycle = 1800
sql_alchemy_reconnect_timeout = 300
Still I can see that airflow always keeps more connections open than the pool size.
How can we limit Airflow to actually use the pool limit?
airflow_version = 1.10.2; postgres_version = 9.6
You have forks of the main process as workers, each of which manage their own thread pool.
Check the implementation of LocalExecutor; as it is using multiprocessing under the hood. SqlAlchemy will close any open connections upon forking the LocalWorker; but the pool size will be equivalent to the parent, so at max, theoretically you'd have k * (n + 1) connections, where n is your parallelism constant and k is your sql_alchemy_pool_size.
People having the same problem with celery executor can try reducing scheduler.max_threads (AIRFLOW__SCHEDULER__MAX_THREADS) .
Theoretically Max no connections from scheduler = (sql_alchemy_pool_size + sql_alchemy_max_overflow) x max_threads + 1[Dag processor Manager] + 1[Main scheduler process]
Check out https://stackoverflow.com/a/57411598/5860608 for further info.
current setting of a pgbouncer server is the following - and what I don't understand is the 'free_servers' info given by the show lists command when connecting to pgbouncer. Is it a (soft or hard) limit on the number of connexion to the postgresql databases used with this instance of pgbouncer ?
configuration :
max_client_conn = 2048
default_pool_size = 1024
min_pool_size = 10
reserve_pool_size = 500
reserve_pool_timeout = 1
server_idle_timeout = 600
listen_backlog = 1024
show lists gives :
pgbouncer=# show lists ;
list | items
---------------+--------
databases | 6
pools | 3
free_clients | 185
used_clients | 15
free_servers | 70
used_servers | 30
it seems that there is a limit at 30 + 70 = 100 servers, but couldn't find it even review configuration values with show config, and documentation doesn't explicit which configuration to change / increase free_servers.
pgbouncer version : 1.7.2
EDIT :
I've just discover that, for a pool of 6 webservers configured to hit the same PG database, 3 of them can have 200 backend connexions (server connexion), and 3 of them can only make and maintain 100 connexions (as described in the first part). But, .. the configuration is exactly the same in pgbouncer configuration file, and the servers are cloned VM. The version of pgbouncer is also the same..
So far, I still haven't found documentation on internet where this limitation come from...
This data is just some internal information for PgBouncer.
Servers information is stored inside an array list data structure which is pre-allocated up to a certain size, in this case that is 100 slots. used_servers = 30, free_servers = 70 means there are 30 slots currently in used, and 70 slots free. PgBouncer will automatically increase the size of the list when it's full, hence there's no configuration for this.
I am doing development now with highcharts on Windows/Eclipse/Jetty/Java environment.
I have the following highcharts-convert.properties:
#### phantomjs properties ####
# the host and port phantomjs listens to
host = 127.0.0.1
port = 7777
# location of the phantomjs executable
exec = phantomjs
# name of the convert script used by phantomjs
script = highcharts-convert.js
#### connect properties used to connect with phantomjs running as HTTP-server ####
# all values in milliseconds
# specifies the timeout when reading from phantomjs when a connection is established
readTimeout = 6000
# timeout to be used when opening a communications link to the phantomjs server
connectTimeout = 500
# the whole request to the phantomjs server is scheduled, max timeout can last to this value.
maxTimeout = 6500
#### Pool properties ####
# number of phantomjs servers you can run in the pool.
poolSize = 6
# The pool is implemented as a BlockingQueue.
maxWait = 500
# Keep files in the temp folder for a certain retentionTime, defined in miliseconds
retentionTime = 30000
I have to constantly start, stop, and restart Jetty within Eclipse. I notice that each time I start jetty, 6 window processes called p"hantomjs.exe *32" are started. However, when I stop Jetty, these process do not disappear. This leads to too many "phantomjs.exe *32" running on my machine.
How can I fix this problem?
I'm attempting to start mongos and fail fast if the config server is unavailable. Right now, with an unavailable config server, I'm seeing:
Tue Feb 12 11:09:13 [mongosMain] can't resolve DNS for [compute-1-3] sleeping and trying 10 more times
How do I configure the 10?
The 10 retries is hard coded, it is not configurable. You can see it here:
https://github.com/mongodb/mongo/blob/master/src/mongo/s/config.cpp#L742
Just in case line numbers change, here's the relevant counter/loop:
for ( int x=10; x>0; x-- ) {
if ( ! hostbyname( host.c_str() ).empty() ) {
ok = true;
break;
}
log() << "can't resolve DNS for [" << host << "] sleeping and trying " << x << " more times" << endl;
sleepsecs( 10 );
Therefore you could, in theory, alter the code and re-build yourself, but then you would have to maintain that for new versions. I would recommend instead that you keep the config server available instead, or at least have it up within ~100 seconds of the mongos starting.