Currently we migrated to Oracle 18c+Tomcat 9 after which we are experiencing max utilization issue both process and session.
INST_ID RESOURCE_NAME CURRENT_UTILIZATION MAX_UTILIZATION LIMIT_VALUE
-------- -------------------- ------------------- --------------- ------------
1 processes 925 1000 1000
1 sessions 945 1020 1522
On further querying most of the utilization are observed from DB,
SELECT machine, count(*) FROM v$session GROUP BY machine ORDER BY count(*) DESC;
MACHINE COUNT(*)
-------- ----------
1825(DB) 591
0058(APP) 332
Could you please assist me how to address this issue and find an permanent solution. As a temporary workaround i'm clearing all inactive sessions from DB using kill session.
Please find my server.xml configurations
<Connector port="8443" protocol="org.apache.coyote.http11.Http11NioProtocol"
connectionTimeout="20000"
acceptCount="500"
maxSavePostSize="0"
disableUploadTimeout="true"
enableLookups="false"
maxThreads="450"
removeAbandonedTimeout="60" removeAbandoned="true"
logAbandoned="true"
SSLEnabled="true" scheme="https" secure="true" />
If your app needs that many connections then you can increase processes parameter.
Related
current setting of a pgbouncer server is the following - and what I don't understand is the 'free_servers' info given by the show lists command when connecting to pgbouncer. Is it a (soft or hard) limit on the number of connexion to the postgresql databases used with this instance of pgbouncer ?
configuration :
max_client_conn = 2048
default_pool_size = 1024
min_pool_size = 10
reserve_pool_size = 500
reserve_pool_timeout = 1
server_idle_timeout = 600
listen_backlog = 1024
show lists gives :
pgbouncer=# show lists ;
list | items
---------------+--------
databases | 6
pools | 3
free_clients | 185
used_clients | 15
free_servers | 70
used_servers | 30
it seems that there is a limit at 30 + 70 = 100 servers, but couldn't find it even review configuration values with show config, and documentation doesn't explicit which configuration to change / increase free_servers.
pgbouncer version : 1.7.2
EDIT :
I've just discover that, for a pool of 6 webservers configured to hit the same PG database, 3 of them can have 200 backend connexions (server connexion), and 3 of them can only make and maintain 100 connexions (as described in the first part). But, .. the configuration is exactly the same in pgbouncer configuration file, and the servers are cloned VM. The version of pgbouncer is also the same..
So far, I still haven't found documentation on internet where this limitation come from...
This data is just some internal information for PgBouncer.
Servers information is stored inside an array list data structure which is pre-allocated up to a certain size, in this case that is 100 slots. used_servers = 30, free_servers = 70 means there are 30 slots currently in used, and 70 slots free. PgBouncer will automatically increase the size of the list when it's full, hence there's no configuration for this.
I am using jboss server with the following configurations timeout:-
<timeout>
<idle-timeout-minutes>1</idle-timeout-minutes>
</timeout>
<min-pool-size>10</min-pool-size>
<max-pool-size>30</max-pool-size>
<prefill>true</prefill>
<use-strict-min>false</use-strict-min>
<flush-strategy>FailingConnectionOnly</flush-strategy>
Now, as soon as the server reaches the max load(30 connections) the datasource details obtained from jboss's CLI reports Active Count = 30 and Available Count = 30.
However, even after reducing the server request to 1, the active count and the Available Count report 30 as their values.
Expected :- The numbers should decrease and ideally only 1 connection from prefilled pool should be used instead of keeping all the connections awake!!
I am seeing the following logs :-
17:34:12,359 DEBUG [org.jboss.jca.core.connectionmanager.pool.idle.IdleRemover] (IdleRemover) Notifying pools, interval: 30000
Please help!
The connection pool implementation (ironjacamar) on WildFly 8 is in FIFO aka round robin manner. So having max-pool-size number of request within time of idle-timeout-minutes is enough to keep the pool from shrinking.
I have to use another decrementer policy to tell the connection pool to shrink for a size n explicitly for every idle-timeout-minutes interval.
Sample setting as below
<pool>
<min-pool-size>5</min-pool-size>
<max-pool-size>20</max-pool-size>
<prefill>false</prefill>
<use-strict-min>true</use-strict-min>
<capacity>
<decrementer class-name="org.jboss.jca.core.connectionmanager.pool.capacity.SizeDecrementer">
<config-property name="Size">
1
</config-property>
</decrementer>
</capacity>
</pool>
http://www.ironjacamar.org/doc/userguide/1.1/en-US/html/ch05.html#deploying_capacity_policies
I am using a cluster computer to compute a "lmem" job. I submitted my job a day ago and while normally the job immediately begins running and I can monitor how long it has been running with qstat, this job remains in the queue.
I used qstat -q to see that
`Queue Memory CPU Time Walltime Node Run Que Lm State
---------------- ------ -------- -------- ---- --- --- -- -----
lmem -- -- -- -- 36 235 -- E R
batch -- -- -- -- 8 0 -- E R
express 4gb -- 06:00:00 -- 17 0 -- E R
test -- -- -- -- 0 0 -- D S
production 16gb -- -- -- 66 157 -- E R
route -- -- -- -- 0 0 -- E R`
Someone must have put in A LOT of lmem jobs. I was wondering if there was a way to see where on that list of 235 my job is in line?
When using a scheduler like Moab or Maui, you can run a commands such as checkjob -v and mdiag -p (or have an admin do it for you) to see if the job has an advance/priority/job reservation, and how many jobs it has in front of it. The default ordering of showq will place jobs with a job reservation at or near the top of the Eligible/Idle list. If you're only using pbs_sched, then the order shown in a plain qstat is the order in which it will run, although that may not make it clear how soon it can run.
I am using Resource Governor on my production database for restriction to maximum CPU usage. But it couldnt work efficiently. I am using same structure on different servers and it work correctly. Only difference between servers is CPU count.
My OS version is Windows Server 2012 R2 and my MS SQL Server version is "Microsoft SQL Server 2014 - 12.0.2495.0 (X64)
Mar 31 2015 09:47:37
Copyright (c) Microsoft Corporation
Enterprise Edition: Core-based Licensing (64-bit) on Windows NT 6.3 (Build 9600: ) (Hypervisor)
"
My classifier function script is below:
USE [master]
GO
/****** Object: UserDefinedFunction [dbo].[rgclassifier_MAX_CPU] Script Date: 28.7.2015 13:24:29 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
ALTER FUNCTION [dbo].[rgclassifier_MAX_CPU]() RETURNS sysname
WITH SCHEMABINDING
AS
BEGIN
DECLARE #workload_group_name AS sysname
IF (ORIGINAL_DB_NAME() = 'DeFacto' )
SET #workload_group_name = 'gMAX_CPU_PERCENT_60'
RETURN #workload_group_name
END;
My resource pool script is below:
USE [master]
GO
/****** Object: ResourcePool [pMAX_CPU_PERCENT_60] Script Date: 28.7.2015 13:24:21 ******/
CREATE RESOURCE POOL [pMAX_CPU_PERCENT_60] WITH(min_cpu_percent=0,
max_cpu_percent=60,
min_memory_percent=0,
max_memory_percent=100,
cap_cpu_percent=100,
AFFINITY SCHEDULER = AUTO
,
min_iops_per_volume=0,
max_iops_per_volume=0)
GO
My final description script is below:
USE [master]
GO
ALTER RESOURCE GOVERNOR WITH (CLASSIFIER_FUNCTION = [dbo].[rgclassifier_MAX_CPU]);
GO
ALTER RESOURCE GOVERNOR WITH (MAX_OUTSTANDING_IO_PER_VOLUME = DEFAULT);
GO
ALTER RESOURCE GOVERNOR RECONFIGURE;
GO
İt doesnt restrict max CPU Usage. I set default max_cpu_percent is %60 but sometimes it use %100 percent. Thus, we can not access the system properly.
This is the error message from my event log:
"Failed to run resource governor classifier user-defined function 'dbo.rgclassifier_MAX_CPU'. Last error 8525, state 1. See previous errors in SQL Server error log from session ID 538 for details. Classifier elapsed time: 0 ms." Could you help me?
Note that the maximum CPU percentage is an opportunistic maximum. If there is available CPU capacity, the workload uses it up to 100 percent. The maximum value only applies when there is contention for CPU resources.
Here is full description
My system run out of memory sometimes. I can see bellow error in logs every time when system is running out of heap memory
Maximum number of threads (200) created for connector with address abc.com/192.168.1.45 and port 8080
Any ideas why this is happening?
JBoss is crashing due to a high number of threads created. When it tries to create a new one, the application stops responding and starts to shutdown the application server.
Increasing the maxThreads parameter will resolve the issue. Do this incrementally; raising the value of maxThreads too much can result in performance problems such as:
High memory usage
General slowness due to the JVM being forced to context switch
between many threads frequently
To increase maxThreads edit in JBOSS_EAP_DIST/jboss-as/server/PROFILE/deploy/jbossweb.sar/server.xml
<!-- A HTTP/1.1 Connector on port 8080 -->
<Connector protocol="HTTP/1.1" port="8080" address="${jboss.bind.address}"
connectionTimeout="20000" redirectPort="8443" maxThreads="3000"
minSpareThreads="2000" maxKeepAliveRequests="-1" />
see also: Performance Tuning Guide - Chapter 2. Connectors