I'm facing some deployment issues on the Websphere Portal version 8.5 running on a clustered environment.
The deployment happens through by calling the xmlaccess.sh script. The deployment is working fine but it is taking too much time.
There are a few xmls which are being imported by the xmlaccess and it connects to the url: http://serverhost:10039/wps/config
webmodule.delete
webmodule.import
portal-content.delete
portal-content.import
The webmodule.import & portal-content.import is taking ~20mins each. Total time 45mins.
The server has federated LDAP repository but there is no issue on the LDAP as the application uses the same LDAP server for login and it's very responsive.
We have increased the jvm heap size to:
Initial Heap Memory - 4096m
Maximum Heap Memory - 8192m
The dmgr has a initial and maximum heap memory of 3072m.
Also there are no exceptions in the sysout logs during the deployment. I have enabled the trace option but still not finding any issue.
Also noticed one thing, when opening the websphere portal admin console, the response is very slow. The page is taking ~1m to load.
The hardware configs for the server are -
OS - Solaris
Ram - 16 GB
4CORE CPU
Increased the heap memory on jvms and dmgr.
Set java.net.preferIPv4Stack=true for the jvms, dmgr and node agents.
Related
I've installed BorPred in local iisexpress on clean server 2019 core. Debug in web.config is disabled, log4net setup changed to show only ERROR/FATAL.
Borpred started with mem usage less than 20M, and then I connect to it mem usage start growing and this is ok.
If I leave borpred alone for 1 hour it keeps running and it is normal too due to the periodic api/admin_WebApi/GetChangesSince calls.
But the mem usage after 1 hour increased up to 600M
I use TASKLIST command to check it.
Question - is it normal behavior or it can be mem leak?
Are there some settings to change/to check that can help to decrease mem usage?
Thank you
New name for this product is MDrivenServer.
The MDrivenServer has client synchronization - this builds up a list of changed identities. It will be expected to see a build up of memory due to update operations building the memory of the recently changed objects.
The MDrivenServer also has internal EcoSpaces to handle its own administration and ServerSide jobs - these will be garbaged and recreated when used a certain period of time.
.NET does not necessarily release memory from processes that have shown a need for the memory in the past - this causes you to see the used memory to equal the worst case need - like if you have a server-side job that pushes memory usage and it run once a day - the memory usage may still reflect the max usage.
We have a wildfly 8.2 app server which allocates 6GB of server RAM. Sometime due to havey transaction count wildfly has stop receiving incoming connections. But when I check server (not app server, it is our VM ) memory, it uses 4GB of RAM. Then I checked Wildfly app server's heap memory it did not use at least 25% of allocate heap size. Why is that? When I restart wildfly App server, All things work normally and when it comes that kind of load, above scenario happen again
Try to increase the connection-limit as suggested in this SO question
You can Dump HTTP requests as given here
Also, are you getting any errors in your console? Please post them as well.
I have deployed my dockerized micro services in AWS server using Elastic Beanstalk which is written using Akka-HTTP(https://github.com/theiterators/akka-http-microservice) and Scala.
I have allocated 512mb memory size for each docker and performance problems. I have noticed that the CPU usage increased when server getting more number of requests(like 20%, 23%, 45%...) & depends on load, then it automatically came down to the normal state (0.88%). But Memory usage keeps on increasing for every request and it failed to release unused memory even after CPU usage came to the normal stage and it reached 100% and docker killed by itself and restarted again.
I have also enabled auto scaling feature in EB to handle a huge number of requests. So it created another duplicate instance only after CPU usage of the running instance is reached its maximum.
How can I setup auto-scaling to create another instance once memory usage is reached its maximum limit(i.e 500mb out of 512mb)?
Please provide us a solution/way to resolve these problems as soon as possible as it is a very critical problem for us?
CloudWatch doesn't natively report memory statistics. But there are some scripts that Amazon provides (usually just referred to as the "CloudWatch Monitoring Scripts for Linux) that will get the statistics into CloudWatch so you can use those metrics to build a scaling policy.
The Elastic Beanstalk documentation provides some information on installing the scripts on the Linux platform at http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-cw.html.
However, this will come with another caveat in that you cannot use the native Docker deployment JSON as it won't pick up the .ebextensions folder (see Where to put ebextensions config in AWS Elastic Beanstalk Docker deploy with dockerrun source bundle?). The solution here would be to create a zip of your application that includes the JSON file and .ebextensions folder and use that as the deployment artifact.
There is also one thing I am unclear on and that is if these metrics will be available to choose from under the Configuration -> Scaling section of the application. You may need to create another .ebextensions config file to set the custom metric such as:
option_settings:
aws:elasticbeanstalk:customoption:
BreachDuration: 3
LowerBreachScaleIncrement: -1
MeasureName: MemoryUtilization
Period: 60
Statistic: Average
Threshold: 90
UpperBreachScaleIncrement: 2
Now, even if this works, if the application will not lower its memory usage after scaling and load goes down then the scaling policy would just continue to trigger and reach max instances eventually.
I'd first see if you can get some garbage collection statistics for the JVM and maybe tune the JVM to do garbage collection more often to help bring memory down faster after application load goes down.
I use wildfly appserver, when deploying a war file using Command-Line Interface (CLI) the process requires JVM heap size greater than 10 times the war file size.
How can I reduce this memory size that is consumed by jboss-cli during the deployment.
Problem detail:
I have to deploy 8 war files with 100 MB for each file, this process is applied in one transaction using "batch" and "batch.run", the memory consumed by this process exceeds 8GB.
I'm using the batch behavior because i have remote injections between wars, and i don't know the deployment order.
My question is how can I reduce the memory size consumed by wildfly when using jboss-cli, and if there is no way to reduce it, how can i know the deployment order between wars. (e.g. if app1 injects a remote session bean from app2, then the app2 must be deployed before app1).
You can define JVM options in $JAVA_OPTS environment variable, which will be loaded by WildFly.
For default JVM behavior take a brief look into bin/standalone.conf or bin/domain.conf.
I am doing load testing on my application using jmeter and I have a situation where the cpu usage by the applications jvm goes to 99% and it stays there. Application still work, I am able to login and do some activity. But, it’s understandably slower.
Details of environment:
Server: AMD Optrom, 2.20 Ghz, 8 Core, 64bit, 24 GB RAM. Windows Server 2008 R2 Standard
Application server: jboss-4.0.4.GA
JAVA: jdk1.6.0_25, Java HotSpot(TM) 64-Bit Server VM
JVM settings:
-Xms1G -Xmx10G -XX:MaxNewSize=3G -XX:MaxPermSize=12G -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+UseCompressedOops -Dsun.rmi.dgc.client.gcInterval=1800000 -Dsun.rmi.dgc.server.gcInterval=1800000
Database: MySql 5.6 (in a different machine)
Jmeter: 2.13
My scenario is that, I make 20 users of my application to log into it and perform normal activity that should not be bringing huge load. Some, minutes into the process, JVM of Jboss goes up and it never comes back. CPU usage will remain like that till JVM is killed.
To help better understand, here are few screen shots.
I found few post which had cup # 100%, but nothing there was same as my situation and could not find a solution.
Any suggestion on what’s to be done will be great.
Regards,
Sreekanth.
To understand the root cause of the high CPU utilization, we need to check the CPU data and thread dumps at same time.
Capture 5-6 thread dumps at the time of the issue. Similarly capture CPU consumption thread-by-thread basis.
Generally the root cause of the CPU issue would be problems with threads like BLOCKED threads, long running threads, dead-lock, long running loops etc. That can be resolved by going through the stacks of the threads.