We have an application on JBoss 7.1.0 istance, usually running ok but sometime the performance degrading, in this case a restart of JBoss is useful.
I have configured an access.log to record the timing of the pages and when the performance are degrading, we have an increase in recorded time on it for some pages.
How can I better identify and troubleshoot the problem? I think can be a problem related to objects in memory of JBoss server.
I exclude can be a memory issue on client because the access.log record the slow time, is it right or can I be misinterpreting the access.log and the problem can be also on client?
The ram / cpu on server are not under load when there are slowness and the xmx is oversized.
The JBoss is on Windows with JDK 1.7.0_80.
Thanks
UPDATE: Our degrading performance problem was related to an APM configured to record too much data and solved by removing it (are needed to be better configured).
I don't think JBOSS server has a problem. By reading your statement, I think the your code is leaking connections. Login into JBOSS Console and see the stats of your connection pools (you need to enable stats if not enabled already). If the connection count grows linearly and never comes down then you need to fix the leakage at code level. You can also use JConsole to monitor JVM memory and detect problems.
Related
I'm starting to move from visualvm to java mission control. I use wildfly 8.2, which doesn't work with JMC out of the box, but have modified the JMC ini to include the required class path as per my answer in the jboss community https://developer.jboss.org/message/955742#955742
I've been able to connect successfully, however the time it takes to connect is ridiculous. Often waiting 1-10 minutes (during which time jmc often becomes unresponsive) when connecting to a local wildfly instance, it's almost instant, the delay happens only when connecting to a remote instance.
Does anyone know what would cause this to be so slow, and if anything can be done to sort it?
Liferay 6.1 takes roughly 60-120 seconds to start on my developer machine. Unfortunately we have to start Liferay very often, so this delay hurts. Actually, most features of Liferay aren't needed by a developer (at least not for us). Is it possible to reduce the amount of time Liferay takes to start? Maybe by removing unused functionality, such as blog, chat and forum?
Please note that I'm not talking about deployment time, which can be reduced a little by following the hints in another StackOverflow thread.
removing unused functionality - that's a good idea, just it will be hard on Liferay 6.1. This is what modularity promises to make possible for Liferay 7, where you should be able to remove the unwanted OSGi bundles. Unfortunately Liferay 7 is not out yet (as of the time of writing this answer beta4 is out) and you're explicitly asking about 6.1.
You could try to deactivate several components - most likely by changing several of the spring configuration files in an ext plugin. However, this has rarely been done, so it might have consequences that nobody has ever seen. Also, it will probably be hard to maintain and upgrade this configuration.
Another way to solve this problem is to ask "Why do you have to restart so often?". In fact, utilizing JRebel for deployments - especially on a developer machine - has shown to be quite effective.
In the answer to Meera Prince's answer you're stating the "generous memory settings" of 1024G - while I agree that this shouldn't limit you in the upstart time, you might want to monitor it, so that you at least don't run into garbage collection during startup time. For a Java application the size of Liferay, 1G memory might be sufficient, but I'd not call it "generous".
Note that 8G RAM on a development machine might be your limitation: As soon as your machine hits virtual memory and starts paging to disk, all bets are off and you'll suffer. And I have no clue what else you're running: Browsers, IDE, Editors, Word Processor, Liferay, random other applications and services. You're hopefully on a 64bit OS.
If you have 8GB of RAM, then put more for Liferay e.g.
CATALINA_OPTS="-Xms4G -Xmx4G -XX:MaxPermSize=512m"
using of CATALINA_OPTS is better then JAVA_OPTS, because JAVA_OPTS will be allocated by shutdown of tomcat too, and CATALINA_OPTS only by startup.
And you can disable not necessary Filters in portal-ext.properties, e.g.
com.liferay.portal.servlet.filters.sso.cas.CASFilter = false
com.liferay.portal.servlet.filters.sso.ntlm.NtlmFilter = false
com.liferay.portal.servlet.filters.sso.ntlm.NtlmPostFilter = false
com.liferay.portal.servlet.filters.sso.opensso.OpenSSOFilter= false
com.liferay.portal.sharepoint.SharepointFilter = false
com.liferay.portal.servlet.filters.gzip.GZipFilter = false
See Liferay: How to configure Liferay Portal
If you are using Tomcat to run Liferay, the following will greatly reduce the time it takes to startup. Modifying your catalina.properties file attributes shown below. The article goes into the details.
org.apache.catalina.startup.ContextConfig.jarsToSkip=*
org.apache.catalina.startup.TldConfig.jarsToSkip=*
tomcat.util.scan.DefaultJarScanner.jarsToSkip=*
http://www.gpickin.com/index.cfm/blog/how-to-get-your-tomcat-to-pounce-on-startup-not-crawl
You may buy an SSD HD
8G RAM must be enough for Liferay 6.1 + Browser + Eclipse, however I have 16G, but I very often have two Liferays running an 20+ open browser windows
A CPU with 4 core can be relevant
Undeploy deployed but not needed portlets
Its depends on resource allocation for liferay JVM means heap and permzen space.
and also depends on your system RAM size
find java script and css related proprieties in liferay portal.proeprties file.
Use Java_OPTS as follows
-Xms1024m -Xmx1024m -XX:MaxPermSize=512m
https://www.liferay.com/community/wiki/-/wiki/Main/Liferay+Developer+Modee
http://www.liferaysavvy.com/2013/12/liferay-environment-challenges-and.html
https://www.liferay.com/documentation/liferay-portal/6.1/development/-/ai/lp-6-1-theme-developer-mode
I think, using ramdisk may worth a try.
Windows 7:
http://www.laptopmag.com/articles/faster-than-an-ssd-how-to-turn-extra-memory-into-a-ram-disk
Linux:
http://www.jamescoyle.net/how-to/943-create-a-ram-disk-in-linux
If you have many deployed plugins, it can shorten the startup time.
Without deployed plugins (only with marketplace) I measured the same starttime than with my hdd.
I want to know how can I speed up RSA 7.5( which is an IDE by IBM having eclipse under the hood with websphere server runtimes) mainly server start. The first time I start it after computer reboot it loads after, but after that it takes for ever to start/stop the server. The debug mode for server takes for ever to start.
I am using server 7 run time for IBM RSA 7.5.
So bascially RAD/RSA has websphere run times which allows to configure the server runtime start/stop within RAD/RSA. The run time allows you to develop webapps and test time on the server on deploy it on the websphere run time.
The problem I am facing is with the websphere runtime which works fine after computer reboot but is very slow after several deployments/publishing of the same web app.
I would be grateful you give performance tips for speed up RSA server start/shutdown and overall performance tips. I have plenty of memory like 12 GB with i7 Core 6 cores on Win7.
Of course of your are running the server in debug mode it's going to be a lot slower, but you have a few options like putting the server in development mode or doing some fine tuning as to which applications should start. Take a look at these articles:
Rational Application Developer Performance Tips- Case study: Tuning WebSphere Application Server V7 and V8 for performance
Performance tuning WebSphere Application Server 7 on AIX 6.1
WebSphere tuning for the impatient: How to get 80% of the performance improvement with 20% of the effort
WebSphere Performance Monitoring & Tuning
Some of these are a bit dated but they have some good information that may still be relevent to your issues; especially the first one.
Make sure that the workspaces are stored on a local disk.
edit - forgot this: buy a SSD disk. It makes a huge difference when developing.
If you have a virus scanner, disable on-access scan in the SDP installation directory including the server plugin, and in all your workspaces.
Uninstall any applications (ears) you are not using - the more you have installed the longer the server takes to startup. If your server is taking too long to start, RAD/RSA will assume it has timed out and stop it before it finishes starting - if this happens then increase the start timeout limit by double-clicking your server in the Servers tab and modifying the values in the Timeouts section.
Oh, and If you have a lot of datasources defined, and autostarting connection pools with alot of connections, it may also take a while to start the pools.
But that can't explain it all... I haven't tested, but since WAS and RSA seems to spend a lot of time doing absolutely nothing, I am starting to suspect it's trying to download schemas or something. If you have the time, you could try to trace and see if you find something like that...
I came across this post while trying to troubleshoot my RSA performance. I figured I would update it with a recommendation for improving performance on RSA 8.0.4.
http://publib.boulder.ibm.com/infocenter/radhelp/v8/index.jsp?topic=/com.ibm.performance.doc/topics/cperformance.html has some excellent tips on improving performance in the "Performance Tips" section. After implementing just some of the "Always" tips I've found my memory reducing significantly and performance being much faster.
You should start with the "Always" tips and then move to the "Sometimes" and "Rarely" ones for finer tuning.
For a variety of unfortunate management reasons (budget constraints etc.) I, the developer, have been put in the position to deploy the app in a production environment. The catch is that I don't have any experience in production EJB application server deployment. That said, they are aware that there are no guarantees of success.
The context:
The dev server runs on the latest version of Netbeans with Glassfish v3, on a mac machine
98% / 99% uptime is ok, there are no financial/critical transactions
It is a client/server EJB 3 app, and the web tier, business tier, and resource tiers currently run on the same machine.
I have the liberty to choose the hw/sw infrastructure
Load estimations: 10 simultaneous connexions avg, rare 200 peaks
The outbound public data is text/small pics (it's for iPhone clients), inbound HTTP text only
Basic maintenance will be taken care of (backup, server reboot, etc)
My questions for production deployment:
What are the must haves infrastructure-wise? Minimum system specs etc?
Is it ok to keep Glassfish v3?
Which configuration aspects of the server should I focus on?
Worst case scenario: if I deploy the same software infrastructure (Netbeans/Glassfish v3) as during the development, would the server keep up?
Any piece of advice would be most welcome. Thanks!
For the architecture, you can start small with just a single GlassFish instance with no front web server (GlassFish has one built in that is very capable). If you can wait for the release of GlassFish 3.1 you'll be able to add instances (clustered or standalone) and offer scalability and centralized admin.
Most production instances of GlassFish I've seen run with 1GB-2GB of JVM heap (-Xmx) but your mileage may vary if you load lots of data in memory or if you use some frameworks. If you want better reliability, having them on separate machines is a plus obviously. With two instances on the same machine you can offer continuity of service if one instance fails (but not if the machine fails).
I'd suggest scripting as much as possible the provisioning of the resources (connection pool , JDBC datasource, etc...) and applications using the "asadmin" command-line tool and try to not use NetBeans on the production platform.
Benchmarking with simulated load sounds like a wise thing to try to put together before going live and this survival guide will probably come in handy.
You don't mention the database. Isn't there one?
I suggest the following:
Not a Mac expert but I'll say go with 6GB or more RAM
HDD space is not a problem these days
Do not know much abt Mac Processors (watever eq of dual core etc)
Personally I have not used GF3 in Production but I hope it's stable now so you should be ok.
System Architecture:
Receive all HTTP requests on some web server (Apache or Sun web server) and load balance with your Glassfish server(s).
Now depending on your physical (or virtual) machines create instance of Glassfish Application Server on each machine. If you just have one machine then create atleast 2 instances of Glassfish. This will help to put one node down for maintaince and other to keep going.
As far as deployment is concern make sure you stop debug logs and fine tune JPA logs etc.
Use Ant or other scripts to deploy code and taking backup of existing code.
I hope this will help to kick start and rest you can ask or solve as you go along.
Good luck.
At a high level, what does JBoss 5's shutdown do? What might go wrong if I just kill the java process instead of gracefully shutting down JBoss?
A graceful JBoss 5 shutdown takes about 6 minutes for my application, which is pretty big and has 305 EJBs. JBoss seems to pause for a long time just before unbinding the EJB LocalHomes from jndi.
Given that, I am considering simply killing the java process. I am wondering about what might go wrong if I do that.
I run JBoss in mostly in *nix, sometimes in Windows.
Killing the process will leave the JBoss files in a potentially inconsistent state, and will certainly leave them in a messy state. When it restarts, it will probably clean up after itself OK, but then again it may not.
On Windows (you didn't say which platform you use), I've seen a killed JBoss process not release locked files properly, and the server won't restart at all. It's pretty rare, though.
In the end, if it works for you, the I wouldn't worry too much about it.
It releases all you connections such as jndi, jdbc, jms.. It also waits for your classes to finish up what it is doing. It also stops other services such as jms and web console