jboss 5 startup time? [duplicate] - jboss5.x

We upgraded from JBoss 4 (and JDK 5) to JBoss 5 (and JDK 6). The problem is that the start time has gone from 1.5 minutes (on JBoss 4) to more than 4 minutes.
18:53:35,444 INFO [ServerImpl] JBoss (Microcontainer) [5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221053)] Started in 3m:9s:262ms
It seems like the component that is taking JBoss the longest time to initialize is the JMX
18:50:41,926 INFO [LogNotificationListener] Adding notification listener for logging mbean "jboss.system:service=Logging,type=Log4jService" to server org.jboss.mx.server.MBeanServerImpl#1adc122[ defaultDomain='jboss' ]
18:52:38,797 INFO [JMXConnectorServerService] JMX Connector server: service:jmx:rmi://lharel2/jndi/rmi://lharel2:1090/jmxconnector
From the DEBUG server log, I get these lines at the problematic time:
2009-12-18 18:51:00,886 DEBUG [org.jboss.deployment.MappedReferenceMetaDataResolverDeployer] (main) vfsfile:/C:/QC/Views/QCDev/jboss-5.1.0.GA/server/default/deploy/jmx-console.war/ endpoint mappings:
2009-12-18 18:51:00,886 DEBUG [org.jboss.deployment.MappedReferenceMetaDataResolverDeployer] (main) Processing unit=jmx-console.war, structure: jmx-console.war
2009-12-18 18:52:35,209 DEBUG [org.jboss.deployment.OptAnnotationMetaDataDeployer] (main) Deployment is metadata-complete, skipping annotation processing, ejbJarMetaData=null, jbossWebMetaData=org.jboss.metadata.web.spec.Web23MetaData#1f, jbossClientMetaData=null, metaDataCompleteIsDefault=false
There is no EJB in the project.
The memory settings are:
-Xms128m -Xmx512m -XX:MaxPermSize=256m
Do you have any idea how JBoss start time can be improved?
Update: so far no luck, I tried shreeni's suggestion (changed the scanning xmls). The server is not running in debug mode so MicSim's suggestion is not relevant

A shot into the blue sky without more information
Network timeouts: 1,5 minutes of delay when deploying jmx-console.war may indicate a network timeout (e.g. 3 x 30 seconds). Try to start JBoss and bind it to a specific IP address using the -b command line argument or the jboss.bind.address system property. Also, try to make sure your host and DNS resolution settings on your system are correct.
JMX is also using RMI and you may want to set the RMI server host name as system property. On some Linux distributions, RMI has problems with looking up the correct hostname and jmx-console.war may try to connect to the 'wrong localhost'. The system property is java.rmi.server.hostname
System tracing: If that does not help, you may want to use strace to start the java process, so you can see the point where the system hangs (if it really does hang due to a network timeout or similar).

That is an awfully big gap in the logs. I suggest changing the log configuration to log everything at DEBUG level, rather than INFO. This will generate an awful lot more log entries, but hopefully will help you narrow it down.
The easiest way to do this is to set the -Djboss.server.log.threshold=DEBUG system property when you start JBoss

I had an issue like this but I found a good improvement by setting the initial and max heap size to same values, I mean:
-Xms512m -Xmx512m
With this, I improved from 4 to 2 minutes the starting time.

Your suspicion about the jmx-console can be misleading. There may be other components doing work in the background unrelated to the jmx-console. In my experience, we had an issue where a small war file appeared to take 3 minutes to load! It was innocent. The culprit was partly due to an EAR file with many wars and jars.
While I'm no expert, I would suggest the following:
Try turning up the logging to TRACE. By doing this, I witnessed one of the deployers (EJBDeployer, I think) unnecessarily scanning WARs in one of our EARs. I then manually disabled the scanning of those WAR files.
Run wireshark during the startup. I discovered some war files were hanging while waiting for a response from an external DTD request. Those websites were either now non-existent or would not properly serve the DTD files to java-based programs. I could speed it up by either having the programmers use local DTD files or mirroring those DTDs locally and having /etc/host loopback locally.

You could refer to this link to avoid unnecessary annotation scanning which could speedup your server start

See http://community.jboss.org/wiki/jboss5xtuningslimming especially the "Tuning" part.

I'm using JBoss 5.1.0 with a Macbook pro (2.26ghz 4gb) without applications it start in 54s
15:00:26,449 INFO [ServerImpl] JBoss (Microcontainer) [5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221634)] Started in 54s:720ms
I made a new configuration based on the "default". The JMS dataosurce points to a Postgres database instead of the "Hypersonic Database" (in memory database)

I suppose you are starting in debug mode. This mode can be up to 3 times slower than normal mode.
But there might be also a problem when switching from JDK5 to JDK6. I found this solution here on the net:
I've solved that. It's a debugging issue. I've changed my debug settings from:
wrapper.java.additional.26=-Xdebug
wrapper.java.additional.27=-Xnoagent
wrapper.java.additional.28=-Djava.compiler=NONE
wrapper.java.additional.27=-Xrunjdwp:transport=dt_socket,address=7199,server=y,suspend=n
to:
wrapper.java.additional.26=-Xdebug
wrapper.java.additional.27=-Xrunjdwp:transport=dt_socket,address=7199,server=y,suspend=n
and JBoss becomes fast again.
Hope this helps.

Do you need the JMX console application? Pragmatic thing would be to un-deploy it from the server, you could still use the jconsole or jvisualvm for basically the same thing.

Turn off annotation scanning and other features you don't need https://community.jboss.org/wiki/jboss5xtuningslimming
When you start/stop JBoss from eclipse, it does not clean up the tmp & work folders correctly. Setup an External Tool configuration and run a batch file to delete everything in tmp & work folders before each startup.
I was able to speedup running the "default" profile from 15/20 minutes to 5 minutes.

Related

Deploy to remote glassfish from Netbeans fails

Trying to deploy to remote glassfish 4.1 (same as on my local machine) from Netbeans fails without any error message (that I could find).
When I do "Run project" from Netbeans 8.0.2, I get a remotetest (run) in the status bar (remotetest is the name of my project, which is just a "Hello from Servlets" default JSF project). When I click on the 1 more spot, it says it's deploying my .war to my remote server. After 20 minutes and 1 second, I get
Deployment error: Deployment timeout has exceeded. See the server log
for details."
(the first time I waited that long... actually right now, it's been "deploying" about 60 minutes and hasn't stopped).
There is nothing in the remote glassfish server log after glassfish server has finished startup, no sign of the deployment trial. This directly after rebooting my AWS EC2 instance.(I read somewhere that the micro instances might not be good for a full blown server, normally it seems that it's taking between 0 and 12% in top. What I do see, right now though, is the java process taking 75% cpu and 40% memory when the "deployment" has been going on for an hour or so, and it remained so after stopping the deployment from Netbeans end.)
What I can do after a dozen hours of setting things up and googling:
Deploy my app to local glassfish.
Connect to my remote glassfish admin page via port 4848
move the .war to remote .../autodeploy, see it deploy and have it visible remotely at myhost:8080/remotetest
connect to remote mysql via Netbeans (not sure if relevant)
ssh a basic java "hello world" from Netbeans to remote machine (not sure if relevant)
What I've tried:
opened up various ports for custom TCP rule from the EC2 management console , including 8080,8181,4848
tinkered with every possible switch from Netbeans and glassfish I could think of
Google my butt off
sleep over it
rinse and repeat
The glassfish server logs show two warnings after startup:
All SSL cipher suites disabled for network-listener(s). Using SSL
implementation specific defaults
and three SEVERE:
java.lang.ClassNotFoundException:
org.glassfish.admin.rest.resources.generatedASM.DomainResource not found by org.glassfish.main.admin.rest-service [97] at
org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1532)
at
org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:75)
at
org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1955)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at
java.lang.Class.forName0(Native Method) at ...
I can't make anything of those. Nothing in the log since I started the deployment a couple of minutes after the server was up and visible in Netbeans.

JBOSS 7 Monitoring Tools

Any good suggestion for Monitoring JBOSS 7 in Production ? I would also like to configure alerts based on certain condition. Of course , It has to be Open source.
Thanks.
You can use standard JConsole that comes with JBoss dependencies added. It's used to monitor your servers state and mbeans, it's very useful.
To test it on localhost start your server and then run the JConsole from your server/bin directory and select JBoss in the Local process selection.
To use it on "remote" server, start your server on "REMOTE_HOST" and then run JConsole from an JBoss/bin directory and connect with the followin string
service:jmx:jmx-remoting://REMOTE_HOST_NAME:9999 (or the port you use) and enter the username and password.
Secondly, for more detailed info of objects creation, memory leaks, CPU% (profiling) there is another one as:
http://jbossprofiler.jboss.org/
You can try to use free open source APM like scouter.
It shows very useful realtime performance information of every request.
And also you can set a threshold of resources and can make plugin for alerting to external.
https://github.com/scouter-project/scouter
JBoss7 need to set module option.
-Djboss.modules.system.pkgs=~~~,scouter

Apache tomcat start up failure

I have been working with netbeans...til yesterday night it worked but now i couldnt run even a simple code its showing the following comments in the output screen
init:
deps-module-jar:
deps-ear-jar:
deps-jar:
library-inclusion-in-archive:
library-inclusion-in-manifest:
compile:
compile-jsps:
Starting of Tomcat failed, the server port 8084 is already in use.
D:\Dhivya\Job\calculator\payodaproject\nbproject\build-impl.xml:683: Deployment error:
Starting of Tomcat failed, the server port 8084 is already in use.
See the server log for details.
BUILD FAILED (total time: 3 seconds)
how to recover from this
First this isn't a constructive question as there is obvious lack of research. You didn't mention whether you tried to find out if any process was using port 8084. You may use netstat command to figure that out.
This usually happens when there is a Tomcat process already running and for whatever reason Netbeans is not able to see it, and tries to start another. You have to kill the Tomcat process manually, then Netbeans will be able to start it again.
It's a netbeans bug for port 8084 and there are few ways to resolve it. One is to remove tcnative-1.dll from the apache-tomcat/bin folder in your installation. For more you can visit netbeans-bugzilla.

FATAL ERROR in native method: JDWP No transports initialized, jvmtiError=AGENT_ERROR_TRANSPORT_INIT(197)

Can you explain what this error is (and secondly why I am getting it)?
FATAL ERROR in native method: JDWP No transports initialized, jvmtiError=AGENT_ERROR_TRANSPORT_INIT(197)
P.S. It may be related to Known Tomcat 6.0 and JDK 1.7.0_02 issues?, as I only started getting it after upgrading from JDK 1.7.0 to 1.7.0 update 2, with no other upgrades to other software.
I am running:
Eclipse Indigo 3.7
JDK 1.7.0_0u2 (JDK 7 update 2)
Tomcat 6.0
Windows 7
Apache HTTP Server (although not using it yet)
When I start Tomcat I started getting this error, but not all the time. Rebooting just now fixed it. Some mornings I come to work without a reboot and it fixes it even though it failed the day before. It's sporadic. To debug this I need to understand. Can you help explain it?
EDIT : I have two Tomcat servers, for two different projects, on the same port. The other (first) server is "stopped" but remains "synchronized", in case this matters. I've quickly tried changing all the ports up one (8080 to 8081) and the error reproduces. This may not be a proper test of changing ports, however.
EDIT 2: I just had this problem, and rebooting "fixed" the issue. The workstation was on all weekend and Tomcat worked on Friday and Eclipse was shutdown at the end of the day. I will keep taking notes like this as I run into it to remove guesswork.
EDIT 3: Today it gave me this error from an unrebooted system that worked yesterday, programs shut down yesterday and restarted today. I rebooted, and the error is gone. Most notably is that the error always occurs at 23% compilation. It hits 23%, waits a bit and this is when I know it won't succeed, and then popups a window. I'll capture what the window says next time. Then it gives the above error to the Console.
EDIT 4: I am running Windows 7 and Apache HTTP Server (although not using it yet). I'll add these to the list above.
EDIT 5: The popup window mentioned in edit #3 is (and note my Tomcat is named Server Tomcat v6.0 Server at localhost):
Problem Occurred
'Server Tomcat v6.0 Server at localhost' has encountered a problem.
Server Tomcat v6.0 Server at localhost failed to start.
OK << Details
Server Tomcat v6.0 Server at localhost failed to start.
EDIT 6: I just got a new problem, which is Cannot connect to VM com.sun.jdi.connect.TransportTimeoutException popup window error and the same main error that this question asks about in the Console window.
EDIT 7: Just restarting Eclipse, not rebooting the whole computer, solved the error this morning.
This error typically comes up when the necessary port is taken by another program.
You said that you have changed the HTTP connector port from 8080 to 8081 so the two Tomcats do not clash, but have you also changed the <Server port="..." in tomcat/conf/server.xml to be different between your Tomcats?
Are there any other connectors ports which may possibly clash?
Does your HOSTS file have an entry for localhost? Some other situations this error is seen in seem to have this as a problem resolution.
Make sure you have 127.0.0.1 localhost set in it...
(from this and this)
Encountered this. all I did was to kill all the java process(Task Manager) and run again. It worked!
Check whether your config string is okay:
Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=9999
I just had this issue today, and in my case it was because there was an invisible character in the jpda config parameter.
To be precise, I had dos line endings in my setenv.sh file on tomcat, causing a carriage-return character right after 'dt_socket'
EDIT these lines in host file and it should work.
Host file usually located in C:\Windows\System32\drivers\etc\hosts
::1 localhost.localdomain localhost
127.0.0.1 localhost
I had the same problem because I set the following in Catalina.sh of my tomcat:
JAVA_OPTS="$JAVA_OPTS -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=9999"
After removing it, my tomcat worked well.
Hope help you.
Encountered this issue and changing the debug port helped. For some reason, the debug port had to be greater than the app port.
Change control panel Java's option about proxy to "direct", change window's internet option to not use proxy and reboot. It worked for me.
This error mostly comes when we forcefully kill the weblogic server ("kill -9 process id"), so before restart kindly check all the ports status which weblogic using e.g. http port , DEBUG_PORT etc by using this command to see which whether this port is active or not.
netstat –an | grep
(Admin: 7001 or something, Managed server- 7002, 7003 etc)
eg: netstat –an | grep 7001
If it returns value then,
option 1: wait for some time, so that background process can release the port
option 2: execute stopweblogic.sh
Option 3: Bounce the server/host or restart the system.
My issue was resolved by option 2.
if your JVM Cli is: -agentlib:jdwp=transport=dt_socket,address=60000,server=n,suspend=n and JDK version is 7, change "server=n" to "server=y" will be OK.
In my project I had the same error, I restarted Tomcat and it worked, withtout killing the java process.
I set 127.0.0.1 localhost, and solve this problem.
I had the same problem in Catalina.sh of my tomcat for JPDA Options:
JPDA_OPTS="-agentlib:jdwp=transport=$JPDA_TRANSPORT,address=$JPDA_ADDRESS,server=y,suspend=$JPDA_SUSPEND"
After removing JPDA option from my command to start the Tomcat server, I was able to start the server on local environment.
I was getting the same error when i switched to STS version 3.8.3
And imported my entire workspace to the new STS.
Apparently the "Boot Spring App" instance was defective. (i run from STS)
If this is your problem,
Simply create the Boot run configuration again.
In case you are working with environments or docker images you can really change /etc/host I recommend just changing the binding from star to 0.0.0.0.
So (basing on my case for instance) instead of:
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"
You would define it as:
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=0.0.0.0:5005"

Configure Zookeeper zoodiscovery centralized and replicated mode

I have a problem configuring Zookeeper to work with zoodiscovery mode centralized and replicated. The guide at http://wiki.eclipse.org/Zookeeper_Based_ECF_Discovery is a little mysterious about that.
I'm working on Windows XP SP3, Java JDK 1.6, Eclipse STS 2.7.2, org.eclipse.osgi 3.7 and a proxied network.
NOTE: Using the standalone configuration mode gives no problem. I use -Dzoodiscovery.flavor.standalone=192.168.23.21:3030;clientPort=3031 on the server and -Dzoodiscovery.flavor.standalone=192.168.23.28:3031;clientPort=3030 and it works nicely.
I will split the question in multiple parts:
1) In a setup with a (one) central server on 192.168.23.28, multiple clients. The clients will both publish and consume services.
I launch the server as:
java -Dzoodiscovery.dataDir=name -Dzoodiscovery.flavor=zoodiscovery.flavor.centralized=192.168.23.28 -jar org.eclipse.osgi.jar -console -consoleLog -clean -configuration c:\temp\osgiserver\configuration
I can see the ZooDiscovery> Discovery Service Activated.
When I launch the clients (in the example there's only one) as:
java -Dzoodiscovery.autoStart=true -Dzoodiscovery.flavor=zoodiscovery.flavor.centralized=192.168.23.28 -jar org.eclipse.osgi.jar -console -consoleLog -clean -configuration c:\temp\osgiclient\configuration
I can see ZooDiscovery> Discovery Service Activated. but then INFO - Attempting connection to server: /192.168.23.28 which goes on and on never succeeding.
I have to start server and clients by configuring Zookeeper from command line, I cannot insert those parameters inside the bundles. I have tried setting the -Dzoodiscovery.clientPort=8888 on the server (8888 is available) and then -Dzoodiscovery.flavor=zoodiscovery.flavor.centralized=192.168.23.28:8888 on the client, but still it changes nothing.
How do I configure such a setup?
2) Plus I'd like to know if it's possible, using centralized, to have multiple central servers talking between them or if I'd have to use the replicated mode.
3) Which leads to.. how do I configure server and clients to use replicated mode by passing VM command line arguments?
4) In replicated mode, if I add a new Zookeeper instance later on, will I have to stop and reconfigure the existing Zookeeper instances to work with the new one or is it sufficient to configure the new one to work with the existing ones?
Thank you very much,
cheers