Problem: A 10 to 15 minute delay in WebSphere application deployments.
Environment/Situation: WebSphere 6.1.0.23, 90MB ear files containing about 19,000 files (ear file contains jar libraries). The ear file, WebSphere, and the automation driving the deployment are all on the same box. No EJBs. There are about 20 deployed applications like this on this box with 10 of them usually running.
Details: The deployment is automated, and the message 'ADMA5013I: Application ... installed successfully' is received. A few moments later, the directory is created (blah.ear/blah.war), but the directory remains empty for 10 to 15 minutes. Except for this specific delay, the performance on the box is fine and CPU utilization is very low. Once the files start getting created, they all show up in under a minute. Steps before and after this step run at an acceptable speed. It's just this one step, waiting for the files to show-up that's the problem.
Additional Details (precipitated by comments, below): WebSphere ND as evidenced by "Deployment Manager", and "Node Agent" in the logs. The ear contains one war file, one application. By using shared a library definition, the size of the ear was reduced to 60MB. WebSphere itself is started with JVM option -XX:MaxPermSize=256M. The deployments are done using the tools in the com.ibm.websphere.management.* packages (jar file supplied by IBM), primary class is "AdminClient". The code is similar to what is in this IBM documentation WS UI entry [System Administration > Console Preferences > "Synchronize changes with nodes"] was checked, but still sits for 15 minutes 'without doing anything'.
Just guesses:
ear files timestamp is in the past. Might be caused by jvm time-zone problems, different time zones when building/deploying (esp. if different machines used).
JDK could determine is there enough disk space
WSAD of some tool do not close file descriptor. Try deploy with anti-viruses paused.
Could you please describe situation in more details:
does build and deploy on the same machine?
does it work the same on all circumstances?
Related
It used to take 3 minutes to deploy our WAR to WebSphere 8.5.5.0.
It takes 3 hours to deploy the same WAR to WebSphere 9.0.0.2.
Is that a manifestation of a bug in WebSphere 9.0.0.2?
Did anybody have similar experience?
UPDATE:
Results of my experiments with settings suggested by IBM:
http://www-01.ibm.com/support/docview.wss?uid=swg21983564
Our WAR file contains 160 JAR files with total size of 160MB.
Time is measured for the "New Application" wizard, from first screen, until the next screen is shown.
WebSphere 8.5.5 - 3 minutes
WebSphere 9 - 3 hours
Added "Ignore-Scanning-Packages: org, net, com, javax" line to manifest file in the WAR - 30 minutes
Added "Enable-CDI: false" line to manifest file in the EAR - 30 minutes
Added lines
Enable-CDI=false
Enable-Implicit-Bean-Archive=false
to C:\Program Files\IBM\WebSphere\AppServer\properties\amm.filter.properties file - 30 minutes
Added JVM properties
com.ibm.ws.cdi.enableImplicitBeanArchives
false
com.ibm.ws.cdi.enableCDI
false
in Servers->Server Types->WebSphere application servers->server1->Server Infrastrucure->Java and Process Management->Process definition->Java Virtual Mashine->Custom properties - 30 minutes
Added JVM properties
-Dcom.ibm.ws.cdi.enableImplicitBeanArchives=false
-Dcom.ibm.ws.cdi.enableCDI=false
in Servers->Server Types->WebSphere application servers->server1->Server Infrastrucure->Java and Process Management->Process definition->Java Virtual Mashine->Generic JVM arguments - 30 minutes
UPDATE: SystemOut.log contains message:
Thread "WebContainer : ... has been active for 668644 milliseconds and may be hung.
Stack Trace contains:
at org.apache.commons.fileupload.MultipartStream.readBodyData
It looks like WebSphere is processing HTTP request to upload a file.
A preliminary check: The application is presumed to be exactly the same for both deployments. This is more-or-less assumed, but there have been a lot of cases where the application was presumed to be the same but it wasn't, so this needs to be verified.
Between IBM WebSphere Application Server v8.5.5.0 and v9.0, a significant upgrade was made to the CDI level which is supported. That is, v9.0 adds support for CDI 1.2, which has new class and annotation scanning requirements. These new requirements, together with some problems in the initial class and scanning implementation which is used by CDI in v9.0 can cause a very large slowdown of application deployments.
The problem has been noted in particular with medium to large web modules. Here "medium to large" means web modules which have many WEB-INF/lib JAR files with a total size of 50MB or more.
In simple cases, there are two custom properties which have been added which modify the default CDI processing. See this page for information about the two properties:
http://www-01.ibm.com/support/docview.wss?uid=swg21983564
The two properties are to (1) disable CDI processing entirely; or (2) disable CDI processing for archives which do not contain a "beans.xml". See the linked page for additional details.
These two properties provide a benefit in many cases. They should be tried first, as other options are more complicated to try.
I want to ask a general question that I guess, people who are using WebSphere App Server have thought.
Can we deploy multiple EAR or WAR files at once by using a Jython script but only use one sync node and save command?
I have some concerns about boundaries like EAR file size and number of EARs but I have found nothing on it online yet.
Of course you can install/update multiple applications and then Save/Sync Node at once by using a wsadmin script like the following:
AdminApp.install(fullapppath1,params1)
AdminApp.install(fullapppath2,params2)
AdminConfig.save()
AdminNodeManagement.syncActiveNodes()
In fact this can have some performance benefits in contrast with save/sync operation for each app.
Regarding EAR max file size I do not know whether there are any WAS-relative limits except two issues:
Really Big EAR/WAR files can severelly slow down the deployment process
Watch for your OS max Number of files per process limits (i.e. nofile parameter in Linux). This can prevent the EAR expansion process in case you have many jars in your app.
I'm working in a JSF project with Eclipse and Glassfish 3.1.2.
Every time I did a minor change and save it, Glassfish do the hot deployment, but is taking too much time to do that, about 20s, at least.
Can I do something do deacrease this time ? Is horrible develop something where you have to wait all time time everytime I change something.
UPDATE
This is how my project settings.
Just open the folder below, this surprise me, is this so many .jar files, is this correct ?
And this one:
My Glassfish configuration :
Any idea ?
There are several ways to deploy a GlassFish application. To speed up development/debuging we need a way to instantly deploy web applications. One of the ways is to use hot deployment feature, another lesser known feature is ‘directory deployment’. You simply point GlassFish to your development directory and let it pick up and deploy application from there. No packaging and re-deoploying hassles. The catch is whenever you want to re-deploy your application you just need to touch a file called .reload which should be present in your web folder.
Following is the command and directory structure you can use.
–|myproj
–|–|src
–|–|web
–|–|–|WEB-INF
–|–|–|–lib
–|–|–|–classes
–|–|–|–web.xml
–|–|.reload
GLASS_FISH_HOME/bin/asadmin deploydir full_path_to_you_web_folder
When deploying my project to SpringSource dm Server, every once in a while a JAR fails to deploy with the following message:
/mnt/myproject/springsource/work/com.springsource.server.deployer/packed/my.project.0.1.10.M.jar' cannot be unpacked.
java.util.zip.ZipException: error in opening zip file
There are 5 .war files in the project. If one of them fails, it's always the same one (which is also the last one to be copied into the pickup directory). However, usually all 5 will deploy without issues. It is the exact same set of files in all instances, taken from a maven repository, just deployed to new server instances.
The file that fails can be opened just fine by 7-Zip. If I stop Spring, clear the pickup directory, start Spring and copy the .war files to pickup again, it will usually work.
The usual deployment process is:
Start Spring
Wait until it reports Open for business with profile 'web'
Copy all 5 projects with a 2 second delay between each copy (scripted).
Similar issues java-util-zip-zipexception-error-in-opening-zip-file and jboss5-cannot-deploy-due-to-java-util-zip-zipexception-error-in-opening-zip-fil do not seem to apply.
You don't say which version of dm Server you are running, so I would recommend upgrading to 2.0.x to pick up fixes if you haven't already. You may also like to upgrade to Eclipse Virgo which is the continuation of the dm Server project.
My guess is that the heuristic in dm Server for determining when a file copy into pickup has terminated is playing up, possibly due to a slow or irratic copy operation. Is there anything unusual about your disk, such as encryption or remote mount, which may interfere with the copy operation?
One way to rule out the heuristic would be to place the files in the pickup directory when dm Server is not running and then start dm Server when the copy operation has definitely completed. If the problem reproduces, then there may be a problem in the JRE you are using.
I need a windows shell script which will redeploy my .ear file. I wrote one:
net stop "JBAS50SVC"
copy /y O:\TEMP\app.ear C:\jboss-4.2.3.GA\server\default\deploy\app.ear
net start "JBAS50SVC"
But the problem is that when I try to stop the JBoss service it does not fully unload itself (my suggestion), so when I start it there's no app deployed in JBoss.
You mentioned that JBoss becomes very slow when you use autoDeploy and keep the server running for a month or more. That's likely to be because your permanent generation memory is filling up over time. See this related question.
I think you'd be best served by using the autoDeploy feature as other suggest, and restarting the entire server periodically to clear out PermGen. There's several ways to track PermGen utilization; I add the following to JAVA_OPTS in run.conf:
-XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -Xloggc:$LOGDIR/gc.log
Typically to redeploy an application that is self-contained in an ear in jboss just requires replacing the ear. If there is more to the application (such as classes that have to go in the jboss lib directory) then it may require a restart.
In terms of how to restart it, it depends what you are using to start jboss as a service. Some service starters do not actually control the jvm after they start it, so restarting the service just starts a second instance. You can shutdown jboss with the shutdown script in the JBOSS_HOME/bin directory (shutdown.bat) and then deploy and restart the service. Be sure to pause when you do this, as the shutdown command will issue the shutdown request, the server may still be running.
You can just COPY/RENAME the EAR file to deploy directory. JBoss will automatically take care of re-deployment for you.
As mentioned earlier jboss automatically deploys everything you copy into the deploy directory by default.
If not, check if you are working on a vhost that doesnt autodeploy ear files.
<Host name="vhost2" autoDeploy="false" .... />
other than that, I feel like this is a totally different error due to the fact that jboss takes care of deployment for you. Please check the log files for startup errors of your application.
You might want to check the JBoss community wiki on this, there are undoubtedly easier and more reliable methods than a batch file.
Stopping the server should not undeploy the application - that wouldn't make much sense.