It used to take 3 minutes to deploy our WAR to WebSphere 8.5.5.0.
It takes 3 hours to deploy the same WAR to WebSphere 9.0.0.2.
Is that a manifestation of a bug in WebSphere 9.0.0.2?
Did anybody have similar experience?
UPDATE:
Results of my experiments with settings suggested by IBM:
http://www-01.ibm.com/support/docview.wss?uid=swg21983564
Our WAR file contains 160 JAR files with total size of 160MB.
Time is measured for the "New Application" wizard, from first screen, until the next screen is shown.
WebSphere 8.5.5 - 3 minutes
WebSphere 9 - 3 hours
Added "Ignore-Scanning-Packages: org, net, com, javax" line to manifest file in the WAR - 30 minutes
Added "Enable-CDI: false" line to manifest file in the EAR - 30 minutes
Added lines
Enable-CDI=false
Enable-Implicit-Bean-Archive=false
to C:\Program Files\IBM\WebSphere\AppServer\properties\amm.filter.properties file - 30 minutes
Added JVM properties
com.ibm.ws.cdi.enableImplicitBeanArchives
false
com.ibm.ws.cdi.enableCDI
false
in Servers->Server Types->WebSphere application servers->server1->Server Infrastrucure->Java and Process Management->Process definition->Java Virtual Mashine->Custom properties - 30 minutes
Added JVM properties
-Dcom.ibm.ws.cdi.enableImplicitBeanArchives=false
-Dcom.ibm.ws.cdi.enableCDI=false
in Servers->Server Types->WebSphere application servers->server1->Server Infrastrucure->Java and Process Management->Process definition->Java Virtual Mashine->Generic JVM arguments - 30 minutes
UPDATE: SystemOut.log contains message:
Thread "WebContainer : ... has been active for 668644 milliseconds and may be hung.
Stack Trace contains:
at org.apache.commons.fileupload.MultipartStream.readBodyData
It looks like WebSphere is processing HTTP request to upload a file.
A preliminary check: The application is presumed to be exactly the same for both deployments. This is more-or-less assumed, but there have been a lot of cases where the application was presumed to be the same but it wasn't, so this needs to be verified.
Between IBM WebSphere Application Server v8.5.5.0 and v9.0, a significant upgrade was made to the CDI level which is supported. That is, v9.0 adds support for CDI 1.2, which has new class and annotation scanning requirements. These new requirements, together with some problems in the initial class and scanning implementation which is used by CDI in v9.0 can cause a very large slowdown of application deployments.
The problem has been noted in particular with medium to large web modules. Here "medium to large" means web modules which have many WEB-INF/lib JAR files with a total size of 50MB or more.
In simple cases, there are two custom properties which have been added which modify the default CDI processing. See this page for information about the two properties:
http://www-01.ibm.com/support/docview.wss?uid=swg21983564
The two properties are to (1) disable CDI processing entirely; or (2) disable CDI processing for archives which do not contain a "beans.xml". See the linked page for additional details.
These two properties provide a benefit in many cases. They should be tried first, as other options are more complicated to try.
Related
We are deploying 3 webapps in a liberty server (16.0.0.4) and frequently get ziperrors and messages like below (this is fairly easy to reproduce):
[ERROR ] CWWKZ0002E: An exception occurred while starting the application XYZ. The exception message was: com.ibm.ws.container.service.state.StateChangeException: java.util.zip.ZipError: jzentry == 0,
jzfile = 693877616,
total = 1148,
name = C:\opt\IBM\wlp\usr\servers\XYZ\workarea\org.eclipse.osgi\220\data\cache\com.ibm.ws.classloading.sharedlibrary_84.cache\lib\db2jcc.jar,
i = 329,
message = null
It seems to be erratic but easier to reproduce on slower machines so I suspect a race condition, but it may be as simple as the cache clearing during a server Clean is non-blocking?
Deploying the webapps as war files rather than linked via xml files back to projects does not experience this problem.
I've used the beans.xml with bean-discovery-mode="all" with no effect. We are using injection of different classes in two of the three web applications.
Note the directory number in the path to the cache differs from run to run.
This has been present since at least version 16.0.0.2 of Liberty. Is there a workaround for this problem or does anyone know if will be fixed in the December release?
The problem has gone away with migrating projects to newer releases of Eclipse, so closing.
I want to ask a general question that I guess, people who are using WebSphere App Server have thought.
Can we deploy multiple EAR or WAR files at once by using a Jython script but only use one sync node and save command?
I have some concerns about boundaries like EAR file size and number of EARs but I have found nothing on it online yet.
Of course you can install/update multiple applications and then Save/Sync Node at once by using a wsadmin script like the following:
AdminApp.install(fullapppath1,params1)
AdminApp.install(fullapppath2,params2)
AdminConfig.save()
AdminNodeManagement.syncActiveNodes()
In fact this can have some performance benefits in contrast with save/sync operation for each app.
Regarding EAR max file size I do not know whether there are any WAS-relative limits except two issues:
Really Big EAR/WAR files can severelly slow down the deployment process
Watch for your OS max Number of files per process limits (i.e. nofile parameter in Linux). This can prevent the EAR expansion process in case you have many jars in your app.
Problem: A 10 to 15 minute delay in WebSphere application deployments.
Environment/Situation: WebSphere 6.1.0.23, 90MB ear files containing about 19,000 files (ear file contains jar libraries). The ear file, WebSphere, and the automation driving the deployment are all on the same box. No EJBs. There are about 20 deployed applications like this on this box with 10 of them usually running.
Details: The deployment is automated, and the message 'ADMA5013I: Application ... installed successfully' is received. A few moments later, the directory is created (blah.ear/blah.war), but the directory remains empty for 10 to 15 minutes. Except for this specific delay, the performance on the box is fine and CPU utilization is very low. Once the files start getting created, they all show up in under a minute. Steps before and after this step run at an acceptable speed. It's just this one step, waiting for the files to show-up that's the problem.
Additional Details (precipitated by comments, below): WebSphere ND as evidenced by "Deployment Manager", and "Node Agent" in the logs. The ear contains one war file, one application. By using shared a library definition, the size of the ear was reduced to 60MB. WebSphere itself is started with JVM option -XX:MaxPermSize=256M. The deployments are done using the tools in the com.ibm.websphere.management.* packages (jar file supplied by IBM), primary class is "AdminClient". The code is similar to what is in this IBM documentation WS UI entry [System Administration > Console Preferences > "Synchronize changes with nodes"] was checked, but still sits for 15 minutes 'without doing anything'.
Just guesses:
ear files timestamp is in the past. Might be caused by jvm time-zone problems, different time zones when building/deploying (esp. if different machines used).
JDK could determine is there enough disk space
WSAD of some tool do not close file descriptor. Try deploy with anti-viruses paused.
Could you please describe situation in more details:
does build and deploy on the same machine?
does it work the same on all circumstances?
I am using version 3.0.0 of STS, and whenever I open a load of projects or do a build it is giving the message:
validating element <element here> with rule '#required property rule'
or similar messages
It is taking around two or three seconds per bean, and is going through every class file in my hierarchy. This can take around 45 minutes for larger projects.
Are there any configuration settings which could help with this?
It could be a lack of memory causing the problem. I'd try bumping up your Xmx to 1024m (default is 768m). You can change this setting in your STS.ini. It is colocated with the STS executable.
When deploying my project to SpringSource dm Server, every once in a while a JAR fails to deploy with the following message:
/mnt/myproject/springsource/work/com.springsource.server.deployer/packed/my.project.0.1.10.M.jar' cannot be unpacked.
java.util.zip.ZipException: error in opening zip file
There are 5 .war files in the project. If one of them fails, it's always the same one (which is also the last one to be copied into the pickup directory). However, usually all 5 will deploy without issues. It is the exact same set of files in all instances, taken from a maven repository, just deployed to new server instances.
The file that fails can be opened just fine by 7-Zip. If I stop Spring, clear the pickup directory, start Spring and copy the .war files to pickup again, it will usually work.
The usual deployment process is:
Start Spring
Wait until it reports Open for business with profile 'web'
Copy all 5 projects with a 2 second delay between each copy (scripted).
Similar issues java-util-zip-zipexception-error-in-opening-zip-file and jboss5-cannot-deploy-due-to-java-util-zip-zipexception-error-in-opening-zip-fil do not seem to apply.
You don't say which version of dm Server you are running, so I would recommend upgrading to 2.0.x to pick up fixes if you haven't already. You may also like to upgrade to Eclipse Virgo which is the continuation of the dm Server project.
My guess is that the heuristic in dm Server for determining when a file copy into pickup has terminated is playing up, possibly due to a slow or irratic copy operation. Is there anything unusual about your disk, such as encryption or remote mount, which may interfere with the copy operation?
One way to rule out the heuristic would be to place the files in the pickup directory when dm Server is not running and then start dm Server when the copy operation has definitely completed. If the problem reproduces, then there may be a problem in the JRE you are using.