weblogic does not find .ear in stage, but it is there on the file system - deployment

When starting the weblogic server I get this error:
Caused By: java.io.IOException: /home/weblogic/Oracle/Middleware/user_projects/domains/mydomain/servers/managed1/stage/myear/myear.ear (No such file or directory) with : /home/weblogic/Oracle/Middleware/user_projects/domains/epersonenstand/servers/managed1/stage/myear/myear.ear
<Failed to create App/Comp MBeans for AppDeploymentMBean myear. Error - weblogic.management.DeploymentException: /home/weblogic/Oracle/Middleware/user_projects/domains/epersonenstand/servers/managed1/stage/myear/myear/myear.ear.
But the ear file is present in the folder.
Tried to redeploy it while the server is running and while it is shut down, I get always the same error(and restarting the server including the OS). The same deployment did work earlier(tried different versions), the problems started after a restart of the server(weblogic was shut down properly).
The exact weblogic version is 12.2.1.3.0 running on SLES 12R2.
Does anybody has an idea how to solve the problem?
thanks in advance,
Ben

Does your domain have multiple machines configured? I've seem similar errors and got confused because checking the filesystem on the admin server machine showed the files existed, but when checking the machines of other servers (eg. managed1) the file didn't exist. The error message doesn't make it clear which machine the file can't be found on.

Related

Weblogic 12c - Managed Server is failing due to not reachable to admin server

Can you please help me how to bring my managed server running
1)created managed server and deployed application for first time.
2)After some time i could find my Managed Server is down, So started buts failing due to below issue .
Note :Admin Server is up and running .
I have weblogic 12.2.1.3 server with two domains as domain3 and domain4
domain3 is working fine, but in domain4 Managed server is failing
To replicate issue in domain4 i have created Machine and attached the Managed server , but in monitoring Node Manager status i could find inactive .
Can i Start Node Manager in two domains even weblogic server is same ?
http://10.25.10.11:7001.>
http://10.15.10.23:7001.>
A MultiException has 4 exceptions. They are:
1. weblogic.security.SecurityInitializationException: Authentication for user weblogic denied.
2. java.lang.IllegalStateException: Unable to perform operation: post construct on weblogic.security.SecurityService
3. java.lang.IllegalArgumentException: While attempting to resolve the dependencies of weblogic.nodemanager.adminserver.NodeManagerMonitorService errors were found
4. java.lang.IllegalStateException: Unable to perform operation: resolve on weblogic.nodemanager.adminserver.NodeManagerMonitorService
at org.jvnet.hk2.internal.Collector.throwIfErrors(Collector.java:89)
at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:250)
at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:358)
at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487)
at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:305)
Truncated. see log file for complete stacktrace
Caused By: weblogic.security.SecurityInitializationException: Authentication for user weblogic denied.
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(CommonSecurityServiceManagerDelegateImpl.java:1158)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.postInitialize(CommonSecurityServiceManagerDelegateImpl.java:1272)
at weblogic.security.service.SecurityServiceManager.postInitialize(SecurityServiceManager.java:586)
at weblogic.security.SecurityService.start(SecurityService.java:130)
at weblogic.server.AbstractServerService.postConstruct(AbstractServerService.java:76)
Truncated. see log file for complete stacktrace
Caused By: javax.security.auth.login.FailedLoginException: [Security:090938]Authentication failure: The specified user failed to log in. weblogic.security.providers.authentication.LDAPAtnDelegateException: [Security:090295]caught unexpected exception, No such object
at com.bea.common.security.utils.ExceptionHandler.throwFailedLoginException(ExceptionHandler.java:62)
at weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.login(LDAPAtnLoginModuleImpl.java:368)
at com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:117)
at java.security.AccessController.doPrivileged(Native Method)
at com.bea.common.security.internal.service.LoginModuleWrapper.login(LoginModuleWrapper.java:114)
Truncated. see log file for complete stacktrace
Each node manager is typically limited to a single machine. One domain can have one admin server and multiple managed servers on multiple machines.
However, you can have multiple machines configured in a single domain, so if you have two machines configured, you can have two node managers.
The following might help solve your issue:
Delete or make a backup of the LDAP folder under the /servers//data/ and restart the server.
Check if a firewall is blocking the ports. Check if the admin-URL is accessible from the managed server machine.
Try starting a managed server from startManagedWebLogic.cmd/sh . Try enabling debug logging in Weblogic console -> server-name -> debug -> weblogic -> security -> atn.

Deploy to remote glassfish from Netbeans fails

Trying to deploy to remote glassfish 4.1 (same as on my local machine) from Netbeans fails without any error message (that I could find).
When I do "Run project" from Netbeans 8.0.2, I get a remotetest (run) in the status bar (remotetest is the name of my project, which is just a "Hello from Servlets" default JSF project). When I click on the 1 more spot, it says it's deploying my .war to my remote server. After 20 minutes and 1 second, I get
Deployment error: Deployment timeout has exceeded. See the server log
for details."
(the first time I waited that long... actually right now, it's been "deploying" about 60 minutes and hasn't stopped).
There is nothing in the remote glassfish server log after glassfish server has finished startup, no sign of the deployment trial. This directly after rebooting my AWS EC2 instance.(I read somewhere that the micro instances might not be good for a full blown server, normally it seems that it's taking between 0 and 12% in top. What I do see, right now though, is the java process taking 75% cpu and 40% memory when the "deployment" has been going on for an hour or so, and it remained so after stopping the deployment from Netbeans end.)
What I can do after a dozen hours of setting things up and googling:
Deploy my app to local glassfish.
Connect to my remote glassfish admin page via port 4848
move the .war to remote .../autodeploy, see it deploy and have it visible remotely at myhost:8080/remotetest
connect to remote mysql via Netbeans (not sure if relevant)
ssh a basic java "hello world" from Netbeans to remote machine (not sure if relevant)
What I've tried:
opened up various ports for custom TCP rule from the EC2 management console , including 8080,8181,4848
tinkered with every possible switch from Netbeans and glassfish I could think of
Google my butt off
sleep over it
rinse and repeat
The glassfish server logs show two warnings after startup:
All SSL cipher suites disabled for network-listener(s). Using SSL
implementation specific defaults
and three SEVERE:
java.lang.ClassNotFoundException:
org.glassfish.admin.rest.resources.generatedASM.DomainResource not found by org.glassfish.main.admin.rest-service [97] at
org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1532)
at
org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:75)
at
org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1955)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at
java.lang.Class.forName0(Native Method) at ...
I can't make anything of those. Nothing in the log since I started the deployment a couple of minutes after the server was up and visible in Netbeans.

Apache tomcat start up failure

I have been working with netbeans...til yesterday night it worked but now i couldnt run even a simple code its showing the following comments in the output screen
init:
deps-module-jar:
deps-ear-jar:
deps-jar:
library-inclusion-in-archive:
library-inclusion-in-manifest:
compile:
compile-jsps:
Starting of Tomcat failed, the server port 8084 is already in use.
D:\Dhivya\Job\calculator\payodaproject\nbproject\build-impl.xml:683: Deployment error:
Starting of Tomcat failed, the server port 8084 is already in use.
See the server log for details.
BUILD FAILED (total time: 3 seconds)
how to recover from this
First this isn't a constructive question as there is obvious lack of research. You didn't mention whether you tried to find out if any process was using port 8084. You may use netstat command to figure that out.
This usually happens when there is a Tomcat process already running and for whatever reason Netbeans is not able to see it, and tries to start another. You have to kill the Tomcat process manually, then Netbeans will be able to start it again.
It's a netbeans bug for port 8084 and there are few ways to resolve it. One is to remove tcnative-1.dll from the apache-tomcat/bin folder in your installation. For more you can visit netbeans-bugzilla.

FATAL ERROR in native method: JDWP No transports initialized, jvmtiError=AGENT_ERROR_TRANSPORT_INIT(197)

Can you explain what this error is (and secondly why I am getting it)?
FATAL ERROR in native method: JDWP No transports initialized, jvmtiError=AGENT_ERROR_TRANSPORT_INIT(197)
P.S. It may be related to Known Tomcat 6.0 and JDK 1.7.0_02 issues?, as I only started getting it after upgrading from JDK 1.7.0 to 1.7.0 update 2, with no other upgrades to other software.
I am running:
Eclipse Indigo 3.7
JDK 1.7.0_0u2 (JDK 7 update 2)
Tomcat 6.0
Windows 7
Apache HTTP Server (although not using it yet)
When I start Tomcat I started getting this error, but not all the time. Rebooting just now fixed it. Some mornings I come to work without a reboot and it fixes it even though it failed the day before. It's sporadic. To debug this I need to understand. Can you help explain it?
EDIT : I have two Tomcat servers, for two different projects, on the same port. The other (first) server is "stopped" but remains "synchronized", in case this matters. I've quickly tried changing all the ports up one (8080 to 8081) and the error reproduces. This may not be a proper test of changing ports, however.
EDIT 2: I just had this problem, and rebooting "fixed" the issue. The workstation was on all weekend and Tomcat worked on Friday and Eclipse was shutdown at the end of the day. I will keep taking notes like this as I run into it to remove guesswork.
EDIT 3: Today it gave me this error from an unrebooted system that worked yesterday, programs shut down yesterday and restarted today. I rebooted, and the error is gone. Most notably is that the error always occurs at 23% compilation. It hits 23%, waits a bit and this is when I know it won't succeed, and then popups a window. I'll capture what the window says next time. Then it gives the above error to the Console.
EDIT 4: I am running Windows 7 and Apache HTTP Server (although not using it yet). I'll add these to the list above.
EDIT 5: The popup window mentioned in edit #3 is (and note my Tomcat is named Server Tomcat v6.0 Server at localhost):
Problem Occurred
'Server Tomcat v6.0 Server at localhost' has encountered a problem.
Server Tomcat v6.0 Server at localhost failed to start.
OK << Details
Server Tomcat v6.0 Server at localhost failed to start.
EDIT 6: I just got a new problem, which is Cannot connect to VM com.sun.jdi.connect.TransportTimeoutException popup window error and the same main error that this question asks about in the Console window.
EDIT 7: Just restarting Eclipse, not rebooting the whole computer, solved the error this morning.
This error typically comes up when the necessary port is taken by another program.
You said that you have changed the HTTP connector port from 8080 to 8081 so the two Tomcats do not clash, but have you also changed the <Server port="..." in tomcat/conf/server.xml to be different between your Tomcats?
Are there any other connectors ports which may possibly clash?
Does your HOSTS file have an entry for localhost? Some other situations this error is seen in seem to have this as a problem resolution.
Make sure you have 127.0.0.1 localhost set in it...
(from this and this)
Encountered this. all I did was to kill all the java process(Task Manager) and run again. It worked!
Check whether your config string is okay:
Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=9999
I just had this issue today, and in my case it was because there was an invisible character in the jpda config parameter.
To be precise, I had dos line endings in my setenv.sh file on tomcat, causing a carriage-return character right after 'dt_socket'
EDIT these lines in host file and it should work.
Host file usually located in C:\Windows\System32\drivers\etc\hosts
::1 localhost.localdomain localhost
127.0.0.1 localhost
I had the same problem because I set the following in Catalina.sh of my tomcat:
JAVA_OPTS="$JAVA_OPTS -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=9999"
After removing it, my tomcat worked well.
Hope help you.
Encountered this issue and changing the debug port helped. For some reason, the debug port had to be greater than the app port.
Change control panel Java's option about proxy to "direct", change window's internet option to not use proxy and reboot. It worked for me.
This error mostly comes when we forcefully kill the weblogic server ("kill -9 process id"), so before restart kindly check all the ports status which weblogic using e.g. http port , DEBUG_PORT etc by using this command to see which whether this port is active or not.
netstat –an | grep
(Admin: 7001 or something, Managed server- 7002, 7003 etc)
eg: netstat –an | grep 7001
If it returns value then,
option 1: wait for some time, so that background process can release the port
option 2: execute stopweblogic.sh
Option 3: Bounce the server/host or restart the system.
My issue was resolved by option 2.
if your JVM Cli is: -agentlib:jdwp=transport=dt_socket,address=60000,server=n,suspend=n and JDK version is 7, change "server=n" to "server=y" will be OK.
In my project I had the same error, I restarted Tomcat and it worked, withtout killing the java process.
I set 127.0.0.1 localhost, and solve this problem.
I had the same problem in Catalina.sh of my tomcat for JPDA Options:
JPDA_OPTS="-agentlib:jdwp=transport=$JPDA_TRANSPORT,address=$JPDA_ADDRESS,server=y,suspend=$JPDA_SUSPEND"
After removing JPDA option from my command to start the Tomcat server, I was able to start the server on local environment.
I was getting the same error when i switched to STS version 3.8.3
And imported my entire workspace to the new STS.
Apparently the "Boot Spring App" instance was defective. (i run from STS)
If this is your problem,
Simply create the Boot run configuration again.
In case you are working with environments or docker images you can really change /etc/host I recommend just changing the binding from star to 0.0.0.0.
So (basing on my case for instance) instead of:
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"
You would define it as:
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=0.0.0.0:5005"

jboss 5 startup time? [duplicate]

We upgraded from JBoss 4 (and JDK 5) to JBoss 5 (and JDK 6). The problem is that the start time has gone from 1.5 minutes (on JBoss 4) to more than 4 minutes.
18:53:35,444 INFO [ServerImpl] JBoss (Microcontainer) [5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221053)] Started in 3m:9s:262ms
It seems like the component that is taking JBoss the longest time to initialize is the JMX
18:50:41,926 INFO [LogNotificationListener] Adding notification listener for logging mbean "jboss.system:service=Logging,type=Log4jService" to server org.jboss.mx.server.MBeanServerImpl#1adc122[ defaultDomain='jboss' ]
18:52:38,797 INFO [JMXConnectorServerService] JMX Connector server: service:jmx:rmi://lharel2/jndi/rmi://lharel2:1090/jmxconnector
From the DEBUG server log, I get these lines at the problematic time:
2009-12-18 18:51:00,886 DEBUG [org.jboss.deployment.MappedReferenceMetaDataResolverDeployer] (main) vfsfile:/C:/QC/Views/QCDev/jboss-5.1.0.GA/server/default/deploy/jmx-console.war/ endpoint mappings:
2009-12-18 18:51:00,886 DEBUG [org.jboss.deployment.MappedReferenceMetaDataResolverDeployer] (main) Processing unit=jmx-console.war, structure: jmx-console.war
2009-12-18 18:52:35,209 DEBUG [org.jboss.deployment.OptAnnotationMetaDataDeployer] (main) Deployment is metadata-complete, skipping annotation processing, ejbJarMetaData=null, jbossWebMetaData=org.jboss.metadata.web.spec.Web23MetaData#1f, jbossClientMetaData=null, metaDataCompleteIsDefault=false
There is no EJB in the project.
The memory settings are:
-Xms128m -Xmx512m -XX:MaxPermSize=256m
Do you have any idea how JBoss start time can be improved?
Update: so far no luck, I tried shreeni's suggestion (changed the scanning xmls). The server is not running in debug mode so MicSim's suggestion is not relevant
A shot into the blue sky without more information
Network timeouts: 1,5 minutes of delay when deploying jmx-console.war may indicate a network timeout (e.g. 3 x 30 seconds). Try to start JBoss and bind it to a specific IP address using the -b command line argument or the jboss.bind.address system property. Also, try to make sure your host and DNS resolution settings on your system are correct.
JMX is also using RMI and you may want to set the RMI server host name as system property. On some Linux distributions, RMI has problems with looking up the correct hostname and jmx-console.war may try to connect to the 'wrong localhost'. The system property is java.rmi.server.hostname
System tracing: If that does not help, you may want to use strace to start the java process, so you can see the point where the system hangs (if it really does hang due to a network timeout or similar).
That is an awfully big gap in the logs. I suggest changing the log configuration to log everything at DEBUG level, rather than INFO. This will generate an awful lot more log entries, but hopefully will help you narrow it down.
The easiest way to do this is to set the -Djboss.server.log.threshold=DEBUG system property when you start JBoss
I had an issue like this but I found a good improvement by setting the initial and max heap size to same values, I mean:
-Xms512m -Xmx512m
With this, I improved from 4 to 2 minutes the starting time.
Your suspicion about the jmx-console can be misleading. There may be other components doing work in the background unrelated to the jmx-console. In my experience, we had an issue where a small war file appeared to take 3 minutes to load! It was innocent. The culprit was partly due to an EAR file with many wars and jars.
While I'm no expert, I would suggest the following:
Try turning up the logging to TRACE. By doing this, I witnessed one of the deployers (EJBDeployer, I think) unnecessarily scanning WARs in one of our EARs. I then manually disabled the scanning of those WAR files.
Run wireshark during the startup. I discovered some war files were hanging while waiting for a response from an external DTD request. Those websites were either now non-existent or would not properly serve the DTD files to java-based programs. I could speed it up by either having the programmers use local DTD files or mirroring those DTDs locally and having /etc/host loopback locally.
You could refer to this link to avoid unnecessary annotation scanning which could speedup your server start
See http://community.jboss.org/wiki/jboss5xtuningslimming especially the "Tuning" part.
I'm using JBoss 5.1.0 with a Macbook pro (2.26ghz 4gb) without applications it start in 54s
15:00:26,449 INFO [ServerImpl] JBoss (Microcontainer) [5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221634)] Started in 54s:720ms
I made a new configuration based on the "default". The JMS dataosurce points to a Postgres database instead of the "Hypersonic Database" (in memory database)
I suppose you are starting in debug mode. This mode can be up to 3 times slower than normal mode.
But there might be also a problem when switching from JDK5 to JDK6. I found this solution here on the net:
I've solved that. It's a debugging issue. I've changed my debug settings from:
wrapper.java.additional.26=-Xdebug
wrapper.java.additional.27=-Xnoagent
wrapper.java.additional.28=-Djava.compiler=NONE
wrapper.java.additional.27=-Xrunjdwp:transport=dt_socket,address=7199,server=y,suspend=n
to:
wrapper.java.additional.26=-Xdebug
wrapper.java.additional.27=-Xrunjdwp:transport=dt_socket,address=7199,server=y,suspend=n
and JBoss becomes fast again.
Hope this helps.
Do you need the JMX console application? Pragmatic thing would be to un-deploy it from the server, you could still use the jconsole or jvisualvm for basically the same thing.
Turn off annotation scanning and other features you don't need https://community.jboss.org/wiki/jboss5xtuningslimming
When you start/stop JBoss from eclipse, it does not clean up the tmp & work folders correctly. Setup an External Tool configuration and run a batch file to delete everything in tmp & work folders before each startup.
I was able to speedup running the "default" profile from 15/20 minutes to 5 minutes.