I'm trying to get JBoss EAP 6.2 running as a window service on Windows Server (64bit). I've define the service with prunsrv as follows:
prunsrv install "JBoss EAP" --DisplayName="JBoss EAP" --LogLevel=DEBUG --LogPath=d:\Java\jboss-eap-6.2\domain\log\ --LogPrefix=service --StdOutput=auto --StdError=auto --StopTimeout=10 --StartMode=exe --StartImage=cmd.exe --StartPath=d:\Java\jboss-eap-6.2\bin ++StartParams="/c;domain.bat" --StopMode=exe --StopImage=cmd.exe --StopPath=d:\Java\jboss-eap-6.2\bin ++StopParams="/c;jboss-cli.bat;--controller=192.168.50.3:8888;--connect;command=/host=master:shutdown"
and I changed the logon user for the service so that it runs under an account that is part of the Administators group.
I'm able to start the service just fine but I can't get it to stop properly. By "properly" I mean when I stop the service, I get the Service Control dialog that says:
"Windows is attempting to stop the following service on the local computer"
Watching the JBoss log file I can see that JBoss shuts down properly and with Task Manager I can see that all the Java.exe processes disappear. However, the Service Control dialog does not close 'normally'. It eventually times out and I get another dialog with "Error 1053: The service did not respond to the start or control request in a timely mannger" and Task Manager shows prunsrv.exe still running. The service is then hung up in a "Stopping" state and I have to use TaskKill to kill the task and reset the service to a state where I can restart it.
When I look in the service.*.log file, I see the following:
[2015-09-11 11:42:55] [debug] ( prunsrv.c:844 ) [25200] reportServiceStatusE: 4, 0, 0, 0
[2015-09-11 11:42:57] [debug] ( prunsrv.c:844 ) [25200] reportServiceStatusE: 4, 0, 0, 0
[2015-09-11 11:42:57] [debug] ( prunsrv.c:844 ) [25200] reportServiceStatusE: 3, 0, 3000, 0
[2015-09-11 11:42:57] [info] ( prunsrv.c:943 ) [10984] Stopping service...
[2015-09-11 11:42:57] [debug] ( prunsrv.c:1057) [10984] Waiting for stop worker to finish...
I have been trying for a couple of days to figure out what the problem is but I've hit a wall and I'm out of ideas. So far, LogLevel=DEBUG hasn't shed any light on the problem so I'm looking for ideas that might help me debug this. Can anyone give me a suggestion?
Thanks
As it turned out, the problem was with the startup command I was using. The startup command needed a "set NOPAUSE=Y" at the front of it. E.g.
++StartParams="/c;set;NOPAUSE=Y;&&;domain.bat"
Once I added that, it did the trick.
Related
i observed strange problem suddenly on one of our QA machine.
We keep tomcat started using sysdeo eclipse plugin on windows server.
Application was in use for few days, working fine. No changes done on server.
Suddenly we observed today http://localhost:8080 stuck and not giving response.
Taskmanager found 2 tomcat process running with same no of threads around 1500.
PID 12309 with command line property :
"..\javaw.exe" -agentlib:jdwp=transport=dt_socket,suspend=y,address=localhost:52742 "-Dcatalina.home=..\apache-tomcat-7.0.50" "-Djava.endorsed.dirs=..\apache-tomcat-7.0.50\endorsed" "-Dcatalina.base=..\apache-tomcat-7.0.50" "-Djava.io.tmpdir=..\apache-tomcat-7.0.50\temp" -Xmx25000m -Dfile.encoding=Cp1252 -classpath "..\apache-tomcat-7.0.50\bin\bootstrap.jar;..\apache-tomcat-7.0.50\bin\tomcat-juli.jar" org.apache.catalina.startup.Bootstrap start
PID 56098 with command line property :
"..\javaw.exe" -agentlib:jdwp=transport=dt_socket,suspend=y,address=localhost:61684 "-Dcatalina.home=..\apache-tomcat-7.0.50" "-Djava.endorsed.dirs=..\apache-tomcat-7.0.50\endorsed" "-Dcatalina.base=..\apache-tomcat-7.0.50" "-Djava.io.tmpdir=..\apache-tomcat-7.0.50\temp" -Xmx25000m -Dfile.encoding=Cp1252 -classpath "..\apache-tomcat-7.0.50\bin\bootstrap.jar;..\apache-tomcat-7.0.50\bin\tomcat-juli.jar" org.apache.catalina.startup.Bootstrap start
Please give pointers on solving this problem.
I have a Camel application deployed on JBoss in a WAR file with a spring configuration for starting the Camel context.
It deploys and runs very nicely on a JBoss EAP 7.0.0.GA.
If I want to change values in a property file that my application depends on and touch the war file, it normally redeploys the application. But in some cases it fails.
I get the following in the server.log:
2017-07-25 12:05:26.671 INFO class=org.apache.camel.impl.DefaultShutdownStrategy thread="ServerService Thread Pool -- 74" Starting to graceful shutdown 12 routes (timeout 300 seconds)
2017-07-25 12:05:26.725 INFO class=org.apache.camel.impl.DefaultShutdownStrategy thread="Camel (interfacedb) thread #2 - ShutdownTask" Waiting as there are still 4 inflight and pending exchanges to complete, timeout in 300 seconds. Inflights per route: [interfacePersistDirect = 1, route1 = 1, pullFromTransferEntityTable = 1, lastScheduledRun = 1]
...
2017-07-25 12:10:26.691 WARN class=org.apache.camel.impl.DefaultShutdownStrategy thread="ServerService Thread Pool -- 74" Timeout occurred during graceful shutdown. Forcing the routes to be shutdown now. Notice: some resources may still be running as graceful shutdown did not complete successfully.
2017-07-25 12:10:26.691 WARN class=org.apache.camel.impl.DefaultShutdownStrategy thread="Camel (interfacedb) thread #2 - ShutdownTask" Interrupted while waiting during graceful shutdown, will force shutdown now.
2017-07-25 12:10:26.694 INFO class=org.apache.camel.impl.DefaultShutdownStrategy thread="ServerService Thread Pool -- 74" Graceful shutdown of 12 routes completed in 300 seconds
After this the application will not start again. JBoss reports the following in the myApp.war.failed file in the deployments folder.
"WFLYDS0022: Did not receive a response to the deployment operation within the allowed timeout period [600 seconds]. Check the server configuration file and the server logs to find more about the status of the deployment."
The application normally deploys a lot quicker than 600 seconds. I can touch the war file or delete the .failed file, which normally triggers a redeployment, but JBoss keeps giving me the error above in the .failed file.
The application starts normally if I restart the JBoss VM, but I would like to avoid restarting the other applications running on the JBoss instance.
Any suggestions?
Environment:
Glassfish 4.0 (only one DAS), Windows Server 2012 R2, Java 1.7.0_51
Create the DAS instance service by using the create-service subcommand.
Issue:
The maximum history files attribute has been set, however, Glassfish Server couldn’t remove the old log files due to the lock file server.log.lck
Path --> C:\glassfish4\glassfish\domains\domain1\config\logging.properties
com.sun.enterprise.server.logging.GFFileHandler.maxHistoryFiles=10
Log Snippet:
[2014-12-10T18:00:39.372+0900] [glassfish 4.0] [SEVERE] [] [] [tid: _ThreadID=16 _ThreadName=Thread-5] [timeMillis: 1418202039372] [levelValue: 1000] [[
java.util.logging.ErrorManager: 0: FATAL ERROR: COULD NOT DELETE LOG FILE.]]
[2014-12-10T18:00:39.372+0900] [glassfish 4.0] [SEVERE] [] [] [tid: _ThreadID=16 _ThreadName=Thread-5] [timeMillis: 1418202039372] [levelValue: 1000] [[
java.io.IOException: Could not delete log file: C:\glassfish4\glassfish\domains\domain1\logs\server.log.lck
at com.sun.enterprise.server.logging.GFFileHandler.cleanUpHistoryLogFiles(GFFileHandler.java:725)
at com.sun.enterprise.server.logging.GFFileHandler$4.run(GFFileHandler.java:802)
at java.security.AccessController.doPrivileged(Native Method)
at com.sun.enterprise.server.logging.GFFileHandler.rotate(GFFileHandler.java:744)
at com.sun.enterprise.server.logging.GFFileHandler$1.run(GFFileHandler.java:301)
at com.sun.enterprise.server.logging.LogRotationTimerTask.run(LogRotationTimerTask.java:68)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)]]
Findings:
1, If the lock file “server.log.lck” exists in the log folder, the issue occurred, and can find the above errors in the log every day when Glassfish server tries to remove the old log files. If there is no “server.log.lck” in the log folder, no any issue and work properly.
2, If the DAS instance is started by the command “asadmin start-domain domain1”, there is no lock file “server.log.lck” generated in the log folder. But if the DAS instance is started in Windows Service, the lock file “server.log.lck” will be generated automatically and keep 0KB until stop the service, this file will be removed automatically.
3, If the DAS instance is started by the command “asadmin start-domain -w domain1” which adds the watchdog option, the lock file “server.log.lck” will be generated automatically and exist until stop the service.
4, When the lock file “server.log.lck” appears, there is always one more java.exe process existing. Therefore, when start the DAS instance from Windows Service, there are two “java.exe” running in the process and “server.log.lck” is using by one of them.
Questions:
1, I’d like to start/stop the DAS instance by Windows Service, not using the subcommand. Moreover, I don’t want to keep all Glassfish logs on my server and it will cause a disk full issue so that I would prefer to turn on the Glassfish Logging Maximum history Files option. Is there any workaround or solution for that?
2, Is this a defect of Glassfish or it’s just a setting issue? I did try to install on other servers and all had the same issue.
3, Why there are two java.exe processes running if started from Windows Server, is the 2nd one used for “watchdog”?
Thank you so much for your help and please let me know if there is any further information you’d like to know or want me to do some other tests.
In case someone is still struggling I found a solution.
When you create a GF service in Windows environment via asadmin create-service GF creates a file domain1Service.xml in glassfish\domains\domain1\bin which contains parametres for server to start.
It looks something like the following
<service>
<id>domain1</id>
<name>domain1 GlassFish Server</name>
<description>GlassFish Server</description>
<executable>C:/Supertel-NMSv3/glassfish-4.1/glassfish/lib/nadmin.bat</executable>
<logpath>C:\\Supertel-NMSv3\\glassfish-4.1\\glassfish\\domains/domain1/bin</logpath>
<logmode>reset</logmode>
<depend>tcpip</depend>
<startargument>start-domain</startargument>
<startargument>--watchdog</startargument>
<startargument>--domaindir</startargument>
<startargument>C:\\Supertel-NMSv3\\glassfish-4.1\\glassfish\\domains</startargument>
<startargument>domain1</startargument>
<stopargument>stop-domain</stopargument>
<stopargument>--domaindir</stopargument>
<stopargument>C:\\Supertel-NMSv3\\glassfish-4.1\\glassfish\\domains</stopargument>
<stopargument>domain1</stopargument>
</service>
the line <startargument>--watchdog</startargument> is responsible for launching watchdog process which prevents log file from being deleted.
You can't just delete this startargument section (the service won't start) but you can switch this off by setting false flag like this
<startargument>--watchdog=false</startargument>
After that the service will start like via manual start-domain command without watchdog process.
You should do it after every service creation and it could be pretty annoying so I did further research.
It turns out that asadmin create OS specific domainService.xml by using templates located in glassfish\lib\install\templates Those templates also OS specific. And template for Windows (named Domain-service-winsw.xml.template) looks like this
<service>
<id>%%%NAME%%%</id>
<name>%%%DISPLAY_NAME%%%</name>
<description>GlassFish Server</description>
<executable>%%%AS_ADMIN_PATH%%%</executable>
<logpath>%%%LOCATION%%%/%%%ENTITY_NAME%%%/bin</logpath>
<logmode>reset</logmode>
<depend>tcpip</depend>
<startargument>%%%START_COMMAND%%%</startargument>
<startargument>--watchdog</startargument>
%%%CREDENTIALS_START%%%%%%LOCATION_ARGS_START%%%<startargument>%%%ENTITY_NAME%%%</startargument>
<stopargument>%%%STOP_COMMAND%%%</stopargument>
%%%CREDENTIALS_STOP%%%%%%LOCATION_ARGS_STOP%%%<stopargument>%%%ENTITY_NAME%%%</stopargument>
</service>
So you can edit template directly by setting param --watchdog=false and this change will reflect in all future created files domainService.xml
Hope it helps.
That’s not the right solution. Watchdog has an important function: it monitors whether the service is running or not. Without watchdog, glassfish is started correctly, but shortly afterwards the system no longer knows if the service is still running or maybe crashed. In the Services GUI, only the “start” button is active (always!). A “stop” and “restart” cannot be used.
A right solution would be the possibility to change the path to the lock file.
I am calling a script within main script to start the jboss sever after releasing the build on server.it is successfully starting the JBOSS but showing the below output in server/log/ server.log file and at the console output which is hanged.
To run the next build i need to kill this manually which is not appropriate.
05:04:17,373 INFO [AjpProtocol] Starting Coyote AJP/1.3 on ajp-0.0.0.0-8209
05:04:17,451 INFO [ServerImpl] JBoss (Microcontainer) [5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221053)] Started in 2m:38s:444ms
05:04:20,912 WARN [PropertyMessageResources] Resource MessageResources_en_US.properties Not Found.
05:04:20,913 WARN [PropertyMessageResources] Resource MessageResources_en.properties Not Found.
Help would be really highly appreciable.
Thanks.
By default, when you start your jboss server, it is not started as a background process and the console just sits there with the logs when server is started, that must be the reason why your script seems to hang , in reality it is just awaiting console output from the server.
To start jboss as background operation, replace the lines of code where you fire the run.sh in startup script with
nohup /path/to/jboss_home/jboss/bin/run.sh -b0.0.0.0 > /tmp/logs/jbosslogs.log &
This should start jboss in the background and redirect all startup logs to jbosslogs.log file. Since it is a background process, it will not hang at all.
I am having web application running in JBOSS AS 4.2.2.
Observed that jboss server automatically shuts down, and the following exception is observed in server.log
14:20:38,048 INFO [Server] Runtime shutdown hook called, forceHalt: true
14:20:38,049 INFO [Server] JBoss SHUTDOWN: Undeploying all packages
I want to enable TRACE for org.jboss.system.server.Server in jboss-log4j.xml, to hopefully get some more info when the server shuts down.
Please let me know how to enable TRACE for org.jboss.system.server.Server in jboss-log4j.xml.
I was able to add trace for server log and i could see the following output when JBOSS AS shuts down automatically:
2010-06-09 19:07:46,631 DEBUG [org.jboss.wsf.stack.jbws.RequestHandlerImpl] END handleRequest: jboss.ws:context=hpnp_lqs,endpoint=APIWebService
2010-06-09 19:07:46,631 DEBUG [org.jboss.ws.core.soap.MessageContextAssociation] popMessageContext: org.jboss.ws.core.jaxws.handler.SOAPMessageContextJAXWS#3290a11e (Thread http-0.0.0.0-8080-1)
2010-06-09 19:07:55,895 INFO [org.jboss.system.server.Server] Runtime shutdown hook called, forceHalt: true
2010-06-09 19:07:55,895 TRACE [org.jboss.system.server.Server] Shutdown caller:
java.lang.Throwable: Here
at org.jboss.system.server.ServerImpl$ShutdownHook.shutdown(ServerImpl.java:1017)
at org.jboss.system.server.ServerImpl$ShutdownHook.run(ServerImpl.java:996)
2010-06-09 19:07:55,895 INFO [org.jboss.system.server.Server] JBoss SHUTDOWN: Undeploying all packages
If anybody, has any clue, on what might be cause for automatic shutdown, pls help me.
Thanks!
There's a JBoss wiki page listing log output for various shutdown causes. It looks like yours was caused by a Ctrl-C. I assume you would have known if you hit Ctrl-C, though.
On unix-type servers, Ctrl-C generates a TERM signal, which could also come from someone or some script running as your jboss user or as root executing "kill <jboss pid>". If you're on linux I'd take a look at this question about the OOM killer.
One possible cause for this behaviour is console logout. We have observed this with our own server.
In brief, by default the Sun JVM listens to the event of the console user logging out, and shuts itself down automatically when that happens. To disable this, start the JVM with the -Xrs parameter.
See here for more details (look for Mysterious shutdowns).
One possible cause for a forced shutdown is if the virtual machine is out of memory.
I had this problem several years ago when a colleague implemented some very nasty bulk loading of objects from a database which caused jboss to shutdown on certain requests.
Try searching for "memory" or similar keywords in the log file and/or monitor the memory usage of the server.