Wildfly not starting properly - deployment

So I have a rather strange issue with wildfly not starting...
If I clean the standalone/deployments of everything but one .war file, wildfly starts perfectly. I can then add in all other .war files(6 in total) and wildfly deploys them without issues.
However if I have all the war files in there and start wildfly it completely fails. It stays in a state where everything is set to .isdeploying for maybe 5 minutes until everything gets set to failed.
The logs that I am getting from service wildfly status
Feb 09 08:49:12 wildfly[2079]: /etc/init.d/wildfly: 3: /etc/default/wildfly: default: not found
Feb 09 08:49:12 wildfly[2079]: * Starting WildFly Application Server wildfly
Feb 09 08:49:43 wildfly[2079]: ...done.
Feb 09 08:49:43 wildfly[2079]: * WildFly Application Server hasn't started within the timeout allowed
Feb 09 08:49:43 wildfly[2079]: * please review file "/var/log/wildfly/console.log" to see the status of the service
Has anyone seen anything like this before?
After looking aroung I found this just before it undeployed everything:
ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated t
he service container was 'add' at address '[
("core-service" => "management"),
("management-interface" => "http-interface")
]'
But I am still not sure what i means...

This happened to me too starting on WildFly 11 and above IIRC.
Are you trying to access to the public or management IP while the server is booting? Basically you have to wait until the server has started to access those IPs.
My workaround was use the marker files that deployment scanner checks.
https://docs.jboss.org/author/display/WFLY/Application+deployment#Applicationdeployment-MarkerFiles
Before you start wildfly you have to put a .skipdeploy file for each .war you want to skip. Then, when the server has started you only have to delete that file to let wildfly start the deployment. You can achieve this making a shell script and calling it from your standalone.sh

This error shows that your IP/Port is being used by another process.
Use below command to check it.
For windows: use netstat -aon | find "port number"

You can configure jboss.as.management.blocking.timeout system property to tune timeout (seconds) waiting for service container stabilityas below :
...
</extensions>
<system-properties>
<property name="jboss.as.management.blocking.timeout" value="900"/>
</system-properties>
<management>
...
Or, if still doesn't work this way, collect a series of thread dumps during your startup period so we can see what it might be getting stuck on.

Related

Timeout deploying an artifact that has deployed many times under JBoss EAP 7.3

I have been running PAM 7.9/JBPM 7.48 for about a year under JBOSS EAP 7.3. My JBPM's KieServer persists using SQL Server. I repeatedly deployed the KieServer yesterday but deploying today fails.
2021-12-16 15:25:53,645 ERROR [org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 1) WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'full-replace-deployment' at address '[]'
2021-12-16 15:26:03,649 ERROR [org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 1) WFLYCTL0190: Step handler org.jboss.as.server.deployment.DeploymentHandlerUtil$4#74e289e9 for operation full-replace-deployment at address [] failed handling operation rollback -- java.util.concurrent.TimeoutException: java.util.concurrent.TimeoutException
at org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:523)
at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1518)
I have already set the property to increase the timeout for the deployment but it still complains about a 5 second timeout that must be controlled by another property
2021-12-16 13:40:47,039 ERROR [org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 1) WFLYCTL0349: Timeout after [5] seconds waiting for service container stability while finalizing an operation. Process must be restarted. Step that first updated the service container was 'deploy' at address '[("deployment" => "kie-server.war")]'
I have changed the logging level to trace in order to gain all the information I can. How else can I debug / solve this issue?
There are two factors that may be contributing to this, but I don't have a good approach for addressing them.
There was a Windows Update yesterday (likely due to the recent Log4j exploit)
Some people at my company are having problems connecting to the SQL Server database. I am not seeing log messages about KieServer being unable to connect to the DB, but when it cannot reaching the DB the KieServer fails to start.

Jasper Report Server not working

I have a problem with the installation of Jasper Server v. 6.4.2 on Redhat 7.2
redhat-release-server-7.5-8.el7.x86_64.
My Tomcat is running and my postgresql
[root# jasperreports-server-cp-6.4.2]# ./ctlscript.sh restart
Using CATALINA_BASE: /opt/jasperreports-server-cp-6.4.2/apache-tomcat
Using CATALINA_HOME: /opt/jasperreports-server-cp-6.4.2/apache-tomcat
Using CATALINA_TMPDIR: /opt/jasperreports-server-cp-6.4.2/apache-tomcat/temp
Using JRE_HOME: /opt/jasperreports-server-cp-6.4.2/java
Using CLASSPATH: /opt/jasperreports-server-cp-6.4.2/apache-tomcat/bin/bootstrap.jar:/opt/jasperreports-server-cp-6.4.2/apache-tomcat/bin/tomcat-juli.jar
Using CATALINA_PID: /opt/jasperreports-server-cp-6.4.2/apache-tomcat/temp/catalina.pid
Tomcat stopped.
/opt/jasperreports-server-cp-6.4.2/apache-tomcat/scripts/ctl.sh : tomcat stopped
waiting for server to shut down.... done
server stopped
/opt/jasperreports-server-cp-6.4.2/postgresql/scripts/ctl.sh : postgresql stopped
waiting for server to start.... done
server started
/opt/jasperreports-server-cp-6.4.2/postgresql/scripts/ctl.sh : postgresql started at port 5432
Using CATALINA_BASE: /opt/jasperreports-server-cp-6.4.2/apache-tomcat
Using CATALINA_HOME: /opt/jasperreports-server-cp-6.4.2/apache-tomcat
Using CATALINA_TMPDIR: /opt/jasperreports-server-cp-6.4.2/apache-tomcat/temp
Using JRE_HOME: /opt/jasperreports-server-cp-6.4.2/java
Using CLASSPATH: /opt/jasperreports-server-cp-6.4.2/apache-tomcat/bin/bootstrap.jar:/opt/jasperreports-server-cp-6.4.2/apache-tomcat/bin/tomcat-juli.jar
Using CATALINA_PID: /opt/jasperreports-server-cp-6.4.2/apache-tomcat/temp/catalina.pid
Tomcat started.
/opt/jasperreports-server-cp-6.4.2/apache-tomcat/scripts/ctl.sh : tomcat started
but went I try to connect to http://localhost:8080/jasperserver-pro I have this Message
HTTP Status 404 – Not Found
Type Status Report
Message /jasperserver-pro
Description The origin server did not find a current representation for the target resource or is not willing to disclose that one exists.
Apache Tomcat/8.5.20
First of all just check the whether your war file (tomcat> webapps> jasperserver-pro) is there or not.
Sometimes it does get corrupt and deleted while you are running a js-install sh script for database setup.
So while setting up jasperserver, before running js-install sh (/buildomatic/) make one directory in at /buildmomatic/install_resources with name "war" and copy your war file from tomcat> webapps> to this directory. otherwise, jasperverserver will through error (/war/jasperserver not found).

Concurrent Timeout exception on starting Jboss Wildfly 9.02 server

I am new to jboss server. When I am trying to deploy .war file on server the following exception gets print on console:
6:38:04,388 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'add' at address '[
("core-service" => "management"),
("management-interface" => "http-interface")
]'
16:38:05,642 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-4) WFLYJCA0019: Stopped Driver service with driver-name = Aerobay.war_com.mysql.jdbc.Driver_5_1
16:38:09,548 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.server.DeployerChainAddHandler$FinalRuntimeStepHandler#5f88823f for operation {"operation" => "add-deployer-chains","address" => []} at address [] failed handling operation rollback -- java.util.concurrent.TimeoutException: java.util.concurrent.TimeoutException
at org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:396)
at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1384)
at org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1332)
at org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1292)
at org.jboss.as.controller.AbstractOperationContext$Step.access$300(AbstractOperationContext.java:1180)
at org.jboss.as.controller.AbstractOperationContext.handleContainerStabilityFailure(AbstractOperationContext.java:964)
at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:590)
at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:354)
at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:330)
at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1183)
at org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:453)
at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:327)
at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:313)
at org.jboss.as.server.ServerService.boot(ServerService.java:384)
at org.jboss.as.server.ServerService.boot(ServerService.java:359)
at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:271)
at java.lang.Thread.run(Thread.java:745)
Thanks in advance for the help !
I had the same problem when I tried to deploy the WAR file on my Red Hat Jboss EAP 7.0.
But the server was integrated into my IDE (Eclipse Neon) and the problem only occured in Debug-Modus.
I was able to solve the problem by removing all breakpoints and after that i started the server again.
Try increasing timeout by adding java option "blocking.timeout". You can do it in bin/standalone.conf.bat (depends on how you configure wildfly) by adding line:
set "JAVA_OPTS=%JAVA_OPTS% -Djboss.as.management.blocking.timeout=600
Change the number if it's not enough.
increasing the timeout doesn't solve the root cause of the problem. You need to check the cause of the time of the block and solve the issue. Maybe in some cases the solution is to increase the timeout.
In most cases, increasing resources is a bad way to solve issues. I had this case, the Wildfly took a lot of time to boot. I increased the timeout to 600 and solved the issue but was still having issue with the wildfly booting time which was so annoying.
2018-03-26 07:50:36,523 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'add' at address '[("path" => "xxxxxxxxxxxxxxxx")]'
Finally I checked the block cause in and found the block was due to network host resolving (NAS storage defined as a path in wildfly).
I jumped to the network setting and found that my local DNS was not set properly. I added the local DNS instead of the public DNS and the block issue was gone. Hope this helps
Regards
Sleem
When i tried to debug and started the server with debug mode got the following error:
16:19:50,096 ERROR [org.jboss.as.controller.management-operation] (management-handler-thread - 1) JBAS013412: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'deploy' at address '[("deployment" => "ViprWeb.war")]'
16:19:50,096 ERROR [org.jboss.as.server] (management-handler-thread - 1) JBAS015870
16:20:00,117 ERROR [org.jboss.as.controller.management-operation] (management-handler-thread - 1) JBAS013413: Timeout after [5000] seconds waiting for service container stability while finalizing an operation.
I removed all my breakpoints and restarted my server jboss and it resolved the issue.
just increase time out in standalone.conf.bat
set as set "JAVA_OPTS=%JAVA_OPTS% -Djboss.as.management.blocking.timeout=600
It worked for me.
I had the same problem running a "dockerized" application locally - turns out increasing the resources fixed the issue. What I finally settled on:
CPUs: 4
Memory: 8GB
Swap: 2GB
Same problem, with netbeans
but I had not break points.
Running jboss by command line, helped me
Stop jboss
Close Netbeans
open command line
Go to jboss folder > bin >
type: standalone.bat (this starts jboss)
open Netbeans
worked fine!
Hope it'll help someone else.
I've been facing the same problem recently with WildFly 18 and 21, trying to run a WAR file containing JSR-352 batch jobs that worked fine on WildFly 14.
Increasing the timeout did not solve the situation, only prolonged the time before the TimeoutException was casted, no matter the value (e.g. 5, 10 or 20 minutes).
I've just found that to turn off microprofile-metrics-smallryesubsystem seems to be a possible solution.
After commenting out this line from the standalone.xml file, the war deploy was successful and much faster (about 2 minutes):
<subsystem xmlns="urn:wildfly:microprofile-metrics-smallrye:2.0" security-enabled="false" exposed-subsystems="*" prefix="${wildfly.metrics.prefix:wildfly}"/>
I am having problem with keycloak server 15.0.2.
WFLYCTL0190: Step handler org.jboss.as.server.DeployerChainAddHandler$FinalRuntimeStepHandler#410c55ac for operation add-deployer-chains at address [] failed
I am using mysql5.7 with jconnect8.0 jar.
I had the same problem. Then I killed the Kaspersky process and it helped!
I tackled a similar problem and had only succeed with undeploy the the apps. This gave a clean environment for Wildfly to restart and start the management and http-service. Then deploy the apps/WARs and identify what got you to this state.
In my case it was transactions that wanted to recover and deleting those from DB solve the problem bot to re-occur.

Jboss server log is showig error

I am calling a script within main script to start the jboss sever after releasing the build on server.it is successfully starting the JBOSS but showing the below output in server/log/ server.log file and at the console output which is hanged.
To run the next build i need to kill this manually which is not appropriate.
05:04:17,373 INFO [AjpProtocol] Starting Coyote AJP/1.3 on ajp-0.0.0.0-8209
05:04:17,451 INFO [ServerImpl] JBoss (Microcontainer) [5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221053)] Started in 2m:38s:444ms
05:04:20,912 WARN [PropertyMessageResources] Resource MessageResources_en_US.properties Not Found.
05:04:20,913 WARN [PropertyMessageResources] Resource MessageResources_en.properties Not Found.
Help would be really highly appreciable.
Thanks.
By default, when you start your jboss server, it is not started as a background process and the console just sits there with the logs when server is started, that must be the reason why your script seems to hang , in reality it is just awaiting console output from the server.
To start jboss as background operation, replace the lines of code where you fire the run.sh in startup script with
nohup /path/to/jboss_home/jboss/bin/run.sh -b0.0.0.0 > /tmp/logs/jbosslogs.log &
This should start jboss in the background and redirect all startup logs to jbosslogs.log file. Since it is a background process, it will not hang at all.

Netbeans: project won't start all of a sudden - Glassfish problems?

I am a first time user of Netbeans, and was just playing around. I followed a very basic tutorial, and when I first ran the project, it opened up my browser (Chrome) and displayed what it should display.
The next day, I want to show it to my buddies, but the project won't start.
The project tab shows the following:
init:
deps-module-jar:
deps-ear-jar:
deps-jar:
library-inclusion-in-archive:
library-inclusion-in-manifest:
compile:
compile-jsps:
Starting GlassFish Server 3.1.2
and the other tab "Java DB Database Process" gives me the following:
Wed May 23 19:52:45 CEST 2012 : Sicherheitsmanager mit einfacher Server-Sicherheitsrichtlinie installiert.
Wed May 23 19:52:46 CEST 2012 : Apache Derby Network Server 10.8.1.2 - (1095077) wurde gestartet und ist bereit, Verbindungen am Port 1527 zu akzeptieren.
(the last line translates to "Server was started and is ready to accept connections in port 1527")
The only thing that was different when I tried it was the fact that I was in a public wireless net in University, and not at home. Windows Firewall asked me, if I wanted to grant permission, and I said "Yes"
So basically Netbeans tells me that my project is currently in the process of being started, but nothing happens for ages.
Any ideas what I could possibly have done wrong? I googled the heck out of it, but found nothing.
EDIT: After 20 minutes I get a timeout and a message that the server couldn't be started. I am clueless...
The different network configuration can indeed be a reason for this issue.
You could verify this by either checking the server.log under:
glassfish-install-dir/glassfish/domains/domain1/config/server.log
Scroll down to the last log entry before server start stopped.
Or you can start your server manually in verbose mode. Then you will see all log messages in the console:
Open a console window and switch to the following directory:
glassfish-install-dir/bin
Then enter the following command:
asadmin start-domain --verbose=true