I have been running PAM 7.9/JBPM 7.48 for about a year under JBOSS EAP 7.3. My JBPM's KieServer persists using SQL Server. I repeatedly deployed the KieServer yesterday but deploying today fails.
2021-12-16 15:25:53,645 ERROR [org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 1) WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'full-replace-deployment' at address '[]'
2021-12-16 15:26:03,649 ERROR [org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 1) WFLYCTL0190: Step handler org.jboss.as.server.deployment.DeploymentHandlerUtil$4#74e289e9 for operation full-replace-deployment at address [] failed handling operation rollback -- java.util.concurrent.TimeoutException: java.util.concurrent.TimeoutException
at org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:523)
at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1518)
I have already set the property to increase the timeout for the deployment but it still complains about a 5 second timeout that must be controlled by another property
2021-12-16 13:40:47,039 ERROR [org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 1) WFLYCTL0349: Timeout after [5] seconds waiting for service container stability while finalizing an operation. Process must be restarted. Step that first updated the service container was 'deploy' at address '[("deployment" => "kie-server.war")]'
I have changed the logging level to trace in order to gain all the information I can. How else can I debug / solve this issue?
There are two factors that may be contributing to this, but I don't have a good approach for addressing them.
There was a Windows Update yesterday (likely due to the recent Log4j exploit)
Some people at my company are having problems connecting to the SQL Server database. I am not seeing log messages about KieServer being unable to connect to the DB, but when it cannot reaching the DB the KieServer fails to start.
I downloaded Drools 7.46.0.Final and extracted the contents to my local drive. When I try to run the examples from the Linux command line using the provided runExamples.sh script I'm getting the following exception. I've tried with Java 8 and Java 11 (the only versions I have installed). Does this really require Java 6 like the message recommends or is there some other problem here?
I'm new to Drools, so I'm afraid I'm not sure how to troubleshoot this.
UPDATE: interestingly I tried version 7.44.0.Final and that runs fine. So downloaded 7.45.0.Final and that one is broken too. So something changed between 7.44 and 7.45 that's causing this.
10:06:44.154 [main] INFO o.k.a.i.utils.ServiceDiscoveryImpl.processKieService:129 - Cannot load service: org.kie.internal.process.CorrelationKeyFactory
10:06:44.157 [main] ERROR o.k.a.i.utils.ServiceDiscoveryImpl.processKieService:131 - Loading failed because There already exists an implementation for service org.drools.core.reteoo.KieComponentFactoryFactory with same priority 0
Exception in thread "main" java.lang.ExceptionInInitializerError
at org.drools.dynamic.DynamicServiceRegistrySupplier.get(DynamicServiceRegistrySupplier.java:32)
at org.drools.dynamic.DynamicServiceRegistrySupplier.get(DynamicServiceRegistrySupplier.java:23)
at org.kie.api.internal.utils.ServiceRegistry$Impl.getServiceRegistry(ServiceRegistry.java:88)
at org.kie.api.internal.utils.ServiceRegistry$ServiceRegistryHolder.<clinit>(ServiceRegistry.java:47)
at org.kie.api.internal.utils.ServiceRegistry.getInstance(ServiceRegistry.java:39)
at org.kie.api.internal.utils.ServiceRegistry.getService(ServiceRegistry.java:35)
at org.kie.api.KieServices$Factory$LazyHolder.<clinit>(KieServices.java:358)
at org.kie.api.KieServices$Factory.get(KieServices.java:365)
at org.kie.api.KieServices.get(KieServices.java:349)
at org.drools.examples.DroolsExamplesApp.<init>(DroolsExamplesApp.java:59)
at org.drools.examples.DroolsExamplesApp.main(DroolsExamplesApp.java:52)
Caused by: java.lang.RuntimeException: Unable to build kie service url = jar:file:/home/davek/apps/drools-distribution-7.46.0.Final/examples/binaries/drools-examples-7.46.0.Final.jar!/META-INF/kie.conf
at org.kie.api.internal.utils.ServiceDiscoveryImpl.registerConfs(ServiceDiscoveryImpl.java:105)
at org.kie.api.internal.utils.ServiceDiscoveryImpl.lambda$getServices$1(ServiceDiscoveryImpl.java:83)
at java.util.Optional.ifPresent(Optional.java:159)
at org.kie.api.internal.utils.ServiceDiscoveryImpl.getServices(ServiceDiscoveryImpl.java:81)
at org.kie.api.internal.utils.ServiceRegistry$Impl.<init>(ServiceRegistry.java:60)
at org.drools.dynamic.DynamicServiceRegistrySupplier$LazyHolder.<clinit>(DynamicServiceRegistrySupplier.java:27)
... 11 more
Caused by: java.lang.RuntimeException: There already exists an implementation for service org.drools.core.reteoo.KieComponentFactoryFactory with same priority 0
at org.kie.api.internal.utils.ServiceDiscoveryImpl$PriorityMap.put(ServiceDiscoveryImpl.java:222)
at org.kie.api.internal.utils.ServiceDiscoveryImpl.processKieService(ServiceDiscoveryImpl.java:124)
at org.kie.api.internal.utils.ServiceDiscoveryImpl.registerConfs(ServiceDiscoveryImpl.java:101)
... 16 more
Unfortunately this is a known issue that I fixed with this commit.
Upcoming Drools 7.47.0.Final (to be released next week) won't suffer of this.
Switching to version 8.16.0.Beta or newer resolved this for me
I am new to jboss server. When I am trying to deploy .war file on server the following exception gets print on console:
6:38:04,388 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'add' at address '[
("core-service" => "management"),
("management-interface" => "http-interface")
]'
16:38:05,642 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-4) WFLYJCA0019: Stopped Driver service with driver-name = Aerobay.war_com.mysql.jdbc.Driver_5_1
16:38:09,548 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.server.DeployerChainAddHandler$FinalRuntimeStepHandler#5f88823f for operation {"operation" => "add-deployer-chains","address" => []} at address [] failed handling operation rollback -- java.util.concurrent.TimeoutException: java.util.concurrent.TimeoutException
at org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:396)
at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1384)
at org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1332)
at org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1292)
at org.jboss.as.controller.AbstractOperationContext$Step.access$300(AbstractOperationContext.java:1180)
at org.jboss.as.controller.AbstractOperationContext.handleContainerStabilityFailure(AbstractOperationContext.java:964)
at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:590)
at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:354)
at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:330)
at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1183)
at org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:453)
at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:327)
at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:313)
at org.jboss.as.server.ServerService.boot(ServerService.java:384)
at org.jboss.as.server.ServerService.boot(ServerService.java:359)
at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:271)
at java.lang.Thread.run(Thread.java:745)
Thanks in advance for the help !
I had the same problem when I tried to deploy the WAR file on my Red Hat Jboss EAP 7.0.
But the server was integrated into my IDE (Eclipse Neon) and the problem only occured in Debug-Modus.
I was able to solve the problem by removing all breakpoints and after that i started the server again.
Try increasing timeout by adding java option "blocking.timeout". You can do it in bin/standalone.conf.bat (depends on how you configure wildfly) by adding line:
set "JAVA_OPTS=%JAVA_OPTS% -Djboss.as.management.blocking.timeout=600
Change the number if it's not enough.
increasing the timeout doesn't solve the root cause of the problem. You need to check the cause of the time of the block and solve the issue. Maybe in some cases the solution is to increase the timeout.
In most cases, increasing resources is a bad way to solve issues. I had this case, the Wildfly took a lot of time to boot. I increased the timeout to 600 and solved the issue but was still having issue with the wildfly booting time which was so annoying.
2018-03-26 07:50:36,523 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'add' at address '[("path" => "xxxxxxxxxxxxxxxx")]'
Finally I checked the block cause in and found the block was due to network host resolving (NAS storage defined as a path in wildfly).
I jumped to the network setting and found that my local DNS was not set properly. I added the local DNS instead of the public DNS and the block issue was gone. Hope this helps
Regards
Sleem
When i tried to debug and started the server with debug mode got the following error:
16:19:50,096 ERROR [org.jboss.as.controller.management-operation] (management-handler-thread - 1) JBAS013412: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'deploy' at address '[("deployment" => "ViprWeb.war")]'
16:19:50,096 ERROR [org.jboss.as.server] (management-handler-thread - 1) JBAS015870
16:20:00,117 ERROR [org.jboss.as.controller.management-operation] (management-handler-thread - 1) JBAS013413: Timeout after [5000] seconds waiting for service container stability while finalizing an operation.
I removed all my breakpoints and restarted my server jboss and it resolved the issue.
just increase time out in standalone.conf.bat
set as set "JAVA_OPTS=%JAVA_OPTS% -Djboss.as.management.blocking.timeout=600
It worked for me.
I had the same problem running a "dockerized" application locally - turns out increasing the resources fixed the issue. What I finally settled on:
CPUs: 4
Memory: 8GB
Swap: 2GB
Same problem, with netbeans
but I had not break points.
Running jboss by command line, helped me
Stop jboss
Close Netbeans
open command line
Go to jboss folder > bin >
type: standalone.bat (this starts jboss)
open Netbeans
worked fine!
Hope it'll help someone else.
I've been facing the same problem recently with WildFly 18 and 21, trying to run a WAR file containing JSR-352 batch jobs that worked fine on WildFly 14.
Increasing the timeout did not solve the situation, only prolonged the time before the TimeoutException was casted, no matter the value (e.g. 5, 10 or 20 minutes).
I've just found that to turn off microprofile-metrics-smallryesubsystem seems to be a possible solution.
After commenting out this line from the standalone.xml file, the war deploy was successful and much faster (about 2 minutes):
<subsystem xmlns="urn:wildfly:microprofile-metrics-smallrye:2.0" security-enabled="false" exposed-subsystems="*" prefix="${wildfly.metrics.prefix:wildfly}"/>
I am having problem with keycloak server 15.0.2.
WFLYCTL0190: Step handler org.jboss.as.server.DeployerChainAddHandler$FinalRuntimeStepHandler#410c55ac for operation add-deployer-chains at address [] failed
I am using mysql5.7 with jconnect8.0 jar.
I had the same problem. Then I killed the Kaspersky process and it helped!
I tackled a similar problem and had only succeed with undeploy the the apps. This gave a clean environment for Wildfly to restart and start the management and http-service. Then deploy the apps/WARs and identify what got you to this state.
In my case it was transactions that wanted to recover and deleting those from DB solve the problem bot to re-occur.
I am currently trying to replace Hypersonic with MS-SQL 2008 R2 in JBoss AS 5.1.0GA.
I have followed the instructions in the JBoss Server Configuration Guide, however the server fails to load with this error:
2013-09-26 17:06:04,479 WARN [org.jboss.resource.adapter.jms.inflow.JmsActivation] (WorkManager(2)-3) Failure in jms activation org.jboss.resource.adapter.jms.inflow.JmsActivationSpec#8bb1eb(ra=org.jboss.resource.adapter.jms.JmsResourceAdapter#c54851 destination=queue/iam/im/jms/queue/wpUtilQueue destinationType=javax.jms.Queue tx=true durable=false reconnect=10 provider=DefaultJMSProvider user=null maxMessages=1 minSession=1 maxSession=15 keepAlive=30000 useDLQ=true DLQHandler=org.jboss.resource.adapter.jms.inflow.dlq.GenericDLQHandler DLQJndiName=queue/DLQ DLQUser=null DLQMaxResent=10)
javax.naming.NameNotFoundException: DLQ not bound
(I left out the stack trace for brevity; it isn't important.)
I have checked, and DLQ is defined in destinations-service.xml
I'm not sure where to proceed from here; every response I can find on Google seems to suggest that defining the queue in destinations-service.xml has solved the issue for almost everyone.
Any help would be appreciated.
It turns out that the instructions in the Configuration Guide aren't 100% complete. The issue was that a ChannelFactory was referenced in the mssql-persistence-service.xml; however, this environment is not clustered, and so there were no ChannelFactory objects defined.
Removing the reference to the ChannelFactory was sufficient to resolve the issue.
On starting jboss I am getting the following error :
--- MBEANS THAT ARE THE ROOT CAUSE OF THE PROBLEM ---
ObjectName: jboss.jca:service=DataSourceBinding,name=DefaultDS
State: NOTYETINSTALLED
Depends On Me:
jboss.ejb:service=EJBTimerService,persistencePolicy=database
jboss:service=KeyGeneratorFactory,type=HiLo
jboss.mq:service=StateManager
jboss.mq:service=PersistenceManager
And for all database connections in the servlet I get the following exception :
org.postgresql.util.PSQLException: FATAL: password a
uthentication failed for user "poll"
It was working fine and all of a sudden I started getting these errors. My password is correct. I even tried changing the password and then tried again it showed the same exception. What is happening here?
The DefaultDS data source is what the name suggests; the default datasource. It ships with JBoss and is configured to use the Hypersonic (ie in-memory) database. JBoss uses the DefaultDS datasource to read/write internal queues, timed events, etc
Check the file ../conf/standardjbosscmp-jdbc.xml to see what you've got configured for the DefaultDS datasource. It sounds like you've edited that file unintentionally. Unless you need to persist internal queues etc across boots, just leave it as shipped using Hypersonic.
See the JBoss doc for more.