We are using the spring cloud gateway to proxy requests to the backend system. Currently, we found the following error. Does anyone know what could possibly cause the following exception?
2019-05-21 09:18:24.792 WARN An exception 'io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 2046820359, max: 2058354688)' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 2046820359, max: 2058354688)
at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:655)
at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:610)
at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:769)
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:745)
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:244)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:226)
at io.netty.buffer.PoolArena.reallocate(PoolArena.java:397)
at io.netty.buffer.PooledByteBuf.capacity(PooledByteBuf.java:120)
at io.netty.buffer.AbstractByteBuf.ensureWritable0(AbstractByteBuf.java:299)
at io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:278)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1103)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1096)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1087)
at io.netty.handler.codec.ByteToMessageDecoder$1.cumulate(ByteToMessageDecoder.java:93)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:677)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:612)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:529)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:491)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
at java.lang.Thread.run(Unknown Source)
spring-boot-version: 2.1.3.RELEASE
spring-cloud-version: Greenwich.SR1
reactor-netty: 0.8.5.RELEASE
Related
I keep getting a OutOfMemoryError while deploying/installing keycloak using bitnami/keycloak in a Kubernetes pod.
2023-02-09 17:33:49,316 ERROR [org.jboss.threads.errors] (build-19) Thread Thread[build-19,5,build group] threw an uncaught exception: java.util.concurrent.RejectedExecutionException
at org.jboss.threads.RejectingExecutor.execute(RejectingExecutor.java:38)
at org.jboss.threads.EnhancedQueueExecutor.rejectException(EnhancedQueueExecutor.java:2049)
at org.jboss.threads.EnhancedQueueExecutor.doStartThread(EnhancedQueueExecutor.java:1713)
at org.jboss.threads.EnhancedQueueExecutor.execute(EnhancedQueueExecutor.java:787)
at io.quarkus.builder.BuildContext.depFinished(BuildContext.java:261)
at io.quarkus.builder.BuildContext.run(BuildContext.java:294)
at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
at java.base/java.lang.Thread.run(Thread.java:829)
at org.jboss.threads.JBossThread.run(JBossThread.java:501)
Suppressed: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
at java.base/java.lang.Thread.start0(Native Method)
at java.base/java.lang.Thread.start(Thread.java:798)
at org.jboss.threads.JBossThread.start(JBossThread.java:558)
at org.jboss.threads.EnhancedQueueExecutor.doStartThread(EnhancedQueueExecutor.java:1709)
... 8 more
I suspect it could be because of the Java Heap size. Please correct me if I am wrong.
I have checked the resources allocated to the machine; everything looks fine. The question is how do I update the heap size in value.yaml file? Is JAVA_OPTS_APPEND in https://github.com/bitnami/charts/blob/main/bitnami/keycloak/templates/configmap-env-vars.yaml is the env variable to be overridden?
If anybody has an example, please share.
I am using Spring WebClient to call a rest service. The code for the post call as mentioned below.
Mono<ClientResponse> response = client.post()
.uri(uriBuilder -> uriBuilder.build())
.headers(httpHeaders -> httpHeaders.setAll(getHeaders()))
.body(BodyInserters.fromPublisher(Mono.just(message), String.class))
.exchange();
response.subscribe(clientResponse -> {
System.out.println(clientResponse.statusCode());
});
I am getting the following exception after posting continuously for a while (after posting 2-3 million requests in 5 mins).
[ parallel-3] r.c.s.Schedulers : Scheduler worker in group main failed with an uncaught exception
reactor.core.Exceptions$ErrorCallbackNotImplemented: reactor.netty.internal.shaded.reactor.pool.PoolAcquireTimeoutException: Pool#acquire(Duration) has been pending for more than the configured timeout of 45000ms
Caused by: reactor.netty.internal.shaded.reactor.pool.PoolAcquireTimeoutException: Pool#acquire(Duration) has been pending for more than the configured timeout of 45000ms
at reactor.netty.internal.shaded.reactor.pool.AbstractPool$Borrower.run(AbstractPool.java:317) ~[reactor-netty-0.9.0.RELEASE.jar!/:0.9.0.RELEASE]
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
|_ checkpoint ⇢ Request to POST https://myrest-service-c12-1-lb-125370128.us-west-2.elb.amazonaws.com/antenna [DefaultWebClient]
Stack trace:
at reactor.netty.internal.shaded.reactor.pool.AbstractPool$Borrower.run(AbstractPool.java:317) ~[reactor-netty-0.9.0.RELEASE.jar!/:0.9.0.RELEASE]
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68) [reactor-core-3.3.0.RELEASE.jar!/:3.3.0.RELEASE]
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28) [reactor-core-3.3.0.RELEASE.jar!/:3.3.0.RELEASE]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
It looks like the pool is exhausted and I need to limit the number of requests. Could some one help me in solving this issue. Thanks in advance.
Seems to be a bug in reactor-netty that has recently been fixed: https://github.com/reactor/reactor-netty/issues/1012
I am using vert.x java client for connecting solace server. While using SSL certificates for connectivity, receiving below SSL handshake error.I am using settrustall(true) in my code. Could someone help with cause for the error and resolution.
Below is the error:
SEVERE: Unhandled exception
java.lang.IllegalStateException: Bridge was not successfully started
at io.vertx.amqpbridge.impl.AmqpBridgeImpl.createConsumer(AmqpBridgeImpl.java:174)
at com.gtaa.nch.amqp.MyErrorResponse.lambda$2(MyErrorResponse.java:241)
at io.vertx.amqpbridge.impl.AmqpBridgeImpl.lambda$startImpl$5(AmqpBridgeImpl.java:157)
at io.vertx.proton.impl.ProtonClientImpl$ConnectCompletionHandler.handle(ProtonClientImpl.java:122)
at io.vertx.proton.impl.ProtonClientImpl.lambda$connectNetClient$1(ProtonClientImpl.java:97)
at io.vertx.core.net.impl.NetClientImpl.lambda$connect$2(NetClientImpl.java:113)
at io.vertx.core.net.impl.NetClientImpl.doFailed(NetClientImpl.java:268)
at io.vertx.core.net.impl.NetClientImpl.lambda$failed$12(NetClientImpl.java:264)
at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:337)
at io.vertx.core.impl.ContextImpl.executeFromIO(ContextImpl.java:195)
at io.vertx.core.net.impl.NetClientImpl.failed(NetClientImpl.java:264)
at io.vertx.core.net.impl.NetClientImpl.lambda$null$4(NetClientImpl.java:208)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122)
at io.netty.handler.ssl.SslHandler.notifyHandshakeFailure(SslHandler.java:1535)
at io.netty.handler.ssl.SslHandler.setHandshakeFailure(SslHandler.java:1521)
at io.netty.handler.ssl.SslHandler.setHandshakeFailure(SslHandler.java:1493)
at io.netty.handler.ssl.SslHandler.handleUnwrapThrowable(SslHandler.java:1186)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1165)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1194)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:141)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Unknown Source)
This is in all likelihood an SSL configuration problem.
Check your bridge configuration targets the AMQP SSL port on the Solace broker, not the plain text port (defaults are 5671 rather than 5672)
Check the broker SSL configuration is valid (is the AMQP SSL service showing Operational UP?
Use sdkperf_jmsamqp to check that your SSL connection, certificates etc are valid.
I'm setting up the Banking application I use(T24) to send and receive messages from IBM MQ. I'm getting errors when I connect my application server(JBOSS 7) to MQ
I've tried altering the MDB's but to no avail.
This is a snippet from my Log file:
2019-03-22 17:00:17,269 INFO [org.jboss.as.connector.deployers.RaXmlDeployer] (default-threads - 13) wmq.jmsra.rar: MQJCA4026:Transaction backed out with reason: 'The method 'xa_end' has failed with errorCode '100'.'.
2019-03-22 17:00:17,269 WARN [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (default-threads - 24) IJ000612: Destroying connection that could not be successfully matched: org.jboss.jca.core.connectionmanager.listener.TxConnectionListener#213a8b6c[state=NORMAL managed connection=com.ibm.mq.connector.outbound.ManagedConnectionImpl#3db98551 connection handles=0 lastReturned=1553274017260 lastValidated=1553274017260 lastCheckedOut=1553274017260 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool#2185ef44 mcp=SemaphoreConcurrentLinkedQueueManagedConnectionPool#52753b07[pool=WMQ Connection Pool] xaResource=XAResourceWrapperImpl#518c200e[xaResource=com.ibm.mq.connector.xa.XARWrapper#67583e6a pad=false overrideRmValue=null productName=IBM MQ productVersion=#(#) MQMBID sn=p910-L180709.TRIAL su=_QGElQYNUEeidSaRkJ_p2Kg pn=com.ibm.mq.connector/src/com/ibm/mq/connector/outbound/ManagedConnectionMetaDataImpl.java jndiName=java:jboss/jms/MQConnectionFactory] txSync=null]
2019-03-22 17:00:17,269 WARN [com.arjuna.ats.jta] (default-threads - 13) ARJUNA016045: attempted rollback of < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffffc0a81701:-783f63b7:5c951441:40a3, node_name=1, branch_uid=0:ffffc0a81701:-783f63b7:5c951441:40b4, subordinatenodename=null, eis_name=java:jboss/jms/MQConnectionFactory > (XAResourceWrapperImpl#44eb1bdd[xaResource=com.ibm.mq.connector.xa.XARWrapper#c57055e pad=false overrideRmValue=null productName=IBM MQ productVersion=#(#) MQMBID sn=p910-L180709.TRIAL su=_QGElQYNUEeidSaRkJ_p2Kg pn=com.ibm.mq.connector/src/com/ibm/mq/connector/outbound/ManagedConnectionMetaDataImpl.java jndiName=java:jboss/jms/MQConnectionFactory]) failed with exception code XAException.XAER_NOTA: javax.transaction.xa.XAException: The method 'xa_rollback' has failed with errorCode '-4'.
at com.ibm.mq.jmqi.JmqiXAResource.rollback(JmqiXAResource.java:874)
at com.ibm.mq.connector.xa.XARWrapper.rollback(XARWrapper.java:605)
at org.jboss.jca.core.tx.jbossts.XAResourceWrapperImpl.rollback(XAResourceWrapperImpl.java:196)
at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelAbort(XAResourceRecord.java:369)
at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:2999)
at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:2978)
at com.arjuna.ats.arjuna.coordinator.BasicAction.Abort(BasicAction.java:1658)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.cancel(TwoPhaseCoordinator.java:127)
at com.arjuna.ats.arjuna.AtomicAction.abort(AtomicAction.java:186)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.rollbackAndDisassociate(TransactionImple.java:1282)
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.rollback(BaseTransaction.java:143)
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.rollback(BaseTransactionManagerDelegate.java:134)
at org.jboss.as.ejb3.inflow.MessageEndpointInvocationHandler.afterDelivery(MessageEndpointInvocationHandler.java:69)
at sun.reflect.GeneratedMethodAccessor46.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jboss.as.ejb3.inflow.AbstractInvocationHandler.handle(AbstractInvocationHandler.java:60)
at org.jboss.as.ejb3.inflow.MessageEndpointInvocationHandler.doInvoke(MessageEndpointInvocationHandler.java:135)
at org.jboss.as.ejb3.inflow.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:73)
at com.temenos.tafj.mdb.TransactedMDB$$$endpoint1.afterDelivery(Unknown Source)
at com.ibm.mq.connector.inbound.AbstractWorkImpl.run(AbstractWorkImpl.java:343)
at org.jboss.jca.core.workmanager.WorkWrapper.run(WorkWrapper.java:223)
at org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:33)
at org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:808)
at org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:45)
at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:849)
at java.lang.Thread.run(Thread.java:748)
at org.jboss.threads.JBossThread.run(JBossThread.java:320)
That error can be safely ignored.
XAER_NOTA is a valid return from xa_rollback()
The XA specification indicates that "[a]n RM can also unilaterally roll back and forget a branch any time except after a successful prepare"
I've created a small project which combines Spring Data, a JPA Repository, EJB/CDI and either Wildfly Swarm or plain Wildfly.
The REST resource (an EJB) calls a CDI bean which has a Spring Data JPA Repository injected.
The initial request to the repository ends in an exception, but subsequent calls work just fine.
2016-12-15 23:03:09,690 WARN [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (default task-1) SQL Error: 0, SQLState: null
2016-12-15 23:03:09,690 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (default task-1) javax.resource.ResourceException: IJ000460: Error checking for a transaction
2016-12-15 23:03:10,268 ERROR [io.undertow.request] (default task-1) UT005023: Exception handling request to /Penny: org.jboss.resteasy.spi.UnhandledException: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
...
Caused by: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
...
Caused by: java.lang.Throwable: setRollbackOnly called from:
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.setRollbackOnly(TransactionImple.java:339)
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.setRollbackOnly(BaseTransaction.java:159)
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.setRollbackOnly(BaseTransactionManagerDelegate.java:143)
at org.hibernate.jpa.spi.AbstractEntityManagerImpl.markForRollbackOnly(AbstractEntityManagerImpl.java:1509)
at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1611)
at org.hibernate.jpa.spi.AbstractEntityManagerImpl.buildQueryFromName(AbstractEntityManagerImpl.java:753)
at org.hibernate.jpa.spi.AbstractEntityManagerImpl.createNamedQuery(AbstractEntityManagerImpl.java:730)
at org.springframework.data.jpa.repository.query.NamedQuery.hasNamedQuery(NamedQuery.java:99)
at org.springframework.data.jpa.repository.query.NamedQuery.lookupFrom(NamedQuery.java:121)
at org.springframework.data.jpa.repository.query.JpaQueryLookupStrategy$DeclaredQueryLookupStrategy.resolveQuery(JpaQueryLookupStrategy.java:162)
at org.springframework.data.jpa.repository.query.JpaQueryLookupStrategy$CreateIfNotFoundQueryLookupStrategy.resolveQuery(JpaQueryLookupStrategy.java:212)
at org.springframework.data.jpa.repository.query.JpaQueryLookupStrategy$AbstractQueryLookupStrategy.resolveQuery(JpaQueryLookupStrategy.java:77)
at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.<init>(RepositoryFactorySupport.java:435)
at org.springframework.data.repository.core.support.RepositoryFactorySupport.getRepository(RepositoryFactorySupport.java:220)
at org.springframework.data.jpa.repository.cdi.JpaRepositoryBean.create(JpaRepositoryBean.java:73)
at org.springframework.data.repository.cdi.CdiRepositoryBean.create(CdiRepositoryBean.java:372)
at org.springframework.data.repository.cdi.CdiRepositoryBean.create(CdiRepositoryBean.java:170)
at org.jboss.weld.context.AbstractContext.get(AbstractContext.java:96)
at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:101)
at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50)
at org.jboss.weld.bean.proxy.ContextBeanInstance.getInstance(ContextBeanInstance.java:99)
at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:99)
at ch.maxant.demo.swarmproblems.EmployeeService.findByName(EmployeeService.java)
at ch.maxant.demo.swarmproblems.EmployeeService$Proxy$_$$_WeldClientProxy.findByName(Unknown Source)
at ch.maxant.demo.swarmproblems.EmployeeResource.get(EmployeeResource.java)
Debugging, I found out that org.springframework.data.jpa.respository.query.NamedQuery#lookupFrom ends up calling
through to org.hibernate.jpa.spi.AbstractentityManagerImpl#buildQueryfromName which seems to throw an
IllegalArgumentException: No query defined for that name [Employee.findByName]. That kind of makes sense, in that
Spring appears to be loading the named query lazily.
Is this a bug in Spring or is the application not setup properly?
A different project which uses JBoss Wildfly (8) has the same problem: https://github.com/maxant/jee7webappwithspringdata.