NServiceBus Worker checking in with capacity of 0 - msmq

I am using NServiceBus 3.3. I am trying to get a new Pre-Prod-Environment setup.
It all works fine in production and in one of my existing Pre-Prod-Environments with my existing configurations.
But in my new Environment, I am getting my workers checking-in with a capacity of 0. (They check-in with a capacity of 1 in the working Environments).
Again, the configs are the same between the Environments. (Except for machine names of course.)
Any idea why this could happen?
This is the output of my log (with Queue names and machine names changed):
2015-05-07 10:53:33,904 [1] INFO NServiceBus.Host [(null)] - Going to activate profile: NServiceBus.Distributor, NServiceBus.Host, Version=3.3.0.0, Culture=neutral, PublicKeyToken=9fc386479f8a226c
2015-05-07 10:53:33,904 [1] INFO NServiceBus.Host [(null)] - Going to activate profile: NServiceBus.Production, NServiceBus.Host, Version=3.3.0.0, Culture=neutral, PublicKeyToken=9fc386479f8a226c
2015-05-07 10:53:33,919 [1] INFO NServiceBus.Host [(null)] - Going to activate profile: NServiceBus.PerformanceCounters, NServiceBus.Host, Version=3.3.0.0, Culture=neutral, PublicKeyToken=9fc386479f8a226c
2015-05-07 10:53:33,935 [1] WARN Distributor.myFromQueue [(null)] - No transport configuration found so the distributor will default to one thread, for production scenarios you would want to adjust this setting
2015-05-07 10:53:33,950 [1] INFO Distributor.myFromQueue [(null)] - Endpoint configured to host the distributor, applicative input queue re routed to myFromQueue.worker#DistributorHost
2015-05-07 11:10:05,015 [Worker.13] INFO Distributor.myFromQueue [(null)] - Worker myFromQueue#WorkerMachine has started up, clearing previous reported capacity
2015-05-07 11:10:05,030 [Worker.13] INFO Distributor.myFromQueue [(null)] - Worker myFromQueue#WorkerMachine checked in with available capacity: 0
And this is the relevant part of my worker config file:
<MsmqTransportConfig NumberOfWorkerThreads="4" MaxRetries="5" />
<MessageForwardingInCaseOfFaultConfig ErrorQueue="error" />
<MasterNodeConfig Node="DistributorHost" />
<UnicastBusConfig>
<MessageEndpointMappings>
<add Messages="Bus.MyMessageAssembly" Endpoint="QueueForTheDistributor#DistributorHost" />
</MessageEndpointMappings>
</UnicastBusConfig>

I had the same problem and was able to resolve it by deleting the queues (myfromqueue and myfromqueue.retries) on the worker agent. NServiceBus automatically recreated the queues and everything started processing again for me.

Related

Infinispan (Red Hat Data Grid) in Openshift, with WebSphere Liberty

We're trying to use Red Hat Data Grid (RHDG)/Infinispan in our OCP (4.5.36) cluster. We have the latest official RHDG Operator installed and a Cache type cluster defined. (Which is apparently a k8s StatefulSet.)
I've then configured a WebSphere Liberty container/Deployment to try to use that Infinispan cluster for its sessions, as described in https://github.com/WASdev/ci.docker#session-caching.
Both the Infinispan cluster and the Liberty Deployment are in the same Project/namespace.
However, the Liberty container fails to connect, and the Infinispan containers are reporting several warnings of their own.
The Liberty container "client" log:
INFINISPAN_SERVICE_NAME(original): session-infinispan
INFINISPAN_SERVICE_NAME(normalized): SESSION_INFINISPAN
INFINISPAN_HOST: 172.30.137.86
INFINISPAN_PORT: 11222
INFINISPAN_USER: developer
INFINISPAN_PASS: <redacted>
Launching defaultServer (WebSphere Application Server 21.0.0.3/wlp-1.0.50.cl210320210309-1101) on Eclipse OpenJ9 VM, version 1.8.0_282-b08 (en_US)
[AUDIT ] CWWKE0001I: The server defaultServer has been launched.
[AUDIT ] CWWKE0100I: This product is licensed for development, and limited production use. The full license terms can be viewed here: https://public.dhe.ibm.com/ibmdl/export/pub/software/websphere/wasdev/license/base_ilan/ilan/21.0.0.3/lafiles/en.html
[AUDIT ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ibm/wlp/usr/servers/defaultServer/configDropins/defaults/keystore.xml
[AUDIT ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ibm/wlp/usr/servers/defaultServer/configDropins/overrides/infinispan-client-sessioncache.xml
[AUDIT ] CWWKZ0058I: Monitoring dropins for applications.
[AUDIT ] CWWKT0016I: Web application available (default_host): http://payment-engine-6dcc5b6d5-jclx2:9080/payment/
[ERROR ] ISPN004007: Exception encountered. Retry 10 out of 10
org.infinispan.client.hotrod.exceptions.TransportException:: ISPN004071: Connection to 172.30.137.86/172.30.137.86:11222 was closed while waiting for response.
[ERROR ] SESN0307E: An exception occurred when initializing the cache. The exception is: org.infinispan.client.hotrod.exceptions.TransportException:: org.infinispan.client.hotrod.exceptions.TransportException:: ISPN004071: Connection to 172.30.137.86/172.30.137.86:11222 was closed while waiting for response.
at org.infinispan.client.hotrod.impl.transport.netty.ActivationHandler.exceptionCaught(ActivationHandler.java:53)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:300)
...
What looks like the relevant part of the Inifinispan container log:
03:40:18,628 WARN (SINGLE_PORT-ServerIO-4-2) [io.netty.handler.ssl.ApplicationProtocolNegotiationHandler] [id: 0xc39380c8, L:/10.254.0.248:11222 ! R:/10.254.2.65:32986] TLS handshake failed: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: a0061e21000003ffffffff0f0000
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1254)
(Actually, there are several Infinispan startup WARNs, mostly about deprecated capabilities. But this is the only one with a stack trace, so I'm jumping to the conclusion that it might be "the culprit")
Also, this is the Infinispan Service, so you can see the IP and port match what the Liberty container is using:
Working through this on the Infinispan chat service, it does appear that there's incorrect or incomplete setup of SSL/TLS.
I had attempted to remove encryption in the Infinispan cluster, but I either didn't sufficiently restart components or you can't change it after the fact. Removing the cluster and recreating with it disabled, though, enabled the Liberty communication to work.
The following CR YAML works:
apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
name: session-infinispan
spec:
replicas: 1
service:
type: Cache
security:
endpointEncryption:
type: None
Now to pursue what's missing from the Liberty setup to make use of SSL correctly. The Infinispan chat conversation says that this Liberty XML setup from the official image:
<server>
<featureManager>
<feature>sessionCache-1.0</feature>
</featureManager>
<httpSessionCache libraryRef="InfinispanLib">
<properties infinispan.client.hotrod.server_list="${INFINISPAN_HOST}:${INFINISPAN_PORT}"/>
<properties infinispan.client.hotrod.marshaller="org.infinispan.commons.marshall.JavaSerializationMarshaller"/>
<properties infinispan.client.hotrod.java_serial_whitelist=".*"/>
<properties infinispan.client.hotrod.auth_username="${INFINISPAN_USER}"/>
<properties infinispan.client.hotrod.auth_password="${INFINISPAN_PASS}"/>
<properties infinispan.client.hotrod.auth_realm="default"/>
<properties infinispan.client.hotrod.sasl_mechanism="DIGEST-MD5"/>
<properties infinispan.client.hotrod.auth_server_name="infinispan"/>
</httpSessionCache>
<httpSessionCache enableBetaSupportForInfinispan="true"/> <!-- TODO remove once no longer gated -->
<library id="InfinispanLib">
<fileset dir="${shared.resource.dir}/infinispan" includes="*.jar"/>
</library>
</server>
Needs the following properties added:
# Encryption
infinispan.client.hotrod.sni_host_name=$SERVICE_HOSTNAME
# Path to the TLS certificate.
# Clients automatically generate trust stores from certificates.
infinispan.client.hotrod.trust_store_path=/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt

kaa data collection doesn't retrieve data mongodb

I installed kaa iot server manually on ubuntu 16.04, and use data collection sample to test how it works.
the code run without any errors, but when I run these commands below nothing happens:
mongo kaa
db.logs_$my_app_token$.find()
I even comment out bind_ip of mongodb.conf and restart mongodb, zookeeper and kaa-node services, but nothings changed.
I also regenerated SDK and rebuild project but that wouldn't help either.
finally this is the kaa log:
2018-06-05 15:03:53,899 [Thread-3] TRACE
o.k.k.s.c.s.l.DynamicLoadManager - DynamicLoadManager recalculate() got 0 redirection rules
2018-06-05 15:03:59,472 [EPS-core-dispatcher-6] DEBUG
o.k.k.s.o.s.a.a.c.OperationsServerActor - Received: org.kaaproject.kaa.server.operations.service.akka.messages.core.stats.StatusRequestMessage#30d61bb1
2018-06-05 15:03:59,472 [EPS-core-dispatcher-6] DEBUG o.k.k.s.o.s.a.a.c.OperationsServerActor - [14fc1a87-8b34-47f6-8f39-d91aff7bfff7] Processing status request
2018-06-05 15:03:59,475 [pool-5-thread-1] INFO o.k.k.s.o.s.l.DefaultLoadBalancingService - Updated load info: {"endpointCount": 0, "loadAverage": 0.02}
2018-06-05 15:03:59,477 [Curator-PathChildrenCache-0] INFO o.k.k.s.c.s.l.DynamicLoadManager - Operations server [-1835393002][localhost:9090] updated
2018-06-05 15:03:59,477 [Curator-PathChildrenCache-4] DEBUG o.k.k.s.o.s.c.DefaultClusterService - Update of node [localhost:9090:1528181889050]-[{"endpointCount": 0, "loadAverage": 0.02}] is pushed to resolver org.kaaproject.kaa.server.hash.ConsistentHashResolver#1d0276a4
2018-06-05 15:04:03,899 [Thread-3] INFO o.k.k.s.c.s.l.LoadDistributionService - Load distribution service recalculation started...
2018-06-05 15:04:03,899 [Thread-3] INFO o.k.k.s.c.s.l.DynamicLoadManager - DynamicLoadManager recalculate() started... lastBootstrapServersUpdateFailed false
2018-06-05 15:04:03,899 [Thread-3] DEBUG o.k.k.s.c.s.l.d.EndpointCountRebalancer - No rebalancing in standalone mode
2018-06-05 15:04:03,899 [Thread-3] TRACE o.k.k.s.c.s.l.DynamicLoadManager - DynamicLoadManager recalculate() got 0 redirection rules
thank you for your help to fix this problem...
After lots of searching and checking, I finally found it!!!
there are multiple reason that this would happen:
if you are using Kaa Sanddbox make sure that you set your network setting into bridge (not NAT).
check your iptables and find out if these ports are open: 9888,9889,9997,9999.
if you are using virtual machine as your server, make sure that hosts firewall system doesn't block the ports.(This is what happened to me...)

Restcomm - Solving SMSC GW 7.2 configuration failures

We configured the latest version (7.2) SMSC-GW to work on on our server with the environment (cassandra and such). However, after setting up everything. Some failures are appearing (which did not appear in previous versions).
Firstly, when connecting the simulators and the gateway using the default settings (JSS7 <-> SMSCGW <-> SMPP)
JSS7 is connected and sending, but no response is received.
SMPP is connected to SMSC-GW and the EMSE is bound. SMPP tries to send to SS7 but receives a response PDU packet failure from the SMSC-GW
I tried configuring DB routing rules, but that did not work.
Also, the log in the SMSC-GW server is frequently displaying the following message:
16:00:28,504 INFO [SchedulerResourceAdaptor] (pool-56-thread-1) Not all SBB are running now: ServicesDownList=[smscTxSmppServerServiceState, smscRxSmppServerServiceState, smscTxSipServerServiceState, smscRxSipServerServiceState, smscTxHttpServerServiceState, moServiceState, homeRoutingServiceState, mtServiceState, alertServiceState, chargingServiceState, ]
And the JSS7 management console GUI is displaying this (which looks wrong):
So are these the source of the SMSC-GW failures?
UPDATE: I found this error in the server.log
2017-02-02 10:57:42,005 WARN [org.mobicents.slee.container.deployment.jboss.SleeContainerDeployerImpl] (SLEE-InternalDeployer-thread-1) SLEE DUs not deployed, due to missing dependencies: file:/home/coreteam/kitchensink/restcomm-smsc-7.2.109/jboss-5.1.0.GA/server/simulator/deploy/smsc-services-du-7.2.109.jar/
Followed by:
EventTypeID[name=org.mobicents.smsc.slee.services.smpp.server.events.SS7_SEND_MT,vendor=org.mobicents,version=1.0]
ResourceAdaptorTypeID[name=PersistenceResourceAdaptorType,vendor=org.mobicents,version=1.0]
ResourceAdaptorTypeID[name=SchedulerResourceAdaptorType,vendor=org.mobicents,version=1.0]
SipRA
EventTypeID[name=org.mobicents.smsc.slee.services.smpp.server.events.SS7_SEND_RSDS,vendor=org.mobicents,version=1.0]
SchedulerResourceAdaptor^M
PersistenceResourceAdaptor^M
EventTypeID[name=org.mobicents.smsc.slee.services.smpp.server.events.SMPP_SM,vendor=org.mobicents,version=1.0]
EventTypeID[name=org.mobicents.smsc.slee.services.smpp.server.events.SS7_SM,vendor=org.mobicents,version=1.0]
EventTypeID[name=org.mobicents.smsc.slee.services.smpp.server.events.SIP_SM,vendor=org.mobicents,version=1.0]
2017-02-02 14:41:17,450 WARN [org.mobicents.slee.container.deployment.jboss.DeploymentManager] (main) Unable to INSTALL smsc-services-du-7.3.0-SNAPSHOT.jar right now. Waiting for dependencies to be resolved.
Solved it quite a while ago, but thought I would share. I just simply installed the SipRA missing dependency by adding the following in the deploy-config.xml file:
<ra-entity
resource-adaptor-id="ResourceAdaptorID[name=JainSipResourceAdaptor,vendor=net.java.slee.sip,version=1.2]"
entity-name="SipRA">
<properties>
<property name="javax.sip.PORT" type="java.lang.Integer" value="5060" />
</properties>
<ra-link name="SipRA" />
In the $JBOSS_HOME/server/profile_name/deploy/restcomm-slee directory.
I set the port to some other value since that number was already taken by some other service.
The smsc-services-du-7.2.109.jar then installed automatically the next time I ran the SMSC-GW.

Azure deployment is cycling for a long time

12:08:41 - Preparing deployment for Windows Azure MSDN - Visual Studio Ultimate with
Subscription ID: xxxxxxxxxxxxxxxxxxxx...
12:08:41 - Connecting...
12:08:41 - Verifying storage account 'cck'...
12:08:43 - Uploading Package...
12:41:00 - Creating...
12:57:14 - Created Deployment ID: xxxxxxxxxxxxxxxxx.
12:57:14 - Starting...
12:57:55 - Initializing...
12:57:55 - Instance 0 of role RIS2048.ConsultaClick.Web is in an unknown state
12:59:03 - Instance 0 of role RIS2048.ConsultaClick.Web is starting the virtual machine
13:00:40 - Instance 0 of role RIS2048.ConsultaClick.Web is in an unknown state
13:01:48 - Instance 0 of role RIS2048.ConsultaClick.Web is busy
13:06:12 - Instance 0 of role RIS2048.ConsultaClick.Web is cycling
It's now 13:31, more than 1 hour stared ago and at 25 minutes the instance is cycling (I don't know what that means). Will it finish? When?
Here's part of my ServiceDefinition.csdef file with 2 VirtualApplications:
<Site name="Web">
<VirtualApplication name="CCKPt"
physicalDirectory=".">
<VirtualDirectory name="images"
physicalDirectory="..\RIS2048.ConsultaClick.WWWPacientes\imgpt" />
</VirtualApplication>
<VirtualApplication name="CCKRo"
physicalDirectory=".">
<VirtualDirectory name="images"
physicalDirectory="..\RIS2048.ConsultaClick.WWWPacientes\imgro" />
</VirtualApplication>
<Bindings>
<Binding name="Endpoint1" endpointName="Endpoint1" />
</Bindings>
</Site>
Is this a new app you just built? And does it run locally in the emulator? First thing I always check is the connection string for diagnostics, along with the session state provider. By default, your diagnostics are written to the emulator storage (meaning SQL Express on your local machine), and SQL Express is used for session state, which doesn't exist in Windows Azure. Rather, you'd need to switch it to Cache or SQL Azure.
Likely means there's a problem. Sometimes, if you're lucky, you can squeeze a little more info about where exactly it's bombing from IntelliTrace:
http://blogs.msdn.com/b/jnak/archive/2010/06/07/using-intellitrace-to-debug-windows-azure-cloud-services.aspx

NServiceBus MSMQ Send question

I have trouble sending a message via NServiceBus. I have an ASP.Net MVC web app, developing on Win7 x64, I have configured my web.config as
<MsmqTransportConfig InputQueue="worker" ErrorQueue="error" NumberOfWorkerThreads="1" MaxRetries="5" />
<UnicastBusConfig>
<MessageEndpointMappings>
<add Messages="PricingInformation.Messages" Endpoint="worker2" />
</MessageEndpointMappings> </UnicastBusConfig>
In application_start I wire up the following:
var bus = NServiceBus.Configure.WithWeb()
.StructureMapBuilder()
.XmlSerializer()
.MsmqTransport()
.IsTransactional(false)
.PurgeOnStartup(false)
.UnicastBus()
.ImpersonateSender(false)
.CreateBus()
.Start();
When the action I'm interested happens in the app I fire
public override void HandleEvent(SupplierPricingUpdatedEvent updatedEvent)
{
bus.Send(new ModelSupplierDetailsUpdatedMessage() {Id = updatedEvent.Id})
return;
}
ModelSupplierDetailsUpdatedMessage is simple class in PricingInformation.Messages using interface marker IMessage and decorated with Serializable attribute.
The MSMQ queues are setup, not transactional, and everyone including NETWORK SERVICE and IIS_IUSRS have full control (in deseperate troubleshooting measures)
log4net shows the following:
DEBUG NServiceBus.Utils.MsmqUtilities 14 - Checking if queue exists: worker.
DEBUG NServiceBus.Utils.MsmqUtilities 14 - Checking if queue exists: error.
DEBUG NServiceBus.Unicast.UnicastBus Worker.15 - Calling 'HandleBeginMessage' on NServiceBus.SagaPersisters.NHibernate.NHibernateMessageModule
INFO NServiceBus.Unicast.UnicastBus Worker.15 - worker initialized.
DEBUG NServiceBus.Unicast.UnicastBus Worker.15 - Calling 'HandleEndMessage' on NServiceBus.SagaPersisters.NHibernate.NHibernateMessageModule
DEBUG NServiceBus.Unicast.UnicastBus 9 - Sending message PricingInformation.Messages.ModelSupplierDetailsUpdatedMessage, PricingInformation.Messages, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null with ID 2c642672-1bf1-48d4-a90f-734e2fdd726d\8267 to destination worker2.
Yet no matter what I try, and what I tweak (been three hours at it already) the message never appears in the queue. I cant find an exception and i have debug level logging on everything.. Its probably something simple help
The problem is that if you manually set up your queues as non-transactional, then it won't work. Setting the NServiceBus "IsTransactional" property to false doesn't mean you can work with non-transactional queues.
Please try deleting the queues and recreating them as transactional or, if you're using the beta of v2.0, just letting NServiceBus create the queues for you.