I am experimenting with Nats streaming server and it looks quite promising so far.
However it appears queues can only be durable after a durable subscription has been created for it.
This certainly makes sense however how does it work in practice in a microservices architecture?
For instance assume you are publishing services and Service1 is pumping messages out to a queue that is not yet durable and has no listeners. Some time later that corresponding service starts and makes that queue durable. Do you just deal with this hopefully short loss or ensure the later service is started first?
Sorry for the delay. In NATS Streaming, any message published to a channel are stored, regardless of subscription interest. You can experiment and publish say 3 messages on "foo". Then, you can start a subscription (even non durable) and replay those messages. It is just a matter of specifying the starting point of the subscription. For instance, there is an option to have deliver "all available". Using the Go nats samples, it would be:
$ go run examples/stan-pub/main.go foo msg1
Published [foo] : 'msg1'
$ go run examples/stan-pub/main.go foo msg2
Published [foo] : 'msg2'
$ go run examples/stan-pub/main.go foo msg3
Published [foo] : 'msg3'
$ go run examples/stan-sub/main.go -id "me" -all foo
Connected to nats://127.0.0.1:4222 clusterID: [test-cluster] clientID: [me]
Listening on [foo], clientID=[me], qgroup=[] durable=[]
[#1] Received: sequence:1 subject:"foo" data:"msg1" timestamp:1583947471103854000
[#2] Received: sequence:2 subject:"foo" data:"msg2" timestamp:1583947472684693000
[#3] Received: sequence:3 subject:"foo" data:"msg3" timestamp:1583947473990567000
Related
We are using the JMSOutboundGateway to send message and receive message using the reply channel within the JMSOutboundGateway. When we run multiple iterations of the same job using the same JMSOutboundGateway, it fails with this error "Message contained wrong job instance id [85] should have been [86]" ( org.springframework.batch.integration.chunk.ChunkMessageChannelItemWriter.getNextResult() ) .
This is due to same JMSOutBoundGateway instance being using when I run the second when the first job is still in progress.
Is there a way I can run parallel execution of the same job type ?
This is a known issue, see https://github.com/spring-projects/spring-batch/issues/1372 and https://github.com/spring-projects/spring-batch/issues/1096.
The workaround is to use a separate instance of the writer for each job to prevent sharing the same reply channel.
I am using sarama(1.27) ClusterAdmin to manage topics in kafka1.1.0. My application that manages kafka topics, is running as a REST service. My application runs fine for a while and I can get/create/delete topic.
But after some time elapses without any activity, a new topic request gets error - write tcp xxxxx:37888->xxxxx:9092: write: broken pipe.
I came across this How to fix broker may not be available after broken pipe.
Since my application is running as a service, how do I prevent broken pipe issue ? I close ClusterAdmin only when application exits. Same ClusterAdmin connection is used to serve all requests. I reinitialize clusterAdmin for each request if for any reason it is nil(Usually it is not nil after first initialization, so same connection is reused).
Should I close clusteradmin after each request is served and open a NewClusterAdmin() for each topic request, or is there a keepalive option that I need to use?
Here is my existing code:
if admin == nil{
admin, err := NewClusterAdmin([]string{"localhost:9092"}, s.config)
..
}
topicMetadata, err := admin.DescribeTopics([]string{topicName})
I also came cross this error. My way to fix this question is try again several times, e.g. 2 to 10 times.
I have a situation where there are 2 services. Service A is exposing query API through HTTP endpoint and also is listening for incomming asynchronous command messages (service A owns both of CQRS contracts).
Service B is using both endpoints of service A: to GET data and to invoke commands.
While implementing contract (stub and tests) for HTTP flow is quite simple, configuring messaging part is a tricky for me and actually I've stucked at this one.
Docs says that there is publisher side test generation what is suitable for publishing event case where publisher owns the contract.
But how to makes it working for situation where message consumer owns the contract??
I can't figure out any solution on that one as I need to have a stub used in service A to verify if service A is properly consuming commands messages and also I need genereated tests on service B that will verify that service B if it is producing compliant command message.
I'd appreciate any help.
Many thanks in advance.
Service A is the producer of the API and the consumer of messages. It owns only contracts for HTTP. The messaging contracts are owned by Service B. Service B is the producer of messages. You should have an HTTP contract defined on the Service A side and a Stub Runner test to test if it can receive the message sent by Service B. Service B should have the messaging contract to assert whether the message is properly sent and Stub Runner test for HTTP
That might lead to a dependency cycle. If you have a cycle between your apps then, yeah, what you have to do is ignore a stub runner test on one side until the jars got uploaded.
You've asked about storing contracts in a separate repository. You can do it - here are the docs https://cloud.spring.io/spring-cloud-static/Edgware.SR3/multi/multi__spring_cloud_contract_faq.html#_common_repo_with_contracts and here is an example https://github.com/spring-cloud-samples/spring-cloud-contract-samples/tree/master/beer_contracts
You've asked about not generating the tests for some reason (IMO that's a wrong thing to do). You can not use <extensions>true</extensions> in Maven but manually provide which goals you want to execute (omit the test generation). In Gradle just disable generateContractTests task AFAIR
I'm trying to get MSMQ 5 working on my two Windows Server 2008 R2 virtual machines.
I can send to local and remote private queues, and I can read from local private queues.
I can't read from remote private queues.
I've read a number of suggestions, especially the ones summarised by John Breakwell at MSMQ Issue reading remote private queues (again).
Things I've already done:
Turned off firewalls on both machines.
Ensured that Everyone and AnonymousLogon have full control of the queues. (If I take away AnonymousLogon access, then I can't remotely send to the queue, and the message ends up with "Access is denied" on the receiving machine.)
Allowed Nonauthenticated Rpc on both machines.
Allowed NewRemoteReadServerAllowNoneSecurityClient on both machines.
the sending code fragment is:
MessageQueue queue = new MessageQueue(queueName, false, false, QueueAccessMode.Send);
Message msg = new Message("Blah");
msg.UseDeadLetterQueue = true;
msg.UseJournalQueue = true;
queue.Send(msg, MessageQueueTransactionType.Automatic);
queue.Close();
The receiving code fragment is:
queueName = String.Format("FormatName:DIRECT=OS:{0}\\private$\\{1}",host,id);
queue = new MessageQueue(queueName, QueueAccessMode.Receive);
queue.ReceiveCompleted += new ReceiveCompletedEventHandler(receive);
queue.BeginReceive();
...
public void receive(object sender, ReceiveCompletedEventArgs e)
{
queue.EndReceive(e.AsyncResult);
Console.WriteLine("Message received");
queue.BeginReceive();
}
My queueName ends up as FormatName:DIRECT=OS:server2\private$\TestQueue
When I call beginReceive() on the queue, I get
Exception: System.Messaging.MessageQueueException (0x80004005)
at System.Messaging.MessageQueue.MQCacheableInfo.get_ReadHandle()
at System.Messaging.MessageQueue.ReceiveAsync(TimeSpan timeout, CursorHandle cursorHandle, Int32 action, AsyncCallback callback, Object stateObject)
at System.Messaging.MessageQueue.BeginReceive()
I've used Wireshark on Server1 to look at the network traffic. Without posting all the detail, it seems to go through the following stages. (Server1 is trying to read from a queue on Server2.)
Server1 contacts Server2, and there is an NTLMSSP challenge/response negotiation. A couple of the responses mention "Unknown result (3), reason: Local limit exceeded".
Server1 sends Server2 an rpc__mgmt_inq_princ_name request, and Server2 replies with a corresponding response.
There's some ldap exchanges looking up the domain, then a referral to ldap://domain/cn=msmq,CN=Server2,CN=Computers,DC=domain which returns a "no such object" response.
Then there's some SASL GSS-API encrypted exchange with the LDAP server
Then connections to the ldap server and Server2 are closed.
I've tried enabling Event Viewer > Applications and Services Logs > Microsoft > Windows > MSMQ > End2End. It shows messages being sent, but no indication of why trying to receive is failing.
How can I debug this further?
The problem was related to domains. Server1 and Server2 were part of a development domain. My login account was part of the corporate domain. The development domain trusts the corporate domain enough for me to log in, be a member of administrators, install features etc. But it seems to be insufficient trust to read remote queues.
I found this by looking into public queues. If I was having trouble reading remote private queues, perhaps I should get more data by trying public queues. After installing the appropriate directory integration feature, I was able to create a public queue, but not see it in the list of public queues. Trying to refresh the list of public queues gave me this error:
Not all
public queues can be displayed. Only public queues cached locally can be
displayed. Error: The object was not found in Active Directory.
Google pointed me to John Breakwell's answer to a similar problem here, which indicates that trust relationships don't work across messaging protocols.
Try to use the standard Receive method instead and specify the transaction type as it seems like BeginReceive does not support receiving from transactional queues.
Message msg = queue.Receive(MessageQueueTransactionType.Automatic);
MSMQ does not always return logical error messages...
System.Messaging.MessageQueueException (0x80004005)
at System.Messaging.MessageQueue.MQCacheableInfo.get_ReadHandle()
This error can also be caused due to the BeginReceive Read on an non-existent queue. Check the configuration to ensure queue path specified exists physically and has "Everyone" full permissions
If I have a Websphere MQ queue defined on another queue manager in the cluster, is there a way I can open it for writing using the Perl interface? The code below brings back mqrc 2085.
$messageQ = MQSeries::Queue->new
(
QueueManager => $qMgr,
Queue => $queue,
Options => $openOpt
) or die ">>>ERROR2: Unable to open the queue: $queue\n";
}
Yes! The Perl modules are a thin veneer over the WMQ API and expose all the basic options and most of the really esoteric stuff as well.
When you open a queue, WebSphere MQ performs name resolution on the values you provide for Queue and QMgr names. If you provide both a Queue and a QMgr name then the object reference is fully qualified and WMQ will attempt to open it as named. So if the name you provide is the local QMgr and the clustered queue does not have a locally defined instance, the open will fail with a 2085 Unknown Object Name.
The trick to opening a clustered queue is to provide a null value for the QMgr name. This causes name resolution to check the local QMgr for a queue of the same name, then finding none it checks the cluster repository and resolve the open to the clustered queue. Note that the queue must be advertised to the cluster for this to work. Specifically, the CLUSTER or CLUSNL attribute of the target queue must be non-blank and refer to a cluster that the source QMgr participates in. Similarly, the destination QMgr must also participate in the same cluster as the source QMgr.
Note also that if you specify a QMgr name on the open that is not the local QMgr, then WMQ will attempt to resolve the QMgr name only. If it can resolve a route to that QMgr then it will send the message there. This means that in a cluster you can send a message to any queue on any QMgr so long as you know the fully-qualified name.
Finally, you can define a local alias over a clustered queue. For example if you are on QMGRA and DEF QA(TARGET.QUEUE) TARGQ(TARGET.QUEUE) and then on QMGRB and QMGRC in the same cluster you DEF QL(TARGET.QUEUE) CLUSTER(MYCLUS) then it is possible to open QMGR=QMGRA QUEUE=TARGET.QUEUE and still have it work as expected. Note that the alias is NOT advertised to the cluster but the target queue is. The only issue with this approach is that the first time it is opened the API call may fail if the cluster query takes too long. When I do this in Production, I always use amqsput on the alias ahead of time to make the QMgr query the repository before the actual application opens the queue. Why would you do this? If security is a concern you probably don't want to authorize all apps directly to the cluster XMitQ because, as noted above, they could then put a message onto any queue on any QMgr in the cluster, including SYSTEM.ADMIN.COMMAND.QUEUE. The alias gives you a place to hang authorizations and restrict the user to specific destinations in the cluster.
So short answer, make sure you provide a null QMgr name on the Open call or set up a local alias over the clustered queue. For more about the security aspects of this, see the WMQ Security presentation at http://t-rob.net/links