Gcloud PubSub Java implementation - java.util.concurrent.RejectedExecutionException - gcloud

I use the sample snippet from GCloud documentation to receive msg as a subscriber. My pubsub gcloud jar version is 0.19.0-alpha
The problem is that I can receive the msg with attribute map but I keep having this exception:
2017-07-12 16:52:25,219 [grpc-default-worker-ELG-1-16] WARN io.netty.util.concurrent.DefaultPromise - An exception was thrown by io.grpc.netty.NettyClientHandler$3.operationComplete()
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#fbf4a6d rejected from java.util.concurrent.ScheduledThreadPoolExecutor#25cbe860[Terminated, pool size = 35, active threads = 0, queued tasks = 0, completed tasks = 2403]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326)
at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
at java.util.concurrent.ScheduledThreadPoolExecutor.execute(ScheduledThreadPoolExecutor.java:622)
at java.util.concurrent.Executors$DelegatedExecutorService.execute(Executors.java:668)
at io.grpc.internal.SerializingExecutor.execute(SerializingExecutor.java:110)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.onReady(ClientCallImpl.java:573)
at io.grpc.internal.DelayedStream$DelayedStreamListener.onReady(DelayedStream.java:398)
at io.grpc.internal.AbstractStream2$TransportState.notifyIfReady(AbstractStream2.java:305)
at io.grpc.internal.AbstractStream2$TransportState.onStreamAllocated(AbstractStream2.java:248)
at io.grpc.netty.NettyClientStream$TransportState.setHttp2Stream(NettyClientStream.java:227)
at io.grpc.netty.NettyClientHandler$3.operationComplete(NettyClientHandler.java:429)
at io.grpc.netty.NettyClientHandler$3.operationComplete(NettyClientHandler.java:417)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
After that, the program shuts and stop listening and getting msg. How to resolve this interruption and I even get rid of finally clause that has subscriber.stopAsync().

There is a bug in the snippet provided by them. You need to call get() on the messaegeIdFuture. Following code resolves the issue:
Publisher publisher = null;
String projectId = ServiceOptions.getDefaultProjectId();
ProjectTopicName topic = ProjectTopicName.of(projectId, "test");
ApiFuture<String> messageIdFuture = null;
try {
publisher = Publisher.newBuilder(topic).build();
ByteString data = ByteString.copyFromUtf8("my-message");
PubsubMessage pubsubMessage = PubsubMessage.newBuilder().setData(data).build();
messageIdFuture = publisher.publish(pubsubMessage);
} catch (IOException e) {
e.printStackTrace();
} finally {
messageIdFuture.get(); //This resolves this issue.
// Wait on any pending requests
if (publisher != null) {
publisher.shutdown();
//publisher.awaitTermination(1, TimeUnit.SECONDS);
}
}

Related

Google PubSub ACK not received after processing few messages

We are using gcloud pub-sub 1.102.0 release.
Problem :- We have around 8K messages to process. We have 3 subscribers running in 3 different k8s pods and we are doing stream pulling. We are receiving the ACK for few messages (around 100) from there on wards we are not receiving the ACKs (Verified in GCP console). But the message process is going on in background. And we saw couple of duplicate messages as well. In our use case to process a single message it takes around 40 secs to 1 min. Ack deadline has been configured to 10 mins while creating the subscription.
Flow control configured with 20L.
ExecutorProvider configured with 3.
public void runSubsciber() throws Exception {
Subscriber subscriber = null;
MessageReceiver receiver = new MessageReceiver() {
public void receiveMessage(PubsubMessage message, AckReplyConsumer consumer) {
messageProcessor.processMessage(message.getData().toStringUtf8());
consumer.ack();
}
};
try {
FlowControlSettings flowControlSettings =
FlowControlSettings.newBuilder()
.setMaxOutstandingElementCount(20L)
.build();
ExecutorProvider executorProvider =
InstantiatingExecutorProvider.newBuilder().setExecutorThreadCount(3).build();
projectId=ServiceOptions.getDefaultProjectId();
ProjectSubscriptionName subscriptionName = ProjectSubscriptionName.of(projectId, subId);
subscriber = Subscriber.newBuilder(subscriptionName, receiver)
.setExecutorProvider(executorProvider)
.setFlowControlSettings(flowControlSettings)
.build();
subscriber.addListener(new SubscriberListener(), MoreExecutors.directExecutor());
subscriber.startAsync().awaitRunning();
} catch (Exception e) {
throw new Exception(e);
} finally {
if (subscriber != null) {
//subscriber.stopAsync();
}
}
}
Please help us out, Thanks in advance.

Kafka async publishing failure case, does it throw exception?

Below is my code for publishing message to kafka.
I need to know what happens in below cases ,
When message is sent async and doesnt get ack from all relicas, does it throw exception in client side ?
What happens when message is in buffer but never sent to kafka, does it throw exception ?
What happens in async publishing with retries =1 and retries =0 ?
After send() call it added to buffer and returned , now if it fails to sent to kafka, does it throw exception ?
I have a case where it failed to publish to kafka but didnt get any exception . Why ?
//Kafka
Properties prop = new Properties();
prop.put("bootstarp.servers","");
prop.put("acks","all");
prop.put("retries",0);
KafkaProducer connection = null;
try {
connection = new KafkaProducer<String, byte[]>(props);
msg.setTopic(topic);
msg.setDate(new Date());
connection.send(msg);
} catch() {
} finally {
connection.close();
}

Kafka Transactional Producer

I am using Kafka 2 and I was going through the following link.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging
Below is my sample code for Transactional producer.
My code:
public void runProducer(final int sendMessageCount) throws Exception {
final Producer<Long, String> producer = createProducer();
producer.initTransactions();
final long time = System.currentTimeMillis();
try {
producer.beginTransaction();
for (long index = time; index < (time + sendMessageCount); index++) {
final ProducerRecord<Long, String> record =
new ProducerRecord<>(TOPIC, index,
"Test " + index);
// send returns Future
producer.send(record).get();
}
producer.commitTransaction();
}
catch (ProducerFencedException | OutOfOrderSequenceException | AuthorizationException e) {
e.printStackTrace();
// We can't recover from these exceptions, so our only option is to close the producer and exit.
producer.close();
}
catch (final KafkaException e) {
e.printStackTrace();
// For all other exceptions, just abort the transaction and try again.
producer.abortTransaction();
}
finally {
producer.flush();
producer.close();
}
}
Questions:
Do we need to call endTransaction after commitTransaction ?
Do we need to call sendOffsetsToTransaction? What will happen if I don't include this?
How does it work when we deploy the same code to multiple servers with same transactionId? Do we need to have a separate transactionId for each instance? Say, machine1 crashes after beginTransaction() and after sending few records? How does machine2 with same transactionId recovers.
Machine1 is using transactionId "test" and it crashed after beginTransaction() and after producing few records. When the same instance comes up how does it resume the same transaction? We will actually again start from init & begin transaction.
How does it work for the same topic which was not involving in transaction and involving in transaction now? I am starting a new consumerGroup with transaction_committed, Will it read the messages which were committed before the transaction? Will the consumer with transaction_uncommitted see the messages which were aborted by transaction?

How to read message from a IBM MessageQueue or remote MQ

Currently I'm doing MQ scripting in load runner using JAVA Vuser protocol.I'm using one Input queue and one Output queue. I'm able to put the message using Input queue successfully, But I'm unable to read the message from the output queue.
Below is the code I'm using to PUT/GET message from MQ. Kindly Let me know how to read message from output MQ.
lr.start_transaction("test_message");
try {
MQQueue destQueue1 = queueMgr.accessQueue(putQueueName, MQC.MQOO_INQUIRE);
pmo.options = MQC.MQPMO_NEW_MSG_ID;
requestMsg.replyToQueueName =getQueueName;
requestMsg.report=MQC.MQRO_PASS_MSG_ID;
requestMsg.format = MQC.MQFMT_STRING;
requestMsg.messageType=MQC.MQMT_REQUEST;
requestMsg.writeString(msgBody);
putQueue.put(requestMsg, pmo);
} catch(Exception e) {
lr.error_message("Error sending message.");
lr.exit(lr.EXIT_VUSER, lr.FAIL);
}
putQueue.close();
// Get the response message object from the response queue
try {
responseMsg.correlationId = requestMsg.messageId;
gmo.matchOptions=MQC.MQMO_MATCH_CORREL_ID;
gmo.options= MQC.MQGMO_NO_SYNCPOINT;
gmo.matchOptions=MQC.MQMO_NONE;
gmo.options= MQC.MQGMO_SYNCPOINT;
gmo.options= MQC.MQGMO_CONVERT;
gmo.options= MQC.MQGMO_WAIT;
gmo.waitInterval=MQC.MQWI_UNLIMITED;
gmo.waitInterval=60000;
getQueue.get(responseMsg, gmo);
System.out.println("QueueDepth for get:"+getQueue.getCurrentDepth());
//Check the message content
byte[] responseMsgData = responseMsg.readStringOfByteLength(responseMsg.getTotalMessageLength()).getBytes();
String msg = new String(responseMsgData);
lr.output_message(msg);
} catch(Exception e) {
lr.error_message("Error receiving message.");
lr.exit(lr.EXIT_VUSER, lr.FAIL);
}
lr.end_transaction("test_message", lr.AUTO);
You seem to be new to MQ. There are multiple problems in your code. Here is a piece of code demonstrating MQ Request/Response scenario. Code is developed using MQ v8. Modify it according to your MQ version and need.
/**
* Reqeust reply scenario
*/
public void mqRequestRespose() {
Hashtable<String, Object> properties;
try {
System.out.println("***Request/Reply Started *** ");
properties = new Hashtable<String, Object>();
properties.put("hostname", "localhost");
properties.put("port", new Integer(1414));
properties.put("channel", "APP.SVRCONN.CHN");
properties.put(MQConstants.USE_MQCSP_AUTHENTICATION_PROPERTY,"true");
properties.put(MQConstants.USER_ID_PROPERTY, "username");
properties.put(MQConstants.PASSWORD_PROPERTY, "password");
/**
* Connect to a queue manager
*/
MQQueueManager queueManager = new MQQueueManager("APPQMGR", properties);
/**
* Now create a subscription by providing our own temporary queue
*/
MQQueue mqRequestQ = queueManager.accessQueue("REQUEST.QUEUE", CMQC.MQOO_FAIL_IF_QUIESCING | CMQC.MQOO_OUTPUT );
MQQueue mqReplyQ = queueManager.accessQueue("REPLY.QUEUE", CMQC.MQOO_FAIL_IF_QUIESCING | CMQC.MQOO_INPUT_AS_Q_DEF);
/**
* Build a request message and send it to request queue.
*/
System.out.println("***Sending a request ***");
MQMessage msgRequest = new MQMessage();
msgRequest.writeUTF("Give me quote for IBM");
mqRequestQ.put(msgRequest);
/**
* Wait for 30 seconds to receive reply from reply queue
*/
System.out.println("*** Waiting for reply ***");
MQGetMessageOptions mqgmo = new MQGetMessageOptions();
mqgmo.options = CMQC.MQGMO_WAIT | CMQC.MQGMO_CONVERT;
mqgmo.waitInterval = 30000;
mqgmo.matchOptions=CMQC.MQMO_MATCH_CORREL_ID;
MQMessage msgReply = new MQMessage();
msgReply.correlationId = msgRequest.messageId;
try {
mqReplyQ.get(msgReply, mqgmo);
System.out.println("***Reply received***");
System.out.println("STOCK QUOTE: USD" + msgReply.readUTF());
}catch (MQException mqex) {
System.out.println("***No reply received in given time***");
}
} catch (Exception e) {
System.err.println(e);
e.printStackTrace();
for (Throwable t = e.getCause(); t != null; t = t.getCause()) {
System.out.println("... Caused by ");
t.printStackTrace();
}
}
}

Cloud Service for incoming TCP connections hangs

I'm developing a cloud service (worker role) for collecting data from a number of instruments. These instruments reports data randomly every minute or so. The service itself is not performance critical and doesn't need to be asynchronous. The instruments are able to resend their data up to an hour on failed connection attempt.
I have tried several implementations for my cloud service including this one:
http://msdn.microsoft.com/en-us/library/system.net.sockets.tcplistener.stop(v=vs.110).aspx
But all of them hang my cloud server sooner or later (sometimes within an hour).
I suspect something is wrong with my code. I have a lot of logging in my code but I get no errors. The service just stops to receive incoming connections.
In Azure portal it seems like the service is running fine. No error logs and no suspicious cpu usage etc.
If I restart the service it will run fine again until it hangs next time.
Would be most grateful if someone could help me with this.
public class WorkerRole : RoleEntryPoint
{
private LoggingService _loggingService;
public override void Run()
{
_loggingService = new LoggingService();
StartListeningForIncommingTCPConnections();
}
private void StartListeningForIncommingTCPConnections()
{
TcpListener listener = null;
try
{
listener = new TcpListener(RoleEnvironment.CurrentRoleInstance.InstanceEndpoints["WatchMeEndpoint"].IPEndpoint);
listener.Start();
while (true)
{
_loggingService.Log(SeverityLevel.Info, "Waiting for connection...");
var client = listener.AcceptTcpClient();
var remoteEndPoint = client.Client != null ? client.Client.RemoteEndPoint.ToString() : "Unknown";
_loggingService.Log(SeverityLevel.Info, String.Format("Connected to {0}", remoteEndPoint));
var netStream = client.GetStream();
var data = String.Empty;
using (var reader = new StreamReader(netStream, Encoding.ASCII))
{
data = reader.ReadToEnd();
}
_loggingService.Log(SeverityLevel.Info, "Received data: " + data);
ProcessData(data); //data is processed and stored in database (all resources are released when done)
client.Close();
_loggingService.Log(SeverityLevel.Info, String.Format("Connection closed for {0}", remoteEndPoint));
}
}
catch (Exception exception)
{
_loggingService.Log(SeverityLevel.Error, exception.Message);
}
finally
{
if (listener != null)
listener.Stop();
}
}
private void ProcessData(String data)
{
try
{
var processor = new Processor();
var lines = data.Split('\n');
foreach (var line in lines)
processor.ProcessLine(line);
processor.ProcessMessage();
}
catch (Exception ex)
{
_loggingService.Log(SeverityLevel.Error, ex.Message);
throw new Exception(ex.InnerException.Message);
}
}
}
One strange observation i just did:
I checked the log recently and no instrument has connected for the last 30 minutes (which indicates that the service is down).
I connected to the service myself via a TCP client i've written myself and uploaded some test data.
This worked fine.
When I checked the log again my test data had been stored.
The strange thing is, that 4 other instruments had connected about the same time and send their data successfully.
Why couldn't they connect by themself before I connected with my test client?
Also, what does this setting in .csdef do for an InputEndpoint, idleTimeoutInMinutes?
===============================================
Edit:
Since a cuple of days back my cloud service has been running successfully.
Unfortunately this morning last log entry was from this line:
_loggingService.Log(SeverityLevel.Info, String.Format("Connected to {0}", remoteEndPoint));
No other connections could be made after this. Not even from my own test TCP client (didn't get any error though, but no data was stored and no new logs).
This makes me think that following code causes the service to hang:
var netStream = client.GetStream();
var data = String.Empty;
using (var reader = new StreamReader(netStream, Encoding.ASCII))
{
data = reader.ReadToEnd();
}
I've read somewhere that StremReader's ReadToEnd() could hang. Is this possible?
I have now changed this piece of code to this:
int i;
var bytes = new Byte[256];
var data = new StringBuilder();
const int dataLimit = 10;
var dataCount = 0;
while ((i = netStream.Read(bytes, 0, bytes.Length)) != 0)
{
data.Append(Encoding.ASCII.GetString(bytes, 0, i));
if (dataCount >= dataLimit)
{
_loggingService.Log(SeverityLevel.Error, "Reached data limit");
break;
}
dataCount++;
}
Another explanation could be something hanging in the database. I use the SqlConnection and SqlCommand classes to read and write to my database. I always close my connection afterwards (finally block).
SqlConnection and SqlCommand should have default timeouts, right?
===============================================
Edit:
After some more debugging I found out that when the service wasn't responding it "hanged" on this line of code:
while ((i = netStream.Read(bytes, 0, bytes.Length)) != 0)
After some digging I found out that the NetStream class and its read methods could actually hang. Even though MS declares otherwise.
NetworkStream read hangs
I've now changed my code into this:
Thread thread = null;
var task = Task.Factory.StartNew(() =>
{
thread = Thread.CurrentThread;
while ((i = netStream.Read(bytes, 0, bytes.Length)) != 0)
{
// Translate data bytes to a ASCII string.
data.Append(Encoding.ASCII.GetString(bytes, 0, i));
}
streamReadSucceeded = true;
});
task.Wait(5000);
if (streamReadSucceeded)
{
//Process data
}
else
{
thread.Abort();
}
Hopefully this will stop the hanging.
I'd say that part of your problem is you are processing your data on the thread that listens for connections from clients. This would prevent new clients from connecting if another client has started a long running operation of some type. I'd suggest you defer your processing to worker threads thus freeing the "listener" thread to accept new connections.
Another problem you could be experiencing, if your service throws an error, then the service will stop accepting connections as well.
private static void ListenForClients()
{
tcpListener.Start();
while (true)
{
TcpClient client = tcpListener.AcceptTcpClient();
Thread clientThread = new Thread(new ParameterizedThreadStart(HandleClientComm));
clientThread.Start(client);
}
}
private static void HandleClientComm(object obj)
{
try
{
using(TcpClient tcpClient = (TcpClient)obj)
{
Console.WriteLine("Got Client...");
using (NetworkStream clientStream = tcpClient.GetStream())
using (StreamWriter writer = new StreamWriter(clientStream))
using(StreamReader reader = new StreamReader(clientStream))
{
//do stuff
}
}
}
catch(Exception ex)
{
}
}