We are using NATS as our eventBus impl and noticed one issue where our msgs got missed from the eventbus with no error or alerts from ARGO :-
Let's say we are using a sqs trigger to trigger our workflows, the whole flow is below:-
1) a web app publishes a msg to a SQS.
2) SQS event source polls the msg from sqs, put it into event-bus and delete the msg from SQS.
3) sensor polls the msg from event-bus and try to hit a workflow trigger but somehow fails to trigger the workflow(even it fails to start the k8s pod for the workflow).
4) Due to step 3, we have lost the polled msg with no error,alert etc.
My question is how we can make this thing resillient i.e how to configure sensor so that msg can only be deleted from the event-bus iff workflow has been triggered and if not how to run some alternate light-weight failover workflow?
Related
I am trying understand how to handle failed consumer records. How to
we know there is record failure. What I am seeing is when the record
processing failed in the consumer with runtime exception consumer is
keep on retrying. But when the next record is available to process it
is commiting offset of the latest record, which is expected. My
question how to we know about failed record. In older messaging
systems failed messages are rolled back to queues and processing stops
there. Then we know the queue is down and we can take action.
I can record the failed record into some db table,but what happens if this recording fails?
I can move failures to error/ dead letter queues, again what happens if this moving fails?
I am using kafka 2.6 with spring boot 2.3.4. Any help would be appreciated
Sounds like you would need to disable auto commits and manually commit the offsets yourself when your scope of "sucessfully processed" is achieved. If you include external processes like a database, then you will also need to increase Kafka client timeouts so it doesnt think the consumer is dead while waiting on error logging/handling.
I would like to by-pass starttime and frequency parameter to get workflow started immediatly when the message is received from the kafka broker.
is is possible to add some plugin/class to oozie to realize this ?
You would need to create a consumer to submit an Oozie workflow since Oozie does not have such an event-driven coordinator.
NiFi would be more inline with what you're trying to do
I have oenwhisk system running and working.
I also of RabbitMQ broker that publish messages via MQTT protocol.
what is the best practice to create subscriber for that rabbitMQ and invoke actions in openwhisk?
is it possible by Webhook? Trigger?
I don't want to create external service that listen and invoke actions, because I'm not get the benefits of openwhisk build in actions and triggers management.
Thanks
Apache OpenWhisk does not have a built-in trigger feed for RabbitMQ event source. It does support Apache Kafka if you can use another general purpose event stream. If this is not possible, there are a few options.
Use RabbitMQ Webhook plugin to enable RabbitMQ to invoke OpenWhisk actions.
Use Alarm feed to poll RabbitMQ queue on a regular interval for new messages.
I am using rabbitmq to send message between 2 services in micro-service.
I am having a problem. Can I setting the queue pause push message to consumer and continue to process message when I want? Or make consumer pause to get message out from queue and continue to get message when I want (But don't use the way stop/start consumer. Because I can't do it in my system.)?
If YES, can I do it by RabbitMQ Management HTTP API?
No you can't do it. These are consuming policies. Maybe you can stop the publish.
Read also thread about: https://groups.google.com/forum/#!topic/rabbitmq-users/68-DPZN4b_Q
I am using Oracle Fusion Middleware 12.1.3. Weblogic Server 12.1.3 and OSB 12.1.3.
I have created 1 connection factory and one topic. I have one producer sending messages to the topic and 3 consumers(subscribers)
I have also set redelivery failure settings (retry 3 times every half hour) in case there is a connection error or network issue the messages will be written back to the topic and will be retried.
but I want to make sure that the messages are retried in the same order that it was received.
i.e. for example there are 3 messages in the topic (message 1, message 2, message 3) and one of the subscriber is not able to consume the message so the message is pending in the topic and it will be retried.
but I want it to be retried in same order i.e. message 1, message2 and message 3.
Is there any specific setting to be done in weblogic or OSB to achieve this behavior
Have you tried using the weblogic-specific Unit of Order feature?
What Is Message Unit-Of-Order?
Message Unit-of-Order is a WebLogic Server value-added feature that enables a stand-alone message producer, or a group of producers acting as one, to group messages into a single unit with respect to the processing order. This single unit is called a Unit-of-Order and requires that all messages from that unit be processed sequentially in the order they were created.
You can configure it programmatically for more control, or administratively (via WLS console, attaching one to connection factories etc) if you don't have control over the messages produced.
For more info about how to attach the JMS headers to enable it, you might find this site helpful.