I'd like to write a catch all eventBus consumer. Is this possible?
eB = vertx.eventBus();
MessageConsumer<JsonObject> consumer = eB.consumer("*"); // What is catch all address ???
consumer.handler(message -> {
Log.info("Received: " + message.body().toString());
});
A solution to your problem might be an interceptor.
vertx.eventBus().addInterceptor( message -> {
System.out.println("LOG: " + message.message().body().toString());
});
This handler will write every message that comes to the event-bus in vertx.
Reference is here:
http://vertx.io/docs/apidocs/io/vertx/rxjava/core/eventbus/EventBus.html#addInterceptor-io.vertx.core.Handler-
Also, version of vertx-core that I'm using is 3.3.2, I think interceptor functionality is not available in older versions (e.g. 3.0.0).
Having looked through the Java code, I don't think this is possible.
Vert.x stores event bus consumers in a MultiMap looking like:
AsyncMultiMap<String, ServerID>
where the String key is the consumer address.
And as you'd guess, Vert.x just does a map.get(address) to find out the relevant consumers.
Update after OP comment
While I think your use case is valid, I think you're going to have to roll something yourself.
As far as I can see, Vert.x doesn't store consumers of send and publish separately. It's all in one MultiMap. So it would be inadvisable to try to register consumers for all events.
If someone does an eventBus.send(), and Vert.x selects your auditing consumer, it will be the only consumer receiving the event, and I'm going to guess that's not what you want.
I dont know if that´s possible but referring to the documentation, you can put a listener to the events to know when a publish, send, open_socket, close_socket is invoked
sockJSHandler.bridge(options, be -> {
if (be.type() == BridgeEvent.Type.PUBLISH || be.type() == BridgeEvent.Type.RECEIVE) {
Log.info("Received: " + message.body().toString());
}
be.complete(true);
});
Related
I'm looking to respond to a REST endpoint with a Success/Failure response that dynamically accepts a topic as a query param. In Quarkus with smallrye reactive messaging the code would look something like below wrapping the payload with OutgoingKafkaRecordMetadata
i.e. https://myendpoint/publishToKafka?topic=myDynamicTopic
#Channel("test")
Emitter<byte []> kafkaEmitter;
#POST
#Path("/publishToKafka")
public CompletionStage<Void> publishRecord(#QueryParam("topic") String topic, byte [] payload){
kafkaEmitter.send(Message.of(payload).addMetadata(OutgoingKafkaRecordMetadata.<String>builder()
.withKey("my-key")
.withTopic("myDynamicTopic")
.build()));
}
From the Quarkus doco "If the endpoint does not return a CompletionStage, the HTTP response may be written before the message is sent to Kafka, and so failures won’t be reported to the user." The example here describes this process when you send a payload directly (i.e. emitter.send(payload) which returns a CompletionStage but emitter.send(message) returns void) but this requires configuring the topic in advance. Is it possible to specify metadata with a Message and still respond to the calling client with a success/failure response? (I don't mind if it's with Emitter and CompletionStage or MunityEmitter and Uni).
Any advice or suggestions would be appreciated.
Because you use a Message (as you need to specify the topic), you need something a bit more convoluted:
#Channel("test")
Emitter<byte []> kafkaEmitter;
#POST
#Path("/publishToKafka")
public CompletionStage<Void> publishRecord(#QueryParam("topic") String topic, byte [] payload){
CompletableFuture<Void> future = new CompletableFuture<>();
Message<byte[]> message = Message.of(payload).addMetadata(OutgoingKafkaRecordMetadata.
<String>builder()
.withKey("my-key")
.withTopic("myDynamicTopic")
.build()));
message = message.withAck(() -> {
future.complete(null));
return CompleteableFuture.completedFuture(null);
}
.withNack(t -> {
future.completeExceptionnaly(t));
return CompleteableFuture.completedFuture(null);
});
kafkaEmitter.send(message);
return future;
}
In this snippet, I also attach the ack and nack handlers called when the message is either acknowledged (accepted by the broker) or rejected (something wrong happened).
These callbacks report to future, a CompletableFuture created in the method. This is the object to return, as it will do what you want: indicate the outcome.
I know the callbacks are slightly complicated. This is mainly due to the spec: We have to return CompleteableFuture.completedFuture(...); to acknowledge that the nack-process was successful. If we were to return future; instead (which we have set to future.completeExceptionnaly(t));), this would be interpreted as a failure during the nack-process. This would basically be the equivalent to a throw within a catch-block in the imperative world.
Fortunately, an easier version will be available soonish (no worries, we won't break).
I'm using Spring Cloud Stream with Kafka Streams. Let's say I have a processor which is a Function which converts a KStream of Strings to a KStream of CityProgrammes. It invokes an API to find the City by name and an other transformation which finds any events near that city.
Now the problem is that any error happens during the transformation, the whole application stops. I want to send that one particular message to a DLQ and move along. I've been reading for days and everyone suggests to handle errors within the called services but that is a nonesense in my opinion, plus I still need to return a KStream: how do I do that within a catch?
I also looked at UncaughtExeptionHandler but it is not aware of the message and only able to restart the processing which won't skip this invalid message.
This might sound like an A-B problem so the question rephrased: how do I maintain the flow in a KStream when an exception occurs and send the invalid item to the DLQ?
When it comes to the application-level errors you have, it is up to the application itself how the error is handled. Kafka Streams and the Spring Cloud Stream binder mainly support deserialization and serialization errors at the framework level. Although that is the case, I think your scenario can be handled. If you are using Kafka Client prior to 2.8, here is an SO answer I gave before on something similar: https://stackoverflow.com/a/66749750/2070861
If you are using Kafka/Streams 2.8, here is an idea that you can use. However, the code below should only be used as a starting point. Adjust it according to your use case. Read more on how branching works in Kafka Streams 2.8. The branching API is significantly refactored in 2.8 from the prior versions.
public Function<KStream<?, String>, KStream<?, Foo>> convert() {
Foo[] foo = new Foo[0];
return input -> {
final Map<String, ? extends KStream<?, String>> branches =
input.split(Named.as("foo-")).branch((key, value) -> {
try {
foo[0] = new Foo(); // your API call for CitiProgramme converion here, possibly.
return true;
}
catch (Exception e) {
Message<?> message = MessageBuilder.withPayload(value).build();
streamBridge.send("to-my-dlt", message);
return false;
}
}, Branched.as("bar"))
.defaultBranch();
final KStream<?, String> kStream = branches.get("foo-bar");
return kStream.map((key, value) -> new KeyValue<>("", foo[0]));
};
}
}
The default branch is ignored in this code because that only contains the records that threw exceptions. Those were handled by the catch statement above in which we send the records to a DLT programmatically. Finally, we get the good records and map them to a new KStream and send it through the outbound.
Is there any other way to get a number of requests waiting to be processed by worker threads when deploying an HTTP server as worker verticle? I need an alternative for https://vertx.io/docs/vertx-dropwizard-metrics/java/#_pool_metrics.
You can try to utilize Asynchronous Counters which you can increment whenever you send an event to specific address on event bus and then decrement when verticle is done processing (or have just picked up an event). If you have lot of verticles and don't want to modify each of them, you can set outbound interceptor:
vertx.eventBus().addOutboundInterceptor(deliveryContext -> {
//you can validate if the address is what you are looking for
if (deliveryContext.message().address().equalsIgnoreCase("http event")) {
//increment counter
}
deliveryContext.next();
});
if you have a lot of addresses to cover you can always add some specific header to the message and then look for it in the interceptor but that would mean you have to modify each .send() call in worst case scenario:
vertx.eventBus().addOutboundInterceptor(deliveryContext -> {
//looking for specific header
if (deliveryContext.message().headers().contains("incrementCounterHeader")) {
//increment counter
}
deliveryContext.next();
});
//later in code
vertx.eventBus().send("http event", message,
new DeliveryOptions().addHeader("incrementCounterHeader", "somevalue"));
last but not least if you decide to use async counter you might want to propagate message only .onComplete() or .onSuccess(), depends on your business logic.
Hope this will help!
I would like to know, if the following Gherkin phrase correspond to BDD rules:
final class KafkaSpec extends BddSpec {
feature("Kafka distribution to SAP server via websocket") {
scenario("Kafka consumer does not receive messages from Kafka server") {
Given("Kafka server is NOT active")
When("consumer client get started")
val ex = SenderActor.run
Then("print message `Failed to connect to Kafka`")
ex.failed map { ex =>
assertThrows[ConnectException](ex)
}
}
scenario("Kafka consumer receives messages from Kafka server") {
Given("Kafka server is ACTIVE")
When("consumer client get started")
Then("print message `Successfully connected to Kafka`")
succeed
}
}
}
Do I use the right tense? Do I use the Given-When-Then correctly?
The Givens (contexts) are fine; we normally use either continuous present or past tense for those:
Given the kafka server is active <-- continuous present
Given the kafka server was started <-- past tense
For the Whens (events), it's better if you can use an active voice. Active voice starts with who did it. Who started the server? (I've corrected the English a bit here too.)
When the consumer client was started <-- passive voice
When our consumer starts their client <-- active voice
For the Thens (outcomes), I really like the word "should". It encourages people to question it; should it really happen? Now? In this release? Is there any context for which this shouldn't happen or something different should happen? Should it still happen, or has this scenario changed?
Then the consumer interface should print the message, `Successfully connected to Kafka`.
One other thing though: the detail in that last step feels a bit too much to me. If the message changed, you'd have to change it everywhere. Instead I keep that in the code (you can abstract the step out) and would say something like:
Then the interface should tell the consumer that the connection was successful.
This is something we usually call "declarative over imperative". It's also OK to have the passive voice here:
Then the consumer should be told that the connection was successful.
Using the word "should" also helps differentiate between the outcomes of one scenario and the givens of another; often these overlap with an outcome forming the context for another scenario:
Given Priscilla has an account
When she enters her username and password correctly
Then she should be on her home page.
Given Priscilla is on her home page...
I wrote more about tenses and language of BDD here, where you'll also find tons of other resources for new BDDers under the BDD category.
How would I handle poison messages when not using WCF? The code below creates a loop, and I was curious if MSMQ provided a system to automatically handle poison messages.
MessageQueue mq = new MessageQueue(#".\Private$\My/Queue");
while (true)
{
using (MessageQueueTransaction _transaction =
new MessageQueueTransaction())
{
_transaction.Begin();
try
{
Message msg = mq.Receive(_transaction);
//HandleMessage(msg);
throw new Exception("Kaboom!");
_transaction.Commit();
}
catch (Exception ex)
{
_transaction.Abort();
}
}
}
I don't believe there is a simple way to handle poison messages using raw System.Messaging classes. I think the simplest solution is to set the "TimeToBeReceived" property on a message, but this is not perfect, because you may end up losing valid messages if the receiver is offline. I read somewhere that you can have real poison message handling in MSMQ using PInvoke, but was unable to find any resources on this.
I found this article with some ideas on how to manually handle poison messages. It might offer some ideas:
http://www.cogin.com/articles/SurvivingPoisonMessages.php