In Quarkus/small rye, we can send message to dynamic topic. Please check the below link for the example.
https://beyondvelocity.blog/2022/01/05/dynamic-kafka-topics-in-quarkus/
Kindly suggest how can we implement same in Helidon
I could not find equivalent api or classes in helidon to send message to dynamic topic.
There's nothing preventing you from using KafkaProducer + ProducerRecord classes directly and calling send method with any topic String parameter
Otherwise, just create the Channel with the topic name when you need a dynamic value
KafkaConnector kafkaConnector = KafkaConnector.create();
messaging = Messaging.builder()
.publisher(
Channel.<String>builder()
.subscriberConfig(KafkaConnector.configBuilder()
.bootstrapServers(kafkaServer)
.topic("some random string")
.keySerializer(StringSerializer.class)
.valueSerializer(StringSerializer.class)
.build()
).build(),
Multi.just("test1", "test2").map(Message::of) // example messages
)
.connector(kafkaConnector)
.build()
.start();
https://helidon.io/docs/v2/#/se/reactivemessaging/04_kafka
Related
I have a spring boot camel app.
I am reading from an incoming Kafka topic queue and running a processor that lookup a new destination topic from the Redis cache. In the processor, I add this as a new camel header.
exchange.getIn().setHeader("destTopic", cachelookupvalue);
LOG.info ("RouterProcessor Dest Topic set to:= {}", exchange.getIn().getHeader("destTopic"));
This seems to work, and the log shows it set.
On returning to the route builder class
public class RouterBuilder extends RouteBuilder {
...
//setup env properties etc
#Override
public void configure() {
from("kafka:" + SRC_TOPIC + "?brokers=" + SRC_BROKER).routeId("myRoute")
.log("Sending message to kafka: ${header.destTopic}" )
.process(processor)
.to("kafka:${header.destTopic}?brokers="+DEST_BROKER) //not working
but
//.to("kafka:webhook-channel_1-P101?brokers="+DEST_BROKER) //this works syntax
If i hardcode the the ${header.destTopic} expression it work but not if i try to use the header in the .to dsl it does not.
The Log .log("Sending message to kafka: ${header.destTopic}" seems to be right topic.
I am not sure if it's a syntax problem or if I'm missing a step.
The error output is
Failed delivery for (MessageId: 972B795CF230E52-0000000000000001 on
ExchangeId: 972B795CF230E52-0000000000000001). Exhausted after
delivery attempt: 1 caught:
org.apache.kafka.common.errors.InvalidTopicException:
${header.destTopic}
Headers , properties and body ..etc are dynamic values in camel so you must use toD(to Dynamic)
.toD("kafka:${headers.destTopic}?brokers="+DEST_BROKER) //not working
Based on the Akka documentation for Cluster, would I like to Publish a Broad message to all nodes in the Cluster except myself. Currently it's always sending the Broad message to me as well.
val mediator = DistributedPubSub(context.system).mediator
mediator ! Subscribe("content", self) // subscribe to the topic named "content"
mediator ! Publish("content", "msg") // sends the msg out Broad to all nodes including myself
How exactly can I set the documentation property "allButSelf"?
https://doc.akka.io/docs/akka/current/distributed-pub-sub.html
You want to do
mediator ! DistributedPubSubMediator.Put(testActor)
mediator ! DistributedPubSubMediator.SendToAll(path, msg, allButSelf=false) // it is false by default
See the example here https://github.com/akka/akka/blob/0e4d41ad33dbeb00b598cb75b4d29899371bdc8c/akka-cluster-tools/src/test/scala/akka/cluster/pubsub/DistributedPubSubMediatorRouterSpec.scala#L56
I'm trying to inject a KafkaTemplate to send a single message. I'm developing a small function that lies outside the reactive approach.
I can only find examples that use #Ingoing and #Outgoing from Smallrye but I don't need a KafkaStream.
I tried with Kafka-CDI but I'm unable to inject the SimpleKafkaProducer.
Any ideas?
For Clement's answer
It seems the right direction, but executing orders.send("hello"); I receive this error:
(vert.x-eventloop-thread-3) Unhandled exception:java.lang.IllegalStateException: Stream not yet connected
I'm consuming from my topic by command line, Kafka is up and running, if I produce manually I can see the consumed messages.
It seems relative to this sentence by the doc:
To use an Emitter for the stream hello, you need a #Incoming("hello")
somewhere in your code (or in your configuration).
I have this code in my class:
#Incoming("orders")
public CompletionStage<Void> consume(KafkaMessage<String, String> msg) {
log.info("Received message (topic: {}, partition: {}) with key {}: {}", msg.getTopic(), msg.getPartition(), msg.getKey(), msg.getPayload());
return msg.ack();
}
Maybe I've forgotten some configurations?
So, you just need to use an Emitter:
#Inject
#Stream("orders") // Emit on the channel 'orders'
Emitter<String> orders;
// ...
orders.send("hello");
And in your application.properties, declare:
## Orders topic (WRITE)
mp.messaging.outgoing.orders.type=io.smallrye.reactive.messaging.kafka.Kafka
mp.messaging.outgoing.orders.topic=orders
mp.messaging.outgoing.orders.bootstrap.servers=localhost:9092
mp.messaging.outgoing.orders.key.serializer=org.apache.kafka.common.serialization.StringSerializer
mp.messaging.outgoing.orders.value.serializer=org.apache.kafka.common.serialization.StringSerializer
mp.messaging.outgoing.orders.acks=1
To avoid Stream not yet connected exception, as suggested by doc:
To use an Emitter for the stream hello, you need a #Incoming("hello")
somewhere in your code (or in your configuration).
Assuming you have something like this in your application.properties:
# Orders topic (READ)
smallrye.messaging.source.orders-r-topic.type=io.smallrye.reactive.messaging.kafka.Kafka
smallrye.messaging.source.orders-r-topic.topic=orders
smallrye.messaging.source.orders-r-topic.bootstrap.servers=0.0.0.0:9092
smallrye.messaging.source.orders-r-topic.key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
smallrye.messaging.source.orders-r-topic.value.deserializer=org.apache.kafka.common.serialization.StringDeserializer
smallrye.messaging.source.orders-r-topic.group.id=my-group-id
Add something like this:
#Incoming("orders-r-topic")
public CompletionStage<Void> consume(KafkaMessage<String, String> msg) {
log.info("Received message (topic: {}, partition: {}) with key {}: {}", msg.getTopic(), msg.getPartition(), msg.getKey(), msg.getPayload());
return msg.ack();
}
Since Clement's answer the #Stream annotation has been deprecated. The #Channel annotation
must be used instead.
You can use an Emitter provided by the quarkus-smallrye-reactive-messaging-kafka dependency to produce message to a Kafka topic.
A simple Kafka producer implementation:
public class MyKafkaProducer {
#Inject
#Channel("my-topic")
Emitter<String> myEmitter;
public void produce(String message) {
myEmitter.send(message);
}
}
And the following configuration must be added to the application.properties file:
mp.messaging.outgoing.my-topic.connector=smallrye-kafka
mp.messaging.outgoing.my-topic.bootstrap.servers=localhost:9092
mp.messaging.outgoing.my-topic.value.serializer=org.apache.kafka.common.serialization.StringSerializer
This will produce string serialized messages to a kafka topic named my-topic.
Note that by default the name of the channel is also the name of the kafka topic in which the data will be produced. This behavior can be changed through the configuration. The supported configuration attributes are described in the reactive Messaging documentation
I have a problem where I have to broadcast messages to different output locations. I am using JMSComponent for configuring my output queues. My output queues configuration have something like this:
ConnectionFactory factory = createOrGetConnectionFactory(brokerUrl);
JmsConfiguration jmsConfiguration = new JmsConfiguration();
jmsConfiguration.setPreserveMessageQos(true);
jmsConfiguration.setConnectionFactory(factory);
counter++;
outputLocations = new StringBuilder("hubOutput"+counter+":queue://queueName");
JmsEndpoint endpoint = new JmsEndpoint();
JmsComponent component = new JmsComponent();
component.setConcurrentConsumers(5);
component.setConfiguration(jmsConfiguration);
component.setConnectionFactory(factory);
//Add new JMS component in the context. This is done so that the output locations having same queue can be differentiated using the component name in camel registry. getContext().addComponent("hubOutput"+counter, component);
endpoint = (JmsEndpoint) component.createEndpoint(outputLocations.toString());
endpoint.setConfiguration(jmsConfiguration);
I have a camel route for broadcasting the messages to the output queues.
from(fromLocation)
.setHeader("hubRoutesList",constant(hubUrl))
.log(urlToLog)
.setExchangePattern(ExchangePattern.InOnly)
.multicast()
.parallelProcessing()
.to(hubUrl.split(","));
All the output queues have different broker URL but same queue name.
The code works normally but if one of the queues is down, then the message is not broadcasted to other queues also.
Kindly help me with this.
Thanks,
Richa
You can use the recipientList instead of .to(hubUrl.split(","));
With the option stopOnException=false, which is the defaultValue, the forwarding of the messages to the other endpoints will not stop even if one of your queues is down.
See http://camel.apache.org/recipient-list.html for more information.
I am sending a file as a message by converting it to a byte array using kafka producer.
I also need to add some headers for the message, for example file name, timestamps etc so at the consumer end I can process the message based on file name and other headers.
What I am currently doing is creating a object and wrapping the raw message and headers in it and sending the object in a byte array as a message.
I would like to know if there is a way by which I can add custom headers while publishing the message?
Kafka v0.11.0.0 adds support for custom headers.
You can add them when creating a ProducerRecord like this:
new ProducerRecord(key, value, headers, ...), where headers is of type Iterable<Header>
For more details see:
https://issues.apache.org/jira/browse/KAFKA-4208
https://cwiki.apache.org/confluence/display/KAFKA/KIP-82+-+Add+Record+Headers
Record level headers were introduced from Kafka 0.11.0. We can send a list of Headers in each record.
List<Header> headers = Arrays.asList(new RecordHeader("header_key", "header_value".getBytes()));
ProducerRecord<String, String> record = new ProducerRecord<>("topic", null, "key", "value", headers);
You can create your own small java application to send the message with headers to kafka.
Write the following code in intellij or any supporting IDE:-
public static void main(String[] args) throws JsonProcessingException,
InterruptedException {
Properties props=new Properties();
props.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,"localhost:9092");
props.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
JsonSerializer.class.getName());
props.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,JsonSerializer.class.getName());
KafkaProducer<String, JsonNode> producer=new KafkaProducer<String, JsonNode>(props);
String json = "{ \"f1\" : \"v1\" } ";
ObjectMapper mapper = new ObjectMapper();
JsonNode jsonNode = mapper.readTree(json);
ProducerRecord<String,JsonNode> record =new ProducerRecord<String,
JsonNode>("test topic", jsonNode);
record.headers().add(new RecordHeader("key","value1".getBytes()));
producer.send(record);
Thread.sleep(10000);
}
String json = "{ \"f1\" : \"v1\" } "- this is the key and value we want to send to kafka using objectMapper and converting it into jsonNode form.
record.headers().add(new RecordHeader("key","value1".getBytes()))-This is the key and value of headers data that we are sending to kafka.
To verify your data you can check the topic in the kafka control center and verify the headers sent.
Kafka is agnostic to the message content and doesn't provide any special means to enrich it so this is something you need to do yourself. A common way of dealing with these things is to use a structured format such as json, avro or similar where you are free to define the necessary fields and could easily add metadata to your message and ship it off to the Kafka brokers.
This answer is outdated as of Kafka 0.11, please see other answers.
Another Solution
ProducerRecord<String, String> producerRecord = new ProducerRecord<>("bizops", "value");
producerRecord.headers().add("client-id", "2334".getBytes(StandardCharsets.UTF_8));
producerRecord.headers().add("data-file", "incoming-data.txt".getBytes(StandardCharsets.UTF_8));
// Details left out for clarity
producer.send(producerRecord);
https://www.confluent.io/blog/5-things-every-kafka-developer-should-know/#adding-headers
I've been through similar problems with projects I've worked on so I created this simple library to help tackle that: https://github.com/leandronunes85/messaging. For now contains an Avro based implementation but it can be extended to use any other serialization framework of your choice.
You just have to create a (de)serializer for the objects you want to have on the stream (Avro based or not) and let AvroMessageSerializer work its magic.
This is still a very young library but I feel it can save many people a lot of time!