In google cloud's pubsub, I can see that on creating a new topic, I have to create a new message. Can I store a protobuf file there instead of having to write the whole structure of message in key-value pairs? For the protobuf code that shall be written, I mean this. If the protobuf isnt meant to be put on the gcloud pubsub, how can I use it with grpc client?
If you want to send a ProtoBuf message via the Publish API, you should do so by serializing it to a ByteString and then setting it as the message's data field. For example, if you are using the Java client library and you have a Publisher and obj of some ProtoBuf type, then you could do the following:
PubsubMessage message = PubsubMessage.newBuilder()
.setData(obj.toByteString())
.build();
ApiFuture<String> response = publisher.publish(message);
...
On the subscribe side, you would decode the message in your MessageReceiver:
public void receiveMessage(PubsubMessage message, AckReplyConsumer consumer) {
ProtoBufMessage obj;
try {
obj = ProtoBufMessage.parseFrom(message.getData());
} catch (Exception e) {
// Handle improperly encoded message
}
...
}
Related
I'm using Spring Cloud Stream with Kafka Streams. Let's say I have a processor which is a Function which converts a KStream of Strings to a KStream of CityProgrammes. It invokes an API to find the City by name and an other transformation which finds any events near that city.
Now the problem is that any error happens during the transformation, the whole application stops. I want to send that one particular message to a DLQ and move along. I've been reading for days and everyone suggests to handle errors within the called services but that is a nonesense in my opinion, plus I still need to return a KStream: how do I do that within a catch?
I also looked at UncaughtExeptionHandler but it is not aware of the message and only able to restart the processing which won't skip this invalid message.
This might sound like an A-B problem so the question rephrased: how do I maintain the flow in a KStream when an exception occurs and send the invalid item to the DLQ?
When it comes to the application-level errors you have, it is up to the application itself how the error is handled. Kafka Streams and the Spring Cloud Stream binder mainly support deserialization and serialization errors at the framework level. Although that is the case, I think your scenario can be handled. If you are using Kafka Client prior to 2.8, here is an SO answer I gave before on something similar: https://stackoverflow.com/a/66749750/2070861
If you are using Kafka/Streams 2.8, here is an idea that you can use. However, the code below should only be used as a starting point. Adjust it according to your use case. Read more on how branching works in Kafka Streams 2.8. The branching API is significantly refactored in 2.8 from the prior versions.
public Function<KStream<?, String>, KStream<?, Foo>> convert() {
Foo[] foo = new Foo[0];
return input -> {
final Map<String, ? extends KStream<?, String>> branches =
input.split(Named.as("foo-")).branch((key, value) -> {
try {
foo[0] = new Foo(); // your API call for CitiProgramme converion here, possibly.
return true;
}
catch (Exception e) {
Message<?> message = MessageBuilder.withPayload(value).build();
streamBridge.send("to-my-dlt", message);
return false;
}
}, Branched.as("bar"))
.defaultBranch();
final KStream<?, String> kStream = branches.get("foo-bar");
return kStream.map((key, value) -> new KeyValue<>("", foo[0]));
};
}
}
The default branch is ignored in this code because that only contains the records that threw exceptions. Those were handled by the catch statement above in which we send the records to a DLT programmatically. Finally, we get the good records and map them to a new KStream and send it through the outbound.
I'm trying to inject a KafkaTemplate to send a single message. I'm developing a small function that lies outside the reactive approach.
I can only find examples that use #Ingoing and #Outgoing from Smallrye but I don't need a KafkaStream.
I tried with Kafka-CDI but I'm unable to inject the SimpleKafkaProducer.
Any ideas?
For Clement's answer
It seems the right direction, but executing orders.send("hello"); I receive this error:
(vert.x-eventloop-thread-3) Unhandled exception:java.lang.IllegalStateException: Stream not yet connected
I'm consuming from my topic by command line, Kafka is up and running, if I produce manually I can see the consumed messages.
It seems relative to this sentence by the doc:
To use an Emitter for the stream hello, you need a #Incoming("hello")
somewhere in your code (or in your configuration).
I have this code in my class:
#Incoming("orders")
public CompletionStage<Void> consume(KafkaMessage<String, String> msg) {
log.info("Received message (topic: {}, partition: {}) with key {}: {}", msg.getTopic(), msg.getPartition(), msg.getKey(), msg.getPayload());
return msg.ack();
}
Maybe I've forgotten some configurations?
So, you just need to use an Emitter:
#Inject
#Stream("orders") // Emit on the channel 'orders'
Emitter<String> orders;
// ...
orders.send("hello");
And in your application.properties, declare:
## Orders topic (WRITE)
mp.messaging.outgoing.orders.type=io.smallrye.reactive.messaging.kafka.Kafka
mp.messaging.outgoing.orders.topic=orders
mp.messaging.outgoing.orders.bootstrap.servers=localhost:9092
mp.messaging.outgoing.orders.key.serializer=org.apache.kafka.common.serialization.StringSerializer
mp.messaging.outgoing.orders.value.serializer=org.apache.kafka.common.serialization.StringSerializer
mp.messaging.outgoing.orders.acks=1
To avoid Stream not yet connected exception, as suggested by doc:
To use an Emitter for the stream hello, you need a #Incoming("hello")
somewhere in your code (or in your configuration).
Assuming you have something like this in your application.properties:
# Orders topic (READ)
smallrye.messaging.source.orders-r-topic.type=io.smallrye.reactive.messaging.kafka.Kafka
smallrye.messaging.source.orders-r-topic.topic=orders
smallrye.messaging.source.orders-r-topic.bootstrap.servers=0.0.0.0:9092
smallrye.messaging.source.orders-r-topic.key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
smallrye.messaging.source.orders-r-topic.value.deserializer=org.apache.kafka.common.serialization.StringDeserializer
smallrye.messaging.source.orders-r-topic.group.id=my-group-id
Add something like this:
#Incoming("orders-r-topic")
public CompletionStage<Void> consume(KafkaMessage<String, String> msg) {
log.info("Received message (topic: {}, partition: {}) with key {}: {}", msg.getTopic(), msg.getPartition(), msg.getKey(), msg.getPayload());
return msg.ack();
}
Since Clement's answer the #Stream annotation has been deprecated. The #Channel annotation
must be used instead.
You can use an Emitter provided by the quarkus-smallrye-reactive-messaging-kafka dependency to produce message to a Kafka topic.
A simple Kafka producer implementation:
public class MyKafkaProducer {
#Inject
#Channel("my-topic")
Emitter<String> myEmitter;
public void produce(String message) {
myEmitter.send(message);
}
}
And the following configuration must be added to the application.properties file:
mp.messaging.outgoing.my-topic.connector=smallrye-kafka
mp.messaging.outgoing.my-topic.bootstrap.servers=localhost:9092
mp.messaging.outgoing.my-topic.value.serializer=org.apache.kafka.common.serialization.StringSerializer
This will produce string serialized messages to a kafka topic named my-topic.
Note that by default the name of the channel is also the name of the kafka topic in which the data will be produced. This behavior can be changed through the configuration. The supported configuration attributes are described in the reactive Messaging documentation
I'm new to Vert.x and trying I am trying to implement a small REST API that stores its data in JSON files on the local file system.
So far I managed to implement the REST API since Vertx is very well documented on that part.
What I'm currently looking for are examples how to build data access objects in Vert.x. How can I implement a Verticle that can perform crud operations on a text file containing JSON?
Can you provide me any examples? Any hints?
UPDATE 1:
By CRUD operations on a file I'm thinking of the following. Imagine there is a REST resource called Records exposed on the the path /api/v1/user/:userid/records/.
In my verticle that starts my HTTP server I have the following routes.
router.get('/api/user/:userid/records').handler(this.&handleGetRecords)
router.post('/api/user/:userid/records').handler(this.&handleNewRecord)
The handler methods handleGetRecords and handleNewRecord are sending a message using the Vertx event bus.
request.bodyHandler({ b ->
def userid = request.getParam('userid')
logger.info "Reading record for user {}", userid
vertx.eventBus().send(GET_TIME_ENTRIES.name(), "read time records", [headers: [userId: userid]], { reply ->
// This handler will be called for every request
def response = routingContext.response()
if (reply.succeeded()) {
response.putHeader("content-type", "text/json")
// Write to the response and end it
response.end(reply.result().body())
} else {
logger.warn("Reply failed {}", reply.failed())
response.statusCode = 500
response.putHeader("content-type", "text/plain")
response.end('That did not work out well')
}
})
})
Then there is another verticle that consumes these messages GET_TIME_ENTRIES or CREATE_TIME_ENTRY. I think of this consumer verticle as a Data Access Object for Records. This verticle can read a file of the given :userid that contains all user records. The verticle is able to
add a record
read all records
read a specific record
update a record
delete a or all records
Here is the example of reading all records.
vertx.eventBus().consumer(GET_TIME_ENTRIES.name(), { message ->
String userId = message.headers().get('userId')
String absPath = "${this.source}/${userId}.json" as String
vertx.fileSystem().readFile(absPath, { result ->
if (result.succeeded()) {
logger.info("About to read from user file {}", absPath)
def jsonObject = new JsonObject(result.result().toString())
message.reply(jsonObject.getJsonArray('records').toString())
} else {
logger.warn("User file {} does not exist", absPath)
message.fail(404, "user ${userId} does not exist")
}
})
})
What I trying to achieve is to read the file like I did above and deserialise the JSON into a POJO (e.g. a List<Records>). This seems much more convenient that working with JsonObject of Vertx. I don't want to manipulate the JsonObject instance.
First of all, your approach using EventBus is fine, in my opinion. It may be a bit slower, because EventBus will serialize/deserialize your objects, but it gives you a very good decoupling.
Example of another approach you can see here:
https://github.com/aesteve/vertx-feeds/blob/master/src/main/java/io/vertx/examples/feeds/dao/RedisDAO.java
Note how every method receives handler as its last argument:
public void getMaxDate(String feedHash, Handler<Date> handler) {
More coupled, but also more efficient.
And for a more classic and straightforward approach, you can see the official examples:
https://github.com/aokolnychyi/vertx-example/blob/master/src/main/java/com/aokolnychyi/vertx/example/dao/MongoDbTodoDaoImpl.java
You can see that here DAO is pretty much synchronous, but since the handlers are still async, it's fine anyway.
I guess the following link will help you out and this is a good example of Vertx crud operations.
Vertx student crud operations using hikari
I am currently learning Camel and i am not sure if we can send messages to a activemq queue/topic from camel at fixed interval.
Currently i have created code in Scala which looks up the database and create a message and sends it to queue after every minute can we do this in camel.
We have a timer component in camel but it does not produce the message. I was thinking something like this.
from("timer://foo?fixedRate=true&period=60000")
.to("customLogic")
.to("jms:myqueue")
Timer will kick after a minute.
Custom logic will do database look up and create a message
Finally send to jms queue
I am very new to Camel so some code will be really helpful thanks
Can you please point me to how can i create this customeLogic method that can create a message and pass it to next ".to("jms:myqueue")". Is there some class that in need to inherit/implement which will pass the the message etc.
I guess your question is about how to hook custom java logic into your camel route to prepare the JMS message payload.
The JMS component will use the exchange body as the JMS message payload, so you need to set the body in your custom logic. There are several ways to do this.
You can create a custom processor by implementing the org.apache.camel.Processor interface and explicitly setting the new body on the exchange:
Processor customLogicProcessor = new Processor() {
#Override
public void process(Exchange exchange) {
// do your db lookup, etc.
String myMessage = ...
exchange.getIn().setBody(myMessage);
}
};
from("timer://foo?fixedRate=true&period=60000")
.process(customLogicProcessor)
.to("jms:myqueue");
A more elegant option is to make use of Camel's bean binding:
public class CustomLogic {
#Handler
public String doStuff() {
// do your db lookup, etc.
String myMessage = ...
return myMessage;
}
}
[...]
CustomLogic customLogicBean = new CustomLogic();
from("timer://foo?fixedRate=true&period=60000")
.bean(customLogicBean)
.to("jms:myqueue");
The #Handler annotation tells Camel which method it should call. If there's only one qualifying method you don't need that annotation.
Camel makes the result of the method call the new body on the exchange that will be passed to the JMS component.
I am sending a file as a message by converting it to a byte array using kafka producer.
I also need to add some headers for the message, for example file name, timestamps etc so at the consumer end I can process the message based on file name and other headers.
What I am currently doing is creating a object and wrapping the raw message and headers in it and sending the object in a byte array as a message.
I would like to know if there is a way by which I can add custom headers while publishing the message?
Kafka v0.11.0.0 adds support for custom headers.
You can add them when creating a ProducerRecord like this:
new ProducerRecord(key, value, headers, ...), where headers is of type Iterable<Header>
For more details see:
https://issues.apache.org/jira/browse/KAFKA-4208
https://cwiki.apache.org/confluence/display/KAFKA/KIP-82+-+Add+Record+Headers
Record level headers were introduced from Kafka 0.11.0. We can send a list of Headers in each record.
List<Header> headers = Arrays.asList(new RecordHeader("header_key", "header_value".getBytes()));
ProducerRecord<String, String> record = new ProducerRecord<>("topic", null, "key", "value", headers);
You can create your own small java application to send the message with headers to kafka.
Write the following code in intellij or any supporting IDE:-
public static void main(String[] args) throws JsonProcessingException,
InterruptedException {
Properties props=new Properties();
props.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,"localhost:9092");
props.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
JsonSerializer.class.getName());
props.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,JsonSerializer.class.getName());
KafkaProducer<String, JsonNode> producer=new KafkaProducer<String, JsonNode>(props);
String json = "{ \"f1\" : \"v1\" } ";
ObjectMapper mapper = new ObjectMapper();
JsonNode jsonNode = mapper.readTree(json);
ProducerRecord<String,JsonNode> record =new ProducerRecord<String,
JsonNode>("test topic", jsonNode);
record.headers().add(new RecordHeader("key","value1".getBytes()));
producer.send(record);
Thread.sleep(10000);
}
String json = "{ \"f1\" : \"v1\" } "- this is the key and value we want to send to kafka using objectMapper and converting it into jsonNode form.
record.headers().add(new RecordHeader("key","value1".getBytes()))-This is the key and value of headers data that we are sending to kafka.
To verify your data you can check the topic in the kafka control center and verify the headers sent.
Kafka is agnostic to the message content and doesn't provide any special means to enrich it so this is something you need to do yourself. A common way of dealing with these things is to use a structured format such as json, avro or similar where you are free to define the necessary fields and could easily add metadata to your message and ship it off to the Kafka brokers.
This answer is outdated as of Kafka 0.11, please see other answers.
Another Solution
ProducerRecord<String, String> producerRecord = new ProducerRecord<>("bizops", "value");
producerRecord.headers().add("client-id", "2334".getBytes(StandardCharsets.UTF_8));
producerRecord.headers().add("data-file", "incoming-data.txt".getBytes(StandardCharsets.UTF_8));
// Details left out for clarity
producer.send(producerRecord);
https://www.confluent.io/blog/5-things-every-kafka-developer-should-know/#adding-headers
I've been through similar problems with projects I've worked on so I created this simple library to help tackle that: https://github.com/leandronunes85/messaging. For now contains an Avro based implementation but it can be extended to use any other serialization framework of your choice.
You just have to create a (de)serializer for the objects you want to have on the stream (Avro based or not) and let AvroMessageSerializer work its magic.
This is still a very young library but I feel it can save many people a lot of time!