I'm implementing a set of Lagom services and I have one that's planned to purely read from a topic and replying back to that topic.
I can't seem to find any docs on how to do this? All of the subscriber examples map from a message to Done and all of the publish examples are mapping from the event store events to publish external messages.
Related
I would like to make a Web Dashboard system and I am facing a problem. I need to get an information that is in the cache of one of the instances of my program, for this I had thought of doing Pub/Sub with Kafka however I don't know how to do to Publish and get a response from one of my Subscriber. Do you know a pattern that allows this and a service that allows me to do this?
EDIT: I would like to design an infrastructure that follows this pattern:
Attached diagram is showing simple request->response flow, Kafka is designed for different types of architecture, so IMHO you should not focus on Kafka in this case.
However, if you still want to use Kafka for some other reasons I can suggest to you two options:
Stick with request->response flow and use ReplyingKafkaTemplate or AggregatingKafkaTemplate to handle it, second one is an extension of first one, this adds functionality to handle more responses then one. You can send a request to Kafka topic from the Dashboard application, then poll the message by one of the Bot instances, next, send reply to reply topic, and then process reply in Dashboard application.
Use Kafka to implement Event-Carried State Transfer pattern, move state (mutual guilds data) from Bot Instances directly to Dashboard application via Kafka topic. You can use several tools to implement this:
Bot applications send events to Kafka topic via simple KafkaProducer or KafkaTemplate, then use one of the Kafka Connect sink connectors to save data in Dashboards database.
Bot applications send events to Kafka topic via simple KafkaProducer or KafkaTemplate. Run Kafka Streams thread in Dashboard application and build a state using Kafka Streams functionalities - grouping, aggregating etc. Then read the state directly from Kafka Streams internal RocksDB database.
I need to build a secure REST API for different services where client services can post and receive messages from other clients( like mail box. but messages are going to be in JSON form. and should be persistent. I am expecting around 5000 client services. With around 50 message per service per day).
My questions are:
Can I use Kafka for this(I think I will be needing some wrapper over
Kafka to manage other task) ?
If yes then outbox and inbox are going to be a separate topic for
each service?( 2 topics per service. 5000*2 topics. My plan is to
create them dynamically as new client joins in)
what are the alternative technologies to write this kind of gateway.
Any help will be appreciated.
You can't use Kafka to implement REST API because REST API implies request/response while Kafka is just a message queue (Kafka doesn't provide a mechanism to respond to messages). You can use Kafka to produce messages to be consumed by other services. The idea of message queues is to decouple producer from consumer and vice versa. When a consumer receives a message it acts on it, that's it. But when you say inbox/outbox you imply that there's a response for a message which means that producers and consumers pace should be similar which couples them which is against the nature of message queues.
It seems like in your case it makes more sense to use http requests/response or even websockets. If you want to save the request/response data (making it persistent) you can save it either in a database, object storage (like S3), log it or send each message to Kafka so that Kafka stores all of your messages, writes to Kafka will actually be very fast because Kafka is roughly-speaking an append-only log. You can then search messages values using ksqldb.
I have a problem, that I want to solve using kafka queues.
I need to process some result, then return it to the user.
As you can see in the picture, the Rest Service, requests something to the Calculator Service.
Both services have a kafka consumer, and a kafka producer.
The rest service receive a request, then produces a message on toAdd queue, then keep consuming the fromAdd queue, until receives a value.
The calculator service keep consuming the toAdd queue, when some message comes, it sum two values, then produces a message on fromAdd queue.
Sometimes the rest service receives old messages from the queue, or more than one message.
I find something about idempotent configuration, but I don't know how to implement right.
Is that diagram, the right way to the communication between two or more services using kafka?
Can someone give a example?
Thanks.
Is that diagram, the right way to the communication between two or more services using kafka?
If you mean "Does it make sense to have two or more services communicate indirectly through Kafka?", then yes, it does.
Can someone give a example?
Here are some good pointers including examples:
Build Services on a Backbone of Events, Confluent blog, May 2017
Commander: Better Distributed Applications through CQRS, Event Sourcing, and Immutable Logs, by Bobby Calderwood, StrangeLoop, Sep 2016
Recorded talk
Reference implementation on GitHub
To answer your question: There is no problem with such communication.
Now referring back to other parts...
Keep in mind that it's an asynchronous communication so you should not keep HTTP connection open and keep user of that service waiting for the response. This is just not the way to go. You can solve this in many ways. For instance: you can use WebSockets, you can send an email/SMS/slack msg to the user with the reply and so on.
I am planning to track all the user activity on my website and kafka seems to be a reliable solution. I am unable to resolve how to send generated events to kafka i.e. how to make my website as a kafka producer.
Which language is your website written in?
In Java/Scala the solution is to import the kafka producer dependencies, create a producer, create messages and send them.
I hope this example will help:
https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Example
How do I get the number of JMS messages waiting to be consumed by a specific JMS message subscriber? I use the Topic model (publish/subscribe) and not the Queue model.
I want my MDB (message driven bean) to be able to figure out this information about the topic it listens to. To be clear; I want my MDB to get the number of messages waiting to be consumed.
I can't find any information in either on Internet or the documentation :(
I use JBoss Messaging 1.4.4.
AFAIK, JMS does not specify anything to count the number of messages in a destination.
You need to use JMX for this. Check out the MBean attributes of the Topic MBean in the documentation and/or the java documentation of TopicMBean#getMessageCounters(). The depth attribute of MessageCounter should be what you're looking for. But, to be honest, I don't know what you're gonna do with this information and if this has a sense for a Topic. A message will stay in a Topic as long as it hasn't been delivered to all subscribers and each subscriber typically hasn't any knowledge of its peers. So how would one MDB interpret a count of messages?
Also note that I couldn't find this MBean in JBoss Messaging 2.0.0.alpha1's javadoc. I don't know if it has been deprecated (according to the code in 1.4, it wasn't) or if the documentation is not up to date (after all, it's the alpha1 javadoc and I couldn't find a link for beta4).
EDIT: As skaffman pointed out, JBoss Messaging has been rebranded as HornetQ. It looks like there have been some changes in the API but concepts still apply. The documentation is here.
You can't, not with the JMS API. The internal JBossMessaging API may expose that information, but you'll have to go looking through that documentation to find it.