I'm writing an application that will trigger a task after getting a message from a queue on IBM Cloud MQ. There will be only one producer, and only one consumer (the app i'm writing. Because of this, I thought it was better to use a point-to-point strategy instead of pub/sub. However, while I can use an HTTP REST Get action to get messages from the IBM MQ queue, I would also need to periodically execute a GET to "monitor" the queue.
Wouldn't be better to simply use a pub/sub strategy if that's the case? I don't see the advantage of using point-to-point, although the documentation advices it for my use case.
Related
I would like to make a Web Dashboard system and I am facing a problem. I need to get an information that is in the cache of one of the instances of my program, for this I had thought of doing Pub/Sub with Kafka however I don't know how to do to Publish and get a response from one of my Subscriber. Do you know a pattern that allows this and a service that allows me to do this?
EDIT: I would like to design an infrastructure that follows this pattern:
Attached diagram is showing simple request->response flow, Kafka is designed for different types of architecture, so IMHO you should not focus on Kafka in this case.
However, if you still want to use Kafka for some other reasons I can suggest to you two options:
Stick with request->response flow and use ReplyingKafkaTemplate or AggregatingKafkaTemplate to handle it, second one is an extension of first one, this adds functionality to handle more responses then one. You can send a request to Kafka topic from the Dashboard application, then poll the message by one of the Bot instances, next, send reply to reply topic, and then process reply in Dashboard application.
Use Kafka to implement Event-Carried State Transfer pattern, move state (mutual guilds data) from Bot Instances directly to Dashboard application via Kafka topic. You can use several tools to implement this:
Bot applications send events to Kafka topic via simple KafkaProducer or KafkaTemplate, then use one of the Kafka Connect sink connectors to save data in Dashboards database.
Bot applications send events to Kafka topic via simple KafkaProducer or KafkaTemplate. Run Kafka Streams thread in Dashboard application and build a state using Kafka Streams functionalities - grouping, aggregating etc. Then read the state directly from Kafka Streams internal RocksDB database.
We want to subscribe to a Kafka topic with a microservice. So that the service does not have to accept API calls from the surrounding systems and Kafka messages at the same time, we would like to interpose an 'importer service'. However, in the end we would have the same problem again because the importer service now has to process both the Kafka messages and the API calls from the aforementioned microservice. As a solution to this problem, we considered giving both services access to the same database. The importer service could then receive the Kafka message, process it and write it to the database. The original microservice would then not go to the importer service, but would get the data directly from the DB. However, the approach seems a bit dirty, since you shouldn't share databases between services. Do you have any ideas how to solve this more elegantly? And if there isn't a better approach, should one service really take care of processing Kafka messages and API calls simultaneously?
One service should not process both kafka and api messages.
You can make a service that wraps the database and both services will communicate with it.
I have a microservices architecture whose logs have to be sent to a remote Kafka topic.
Next to it, the consumer of this topic will send the logs to an ELK stack (an other team)
I want to have a dedicated microservice (fwk-proxy-elasticsearch) whose responsability is to collec the logs from the others one and send them to the remote kafka topic.
what's the best protocol to dispatch all the logs aggregated from my microservices to the fwk-proxy-elasticsearch microservice ?
I want this pattern to not duplicate the security configuration of the remote kafka topic. I want to centralize it in a single place.
May I use vertx event bus for that ? or kafka is beter ? or someother tool ?
May I use vertx to send message from jvm to jvm ?
Moreover, in a microservice architecture, is it a good pattern to centralize a use case in a dedicated microservice? (remote http connection for example)
On my point of view, it allows business microservices to focus on a business issue and not to worry over the protocol that the result has to be sent.
Thanks!
I believe you can use both Vert.x event bus and Kafka to propagate the logs, there are pros and cons on each approach.
While I understand the reasoning behind this decision, I would still consider a dedicated solution built for this purpose, like Fluentd, which is able to aggregate the logs and push them into multiple sources (including Kafka, via the dedicated plugin). I'm sure there are other similar solutions.
There are a couple of important benefits that I see if you use a dedicated solution, instead of building it yourself:
The level of configurability, which is definitely useful in the future (in a dedicated solution, you need to write code each time you want to build something new)
The number of destinations where you can export the logs
Support for a hybrid architecture - with a few config updates, you will be able to grab logs from non-JVM microservices
Anypoint MQ Connector has two modules, Consume and Subscribers that could be used to retrieve messages from the queue. I cannot find any good documentation about what is the difference between the two and uses cases where one should be preferred over the other?
Consume is an operation and Subscribe is a message source, both provided by the Anypoint MQ connector.
Subscribe polls the queue to find if there are new messages available, and then starts the flow with the message read. You can not use Subscribe inside a flow.
Consume is an operation that is used inside a flow. Because it is not a message source it can not start a flow, like Subscribe. Instead you use when the flow is started by some other source, like an HTTP request, then you want to read a message from a queue in some place of the flow execution.
We have developed modular web app with very powerful API and now we need queuing tool for delayed|time consuming jobs. We are looking at RabbitMQ or AWS SQS. But these two just store messages, and you have to manually get messages from them or I misunderstood it?
We would like to channel all messages through our API, so when message is published to Queue in should be POST-ed (after some delay) to to our Interface.
So my question:
Is there any tool for queuing that support http post (with oauth2)?
If not, is this approach somehow valid:
Create worker that poll messages from queue
and POST them to API with some client?
(we have to maintain cli tool, and we want to avoid that).
Are there any alternatives?
When using SQS polling is the only way out.
To make things easier you can write this polling logic in AWS Lambda because lambda functions do not have the overhead of maintaining infrastructure and servers