Is it possÃble to setup envoy to receive HTTP request send it to a kafka topic wait for response related to that message from another topic(s) and create http response based on the kafka response?
I found this implementation for bridging Sync HTTP to Async Kafka but I think there is some more enterprise solution for such a generic problem:
kafka - http bridge
Related
Is there any way to create a Interceptor for a INCOMING message from Kafka, using the smallrye connector in quarkus ? I need this to read the headers and switch Tenants for persisting data.
In spring we can create a consumer interceptor and register it in the app.
For quarkus i see there is a OUTGOING interceptor, but none for a incoming kafka message.
Currently, I am using Confluent Rest Proxy 5.5.1 for collecting data in production. Duplicated events can come. I found the solution for de-duplication using Kafka Stream API. Is it possible to get an idempotent producer guarantee in Rest Proxy?
You can modify the REST Proxy's producer client with the producer. prefix
producer.enable.idempotence=true
Idempotent producers are not supported in Kafka REST. Ignoring 'enable.idempotence'. (io.confluent.kafkarest.KafkaRestConfig:880)
I am writing HTTP Sink plugin for Kafka connect. Its purpose is to send HTTP requests for each message in the configured Kafka topic. I want to send the message to dead letter queue in case HTTP request fails. Can I make use of dead letter queue configuration provided in sink plugin ?
The reason for this question is that, it has been mentioned in kafka connect documentation and several blogs that only errors in transformer and converter will be send to dead letter queue and not the ones during PUT. Since the task of sending the http request is done in PUT. So I am think, is there a way to send failed http messages to DLQ ? If not, is it possible to send the message to some other kafka topic for further processing ?
According to #Neil this might be informative,
Kip 610 (implemented in Kafka 2.6) added DLQ support for issues when interacting with end system. Kip 298 added DLQ but only on issues prior to sink system interaction.
Check the versions of your connect cluster and sink version and see if it supports it.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-610%3A+Error+Reporting+in+Sink+Connectors
I am new to Microservices architecture.
I want to create a microservice using Netflix OSS.
I want my architecture to look some thing like the one described here.
http://callistaenterprise.se/blogg/teknik/2017/09/13/building-microservices-part-8-logging-with-ELK/
However I want one of my microservice, (which is behind the Zuul Reverse proxy) to consume events from a Kafka events(which is from some other team).
I am not sure If this is a good idea, since this will expose my microservices, which is supposed to be abstracted from outside world behind my Zuul wall.
Is there any other way. Can I use my Zuul to consume event streams from kafka and push to my microservice. If yes, how do I stream from my Zuul to microservice?
Zuul will redirect your request to A service HTTP XXXX port /api/v1/input. This microservice as a producer will put message to kafka channel. After kafka consumer will get message and store or analyze. Another microservice can read from database and return response after frontend request or do push using Server Sent Events or Vertx message bus....
I have a rest based application deployed in server(tomcat) ,
Every request comes to server it takes 1 second of time to serve, now I have a issue, sometimes Server receive more request then it is capable of serving which making server non responsive. Now I was thinking if I can store the requests in a queue so that server can pull request and serve that request and handle the pick time issue.
Now I was thinking can Kafka be helpful for this, if yes any pointer where I can start.
You can use Kafka (or any other messaging system for this ex- ActiveMQ, RabbitMQ etc).
When WebService receives request, add request (with all details required to process it) in Kafka queue (using Kafka message producer details)
Separate service (having Kafka consumer details) will read from topic(queue) and process it.
In case need to send message to client when request is processed, server can push information to client using WebSocket (Or client can poll for request status however this need request status endpoint and will cause load on that endpoint).
Apache Kafka would be helpful in your case. If you use a Kafka broker, it will allow you to face with a peak of requests. The requests will be stored in a queue as you mentionned and be treated by your server at its own speed.
As your are using tomcat, I guess you developped your server in Java. Apache Kafka propose a Java API which is quite easy to use.