Currently I am using spring cloud Netflix stack for development of my microservices based application. where I have used different services provided by Netflix stack like
EUREKA : for service registration and discovery
ZUUL : for proxy gateway and
RIBBON : for load balancing
now we want to use Apache kafka for inter-communication between microservices where kafka will have different topics to which our microservices will subscribe,
now real problem starts here is how the load balancing, proxy gateway and eureka will work here.
few questions I have in mind which are really confusing me are
1. how ribbon load balancing will work here, since while using HTTP we are using #loadbalanced restTemplate but now this will not be the case.
2. how service calls will be distributed among different instances of a same service
3. whether Zuul and Eureka are needed here or we just need to ignore these and go with kafka load balancing and service discovery through topics.
I gooled but not found any satisfying answer if any expert here can help me will be appreciated, if there is any similar questions that can also help.
Thanks in advance.
For your above use-case , if you are going to use kafka for inter microservices communication , there is no need for any spring-cloud-netflix component. You can publish to a topic and have consumers in microservices consume from the topic. Load balancing will automatically happen depending upon number of partitions in the topic.
For example , lets consider your topic name is test and it has 4 partitions . If you have deployed 4 microservices with each consisting of a single Kafka Consumer consuming from topic test then each consumer will consume from 1 partition on the topic test. So , load balancing will automatically happen.
The spring-cloud-netflix components are mainly meant for inter-microservices communication when there is network calls involved between microservices .
For example -
Lets consider two applications A and B . You have 4 instances of Application A deployed and 4 instances of Application B deployed . The entry-point for the consumers of your system is Application A . Here , you will use an Api Gateway like Zuul . As for Eureka , you will have all your instances deployed register in it. When a request comes into Zuul which needs to be forwarded to Application A . All the instances of Application A will be fetched from Eureka (4 in our case) and then being provided to a load balancer Ribbon who will chose which url should be called . Zuul will then forward the request to that instance of Application A.
Some links you should look at are-
https://github.com/Netflix/zuul/wiki/How-it-Works
https://github.com/Netflix/ribbon/wiki/Working-with-load-balancers
https://blog.asarkar.org/technical/netflix-eureka/
Related
If I were to create a public Kafka cluster that accepts messages from multiple clients, but are purely processed by a separate backend. What would be the right way to design it?
A bit more concrete example, let's say I have 50 kafka brokers. How do I:
Configure clients without the manually adding in IPs of the 50 kafka brokers.?
Loadbalancing messages to kafka broker based on load if possible.
Easier/automated way to setup additional clients with quota.
You can use hashicorp consul which is one of the open source service discovery tools to get your kafka brokers on, ultimately you will have single endpoint and you don't need to add multiple brokers in your clients. There are several other open source told available
There are few ways, use kafka assigner tool to balance the traffic or kafka cruise control open source tool to automatically balance the cluster for you
I have just started learning Kafka. So trying to build a social media web application. I am fairly clear on how to use Kafka for my backend ( communicating from backend to databases and other services).
However, I am not sure how should frontend communicate with backend. I was considering an architecture as: Frontend -> Kafka -> Backend.
Frontend acts as producer and backend as consumer. In this case, frontend would supposedly have all required resources to publish to Kafka broker (even if I implement security on Kafka). Now, is this scenario possible:
Lets say I impersonate the frontend and send absurd/invalid messages to my Kafka broker. Now I can handle and filter these messages when they reach to my backend. But I know that Kafka stores these messages temporarily. Wouldn't my Kafka server face DDOS problems if such "fake" messages are published to it in high volume, since it is gonna store them anyway as they dont get filtered out until they actually get consumed by backend?
If so, how can I prevent this?
Or is this not a good option? I can also try using REST for frontend/backend communication and then Kafka will be used from backend to communicate with database(s) and other stuff.
Or I can have a middleware (again, REST) that detects and filters out such messages.
Easiest way is to have the front end produce to the Kafka REST Proxy
See details here https://docs.confluent.io/1.0/kafka-rest/docs/intro.html
That way there is no kafka client code required in your front end and you can use HTTP(S) with standard off the shelf load balancers, and API Management tools.
Could you not consider the other direction, to use Kafka as a transport system for updating assets available to frontend ? This has been proposed for hybrid React / NodeJS/Express solutions.
I am new to Microservices architecture.
I want to create a microservice using Netflix OSS.
I want my architecture to look some thing like the one described here.
http://callistaenterprise.se/blogg/teknik/2017/09/13/building-microservices-part-8-logging-with-ELK/
However I want one of my microservice, (which is behind the Zuul Reverse proxy) to consume events from a Kafka events(which is from some other team).
I am not sure If this is a good idea, since this will expose my microservices, which is supposed to be abstracted from outside world behind my Zuul wall.
Is there any other way. Can I use my Zuul to consume event streams from kafka and push to my microservice. If yes, how do I stream from my Zuul to microservice?
Zuul will redirect your request to A service HTTP XXXX port /api/v1/input. This microservice as a producer will put message to kafka channel. After kafka consumer will get message and store or analyze. Another microservice can read from database and return response after frontend request or do push using Server Sent Events or Vertx message bus....
So far, I have been using Spring Boot apps (with Spring Cloud Stream) and Kafka running without any supporting infrastructure (PaaS).
Since our corporate platform is running on Kubernetes we need to move those Spring Boot apps into K8s to allow the apps to scale and so on. Obviously there will be more than one instance of every application so we will define a consumer group per application to ensure the unique delivery and processing of every message.
Kafka will be running outside Kubernetes.
Now my doubt is: since the apps deployed on k8s are accessed through the k8s service that abstracts the underlying pods, and individual application pods can't be access directly outside of the k8s cluster, Kafka won't know how to call individual instances of the consumer group to deliver the messages, will it?
How can I make them work together?
Kafka brokers do not push data to clients. Rather clients poll() and pull data from the brokers. As long as the consumers can connect to the bootstrap servers and you set the Kafka brokers to advertise an IP and port that the clients can connect to and poll() then it will all work fine.
Can Spring Cloud Data Flow solve your requirement to control the number of instances deployed?
and, there is a community released Spring Cloud Data Flow server for OpenShift:
https://github.com/donovanmuller/spring-cloud-dataflow-server-openshift
I need to use Apache Kafka in my project of Micro-Services. I need my one micro-service to produce data and another to consume the same data. How can I make Kafka do the same between the two services
I would recommend you to take a look at Spring Cloud Stream as it does exactly what you need.
From docs:
Framework for building message-driven microservices. Spring Cloud Stream builds upon Spring Boot to create DevOps friendly microservice applications and Spring Integration to provide connectivity to message brokers. Spring Cloud Stream provides an opinionated configuration of message brokers, introducing the concepts of persistent pub/sub semantics, consumer groups and partitions across several middleware vendors. This opinionated configuration provides the basis to create stream processing applications.
By adding #EnableBinding to your main application, you get immediate connectivity to a message broker and by adding #StreamListener to a method, you will receive events for stream processing.