My application has rabbitMq which has multiple consumers[3] and i do round robing for consuming the messages .
At the moment i do canary deployment while deploying new version but this lands me in situation where the message can be consumed by any consumer [not just the canary one] and it creates problem .
[Also i want it to behave like this only while deployment all other times round robin is needed]
I know about green blue deployment process already but is there any other way to go about this problem ?
Related
I would like to create a cronjob via Knative that sends healthcheck messages to my Kafka topic every 10 minutes.
Then we will have a separate endpoint created, that will receive these messages and pass some response to a receiver (healthcheck).
Tried using KafkaBinding, which seems suitable, but cannot apply this template (TLS): https://github.com/knative/docs/tree/main/code-samples/eventing/kafka/binding
I also find it odd, that regular KafkaBinding template contains apiVersion: bindings.knative.dev/v1beta1, while the one with TLS: sources.knative.dev/v1beta1.
Haven't found much documentation on how to create a Cronjob sending messages on Kafka, which is then grabbed using KafkaSource and passed using broker+trigger to my service on K8s.
Anyone got it implemented right? Or found another way for this?
The idea is test the whole flow including KafkaSource, which seems a little bit unstable.
I’m running a k8s cluster on aws-eks.
I have two services A and B.
Service A listens to a rabbit queue and sends http request to service B (which takes a couple of seconds).
Both services scale based on the number of message in the queue.
The problem is that the requests from A to B are not balanced.
When scaled to about 100 pods each, I see that service A pods sends requests to only about 60% of service B pods at a given time,
Meaning, eventually all pods gets messages, but some pods are at 100 cpu receiving 5 messages at a time, while others at 2 cpu
Receiving 1 message every minute or so..
That obviously causes low performance and timeouts.
I’ve read that it should work in round robin, but when I tried to set 10 fixed replicas of each service (all pods already up and running)
And pushing 10 messages to queue, I’ve seen that all service A pods pulled a message to send to service B, but some of service B pods never got any requests while other got more than one - resulting in one whole process to finish within 4 second while another took about 12 second.
Any ideas for why it’s working like that and how to change it to be more balanced?
Thanks
Our Services utilize Kafka to publish and consume messages. We deploy our services using Blue-Green deployment strategy.
Consider below scenario :
- Suppose we have App1.0 service in Blue(which is live and taking traffic) currently ,consuming from topic,
- when we start deploying a new version App1.1, it will get deployed to Green first,so now
Green has : App1.1 (not consuming)
Blue has : App1.0 (consuming messages)
- once we switch green to blue , Blue has App1.1 , and green has App1.0.
Our issue here is that once the switch happens, our green pods(in which we have older code App1.0) are still consuming messages from that kafka topic . Ideally green pods should now stop consuming from the topic.
We are looking for a solution wherein when we deploy ,our green pods should stop consuming .
I have been experimenting with blue green deployment in kubernetes using nginx-ingress. I created few concurrent http request to hit v1 version of my application. In the meanwhile I switched the router to point to v2 version of my application. As expected v2 version was serving the requests after the swap ,but what made me curious is that all the request were success. It is highly probable that there were some in-progress request in v1 while I made the swap. Why didn't such request fail?
I also tried the same by introducing some delay in my service so that the request take longer time to process.Still none of the request failed.
Usually in-flight requests are allowed to complete, just no new requests will be sent by the proxy.
Fowler says a small percentage of traffic is routed to the Canary version while the old version is still running.
This is assuming that the routing can be controlled at the Load balancer/router level.
We have a use case where a micro-service consumes off a queue and does some processing. We were wondering how the routing can be controlled to direct a subset of traffic to the canary consumer.
One of the options we considered is to have a separate "Canary queue" but the problem is that the producers now have to be aware of this queue which sounds like a smell.
This seems like a common problem where queues are involved. Any ideas on how Canary was adopted for such applications?
As you wrote, the goal of the canary release is to drive a small fraction of live traffic through a new deployment to minimize the potential impact of flaws in the new deployment. When you do not control the routing to the service under deployment, you can adjust the percentage of traffic handled by the new deployment by adjusting the percentage of new version services to current version services.
For example, your queue is being processed by a pool of 100 service instances at v1. To canary test the next version, deploy 1 to 10 of v2 and turn off 1 to 10 of v1. This will approximate routing 1 to 10% of the traffic to the new service.
If expected throughput of the new version of the service is significantly different, consider adjusting the ratio of new services to old.
If you current deployment of services is very small, consider temporarily increasing the total number of deployed current services before deploying an instance of the new service. For example, assume your active deployment is 3 services. Deploying 6 more of your current service before deploying 1 of your new version service will allow you to keep the traffic to the canary closer to 10%.
There are 2 approaches for canary deployment of queue workers:
A dedicated canary queue
A common queue
Both these approaches have their pros and cons which are covered in detail here: http://www.varlog.co.in/blog/canary-deployment-workers/