I am unable to get circuit breaking configuration to work on my elb through egress config.
ELB
elb has success rate of 25% (75% 500 error & 25% with status 200),
the elb has 4 instances, only 1 returns a successful response, other instances are configured to returns 500 error for testing purpose.
Setup
k8s: v1.7.4
istio: 0.5.0
env: k8s on aws
Egress rule
apiVersion: config.istio.io/v1alpha2
kind: EgressRule
metadata:
name: elb-egress-rule
spec:
destination:
service: xxxx.us-east-1.elb.amazonaws.com
ports:
- port: 80
protocol: http
Destination Policy
kind: DestinationPolicy
metadata:
name: elb-circuit-breaker
spec:
destination:
service: xxxx.us-east-1.elb.amazonaws.com
loadBalancing:
name: RANDOM
circuitBreaker:
simpleCb:
maxConnections: 100
httpMaxPendingRequests: 100
sleepWindow: 3m
httpDetectionInterval: 1s
httpMaxEjectionPercent: 100
httpConsecutiveErrors: 3
httpMaxRequestsPerConnection: 10
Route rules: not set
Testing
apiVersion: v1
kind: Service
metadata:
name: sleep
labels:
app: sleep
spec:
ports:
- port: 80
name: http
selector:
app: sleep
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: sleep
spec:
replicas: 1
template:
metadata:
labels:
app: sleep
spec:
containers:
- name: sleep
image: tutum/curl
command: ["/bin/sleep","infinity"]
imagePullPolicy: IfNotPresent
.
export SOURCE_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})
kubectl exec -it $SOURCE_POD -c sleep bash
Sending requests in parallel from the pod
#!/bin/sh
set -m # Enable Job Control
for i in `seq 100`; do # start 100 jobs in parallel
curl xxxx.us-east-1.elb.amazonaws.com &
done
Response
Currently, Istio considers an Egress Rule to designate a single host. This single host will not be ejected due to the load balancer's panic threshold of Envoy (the sidecar proxy implementation of Istio). The default panic threshold of Envoy is 50%. This means that at least two hosts are required for one host to be ejected, so the single host of an Egress Rule will not be ejected.
This practically means that httpConsecutiveErrors does not effect the external services. This lack of functionality should be partially resolved with External Services of Istio that will replace the Egress Rules.
See documentation of the Istio External Services backed by multiple endpoints -https://github.com/istio/api/blob/master/routing/v1alpha2/external_service.proto#L113
Related
I created two replicas of nginx with following yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
name: nginx
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.20-alpine
ports:
- containerPort: 80
And I created service with:
apiVersion: v1
kind: Service
metadata:
name: nginx-test-service
spec:
selector:
app: nginx
ports:
- port: 8082
targetPort: 80
Everything looks good. But when I do
minikube service nginx-test-service
I am able to access the nginx. But when I see the two pods logs, the request is always going to single pod. The other pod is not getting any request.
But, kubernetes service should do the load balancing right?
Am I missing anything?
One way to get load balancing on-premise running is with ip virtual services. (ipvs). It;s a service which hands out ip's of the next pod to schedule/call
it's likely installed already.
lsmod | grep ip_vs
ip_vs_sh 16384 0
ip_vs_wrr 16384 0
ip_vs_rr 16384 19
Have your cni properly setup and run
kubectl edit cm -n kube-system kube-proxy
edit the ipvs section
set mode to ipvs
mode: "ipvs"
and the ipvs section
ipvs:
excludeCIDRs: null
minSyncPeriod: 0s
scheduler: "rr"
As always there are lots of variables biting each other with k8s, but it is possible with ipvs.
https://kubernetes.io/blog/2018/07/09/ipvs-based-in-cluster-load-balancing-deep-dive/
Currently, we successfully setup Istio to create a couple ingress-gateways like api.example.com and app.example.com, that route traffic to a variety of services with destination rules, etc. In addition to this, we would love to use Istio's features for internal-only APIs, but we are unsure of how to set something like this up. Is it possible to use Istio's Gateway and VirtualServices CRDs to route traffic without exiting the cluster? If so, how would we go about setting that up?
Istio gateways are for traffic coming into the cluster or traffic leaving out the cluster. For traffic inside the cluster you should not use ingress/egress gateways. If you have configured Istio in the cluster to create a service mesh then you get all these benefits because Istio will inject a sidecar envoy for all your services inside the cluster. It's this sidecars which provides all the benefits of the mesh.When you use ingress or egress gateway you are actually using the sidecar deployed as ingress or egress gatway.You can use virtual service, destination rule, service entries, sidecars etc without a gateway because at that point you will using the sidecars deployed alongside your application pods.
Here is an example of virtual service:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: reviews
spec:
hosts:
- reviews
http:
- match:
- headers:
end-user:
exact: jason
route:
- destination:
host: reviews
subset: v2
- route:
- destination:
host: reviews
subset: v3
The virtual service hostname can be an IP address, a DNS name, or, depending on the platform, a short name (such as a Kubernetes service short name) that resolves, implicitly or explicitly, to a fully qualified domain name (FQDN). You can also use wildcard (”*”) prefixes, letting you create a single set of routing rules for all matching services. Virtual service hosts don’t actually have to be part of the Istio service registry, they are simply virtual destinations. This lets you model traffic for virtual hosts that don’t have routable entries inside the mesh.
I would add some things to Arghya Sadhu answer.
I think my example in another post is the answer to your question, specifically virtual service gateways and hosts. This example need additional Destination Rule since we have subsets which mark the route to proper subset of nginx here and they're defined in destination rule.
So, as an example, I would call something like internal-gateway/a or internal-gateway/b, and they would get routed to services A or B
I made something like that
2 nginx pods -> 2 services -> virtual service
Deployment1
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx1
spec:
selector:
matchLabels:
run: nginx1
replicas: 1
template:
metadata:
labels:
run: nginx1
app: frontend
spec:
containers:
- name: nginx1
image: nginx
ports:
- containerPort: 80
lifecycle:
postStart:
exec:
command: ["/bin/sh", "-c", "echo Hello nginx1 > /usr/share/nginx/html/index.html"]
Deployment2
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx2
spec:
selector:
matchLabels:
run: nginx2
replicas: 1
template:
metadata:
labels:
run: nginx2
app: frontend2
spec:
containers:
- name: nginx2
image: nginx
ports:
- containerPort: 80
lifecycle:
postStart:
exec:
command: ["/bin/sh", "-c", "echo Hello nginx2 > /usr/share/nginx/html/index.html"]
Service1
apiVersion: v1
kind: Service
metadata:
name: nginx
labels:
app: frontend
spec:
ports:
- port: 80
protocol: TCP
selector:
app: frontend
Service2
apiVersion: v1
kind: Service
metadata:
name: nginx2
labels:
app: frontend2
spec:
ports:
- port: 80
protocol: TCP
selector:
app: frontend2
Virtual Service
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: nginxvirt
spec:
hosts:
- nginx.default.svc.cluster.local
- nginx2.default.svc.cluster.local
http:
- name: a
match:
- uri:
prefix: /a
rewrite:
uri: /
route:
- destination:
host: nginx.default.svc.cluster.local
port:
number: 80
- name: b
match:
- uri:
prefix: /b
rewrite:
uri: /
route:
- destination:
host: nginx2.default.svc.cluster.local
port:
number: 80
Above virtual service works only internal in mesh gateway.
You have 2 matches for 2 nginx services.
root#ubu1:/# curl nginx/a
Hello nginx1
root#ubu1:/# curl nginx/b
Hello nginx2
I would recommend to check istio documentation and read about :
Gateways
Virtual Services
Destination Rules
And istio examples:
bookinfo
httpbin
So I can make up a DNS name or IP address that doesn't really exist
I think You misunderstood, it must exist, but not in the mesh. For example some database which is not in the mesh but You still can use, for example service entry to connect it to the mesh.
There is example with wikipedia in istio documentation and whole external services documentation.
I hope it will help You. Let me know if You have any more questions.
My application has to deployments with a POD.
Can I create a Service to distribute load across these 2 PODs, part of different deployments ?
If so, How ?
Yes it is possible to achieve. Good explanation how to do it can be found on Kubernete documentation. However, keep in mind that both deployments should provide the same functionality, as the output should have the same format.
A Kubernetes Service is an abstraction which defines a logical set of Pods running somewhere in your cluster, that all provide the same functionality. When created, each Service is assigned a unique IP address (also called clusterIP). This address is tied to the lifespan of the Service, and will not change while the Service is alive. Pods can be configured to talk to the Service, and know that communication to the Service will be automatically load-balanced out to some pod that is a member of the Service.
Based on example from Documentation.
1. nginx Deployment. Keep in mind that Deployment can have more than 1 label.
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
selector:
matchLabels:
run: nginx
env: dev
replicas: 2
template:
metadata:
labels:
run: nginx
env: dev
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
2. nginx-second Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-second
spec:
selector:
matchLabels:
run: nginx
env: prod
replicas: 2
template:
metadata:
labels:
run: nginx
env: prod
spec:
containers:
- name: nginx-second
image: nginx
ports:
- containerPort: 80
Now to pair Deployments with Services you have to use Selector based on Deployments labels. Below you can find 2 service YAMLs. nginx-service which pointing to both deployments and nginx-service-1 which points only to nginx-second deployment.
## Both Deployments
apiVersion: v1
kind: Service
metadata:
name: nginx-service
spec:
ports:
- port: 80
protocol: TCP
selector:
run: nginx
---
### To nginx-second deployment
apiVersion: v1
kind: Service
metadata:
name: nginx-service-1
spec:
ports:
- port: 80
protocol: TCP
selector:
env: prod
You can verify that service binds to deployment by checking the endpoints.
$ kubectl get pods -l run=nginx -o yaml | grep podIP
podIP: 10.32.0.9
podIP: 10.32.2.10
podIP: 10.32.0.10
podIP: 10.32.2.11
$ kk get ep nginx-service
NAME ENDPOINTS AGE
nginx-service 10.32.0.10:80,10.32.0.9:80,10.32.2.10:80 + 1 more... 3m33s
$ kk get ep nginx-service-1
NAME ENDPOINTS AGE
nginx-service-1 10.32.0.10:80,10.32.2.11:80 3m36s
Yes, you can do that.
Add a common label key pair to both the deployment pod spec and use that common label as selector in service definition
With the above defined service the requests would be load balanced across all the matching pods.
My goal is to monitor services with Prometheus, so I was following a guide located at:
https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/getting-started.md
I am relatively new to all of this, so please forgive my naiveness. I tried looking into the error, but all the answers were convoluted. I have no idea where to start on the debug process (perhaps look into the YAMLs?)
I wanted to monitor a custom Service. So, I deployed a service.yaml of the following into a custom namespace (t):
kind: Service
apiVersion: v1
metadata:
namespace: t
name: example-service-test
labels:
app: example-service-test
spec:
selector:
app: example-service-test
type: NodePort
ports:
- name: http
nodePort: 30901
port: 8080
protocol: TCP
targetPort: http
---
apiVersion: v1
kind: Pod
metadata:
name: example-service-test
namespace: t
labels:
app: example-service-test
spec:
containers:
- name: example-service-test
image: python:2.7
imagePullPolicy: IfNotPresent
command: ["/bin/bash"]
args: ["-c", "echo \"<p>This is POD1 $(hostname)</p>\" > index.html; python -m SimpleHTTPServer 8080"]
ports:
- name: http
containerPort: 8080
And deployed a service monitor into the namespace:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: example-service-test
labels:
team: frontendtest1
namespace: t
spec:
selector:
matchLabels:
app: example-service-test
endpoints:
- port: http
So far, the service monitor is detecting the service, as shown:
Prometheus Service Discovery.
However, there is an error with obtaining the metrics from the service: Prometheus Targets.
From what I know, prometheus isn't able to access the /metrics on the sample service - in that case, do I need to expose the metrics? If so, could I get a step by step guide solution to how to expose metrics? If not, what route should I take?
I'm afraid you could miss the key thing from the tutorial you're following on CoreOS website, about how a metrics from an app are getting to Prometheus:
First, deploy three instances of a simple example application, which
listens and exposes metrics on port 8080
Yes, your application (website) listens on port 8080, but does not expose any metrics on '/metrics' endpoint in the known to Prometheus format.
You can verify about what kind of metrics I'm talking about by hiting the endpoint from inside of Pod/Conatiner where it's hosted.
kubectl exec -it $(kubectl get po -l app=example-app -o jsonpath='{.items[0].metadata.name}') -c example-app -- curl localhost:8080/metrics
You should see similar output to this one:
# HELP codelab_api_http_requests_in_progress The current number of API HTTP requests in progress.
# TYPE codelab_api_http_requests_in_progress gauge
codelab_api_http_requests_in_progress 1
# HELP codelab_api_request_duration_seconds A histogram of the API HTTP request durations in seconds.
# TYPE codelab_api_request_duration_seconds histogram
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.0001"} 0
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.00015000000000000001"} 0
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.00022500000000000002"} 0
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.0003375"} 0
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.00050625"} 0
codelab_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.000759375"} 0
Please read more here on ways of exposing metrics.
I am for this example running the "echoheaders" Nginx in a deployment with 2 replicas. When I delete 1 pod, I sometimes get slow responses and errors for ~40 seconds.
We are running our API-gateway in Kubernetes, and need to be able to allow the Kubernetes scheduler to handle the pods as it sees fit.
We recently wanted to introduce session affinity, and for that, we wanted to migrate to the new and shiny NEG's: Network Endpoint Groups:
https://cloud.google.com/load-balancing/docs/negs/
When using NEG we experience problems during failover. Without NEG we're fine.
deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: echoheaders
labels:
app: echoheaders
spec:
replicas: 2
selector:
matchLabels:
app: echoheaders
template:
metadata:
labels:
app: echoheaders
spec:
containers:
- image: brndnmtthws/nginx-echo-headers
imagePullPolicy: Always
name: echoheaders
readinessProbe:
httpGet:
path: /
port: 8080
lifecycle:
preStop:
exec:
# Hack: wait for kube-proxy to remove endpoint before exiting, and
# gracefully shut down
command: ["bash", "-c", "sleep 10; nginx -s quit; sleep 40"]
restartPolicy: Always
terminationGracePeriodSeconds: 60
service.yaml
apiVersion: v1
kind: Service
metadata:
name: echoheaders
labels:
app: echoheaders
annotations:
cloud.google.com/neg: '{"ingress": true}'
spec:
ports:
- port: 80
protocol: TCP
targetPort: 8080
selector:
app: echoheaders
ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.global-static-ip-name: echoheaders-staging
name: echoheaders-staging
spec:
backend:
serviceName: echoheaders
servicePort: 80
When deleting a pod I get errors as shown in this image of
$ httping -G -K 35.190.69.21
(https://i.imgur.com/u14MvHN.png)
This is new behaviour when using NEG. Disabling NEG gives the old behaviour with working failover.
Any way to use Google LB, ingress, NEG and Kubernetes without errors during pod deletion?
In GCP load balancers a GET request will only be served a 502 after two subsequent backends fail to meet the response timeout or a impactful error occurred which seems more plausible.
What is possibly happening may be an interim period, wherein a Pod was due to be terminated and had received its SIGTERM, but was still considered healthy by the load balancer and was sent a request. Since this period was so brief, it wasn't able to complete the request and closed the connection.
A graceful service stop[1] in the machine will make that after receiving a SIGTERM, your service would continue to serve in-flight requests, but refuse new connections. This may solve your issue, but keep in mind that there is no guarantee of zero-downtime.
[1] https://landing.google.com/sre/sre-book/chapters/load-balancing-datacenter/#robust_approach_lame_duck