My question about Istio in Kubernetes. I have Istio sample rate of 1% and I have error which is not included in 1%. Would I see in Jaeger trace for this error?
I kind of new to Kubernetes and Istio. That's why can't tested on my own. I have been playing with Istio's example of Book Application and I wonder would I see trace with error which not included in 1% of sample rate.
Configure Istio when installing with:
pilot.traceSampling=1
As result want to know can I see error which not included in sample rate. If no, how I configure Istio to see it if possible?
If you have sampling rate set to 1% then error will be seen in Jaeger once it occurs 100 times.
This is mentioned at Distributed Tracing - Jaeger:
To see trace data, you must send requests to your service. The number of requests depends on Istio’s sampling rate. You set this rate when you install Istio. The default sampling rate is 1%. You need to send at least 100 requests before the first trace is visible. To send a 100 requests to the productpage service, use the following command:
$ for i in `seq 1 100`; do curl -s -o /dev/null http://$GATEWAY_URL/productpage; done
If you are not seeing the error in the current sample, I would advice make the sample higher.
You can read about Tracing context propagation which is being done by Envoy.
Envoy automatically sends spans to tracing collectors
Alternatively the trace context can be manually propagated by the service:
When using the LightStep tracer, Envoy relies on the service to propagate the x-ot-span-context HTTP header while sending HTTP requests to other services.
When using the Zipkin tracer, Envoy relies on the service to propagate the B3 HTTP headers ( x-b3-traceid, x-b3-spanid, x-b3-parentspanid, x-b3-sampled, and x-b3-flags). The x-b3-sampled header can also be supplied by an external client to either enable or disable tracing for a particular request. In addition, the single b3 header propagation format is supported, which is a more compressed format.
When using the Datadog tracer, Envoy relies on the service to propagate the Datadog-specific HTTP headers ( x-datadog-trace-id, x-datadog-parent-id, x-datadog-sampling-priority).
Create yaml-bomb.yaml file:
apiVersion: v1
data:
a: &a ["web","web","web","web","web","web","web","web","web"]
b: &b [*a,*a,*a,*a,*a,*a,*a,*a,*a]
c: &c [*b,*b,*b,*b,*b,*b,*b,*b,*b]
d: &d [*c,*c,*c,*c,*c,*c,*c,*c,*c]
e: &e [*d,*d,*d,*d,*d,*d,*d,*d,*d]
f: &f [*e,*e,*e,*e,*e,*e,*e,*e,*e]
g: &g [*f,*f,*f,*f,*f,*f,*f,*f,*f]
h: &h [*g,*g,*g,*g,*g,*g,*g,*g,*g]
i: &i [*h,*h,*h,*h,*h,*h,*h,*h,*h]
kind: ConfigMap
metadata:
name: yaml-bomb
namespace: default
Send ConfigMap creation request to Kubernetes API by cmd kubectl apply -f yaml-bomb.yaml.
kube-api CPU/memory usage are very high, even later are getting restarted.
How do we prevent such yaml-bomb?
This is a billion laughts attack and can only be fixed in the YAML processor.
Note that the Wikipedia is wrong here when it says
A "Billion laughs" attack should exist for any file format that can contain references, for example this YAML bomb:
The problem is not that the file format contains references; it is the processor expanding them. This is against the spirit of the YAML spec which says that anchors are used for nodes that are actually referred to from multiple places. In the loaded data, anchors & aliases should become multiple references to the same object instead of the alias being expanded to a copy of the anchored node.
As an example, compare the behavior of the online PyYAML parser and the online NimYAML parser (full disclosure: my work) when you paste your code snippet. PyYAML won't respond because of the memory load from expanding aliases, while NimYAML doesn't expand the aliases and therefore responds quickly.
It's astonishing that Kubernetes suffers from this problem; I would have assumed since it's written in Go that they are able to properly handle references. You have to file a bug with them to get this fixed.
There's a couple of possible mitigations I could think of although as #flyx says the real fix here would be in the YAML parsing library used by Kubernetes.
Interestingly running this on a Kubernetes cluster on my local machine showed the CPU spike to be client-side (it's the kubectl process churning CPU) rather than server side.
If the issue was server side, then possible mitigations would be to use RBAC to minimize access to ConfigMap creation, and potentially to use an admission controller like OPA to review manifests before they are applied to the cluster.
This should probably be raised with the Kubernetes security vulnerability response team so that a proper fix can be implemented.
EDIT - I think where the problem manifests, might be down to the cluster version used. Server-side apply graduated to beta (should be enabled by default) in 1.16. So on a 1.16 cluster perhaps this would hit server side instead of client side.
EDIT - Just setup a 1.16 cluster, still showing the CPU usage as client-side in kubectl...
EDIT - I've filed an issue for this here also confirmed that the DoS can be achieved server-side by using curl instead of kubectl
Final EDIT - This got assigned a CVE (CVE-2019-11253) and is being fixed in Kubernetes 1.13+ . The fix has also been applied to the underlying YAML parsing lib here so any other Go programs should be ok as long as they're using an up to date version.
There was a TrustCom19 paper studying vulnerabilities in YAML parsers for different languages, it found that most parsers have some issues, so this is common and there are several recent CVEs in this space (details in paper: Laughter in the Wild: A Study into DoS Vulnerabilities in YAML Libraries, TrustCom19.
Preprint: https://www.researchgate.net/publication/333505459_Laughter_in_the_Wild_A_Study_into_DoS_Vulnerabilities_in_YAML_Libraries
My application (hosted in a Kubernetes cluster with Istio installed) does NOT propagate distributed tracing headers (as described here). My expectation is that istio-proxy should still generate a trace (consisting of a single call) that would be visible in Jaeger, even though of course the entire chain of calls would not be stitched together. However, that doesn't appear to be the case, as I'm not seeing any calls to my application in Jaeger.
In attempt to troubleshoot I have tried the following:
Logs for the istio-proxy container deployed as a side-car to my application's container look good, I can see incoming requests to the application being registered by Envoy:
kubectl logs -f helloworld-69b7f5b6f8-chp9n -c istio-proxy
[2019-01-29T21:29:18.925Z] - 444 289 45 "127.0.0.1:80" inbound|81||helloworld.default.svc.cluster.local 127.0.0.1:45930 10.244.0.54:80 10.244.0.1:33733
[2019-01-29T21:29:29.922Z] - 444 289 25065 "127.0.0.1:80" inbound|81||helloworld.default.svc.cluster.local 127.0.0.1:46014 10.244.0.54:80 10.240.0.5:56166
[2019-01-29T21:30:05.922Z] - 444 289 15051 "127.0.0.1:80" inbound|81||helloworld.default.svc.cluster.local 127.0.0.1:46240 10.244.0.54:80 10.240.0.6:48053
[2019-01-29T21:30:31.922Z] - 444 289 36 "127.0.0.1:80" inbound|81||helloworld.default.svc.cluster.local 127.0.0.1:46392 10.244.0.54:80 10.240.0.6:47009
I have enabled tracing in Mixer's configuration, and I can now see Mixer's activity in Jaeger UI (but no traces of calls to my application still).
I'm new to Istio, and it appears I have run out of option.
First off, is my expectation correct? Am I supposed to be seeing traces - each consisting of a single call - in Jaeger UI when the application doesn't propagate distributed tracing headers?
If my expectation is correct, how can I troubleshoot further? Can I somehow verify Envoy configuration and check that it's indeed tracing data to Mixer?
If my expectation is incorrect, can Istio's behavior be overridden so that I get what I need?
Thank you.
I run kubernetes cluster in AWS, CoreOS-stable-1745.6.0-hvm (ami-401f5e38), all deployed by kops 1.9.1 / terraform.
etcd_version = "3.2.17"
k8s_version = "1.10.2"
This Prometheus alert method=QGET alertname=HighNumberOfFailedHTTPRequests is coming from coreos kube-prometheus monitoring bundle. The alert started to fire from the very beginning of the cluster lifetime and now exists for ~3 weeks without visible impact.
^ QGET fails - 33% requests.
NOTE: I have the 2nd cluster in other region built from scratch on the same versions and it has exact same behavior. So it's reproducible.
Anyone knows what might be the root cause, and what's the impact if ignored further?
EDIT:
Later I found this GH issue which describes my case precisely: https://github.com/coreos/etcd/issues/9596
From CoreOS documentation:
For alerts to not appear on arbitrary events it is typically better not to alert directly on a raw value that was sampled, but rather by aggregating and defining a relative threshold rather than a hardcoded value. For example: send a warning if 1% of the HTTP requests fail, instead of sending a warning if 300 requests failed within the last five minutes. A static value would also require a change whenever your traffic volume changes.
Here you can find detailed information on how to Develop Prometheus alerts for etcd.
I got the explanation in GitHub issue thread.
HTTP metrics/alerts should be replaced with GRPC.
We're building a microservice architecture on Spring Cloud + NetflixOSS and so far IT'S AMAZING. However we have noticed that when we first spin up Eureka + 2 separate microservices, Service1 cannot connect to Service2 for a good ~30 seconds, even though both register with Eureka very quickly and appear up individually. This happens locally or deployed (on AWS via Tutum)
The error we get is:
Load balancer does not have available server for client: Service1
Pretty much all of our timeouts and settings are just specified as the default, so we feel like there's something wrong. What we're worried about is rolling an update to a Service1 and having it register to Eureka & then having 30 seconds of downtime (or possibly downtime on Service2 that depends on Service1)
Has anyone seen this behavior before? Is there a way to inspect / debug / log the client-side load balancing candidate? We've fished through a lot of the Netflix code (both with / without the debugger) but we're not seeing a lot of logging and introspection points that will be useful (especially since the Observables tend to timeout during deep debugging sessions)
Advice? Suggestions? Pagan gods whom desire sacrifices?
Thanks
ribbon.ServerListRefreshInterval=2000 will set the client side to refresh every 2 seconds.