In my cluster, I am interested in service time at the Pod level (time difference between request arrival and departure), thus it would be integral to log incoming and outgoing requests (IP address or request ID, event timestamp).
I am aware of the capability to define custom logs using containers (from Logging Architecture), however I'm looking for a more dynamic solution with a smaller over head (hopefully exposed over the REST API).
Related
I am trying to understand how client response time behaves during a traffic surge. Lets say we have a concert ticket website (nginx HTTP2 and a CDN provider for handling requests) which shall release special tickets at a certain (pre-defined) time of the day. Naturally at that point of time, there would be a huge traffic surge for ticket buyers. Is there a way a client (ticket buyer) can always ensure to be first in the queue and get the ticket ? Won't constant requesting the ticket webpage constitute DoS (denial-of-service) attack ?
Tried to read about M/M/1 queue but don't think its used in practice.
My goal is to: Limit no of requests (not connection) for API per backend server, so that server won't get loaded with too many requests.
I can do this in middleware of each server, but problem is if server goes in stuck state, and won't able to perform any action on request, Request will go in wait state and that will impact client.
So I won't to perform this using haproxy, where haproxy based on requests on each server, will transfer request to next available node
I read documentation on haproxy.
But it has connection based rate limiting on each server. Or total request rate limiter on frontend, problem with this is if no of servers increase no of allowed request should increase and this does not limit request on each server instead its for service
https://www.haproxy.com/blog/four-examples-of-haproxy-rate-limiting/
Any help will be appreciated
I see istio is adding x-b3-traceid, x-b3-spanid and other headers to the incoming request when tracing is enabled. But none of them are returned to the caller.
I am able to capture the x-b3-traceid in the log and can find it out in Tempo/Grafana.
I can see the traceid at the istio envoy proxy (sidecar), I am able to access the header using EnvoyFilter.
Can someone let me know where it is filtered?
TL;DR
these are the headers so that jaeger or zipkin can track individual requests. This application is responsible for their proper propagation. Additionally, they are request headers and not response header, so everything works fine.
Explanation:
It looks okay. First, let's start with what these requests are:
Field Name Request/ Response Type Description
x-request-id request The x-request-idheader is used by Envoy to uniquely identify a request as well as perform stable access logging and tracing
x-b3-traceid request The x-b3-traceidHTTP header is used by the Zipkin tracer in Envoy. The TraceId is 64-bit in length and indicates the overall ID of the trace. Every span in a trace shares this ID
x-b3-spanid request The x-b3-spanidHTTP header is used by the Zipkin tracer in Envoy. The SpanId is 64-bit in length and indicates the position of the current operation in the trace tree
x-b3-sampled request The x-b3-sampledHTTP header is used by the Zipkin tracer in Envoy. When the Sampled flag is either not specified or set to 1, the span will be reported to the tracing system
On the github you can find the question: What is the usage of X-B3-TraceId, traceId, parentSpanId and spanId ?:
http headers contain only a small amount of data: IDs and sampling flags these go synchronously with your application requests, and allow the other side to continue the trace. https://github.com/openzipkin/b3-propagation#overall-process If zipkin is enabled, details including these IDs and other data like duration send asynchronously to zipkin after a request completes. you can see a diagram about that here under "Example Flow" http://zipkin.io/pages/architecture.html
X-B3-TraceId is same or different from every call of the same client? different per overall request. each top-level request into your system has
a different trace id. Each step in that request has a different span id
X-B3-SpanId is not send back to the caller, then how could i set the parent(which show be the X-B3-SpanId of the the present call) of the next call? Here is a response shows the absent of X-B3-SpanId in the header： I don't quite understand. The parent is only used when creating a span.
The span is created before a request is sent. So, there's no relevance to response headers in span creation.
In this doc you can find information about headers from istio site:
Distributed tracing enables users to track a request through mesh that is distributed across multiple services. This allows a deeper understanding about request latency, serialization and parallelism via visualization.
Istio leverages Envoy’s distributed tracing feature to provide tracing integration out of the box. Specifically, Istio provides options to install various tracing backend and configure proxies to send trace spans to them automatically. See Zipkin, Jaeger and Lightstep task docs about how Istio works with those tracing systems.
If you want to full understand how it works you should read this envoyproxy documentation:
Distributed tracing allows developers to obtain visualizations of call flows in large service oriented architectures. It can be invaluable in understanding serialization, parallelism, and sources of latency. Envoy supports three features related to system wide tracing:
Request ID generation: Envoy will generate UUIDs when needed and populate the x-request-id HTTP header. Applications can forward the x-request-id header for unified logging as well as tracing.
Client trace ID joining: The x-client-trace-id header can be used to join untrusted request IDs to the trusted internal x-request-id.
External trace service integration: Envoy supports pluggable external trace visualization providers, that are divided into two subgroups:
External tracers which are part of the Envoy code base, like LightStep, Zipkin or any Zipkin compatible backends (e.g. Jaeger), and Datadog.
External tracers which come as a third party plugin, like Instana.
Answering your question:
Can someone let me know where it is filtered?
They do this by default. It is the application (zipkin, jaeger) that is responsible for acting on these headers. Additionally, they are request headers and not response header, so everything works fine.
I have two google cloud instances running on different regions (West and East). Each instances have their own database. I am using Google Load balancer to route the traffic based upon the client's IP address (This is what Google load balancer is doing internally on Network load balancing).
For example, Bob is requesting from the east region and GLB will route the request to east region node only. Similarly, Dave is requesting from the west region and GLB will route the request to west region node.
Scenarios:
1. Bob just sign up and a new entry added to the East region database.
2. Bob tries to fetch his profile but somehow request went to the West (Bob now using a VPN) region and there is no information available.
Is there a way that I can customize GLB? If yes, then I can solve this problem by applying consistency hashing on the load balancer (using userId as a hash function) which will make sure that the request coming from Bob will always go to East region.
You can use the HTTP Load Balancer options: Session Affinity and Client IP Affinity.
There are subtle issues with any method, read the documentation carefully. The biggest issue is for clients behind a NAT (airport, hotel, Starbucks, etc.). Their public IP address is the same for all clients behind the NAT, therefore all traffic will go to the same backend for Client IP based affinity. I recommend using cookies.
Session affinity uses Client IP or Generated Cookie to make traffic decisions. This can keep traffic routed to the same backend.
Session Affinity
Client IP affinity directs requests from the same client IP address to the same backend instance based on a hash of the client's IP address.
Using Client IP Affinity
We have a SaaS. It consists of Single Page application (client), Gateway, Data Service 1, Data Service 2 and Notification Service.
Client talk with Gateway (using REST) and service route the request to appropriate Data Service (1 or 2) or do own calculations.
One request from the client can be split on multiple at Gateway service. The result is an aggregation of responses from the sub-services.
Notification Service - is a service which pushing information about changes made by other users using MQ and WebSocket connection to the client. Notification can be published by any service.
With enginers, we had a discussion how the process can be optimized.
Currently, the problem that Gateway spending a lot of time just waiting for the response from Data Services.
One of the proposals is letting Gateway service response 200 Ok as soon as message pushed to the Data Service and let client wait for operation progress throw Notification channel (WebSocket connection).
It means that client always sends HTTP request for operation and get confirmation that operation is executed by WebSocket from the different endpoint.
This schema can be hidden by providing JS client library which will hide all this internal complexity.
I think something wrong with this approach. I have never seen such design. But I don't have valuable arguments against it, except complexity and two points of failure (instead of one).
What do you think about this design approach?
Do you see any potential problems with it?
Do you know any public solutions with
such approach?
Since your service is slow it might makes sense to treat it more like a batch job.
Client sends a job request to Gateway.
Gateway returns a job ID immediately after accepting it from the Client.
Client periodically polls the Gateway for results for that job ID.