AWS App Mesh request timeout for file stream which takes longer than 30 seconds - aws-app-mesh

My network setup on AWS looks like the following:
ECS Fargate services with App Mesh, Envoy Proxy and ELB.
Everything is working fine, except when a (download) request is took longer than 30 seconds. One of our service creates a zip file on request and sends a download link to the user. If the zip is small, everything is working fine, the user can download it successfully. If the zip is bigger and the download takes more than 30 seconds, it will fail.
The bug has been tracked down to App Mesh - Virtual Node Listener timeouts.
Timeouts was on default settings (empty/unset) and the 30 sec bug happened.
When Request timeout was set to a big enough number, the download was successful, but a fixed timeout, like 600s still had the chance to produce the same bug for really big files.
When Request timeout was set to 0s (expected to this will work as "unlimited"), bigger downloads was successful too, but not sure if it is a right thing to do or not.
My question is:
App mesh Listener with 0s Request timeout is a good practice or it will produce different problems what I'm not aware?
If it is a bad practice, how can I force App Mesh, to don't kill my file stream after 30 seconds?
Example response header for the file download:
HTTP/2 200 OK
date: Wed, 05 Oct 2022 09:06:45 GMT
content-type: application/octet-stream
content-length: 17325639
content-disposition: attachment; filename="a08c94a3-068e-486f-92c7-371d00984ddc.zip"
expires: Wed, 05 Oct 2022 09:07:45 GMT
cache-control: private, max-age=60
last-modified: Wed, 05 Oct 2022 07:11:28 GMT
access-control-allow-headers: Cache-Control, X-CSRF-Token, X-Requested-With
access-control-allow-origin: *
server: envoy
x-envoy-upstream-service-time: 55
X-Firefox-Spdy: h2
The following header is set by the server but removed possibly by envoy:
connection: keep-alive

30 secs is the default timeout, it is possible to define higher timeouts values for the listeners.
An example of a VirtualNode using 120 secs as timeout for the listener:
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualNode
metadata:
name: myapp
namespace: myns
spec:
podSelector:
matchLabels:
app: myapp
listeners:
- portMapping:
port: 80
protocol: http
timeout:
http:
perRequest:
value: 120
unit: s
backends:
- virtualService:
virtualServiceRef:
name: myapp
serviceDiscovery:
dns:
hostname: myapp.myns.svc.cluster.local

Related

How to access pod by IP when using Istio

I need to use spring boot admin to manage my pods. And I found that spring boot admin failed to call pod actuator endpoints.
I think it may be caused by Istio. Istio might block direct IP address visit. The following are my tests:
If I call a page by POD IP address, I got the following error:
# curl -I http://10.40.133.41
HTTP/1.1 503 Service Unavailable
content-length: 95
content-type: text/plain
date: Thu, 03 Nov 2022 03:10:54 GMT
server: envoy
If I call the same page by service name, it's fine:
# curl -I http://httpbin
HTTP/1.1 200 OK
server: envoy
date: Thu, 03 Nov 2022 03:09:02 GMT
content-type: text/html; charset=utf-8
content-length: 9593
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 3
Does anyone know how to enable a pod to access another pod by IP address directly when using Istio? Or is there any other solution to using spring boot admin when using Istio?
I tried to add service name as the Host request header when calling a web page. But it seems that Istio would find a pod by service name instead of using the IP address.
The following is an example that I used a wrong IP, and the system returns success status:
# curl -I -H "Host: httpbin" http://100.40.133.41
HTTP/1.1 200 OK
server: envoy
date: Thu, 03 Nov 2022 03:39:00 GMT
content-type: text/html; charset=utf-8
content-length: 9593
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 2

Ingress affinity session max age

I am using ingress affinity session in order to keep communication between a client and a pod. Because sticky session could cause some overloading to a pod (the clients keep same pod).
I'm looking for best practices about the parameter nginx.ingress.kubernetes.io/session-cookie-max-age.
The example value is 172 800 (second) which mean 48 hours.
Why? It's a huge duration, is it possible to set up it to 30 minutes?
By the way, what happens when the application session has expired? Does the ingress rebalance the client or keep the same pod?
This is an example documentation, you don't need to use the exact values provided in it.
You can set it up to any value you want, however setting up max-age and expires to too short periods of time, backend will be rebalanced too often. This the answer to another question - yes, ingress will rebalance the client.
There are two optional attributes you can use related to its age:
Expires=<date>
Indicates the maximum lifetime of the cookie as an HTTP-date timestamp. In case of ingress, it's set up as a number.
Max-Age=<number>
Indicates the number of seconds until the cookie expires. A zero or negative number will expire the cookie immediately.
Important! If both Expires and Max-Age are set, Max-Age has precedence.
Below is a working example with cookie max-age and expires set to 30 minutes:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-cookie-test
annotations:
nginx.ingress.kubernetes.io/affinity: "cookie"
nginx.ingress.kubernetes.io/session-cookie-name: "test-cookie"
nginx.ingress.kubernetes.io/session-cookie-expires: "1800"
nginx.ingress.kubernetes.io/session-cookie-max-age: "1800"
spec:
ingressClassName: nginx
rules:
- host: example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: service-name
port:
number: 80
And checking that it works performing a curl request (removed unnecessary details):
$ curl -I example.com
HTTP/1.1 200 OK
Date: Mon, 14 Mar 2022 13:14:42 GMT
Set-Cookie: test-cookie=1647263683.046.104.525797|ad50b946deebe30052b8573dcb9a2339; Expires=Mon, 14-Mar-22 13:44:42 GMT; Max-Age=1800; Path=/; HttpOnly

Can't access grafana through kong ingress controller for kubernetes

Iā€™m trying to expose grafana application to the internet following the below steps:
Applying helm chart as refrenced here https://github.com/Kong/kubernetes-ingress-controller
helm install kong/kong --generate-name --set ingressController.installCRDs=false
Then applying ingress rule as below
echo "
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: demo
annotations:
kubernetes.io/ingress.class: kong
spec:
rules:
- http:
paths:
- path: /grafana
backend:
serviceName: prom-stack-grafana
servicePort: 3000
" | kubectl apply -f -
Services are up and running correctly and pointing to the right end points
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
kong-1602803623-kong-proxy LoadBalancer 10.0.X.Y W.X.Y.Z 80:32218/TCP,443:30596/TCP
prom-stack-grafana ClusterIP 10.0.X.Y 3000/TCP
NAME ENDPOINTS
kong-1602803623-kong-proxy 10.244.X.Y:8443,10.244.X.Y:8000
prom-stack-grafana 10.244.X.Y:3000
Kong controller and ingress are running in the same namespace
Now the issue is that when I;m trying to access grafana through
curl -i $PROXY_IP/grafana and the same from the browser"empty page after redirection to /login"
I got redirected to /login with no output:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 29 100 29 0 0 82 0 --:ā€“:-- --:ā€“:-- --:ā€“:-- 82HTTP/1.1 302 Found
Content-Type: text/html; charset=utf-8
Content-Length: 29
Connection: keep-alive
Cache-Control: no-cache
Expires: -1
Location: /login
Pragma: no-cache
Set-Cookie: redirect_to=%2Fgra; Path=/; HttpOnly; SameSite=Lax
X-Content-Type-Options: nosniff
X-Frame-Options: deny
X-Xss-Protection: 1; mode=block
Date: Thu, 15 Oct 2020 23:31:30 GMT
X-Kong-Upstream-Latency: 2
X-Kong-Proxy-Latency: 0
Via: kong/2.1.4
Found
Need to know what is missing here to get redirected to the home page of grafana
Have you configured the root url in Grafana?
You have configured the Ingress to serve Grafana at /grafana, so you need to let Grafana know it needs to add this as a prefix to it's paths.
Configuration slightly changes based on the version of Grafana you are using, but you will need to set root_url. If you are using a newer version of Grafana you may also need to set serve_from_sub_path and domain.

control headers and routing depcrecated in istio 1.6

I need to route my traffic based on the headers using istio but this option is deprecated in istio1.6 version. Why control headers and routing is deprecated in istio?
As mentioned in istio documentation
The mixer policy is deprecated in Istio 1.5 and not recommended for production usage.
Consider using Envoy ext_authz filter, lua filter, or write a filter using the Envoy-wasm sandbox.
Control headers and routing are not deprecated, it's just mixer which was used to do that. There are different ways to do that now, as mentioned above.
I'm not sure what exactly you want to do, but take a look at envoy filter and virtual service.
Envoy filter
There is envoy filter which add some custom headers to all the outbound responses
kind: EnvoyFilter
metadata:
name: lua-filter
namespace: istio-system
spec:
workloadSelector:
labels:
istio: ingressgateway
configPatches:
- applyTo: HTTP_FILTER
match:
context: GATEWAY
listener:
filterChain:
filter:
name: "envoy.http_connection_manager"
subFilter:
name: "envoy.router"
patch:
operation: INSERT_BEFORE
value:
name: envoy.lua
typed_config:
"#type": "type.googleapis.com/envoy.config.filter.http.lua.v2.Lua"
inlineCode: |
function envoy_on_response(response_handle)
response_handle:logInfo(" ========= XXXXX ========== ")
response_handle:headers():add("X-User-Header", "worked")
end
And tests from curl
$ curl -s -I -X HEAD x.x.x.x/
HTTP/1.1 200 OK
server: istio-envoy
date: Mon, 06 Jul 2020 08:35:37 GMT
content-type: text/html
content-length: 13
last-modified: Thu, 02 Jul 2020 12:11:16 GMT
etag: "5efdcee4-d"
accept-ranges: bytes
x-envoy-upstream-service-time: 2
x-user-header: worked
Few links worth to check about that:
https://blog.opstree.com/2020/05/27/ip-whitelisting-using-istio-policy-on-kubernetes-microservices/
https://github.com/istio/istio/wiki/EnvoyFilter-Samples
https://istio.io/latest/docs/reference/config/networking/envoy-filter/
Virtual Service
Another thing worth to check here would be virtual service, you can do header routing based on matches here.
Take a look at example from istio documentation
HttpMatchRequest specifies a set of criterion to be met in order for the rule to be applied to the HTTP request. For example, the following restricts the rule to match only requests where the URL path starts with /ratings/v2/ and the request contains a custom end-user header with value jason.
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: ratings-route
spec:
hosts:
- ratings.prod.svc.cluster.local
http:
- match:
- headers:
end-user:
exact: jason
uri:
prefix: "/ratings/v2/"
ignoreUriCase: true
route:
- destination:
host: ratings.prod.svc.cluster.local
Additionally there is my older example with header based routing in virtual service.
Let me know if you have any more questions.

Does Istio really can't perform filter HTTPS requests based on the destination domains?

I've read this article about TLS Origination problem in istio. Let me quote it here:
There is a caveat to this story. In HTTPS, all the HTTP details (hostname, path, headers etc.) are encrypted, so Istio cannot know the destination domain of the encrypted requests. Well, Istio could know the destination domain by the SNI (Server Name Indication) field. This feature, however, is not yet implemented in Istio. Therefore, currently Istio cannot perform filtering of HTTPS requests based on the destination domains.
I want to understand, what does the bold statement really mean? Because, I've tried this:
Downloaded the
istio-1.0.0 here to get
the samples yaml code.
kubectl apply -f <(istioctl kube-inject -f samples/sleep/sleep.yaml)
apiVersion: v1
kind: Service
metadata:
name: sleep
labels:
app: sleep
spec:
ports:
- port: 80
name: http
selector:
app: sleep
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: sleep
spec:
replicas: 1
template:
metadata:
labels:
app: sleep
spec:
containers:
- name: sleep
image: tutum/curl
command: ["/bin/sleep","infinity"]
imagePullPolicy: IfNotPresent
And apply this ServiceEntry
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: cnn
spec:
hosts:
- "*.cnn.com"
ports:
- number: 80
name: http-port
protocol: HTTP
- number: 443
name: https-port
protocol: HTTPS
resolution: NONE
And exec this curl command inside the pod
export SOURCE_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})
kubectl exec -it $SOURCE_POD -c sleep -- curl -s -o /dev/null -D - https://edition.cnn.com/politics
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
x-servedByHost: ::ffff:172.17.128.31
access-control-allow-origin: *
cache-control: max-age=60
content-security-policy: default-src 'self' blob: https://*.cnn.com:* http://*.cnn.com:* *.cnn.io:* *.cnn.net:* *.turner.com:* *.turner.io:* *.ugdturner.com:* courageousstudio.com *.vgtf.net:*; script-src 'unsafe-eval' 'unsafe-inline' 'self' *; style-src 'unsafe-inline' 'self' blob: *; child-src 'self' blob: *; frame-src 'self' *; object-src 'self' *; img-src 'self' data: blob: *; media-src 'self' data: blob: *; font-src 'self' data: *; connect-src 'self' *; frame-ancestors 'self' https://*.cnn.com:* http://*.cnn.com https://*.cnn.io:* http://*.cnn.io:* *.turner.com:* courageousstudio.com;
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
Via: 1.1 varnish
Content-Length: 1554561
Accept-Ranges: bytes
Date: Wed, 08 Aug 2018 04:59:07 GMT
Via: 1.1 varnish
Age: 105
Connection: keep-alive
Set-Cookie: countryCode=US; Domain=.cnn.com; Path=/
Set-Cookie: geoData=mountain view|CA|94043|US|NA; Domain=.cnn.com; Path=/
Set-Cookie: tryThing00=3860; Domain=.cnn.com; Path=/; Expires=Mon Jul 01 2019 00:00:00 GMT
Set-Cookie: tryThing01=4349; Domain=.cnn.com; Path=/; Expires=Fri Mar 01 2019 00:00:00 GMT
Set-Cookie: tryThing02=4896; Domain=.cnn.com; Path=/; Expires=Wed Jan 01 2020 00:00:00 GMT
X-Served-By: cache-iad2150-IAD, cache-sin18022-SIN
X-Cache: HIT, MISS
X-Cache-Hits: 1, 0
X-Timer: S1533704347.303019,VS0,VE299
Vary: Accept-Encoding
As you can see, I can access the edition.cnn.com with HTTPS (ssl) protocol. Am I misunderstand the bold statement meaning?
The cited blog post is from January 31, 2018, and the statement was correct then. Now (1.0) Istio supports traffic routing by SNI, see https://istio.io/docs/tasks/traffic-management/egress/.
This reminds me to update that blog post, will do it by the end of this week. Sorry for the confusion, thank you for pointing to the issue.
What you're showing here is an https connection/request, and that has no reason not to work. Filtering in this case means taking a specific action (ie. deny access) based on the destination Host in http terms (the one that makes hosting multiple sites on the same server IP possible) and that is what that statement refers to.
SNI is the way to identify what Host you're connecting to, before TLS connection is established.