Nginx Ingress getting 504 gateway time-out - kubernetes

Question

I’m quite new to k8s in general, only been using for smaller projects but made it work. I hope btw this is the right channel to ask questions (in this case about ingress-nginx). I’m trying to setup a cluster with a gateway-api and a few microservices (all written in NestJs). To give a little background, I first had everything in docker-compose and my entry was also a Nginx container with letsencrypt. The whole docker, works great locally.
This was the config used for my NGinx Docker:
upstream equmedia-api {
server equmedia-api:3000;
}
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name localhost;
return 301 https://$server_name$request_uri;
}
server {
listen 80;
listen 443 ssl http2 default_server;
listen [::]:443 ssl http2 default_server;
keepalive_timeout 70;
server_name subdomain.example.com;
ssl_session_cache shared:SSR:10m;
ssl_session_timeout 10m;
ssl_certificate /etc/letsencrypt/live/equmedia.pixeliner.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/equmedia.pixeliner.com/privkey.pem;
access_log /var/log/nginx/nginx.access.log;
error_log /var/log/nginx/nginx.error.log;
location / {
proxy_pass http://equmedia-api;
# proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
As you can see, it upstreamed to my api container.
Eventually I wanted to turn the whole deployment into k8s. Seemed like a good followup practice after the small projects.
I learned about ingress-nginx and gave it my first try, but I seem to have struck a wall.
Here is the setup I'm trying to achieve:
Through DigitalOcean the setup will be behind a LoadBalancer.
Here is my Ingress NGinx controller:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: equmedia-ingress-api
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/issuer: "letsencrypt-prod"
nginx.ingress.kubernetes.io/rewrite-target: "/"
nginx.ingress.kubernetes.io/proxy-protocol: "true"
nginx.ingress.kubernetes.io/ssl-proxy-headers: "X-Forwarded-Proto: https"
spec:
tls:
- hosts:
- subdomain.example.com
secretName: quickstart-example-tls
rules:
- host: subdomain.example.com
http:
paths:
- path: /api
pathType: Prefix
backend:
service:
name: equmedia-api
port:
number: 3000
And my api service:
apiVersion: v1
kind: Service
metadata:
annotations:
kompose.cmd: kompose convert
kompose.version: 1.22.0 (955b78124)
creationTimestamp: null
labels:
io.kompose.service: equmedia-api
name: equmedia-api
spec:
ports:
- port: 3000
targetPort: 3000
selector:
io.kompose.service: equmedia-api
status:
loadBalancer: {}
When I try to access "https://subdomain.example.com/api/health", I get a 504 Gateway Time-out. Looking at the ingress controller logs I get the following:
2023/02/17 15:51:44 [error] 356#356: *336457 upstream timed out (110: Operation timed out) while connecting to upstream, client: 164.92.221.107, server: subdomain.example.com, request: "GET /api/health HTTP/2.0", upstream: "http://10.244.0.228:3000/", host: "subdomain.example.com"
2023/02/17 15:51:49 [error] 356#356: *336457 upstream timed out (110: Operation timed out) while connecting to upstream, client: 164.92.221.107, server: subdomain.example.com, request: "GET /api/health HTTP/2.0", upstream: "http://10.244.0.228:3000/", host: "subdomain.example.com"
2023/02/17 15:51:54 [error] 356#356: *336457 upstream timed out (110: Operation timed out) while connecting to upstream, client: 164.92.221.107, server: subdomain.example.com, request: "GET /api/health HTTP/2.0", upstream: "http://10.244.0.228:3000/", host: "subdomain.example.com"
Anyone that can point me into the right direction, to fix this issue?
EDIT
The outcome for
kubectl get pods -l io.kompose.service=equmedia-api:
NAME READY STATUS RESTARTS AGE
equmedia-api 1/1 Running 0 2d2h
kubectl get svc:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
equmedia-api ClusterIP 10.245.173.11 <none> 3000/TCP 23h
equmedia-api-rabbitmq ClusterIP 10.245.17.225 <none> 5672/TCP,15673/TCP 2d17h
equmedia-api-redis ClusterIP 10.245.120.11 <none> 6379/TCP 2d17h
equmedia-auth-db ClusterIP 10.245.94.21 <none> 5432/TCP 2d17h
kubernetes ClusterIP 10.245.0.1 <none> 443/TCP 2d17h
quickstart-ingress-nginx-controller LoadBalancer 10.245.36.216 179.128.139.106 80:31194/TCP,443:31609/TCP 2d16h
quickstart-ingress-nginx-controller-admission ClusterIP 10.245.232.77 <none> 443/TCP 2d16h
EDIT2:
I've requested my domain https://subdomain.example.com/api/health through browser, curl and postman. All of them return timeouts.
kubectl get pods -A -o wide | grep 10.244.0.228 returns:
default equmedia-api 1/1 Running 0 2d4h 10.244.0.228 temp-pool-qyhii <none> <none>
kubectl get svc -A | grep 10.244.0.228 returns nothing
EDIT3:
Here is the logs of my API:
[Nest] 1 - 02/17/2023, 10:52:27 AM LOG [NestFactory] Starting Nest application...
[Nest] 1 - 02/17/2023, 10:52:27 AM LOG [InstanceLoader] AppModule dependencies initialized +136ms
[Nest] 1 - 02/17/2023, 10:52:27 AM LOG [InstanceLoader] RedisCacheModule dependencies initialized +1ms
[Nest] 1 - 02/17/2023, 10:52:27 AM LOG [InstanceLoader] UtilsModule dependencies initialized +1ms
[Nest] 1 - 02/17/2023, 10:52:27 AM LOG [InstanceLoader] AxiosWrapperModule dependencies initialized +1ms
[Nest] 1 - 02/17/2023, 10:52:27 AM LOG [InstanceLoader] PassportModule dependencies initialized +32ms
[Nest] 1 - 02/17/2023, 10:52:27 AM LOG [InstanceLoader] JwtModule dependencies initialized +3ms
[Nest] 1 - 02/17/2023, 10:52:27 AM LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms
[Nest] 1 - 02/17/2023, 10:52:27 AM LOG [InstanceLoader] TerminusModule dependencies initialized +2ms
[Nest] 1 - 02/17/2023, 10:52:27 AM LOG [InstanceLoader] DiscoveryModule dependencies initialized +1ms
[Nest] 1 - 02/17/2023, 10:52:27 AM LOG [InstanceLoader] ConfigModule dependencies initialized +2ms
[Nest] 1 - 02/17/2023, 10:52:27 AM LOG [InstanceLoader] ConfigModule dependencies initialized +1ms
[Nest] 1 - 02/17/2023, 10:52:27 AM LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 1 - 02/17/2023, 10:52:27 AM LOG [InstanceLoader] ScheduleModule dependencies initialized +1ms
[Nest] 1 - 02/17/2023, 10:52:27 AM LOG [InstanceLoader] BullModule dependencies initialized +61ms
[Nest] 1 - 02/17/2023, 10:52:27 AM LOG [InstanceLoader] ClientsModule dependencies initialized +17ms
[Nest] 1 - 02/17/2023, 10:52:27 AM LOG [InstanceLoader] ClientsModule dependencies initialized +1ms
[Nest] 1 - 02/17/2023, 10:52:27 AM LOG [InstanceLoader] ClientsModule dependencies initialized +1ms
[Nest] 1 - 02/17/2023, 10:52:27 AM LOG [InstanceLoader] ClientsModule dependencies initialized +1ms
[Nest] 1 - 02/17/2023, 10:52:27 AM LOG [InstanceLoader] ClientsModule dependencies initialized +7ms
[Nest] 1 - 02/17/2023, 10:52:27 AM LOG [InstanceLoader] ClientsModule dependencies initialized +1ms
[Nest] 1 - 02/17/2023, 10:52:27 AM LOG [InstanceLoader] HealthModule dependencies initialized +8ms
[Nest] 1 - 02/17/2023, 10:52:27 AM LOG [InstanceLoader] CacheModule dependencies initialized +2ms
[Nest] 1 - 02/17/2023, 10:52:27 AM LOG [InstanceLoader] MailModule dependencies initialized +1ms
[Nest] 1 - 02/17/2023, 10:52:27 AM LOG [InstanceLoader] HttpModule dependencies initialized +3ms
[Nest] 1 - 02/17/2023, 10:52:27 AM LOG [InstanceLoader] BullModule dependencies initialized +24ms
[Nest] 1 - 02/17/2023, 10:52:27 AM LOG [InstanceLoader] BullQueueModule dependencies initialized +7ms
[Nest] 1 - 02/17/2023, 10:52:27 AM LOG [InstanceLoader] PaymentModule dependencies initialized +8ms
[Nest] 1 - 02/17/2023, 10:52:27 AM LOG [InstanceLoader] CustomerModule dependencies initialized +1ms
[Nest] 1 - 02/17/2023, 10:52:27 AM LOG [InstanceLoader] ContentModule dependencies initialized +2ms
[Nest] 1 - 02/17/2023, 10:52:27 AM LOG [InstanceLoader] AdserveModule dependencies initialized +3ms
[Nest] 1 - 02/17/2023, 10:52:27 AM LOG [InstanceLoader] AuthModule dependencies initialized +2ms
[Nest] 1 - 02/17/2023, 10:52:27 AM LOG [InstanceLoader] OpenIdModule dependencies initialized +65ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RoutesResolver] HealthController {/api/health}: +18ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/health, GET} route +5ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/health/check-ping, GET} route +2ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/health/check-disk, GET} route +2ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/health/check-memory, GET} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/health/check-microservice/:name, GET} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RoutesResolver] OpenIdController {/api/open-id}: +0ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/open-id/login, GET} route +2ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/open-id/user, GET} route +2ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/open-id/callback, GET} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/open-id/logout, GET} route +2ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RoutesResolver] AuthController {/api/auth}: +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/auth, GET} route +2ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/auth/signup, POST} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/auth/signin, POST} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/auth/signout, POST} route +2ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/auth/refresh, GET} route +0ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RoutesResolver] UserController {/api/user}: +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/user/get-user-id/email?, GET} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/user/get-authenticated-user, GET} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/user/:id/change-user-password, PUT} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/user/:id/delete-user-account, DELETE} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/user/confirm/:token, GET} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/user/forgot-password, POST} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/user/set-new-password/:token, POST} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RoutesResolver] UsersController {/api/users}: +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/users, GET} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RoutesResolver] PaymentController {/api/payment}: +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/payment/:id, GET} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/payment/create/:id, POST} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/payment/:id, PUT} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RoutesResolver] CustomerController {/api/customer}: +0ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/customer, GET} route +0ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/customer/profile/:id, GET} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/customer/create, POST} route +2ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/customer/delete/:id, DELETE} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/customer/update/:id, PUT} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RoutesResolver] ContentController {/api/content}: +0ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/content, GET} route +2ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/content/create, POST} route +0ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/content/update/:contentId, PUT} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/content/delete/:contentId, DELETE} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/content/category/:categoryId, GET} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/content/slug/:slug, GET} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RoutesResolver] CategoryController {/api/category}: +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/category, POST} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/category/create, POST} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/category/update/:categoryId, PUT} route +0ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/category/delete/:categoryId, DELETE} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RoutesResolver] WidgetController {/api/widget}: +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/widget, GET} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/widget/create, POST} route +0ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/widget/update/:widgetId, PUT} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/widget/delete/:widgetId, DELETE} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RoutesResolver] AdvertiserController {/api/adserve/advertiser}: +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/adserve/advertiser, GET} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/adserve/advertiser/:advertiserId, GET} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/adserve/advertiser/create, POST} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/adserve/advertiser/:advertiserId/campaigns/create, POST} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/adserve/advertiser/:advertiserId/campaigns/:campaignId, POST} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/adserve/advertiser/:advertiserId/campaigns/:campaignId/create, POST} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/adserve/advertiser/:advertiserId/campaigns/:campaignId/assign, POST} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RoutesResolver] AdserveController {/api/adserve}: +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/adserve/serve, GET} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/adserve/redirect, GET} route +0ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RoutesResolver] PublisherController {/api/adserve}: +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/adserve/publisher, GET} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/adserve/publisher/:publisherId, GET} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/adserve/publisher/create, POST} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/adserve/publisher/:publisherId/zone/create, POST} route +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RoutesResolver] ReportController {/api/adserve/report}: +1ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [RouterExplorer] Mapped {/api/adserve/report, GET} route +0ms
[Nest] 1 - 02/17/2023, 10:52:28 AM LOG [NestApplication] Nest application successfully started +58ms
-- API GATEWAY RUNNING - PORT: 3000 --
No errors are logged, and through a port-forward I also see my api working.
EDIT4:
Here is the gist with all pods/services/claims/...
https://gist.github.com/pixeliner/2c89048294197155b0d4833ab4045f3c

Answer 1

Your output text:
2023/02/17 15:51:44 [error] 356#356: *336457 upstream timed out (110: Operation timed out) while connecting to upstream, client: 164.92.221.107, server: subdomain.example.com, request: "GET /api/health HTTP/2.0", upstream: "http://10.244.0.228:3000/", host: "subdomain.example.com"
2023/02/17 15:51:49 [error] 356#356: *336457 upstream timed out (110: Operation timed out) while connecting to upstream, client: 164.92.221.107, server: subdomain.example.com, request: "GET /api/health HTTP/2.0", upstream: "http://10.244.0.228:3000/", host: "subdomain.example.com"
2023/02/17 15:51:54 [error] 356#356: *336457 upstream timed out (110: Operation timed out) while connecting to upstream, client: 164.92.221.107, server: subdomain.example.com, request: "GET /api/health HTTP/2.0", upstream: "http://10.244.0.228:3000/", host: "subdomain.example.com"
Implies the request is timing out on the IP 10.244.0.228:3000
Things to check:
Is the service IP 10.244.0.228: kubectl get svc equmedia-api (it will likely be of type ClusterIP)
Port forward to the service directly: kubectl port-forward svc/equmedia-api 3000:3000 and then try to access localhost:3000 in another terminal or in your browser. Does it respond, does it error or does it timeout?
Check the pods your service is trying to match: kubectl get pods -l io.kompose.service=equmedia-api -- does this return any pods? If so, are they in Ready state or are they erroring? Do they have a value greater than 0 in the Restarts count?
Check the logs of the pod(s) kubectl logs -f {pod-name} and see if it is unable to start up, or if it is repeatedly starting.
UPDATE 1
Please add the output of the following commands to your question. Wrap the output with three backticks (`) on a single line before and after to preserve formatting:
kubectl get pods -l io.kompose.service=equmedia-api
kubectl get svc
UPDATE 2
Since the IP that your controller is 10.244.0.228 see if any of your pods or services actually have that IP. Please add the output of these commands:
kubectl get pods -A -o wide | grep 10.244.0.228
kubectl get svc -A | grep 10.244.0.228
UPDATE 3
I've yet to try deploying the gist, but I have noticed something
You have networkpolicies setup and you have labelled your pod
apiVersion: v1
kind: Pod
metadata:
annotations:
kompose.cmd: kompose convert
kompose.version: 1.22.0 (955b78124)
creationTimestamp: null
labels:
io.kompose.network/backend: "true" # <<--- HERE
io.kompose.service: equmedia-api
name: equmedia-api-pod
spec:
...
This matches your network policy here:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
creationTimestamp: null
name: backend
spec:
ingress:
- from:
- podSelector:
matchLabels:
io.kompose.network/backend: "true"
podSelector:
matchLabels:
io.kompose.network/backend: "true"
Now, this network policy reads (based in the information off this link)
"Allow connections from Pods with the label io.kompose.network/backend="true" (last three lines) to pods that match the labels io.kompose.network/backend="true" (the ingress.from.podSelector bit)
Sooo.... assuming I'm reading this correct, the reason the ingress controller is not able to talk to the pod, is because the controller pod does not have a label io.kompose.network/backend="true", and since you did not include that in your gist, I'm assuming you're using the ingress controller chart as a subchart/dependency. And if so, then out of the box, that chart won't have this label. This would explain why we were able to port-forward to the pod and the service directly, but the controller pod was not able to talk to the pod.
And easy way to verify this is to either delete the backend networkpolicy, or modify it to allow all ingress traffic as a test (something like the example here)
If this works, it will confirm the networkpolicy is blocking the traffic.