Incorrect Observability in GCP ASM (Managed Istio) - kubernetes

New to the world of Istio, we are using managed Anthos service mesh on our GKE cluster. We have a service called pgbouncer deployed which is a connection pooler for PostgreSQL, we have few internal applications which connect to the pgbouncer service (pgbouncer.pgbouncer.svc.cluster.local) to access PostgreSQL DB.
Istio-proxy logs on pgbouncer pod:
[2023-02-02T17:30:11.633Z] "- - -" 0 - - - "-" 1649 1970 7 - "-" "-" "-" "-" "10.243.34.74:5432" inbound|5432|| 127.0.0.6:58765 10.243.34.74:5432 10.243.36.173:59516 outbound_.5432_._.pgbouncer.pgbouncer.svc.cluster.local -
[2023-02-02T17:30:11.654Z] "- - -" 0 - - - "-" 1645 1968 8 - "-" "-" "-" "-" "10.243.34.74:5432" inbound|5432|| 127.0.0.6:56153 10.243.34.74:5432 10.243.38.39:56404 outbound_.5432_._.pgbouncer.pgbouncer.svc.cluster.local -
[2023-02-02T17:30:11.674Z] "- - -" 0 - - - "-" 1647 1970 7 - "-" "-" "-" "-" "10.243.34.74:5432" inbound|5432|| 127.0.0.6:38471 10.243.34.74:5432 10.243.38.39:56414 outbound_.5432_._.pgbouncer.pgbouncer.svc.cluster.local -
[2023-02-02T17:30:11.696Z] "- - -" 0 - - - "-" 1647 1968 7 - "-" "-" "-" "-" "10.243.34.74:5432" inbound|5432|| 127.0.0.6:35135 10.243.34.74:5432 10.243.33.184:52074 outbound_.5432_._.pgbouncer.pgbouncer.svc.cluster.local -
[2023-02-02T17:30:11.716Z] "- - -" 0 - - - "-" 1646 1970 8 - "-" "-" "-" "-" "10.243.34.74:5432" inbound|5432|| 127.0.0.6:45277 10.243.34.74:5432 10.243.32.36:47044 outbound_.5432_._.pgbouncer.pgbouncer.svc.cluster.local -
[2023-02-02T17:30:11.738Z] "- - -" 0 - - - "-" 1644 1968 7 - "-" "-" "-" "-" "10.243.34.74:5432" inbound|5432|| 127.0.0.6:43099 10.243.34.74:5432 10.243.36.99:33514 outbound_.5432_._.pgbouncer.pgbouncer.svc.cluster.local -
[2023-02-02T17:30:11.757Z] "- - -" 0 - - - "-" 1649 1970 7 - "-" "-" "-" "-" "10.243.34.74:5432" inbound|5432|| 127.0.0.6:54943 10.243.34.74:5432 10.243.36.173:59530 outbound_.5432_._.pgbouncer.pgbouncer.svc.cluster.local -
[2023-02-02T17:30:11.777Z] "- - -" 0 - - - "-" 1644 1968 9 - "-" "-" "-" "-" "10.243.34.74:5432" inbound|5432|| 127.0.0.6:49555 10.243.34.74:5432 10.243.36.99:33524 outbound_.5432_._.pgbouncer.pgbouncer.svc.cluster.local -
[2023-02-02T17:30:11.800Z] "- - -" 0 - - - "-" 1646 1970 8 - "-" "-" "-" "-" "10.243.34.74:5432" inbound|5432|| 127.0.0.6:51239 10.243.34.74:5432 10.243.32.36:47056 outbound_.5432_._.pgbouncer.pgbouncer.svc.cluster.local -
10.243.34.74 --> pgbouncer pod IP
10.243.32.36 --> ingress gateway Pod IP (not sure how the gateway is used here, as the internal apps hit pgbouncer.pgbouncer.svc.cluster.local)
Logs clearly show that there are inbound requests from internal apps.
but when we visualise the kaili kinda view provided by GCP we notice that the source to the pgbouncer service is unknown.
We were in the notion that the sources will be the list of internal apps hitting the pgbouncer to reflect in the above connected graph for pgbouncer service.
Also checked the PromQL istio_requests_total{ app_kubernetes_io_instance="pgbouncer"} to get the number of requests and source.
istio_requests_total{app_kubernetes_io_instance="pgbouncer", app_kubernetes_io_name="pgbouncer", cluster="gcp-np-001", connection_security_policy="none", destination_app="unknown", destination_canonical_revision="latest", destination_canonical_service="pgbouncer", destination_cluster="cn-g-asia-southeast1-g-gke-non-prod-001", destination_principal="unknown", destination_service="pgbouncer", destination_service_name="InboundPassthroughClusterIpv4", destination_service_namespace="pgbouncer", destination_version="unknown", destination_workload="pgbouncer", destination_workload_namespace="pgbouncer", instance="10.243.34.74:15020", job="kubernetes-pods", kubernetes_namespace="pgbouncer", kubernetes_pod_name="pgbouncer-86f5448f69-qgpll", pod_template_hash="86f5448f69", reporter="destination", request_protocol="http", response_code="200", response_flags="-", security_istio_io_tlsMode="istio", service_istio_io_canonical_name="pgbouncer", service_istio_io_canonical_revision="latest", source_app="unknown", source_canonical_revision="latest", source_canonical_service="unknown", source_cluster="unknown", source_principal="unknown", source_version="unknown", source_workload="unknown", source_workload_namespace="unknown"}
Here source is again unknown, we have many request coming in from the internal apps which doesn't reflect in the promql or kaili kinda view. Not sure why the destination_service_name="InboundPassthroughClusterIpv4" is mentioned as passthrough ?
Any insights is appreciated !

Related

nginx-ingress - eks 400 bad request

I've got an AWS EKS environment that i'm just setting up and I'm getting 400 bad request.
Below is my config and I wanted to ask if anyone saw anything I should change.
I can see the requests are getting through the aws NLB as it reaches the nginx ingress controller but I can't see any decent information in the ingress controller logs. will add some below.
I'm terminating 443 at the NLB so sending http/80 to controller...
W0408 01:06:35.428413 7 controller.go:1094] Service "dating/photoapi-prod" does not have any active Endpoint.
W0408 01:06:35.428682 7 controller.go:1094] Service "dating/photoapi-test" does not have any active Endpoint.
192.168.119.209 - - [08/Apr/2022:01:50:55 +0000] "\x00" 400 150 "-" "-" 0 0.000 [] [] - - - - 1d65f0090db1addb14e870d877977bfc
192.168.119.209 - - [08/Apr/2022:01:50:59 +0000] "\x00" 400 150 "-" "-" 0 0.000 [] [] - - - - b853574052dfd56745839f72a6fc5ed1
192.168.90.222 - - [08/Apr/2022:01:50:59 +0000] "\x00" 400 150 "-" "-" 0 0.000 [] [] - - - - c38d59e8ffcb008cf01ab4bb3ea4cd39
192.168.90.222 - - [08/Apr/2022:01:51:00 +0000] "\x00" 400 150 "-" "-" 0 0.000 [] [] - - - - 3ca1bfdbb1f35c7b89d213636a12e864
192.168.119.209 - - [08/Apr/2022:01:51:05 +0000] "\x00" 400 150 "-" "-" 0 0.000 [] [] - - - - 338a2445058641d71c32e9acdf467118
As per error it says the service behind the ingress controller doesn’t have active container running behind it.
Do you have container running behind the services mentioned in error?
in my case, it fixed after remove unused listening port in LB.
I thinks it occured by health check.

aws-load-balancer-controller error: cannot get the logs from *v1.Ingress: selector for *v1.Ingress not implemented

I get this error when trying to get ALB logs:
root#b75651fde30e:/apps/tekton/deployment# kubectl logs -f ingress/tekton-dashboard-alb-dev
error: cannot get the logs from *v1.Ingress: selector for *v1.Ingress not implemented
The load balancer YAML:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: tekton-dashboard-alb-dev
namespace: tekton-pipelines
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/load-balancer-name: tekton-dashboard-alb-dev
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/backend-protocol: HTTP
alb.ingress.kubernetes.io/tags: "Cost=SwiftALK,VantaOnwer=foo#bar.com,VantaNonProd=true,VantaDescription=ALB Ingress for Tekton Dashboard,VantaContainsUserData=false,VantaUserDataStored=None"
alb.ingress.kubernetes.io/security-groups: sg-034ca9846b81fd721
kubectl.kubernetes.io/last-applied-configuration: ""
spec:
defaultBackend:
service:
name: tekton-dashboard
port:
number: 9097
Note: sg-034ca9846b81fd721 restricts access to our VPN CIDRs
Ingress is up as revealed from:
root#b75651fde30e:/apps/tekton/deployment# kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
tekton-dashboard-alb-dev <none> * tekton-dashboard-alb-dev-81361211.us-east-1.elb.amazonaws.com 80 103m
root#b75651fde30e:/apps/tekton/deployment# kubectl describe ingress/tekton-dashboard-alb-dev
Name: tekton-dashboard-alb-dev
Namespace: tekton-pipelines
Address: tekton-dashboard-alb-dev-81361211.us-east-1.elb.amazonaws.com
Default backend: tekton-dashboard:9097 (172.18.5.248:9097)
Rules:
Host Path Backends
---- ---- --------
* * tekton-dashboard:9097 (172.18.5.248:9097)
Annotations: alb.ingress.kubernetes.io/backend-protocol: HTTP
alb.ingress.kubernetes.io/load-balancer-name: tekton-dashboard-alb-dev
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/security-groups: sg-034ca9846b81fd721
alb.ingress.kubernetes.io/tags:
Cost=SwiftALK,VantaOnwer=swiftalkdevteam#digite.com,VantaNonProd=true,VantaDescription=ALB Ingress for SwifTalk Web Microservices,VantaCon...
alb.ingress.kubernetes.io/target-type: ip
kubernetes.io/ingress.class: alb
Events: <none>
The error you received means that the logs for your object are not implemented. It looks like you're trying to get logs from the wrong place.
I am not able to reproduce your problem on AWS, but I tried to do it on GCP and the situation was very similar. You cannot get logs from ingress/tekton-dashboard-alb-dev, and this is normal bahaviour. If you want to get logs of your ALB, you have to find the appropriate pod and then extract the logs from it. Let me show you how I did it on GCP. The commands are the same, but the pod names will be different.
First I have executed:
kubectl get pods --all-namespaces
Output:
NAMESPACE NAME READY STATUS RESTARTS AGE
ingress-nginx ingress-nginx-controller-57cb5bf694-722ml 1/1 Running 0 18d
-----
and many other not related pods in other namespaces
You can find directly your pod with command:
kubectl get pods -n ingress-nginx
Output:
NAME READY STATUS RESTARTS AGE
ingress-nginx-controller-57cb5bf694-722ml 1/1 Running 0 18d
Now you can get logs from ingress controller by command:
kubectl logs -n ingress-nginx ingress-nginx-controller-57cb5bf694-722ml
in your situation:
kubectl logs -n <your namespace> <your ingress controller pod>
The output should be similar to this:
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: v0.46.0
Build: 6348dde672588d5495f70ec77257c230dc8da134
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.19.6
-------------------------------------------------------------------------------
I0923 05:26:20.053561 8 flags.go:208] "Watching for Ingress" class="nginx"
W0923 05:26:20.053753 8 flags.go:213] Ingresses with an empty class will also be processed by this Ingress controller
W0923 05:26:20.054185 8 client_config.go:614] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0923 05:26:20.054502 8 main.go:241] "Creating API client" host="https://10.16.0.1:443"
I0923 05:26:20.069482 8 main.go:285] "Running in Kubernetes cluster" major="1" minor="20+" git="v1.20.9-gke.1001" state="clean" commit="1fe18c314ed577f6047d2712a9d1c8e498e22381" platform="linux/amd64"
I0923 05:26:20.842645 8 main.go:105] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0923 05:26:20.846132 8 main.go:115] "Enabling new Ingress features available since Kubernetes v1.18"
W0923 05:26:20.849470 8 main.go:127] No IngressClass resource with name nginx found. Only annotation will be used.
I0923 05:26:20.866252 8 ssl.go:532] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0923 05:26:20.917594 8 nginx.go:254] "Starting NGINX Ingress controller"
I0923 05:26:20.942084 8 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"42dc476e-3c5c-4cc9-a6a4-266edecb2a4b", APIVersion:"v1", ResourceVersion:"5600", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I0923 05:26:22.118459 8 nginx.go:296] "Starting NGINX process"
I0923 05:26:22.118657 8 leaderelection.go:243] attempting to acquire leader lease ingress-nginx/ingress-controller-leader-nginx...
I0923 05:26:22.119481 8 nginx.go:316] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I0923 05:26:22.120266 8 controller.go:146] "Configuration changes detected, backend reload required"
I0923 05:26:22.126350 8 status.go:84] "New leader elected" identity="ingress-nginx-controller-57cb5bf694-8c9tn"
I0923 05:26:22.214194 8 controller.go:163] "Backend successfully reloaded"
I0923 05:26:22.214838 8 controller.go:174] "Initial sync, sleeping for 1 second"
I0923 05:26:22.215234 8 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-57cb5bf694-722ml", UID:"b9672f3c-ecdf-473e-80f5-529bbc5bc4e5", APIVersion:"v1", ResourceVersion:"59016530", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0923 05:27:00.596169 8 leaderelection.go:253] successfully acquired lease ingress-nginx/ingress-controller-leader-nginx
I0923 05:27:00.596305 8 status.go:84] "New leader elected" identity="ingress-nginx-controller-57cb5bf694-722ml"
157.230.143.29 - - [23/Sep/2021:08:28:25 +0000] "GET / HTTP/1.1" 400 248 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:54.0) Gecko/20100101 Firefox/70.0" 165 0.000 [] [] - - - - d47be1e37ea504aca93d59acc7d36a2b
157.230.143.29 - - [23/Sep/2021:08:28:26 +0000] "\x00\xFFK\x00\x00\x00\xE2\x00 \x00\x00\x00\x0E2O\xAAC\xE92g\xC2W'\x17+\x1D\xD9\xC1\xF3,kN\x17\x14" 400 150 "-" "-" 0 0.076 [] [] - - - - c497187f4945f8e9e7fa84d503198e85
157.230.143.29 - - [23/Sep/2021:08:28:26 +0000] "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00" 400 150 "-" "-" 0 0.138 [] [] - - - - 4067a2d34d0c1f2db7ffbfc143540c1a
167.71.216.70 - - [23/Sep/2021:12:02:23 +0000] "\x16\x03\x01\x01\xFC\x01\x00\x01\xF8\x03\x03\xDB\xBBo*K\xAE\x9A&\x8A\x9B)\x1B\xB8\xED3\xB7\xE16N\xEA\xFCS\x22\x14V\xF7}\xC8&ga\xDA\x00\x01<\xCC\x14\xCC\x13\xCC\x15\xC00\xC0,\xC0(\xC0$\xC0\x14\xC0" 400 150 "-" "-" 0 0.300 [] [] - - - - ff6908bb17b0da020331416773b928b5
167.71.216.70 - - [23/Sep/2021:12:02:23 +0000] "\x16\x03\x01\x01\xFC\x01\x00\x01\xF8\x03\x03a\xBF\xFB\xC1'\x03S\x83D\x5Cn$\xAB\xE1\xA6%\x93G-}\xD1C\xB2\xB0E\x8C\x8F\xA8q-\xF7$\x00\x01<\xCC\x14\xCC\x13\xCC\x15\xC00\xC0,\xC0(\xC0$\xC0\x14\xC0" 400 150 "-" "-" 0 0.307 [] [] - - - - fee3a478240e630e6983c60d1d510f52
66.240.205.34 - - [23/Sep/2021:12:04:11 +0000] "145.ll|'|'|SGFjS2VkX0Q0OTkwNjI3|'|'|WIN-JNAPIER0859|'|'|JNapier|'|'|19-02-01|'|'||'|'|Win 7 Professional SP1 x64|'|'|No|'|'|0.7d|'|'|..|'|'|AA==|'|'|112.inf|'|'|SGFjS2VkDQoxOTIuMTY4LjkyLjIyMjo1NTUyDQpEZXNrdG9wDQpjbGllbnRhLmV4ZQ0KRmFsc2UNCkZhbHNlDQpUcnVlDQpGYWxzZQ==12.act|'|'|AA==" 400 150 "-" "-" 0 0.086 [] [] - - - - 365d42d67e7378359b95c71a8d8ce983
147.182.148.98 - - [23/Sep/2021:12:04:17 +0000] "\x16\x03\x01\x01\xFC\x01\x00\x01\xF8\x03\x03\xABA\xF4\xD5\xB7\x95\x85[.v\xDB\xD1\x1B\x04\xE7\xB4\xB8\x92\x82\xEC\xCC\xDDr\xB7/\xBD\x93/\xD0f4\xB3\x00\x01<\xCC\x14\xCC\x13\xCC\x15\xC00\xC0,\xC0(\xC0$\xC0\x14\xC0" 400 150 "-" "-" 0 0.152 [] [] - - - - 858c2ad7535de95c84dd0899708a3801
164.90.203.66 - - [23/Sep/2021:12:08:19 +0000] "\x16\x03\x01\x01\xFC\x01\x00\x01\xF8\x03\x03\x93\x81+_\x95\xFA\xEAj\xA7\x80\x15 \x179\xD7\x92\xAE\xA9i+\x9D`\xA07:\xD2\x22\xB3\xC6\xF3\x22G\x00\x01<\xCC\x14\xCC\x13\xCC\x15\xC00\xC0,\xC0(\xC0$\xC0\x14\xC0" 400 150 "-" "-" 0 0.237 [] [] - - - - 799487dd8ec874532dcfa7dad1c02a27
164.90.203.66 - - [23/Sep/2021:12:08:20 +0000] "\x16\x03\x01\x01\xFC\x01\x00\x01\xF8\x03\x03\xB8\x22\xCB>1\xBEM\xD4\x92\x95\xEF\x1C0\xB5&\x1E[\xC5\xC8\x1E2\x07\x1C\x02\xA1<\xD2\xAA\x91F\x00\xC6\x00\x01<\xCC\x14\xCC\x13\xCC\x15\xC00\xC0,\xC0(\xC0$\xC0\x14\xC0" 400 150 "-" "-" 0 0.193 [] [] - - - - 4604513713d4b9fb5a7199b7980fa7f2
164.90.203.66 - - [23/Sep/2021:12:16:10 +0000] "\x16\x03\x01\x01\xFC\x01\x00\x01\xF8\x03\x03[\x16\x02\x94\x98\x17\xCA\xB5!\xC11#\x08\xD9\x89RE\x970\xC2\xDF\xFF\xEBh\xA0i\x9Ee%.\x07{\x00\x01<\xCC\x14\xCC\x13\xCC\x15\xC00\xC0,\xC0(\xC0$\xC0\x14\xC0" 400 150 "-" "-" 0 0.116 [] [] - - - - 23019f0886a1c30a78092753f6828e74
77.247.108.81 - - [23/Sep/2021:14:52:51 +0000] "GET /admin/config.php HTTP/1.1" 400 248 "-" "python-requests/2.26.0" 164 0.000 [] [] - - - - 04630dbf3d0ff4a4b7138dbc899080e5
209.141.48.211 - - [23/Sep/2021:16:17:46 +0000] "" 400 0 "-" "-" 0 0.057 [] [] - - - - 3c623b242909a99e18178ec10a814d7b
209.141.62.185 - - [23/Sep/2021:18:13:11 +0000] "GET /config/getuser?index=0 HTTP/1.1" 400 248 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:76.0) Gecko/20100101 Firefox/76.0" 353 0.000 [] [] - - - - 2640cf06912615a7600e814dc893884b
125.64.94.138 - - [23/Sep/2021:19:49:08 +0000] "GET / HTTP/1.0" 400 248 "-" "-" 18 0.000 [] [] - - - - b633636176888bc3b7f6230f691e0724
2021/09/23 19:49:20 [crit] 39#39: *424525 SSL_do_handshake() failed (SSL: error:141CF06C:SSL routines:tls_parse_ctos_key_share:bad key share) while SSL handshaking, client: 125.64.94.138, server: 0.0.0.0:443
125.64.94.138 - - [23/Sep/2021:19:49:21 +0000] "GET /favicon.ico HTTP/1.1" 400 650 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4 240.111 Safari/537.36" 197 0.000 [] [] - - - - ede08c8fb12e8ebaf3adcbd2b7ea5fd5
125.64.94.138 - - [23/Sep/2021:19:49:22 +0000] "GET /robots.txt HTTP/1.1" 400 650 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4 240.111 Safari/537.36" 196 0.000 [] [] - - - - fae50b56a11600abc84078106ba4b008
125.64.94.138 - - [23/Sep/2021:19:49:22 +0000] "GET /.well-known/security.txt HTTP/1.1" 400 650 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4 240.111 Safari/537.36" 210 0.000 [] [] - - - - ad82bcac7d7d6cd9aa2d044d80bb719d
87.251.75.145 - - [23/Sep/2021:21:29:10 +0000] "\x03\x00\x00/*\xE0\x00\x00\x00\x00\x00Cookie: mstshash=Administr" 400 150 "-" "-" 0 0.180 [] [] - - - - 8c2b62bcdf26ac1592202d0940fc30b8
167.71.102.181 - - [23/Sep/2021:21:54:58 +0000] "\x00\x0E8K\xA3\xAAe\xBCn\x14\x1B\x00\x00\x00\x00\x00" 400 150 "-" "-" 0 0.027 [] [] - - - - 65b8ee37a2c6bf8368843e4db3b90b2a
185.156.72.27 - - [23/Sep/2021:22:03:55 +0000] "\x03\x00\x00/*\xE0\x00\x00\x00\x00\x00Cookie: mstshash=Administr" 400 150 "-" "-" 0 0.139 [] [] - - - - 92c6ad2d71b961bf7de4e345ff69da10
185.156.72.27 - - [23/Sep/2021:22:03:55 +0000] "\x03\x00\x00/*\xE0\x00\x00\x00\x00\x00Cookie: mstshash=Administr" 400 150 "-" "-" 0 0.140 [] [] - - - - fe0424f8ecf9afc1d0154bbca2382d13
34.86.35.21 - - [23/Sep/2021:22:54:41 +0000] "\x16\x03\x01\x00\xE3\x01\x00\x00\xDF\x03\x03\x0F[\xA9\x18\x15\xD3#4\x7F\x7F\x98'\xA9(\x8F\xE7\xCCDd\xF9\xFF`\xE3\xCE\x9At\x05\x97\x05\xB1\xC3}\x00\x00h\xCC\x14\xCC\x13\xC0/\xC0+\xC00\xC0,\xC0\x11\xC0\x07\xC0'\xC0#\xC0\x13\xC0\x09\xC0(\xC0$\xC0\x14\xC0" 400 150 "-" "-" 0 2.039 [] [] - - - - c09d38bf2cd925dac4d9e5d5cb843ece
2021/09/24 02:41:15 [crit] 40#40: *627091 SSL_do_handshake() failed (SSL: error:141CF06C:SSL routines:tls_parse_ctos_key_share:bad key share) while SSL handshaking, client: 184.105.247.252, server: 0.0.0.0:443
61.219.11.151 - - [24/Sep/2021:03:40:51 +0000] "dN\x93\xB9\xE6\xBCl\xB6\x92\x84:\xD7\x03\xF1N\xB9\xC5;\x90\xC2\xC6\xBA\xE1I-\x22\xDDs\xBA\x1FgC:\xB1\xA7\x80+\x00\x00\x00\x00%\xFDK:\xAAW.|J\xB2\xB5\xF5'\xA5l\xD3V(\xB7\x01%(CsK8B\xCE\x9A\xD0z\xC7\x13\xAD" 400 150 "-" "-" 0 0.203 [] [] - - - - 190d00221eefc869b5938ab6380f835a
46.101.155.106 - - [24/Sep/2021:04:56:37 +0000] "HEAD / HTTP/1.0" 400 0 "-" "-" 17 0.000 [] [] - - - - e8c108201c37d7457e4578cf68feacf8
46.101.155.106 - - [24/Sep/2021:04:56:38 +0000] "GET /system_api.php HTTP/1.1" 400 650 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36" 255 0.000 [] [] - - - - b3032f9a9b3f4f367bdee6692daeb05c
46.101.155.106 - - [24/Sep/2021:04:56:39 +0000] "GET /c/version.js HTTP/1.1" 400 650 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36" 253 0.000 [] [] - - - - 9104ab72a0232caf6ff98da57d325144
46.101.155.106 - - [24/Sep/2021:04:56:40 +0000] "GET /streaming/clients_live.php HTTP/1.1" 400 650 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36" 267 0.000 [] [] - - - - 341cbb6cf424b348bf8b788f79373b8d
46.101.155.106 - - [24/Sep/2021:04:56:41 +0000] "GET /stalker_portal/c/version.js HTTP/1.1" 400 650 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36" 268 0.000 [] [] - - - - 9954fd805fa092595057dbf83511bd92
46.101.155.106 - - [24/Sep/2021:04:56:42 +0000] "GET /stream/live.php HTTP/1.1" 400 248 "-" "AlexaMediaPlayer/2.1.4676.0 (Linux;Android 5.1.1) ExoPlayerLib/1.5.9" 209 0.000 [] [] - - - - 3c9409419c1ec59dfc08c10cc3eb6eef
It is as the message says: there is no such thing as "logs" for an Ingress resource. Specifically, what it is telling you is that running kubectl -n tekton-pipelines get pods -l $(kubectl -n tekton-pipelines get -o jsonpath={.metadata.labels} ingress/tekton-dashboard-alb-dev) correctly produced no Pods from which one could gather logs
Perhaps you actually wanted the logs for the Pods behind service/tekton-dashboard but asking for the logs of an Ingress is nonsense

Istio envoy proxy issue

We have configured the istio 1.4.0 with demo profile on Kubernetes cluster 1.15.1. It was working as expected but after some time facing issues with the application which are connecting to backend servers like mongo DB. Application pod is going in crashloopbackup and if i disabled istio it works properly.
Upon checking the istio-proxy logs found lines stating http/1.1 DPE and mongo DB IP and port number
Below is the Istio-proxy logs (sidecar),
#
[2020-03-11T13:40:28.504Z] "- - HTTP/1.1" 0 DPE "-" "-" 0 0 0 - "-" "-" "-" "-" "-" - - <mongo IP>:27017 10.233.92.103:49412 - -
[2020-03-11T13:40:28.508Z] "- - HTTP/1.1" 0 DPE "-" "-" 0 0 0 - "-" "-" "-" "-" "-" - - <mongo IP>:27017 10.233.92.103:52062 - -
[2020-03-11T13:40:28.528Z] "- - HTTP/1.1" 0 DPE "-" "-" 0 0 0 - "-" "-" "-" "-" "-" - - <mongo IP>:27017 10.233.92.103:37182 - -
[2020-03-11T13:40:28.529Z] "- - HTTP/1.1" 0 DPE "-" "-" 0 0 0 - "-" "-" "-" "-" "-" - - <mongo IP>:27017 10.233.92.103:49428 - -
[2020-03-11T13:40:28.530Z] "- - HTTP/1.1" 0 DPE "-" "-" 0 0 0 - "-" "-" "-" "-" "-" - - 10.26.61.18:27017 10.233.92.103:52078 - -
[2020-03-11T13:40:28.569Z] "POST /intake/v2/events HTTP/1.1" 202 - "-" "-" 941 0 3 1 "-" "elasticapm-node/3.3.0 elastic-apm-http-client/9.3.0 node/10.12.0" "8954f0a1-709b-963c-a480-05b078955c89" "<apm>:8200" "10.26.61.45:8200" PassthroughCluster - <apm>:8200 10.233.92.103:49992 - -
[2020-03-11T13:40:28.486Z] "- - -" 0 - "-" "-" 47 3671 98 - "-" "-" "-" "-" "<redis>:6379" PassthroughCluster 10.233.92.103:37254 <redis>:6379 10.233.92.103:37252 - -
[2020-03-11T13:40:30.168Z] "- - -" 0 - "-" "-" 632 1212236 104 - "-" "-" "-" "-" "104.16.25.35:443" PassthroughCluster 10.233.92.103:60760 104.16.25.35:443 10.233.92.103:60758 - -```
#
and application logs giving error as below
{ err: 'socketHandler', trace: '', bin: undefined, parseState: { sizeOfMessage: 1347703880, bytesRead: undefined, stubBuffer: undefined } }
The issue has been resolved.
RCA :-
I have manually created service and endpoint of MongoDB with port name as http
After that when i checked the listeners in proxy-config via istioctl command i found an entry with address 0.0.0.0 and port 27017
ADDRESS PORT TYPE
0.0.0.0 27017 TCP
In json output, i interpret that its going into blackholecluster even if i set allow_any in passthroughcluster.
The output in istio-proxy always give me DPE error.
After understanding the issue i changed the name from http to http1 and it worked properly.
Now need to understand why the name http was creating so much issue
Might be this issue: https://github.com/kubernetes/enhancements/issues/753
Basically, about ordering of containers. At startup time if your application container required networking and sidecar container(envoy proxy) not fully started, then application will raise an networking error.
Reference: https://discuss.istio.io/t/k8s-istio-sidecar-injection-with-other-init-containers/845

Using Couchbase operator 1.1 with Istio Service Mesh

I am trying to create a Couchbase cluster in GKE with Istio (envoy proxy) using the Autonomous Operator 1.1.
The operator starts up fine, and after running the yaml to create the couchbasecluster the first node starts up and then the 2nd node starts. Issue is that the 2nd node appears to fail to join the cluster and additional nodes are not being started.
I am not sure how to debug what is happening or what needs to be done to get the cluster to start up in my gke cluster. Any assistance is appreceated.
Thank you
Here are some of the logs from one of the couchbase node pods:
I [2019-04-02T14:58:00.706Z] "POST /engageCluster2HTTP/1.1" 404 NR 0 0 0 - "-" "-" "782bde60-c611-4bfb-a0f4-9975300c71a4" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.8.13:37221
I [2019-04-02T14:58:05.706Z] "POST /engageCluster2HTTP/1.1" 404 NR 0 0 0 - "-" "-" "382b6163-e8bc-4259-baaa-e854c36af1bd" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.8.13:55515
I [2019-04-02T14:58:10.707Z] "POST /engageCluster2HTTP/1.1" 404 NR 0 0 0 - "-" "-" "390e417e-b179-4bbf-81d8-02cc28d2bc98" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.8.13:34377
I [2019-04-02T14:53:13.605Z] - 210 4281 300015 "127.0.0.1:8091" inbound|8091||cb-example.couchbase.svc.cluster.local 127.0.0.1:45756 10.36.8.13:8091 10.36.9.12:49792
I [2019-04-02T14:58:15.709Z] "POST /engageCluster2HTTP/1.1" 404 NR 0 0 0 - "-" "-" "037d1791-9feb-47be-b699-10269aaf36e9" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.8.13:55307
I [2019-04-02T14:58:20.708Z] "POST /engageCluster2HTTP/1.1" 404 NR 0 0 0 - "-" "-" "5ca29b59-ff25-4a13-a0c1-62668d40c681" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.8.13:51205
I [2019-04-02T14:58:25.706Z] "POST /engageCluster2HTTP/1.1" 404 NR 0 0 0 - "-" "-" "9e21bc4d-1367-4d25-b674-39ae6341c9b4" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.8.13:41435
I [2019-04-02T14:58:30.710Z] "POST /engageCluster2HTTP/1.1" 404 NR 0 0 0 - "-" "-" "c2f8e866-e0a5-43ff-b54f-e5c504b17cdf" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.8.13:40203
I [2019-04-02T14:58:35.708Z] "POST /engageCluster2HTTP/1.1" 404 NR 0 0 0 - "-" "-" "4b02e855-cc72-49dc-99e1-a8644fdf1af8" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.8.13:56433
I [2019-04-02T14:53:13.641Z] - 16628 40061 324989 "127.0.0.1:8091" inbound|8091||cb-example.couchbase.svc.cluster.local 127.0.0.1:45760 10.36.8.13:8091 10.36.9.12:49796
I [2019-04-02T14:56:45.698Z] - 9490 13635 112934 "127.0.0.1:8091" inbound|8091||cb-example.couchbase.svc.cluster.local 127.0.0.1:46218 10.36.8.13:8091 10.36.9.12:50534
I [2019-04-02T14:56:45.665Z] - 210 4281 112967 "127.0.0.1:8091" inbound|8091||cb-example.couchbase.svc.cluster.local 127.0.0.1:46216 10.36.8.13:8091 10.36.9.12:50528
And a portion of the error.log from inside the couchbase container.
[ns_server:error,2019-04-03T16:09:47.398Z,ns_1#cb-example-0000.cb-example.couchbase.svc:service_agent-index<0.24974.68>:service_agent:handle_call:182]Got rebalance-only call {if_rebalance,<0.23572.68>,unset_rebalancer} that doesn't match rebalancer pid undefined
[ns_server:error,2019-04-03T16:09:47.398Z,ns_1#cb-example-0000.cb-example.couchbase.svc:service_rebalancer-index<0.23572.68>:service_agent:process_bad_results:810]Service call unset_rebalancer (service index) failed on some nodes:
[{'ns_1#cb-example-0000.cb-example.couchbase.svc',nack}]
[ns_server:error,2019-04-03T16:09:47.398Z,ns_1#cb-example-0000.cb-example.couchbase.svc:cleanup_process<0.23562.68>:service_janitor:maybe_init_topology_aware_service:87]Initial rebalance for `index` failed: {error,
{initial_rebalance_failed,index,
{linked_process_died,<0.23516.68>,
{no_connection,
"index-service_api"}}}}
[ns_server:error,2019-04-03T16:10:47.399Z,ns_1#cb-example-0000.cb-example.couchbase.svc:<0.24979.68>:service_agent:wait_for_connection_loop:299]No connection with label "index-service_api" after 60000ms. Exiting.
[ns_server:error,2019-04-03T16:10:47.399Z,ns_1#cb-example-0000.cb-example.couchbase.svc:service_agent-index<0.24974.68>:service_agent:handle_info:231]Linked process <0.24979.68> died with reason {no_connection,
"index-service_api"}. Terminating
[ns_server:error,2019-04-03T16:10:47.399Z,ns_1#cb-example-0000.cb-example.couchbase.svc:service_agent-index<0.24974.68>:service_agent:terminate:260]Terminating abnormally
[ns_server:error,2019-04-03T16:10:47.399Z,ns_1#cb-example-0000.cb-example.couchbase.svc:service_rebalancer-index<0.25043.68>:service_rebalancer:run_rebalance:82]Agent terminated during the rebalance: {'DOWN',#Ref<0.0.48.97712>,process,
<0.24974.68>,
{linked_process_died,<0.24979.68>,
{no_connection,"index-service_api"}}}
[ns_server:error,2019-04-03T16:10:47.400Z,ns_1#cb-example-0000.cb-example.couchbase.svc:service_agent-index<0.26461.68>:service_agent:handle_call:182]Got rebalance-only call {if_rebalance,<0.25043.68>,unset_rebalancer} that doesn't match rebalancer pid undefined
[ns_server:error,2019-04-03T16:10:47.400Z,ns_1#cb-example-0000.cb-example.couchbase.svc:service_rebalancer-index<0.25043.68>:service_agent:process_bad_results:810]Service call unset_rebalancer (service index) failed on some nodes:
[{'ns_1#cb-example-0000.cb-example.couchbase.svc',nack}]
[ns_server:error,2019-04-03T16:10:47.400Z,ns_1#cb-example-0000.cb-example.couchbase.svc:cleanup_process<0.25042.68>:service_janitor:maybe_init_topology_aware_service:87]Initial rebalance for `index` failed: {error,
{initial_rebalance_failed,index,
{linked_process_died,<0.24979.68>,
{no_connection,
"index-service_api"}}}}
[ns_server:error,2019-04-03T16:11:47.401Z,ns_1#cb-example-0000.cb-example.couchbase.svc:<0.26456.68>:service_agent:wait_for_connection_loop:299]No connection with label "index-service_api" after 60000ms. Exiting.
[ns_server:error,2019-04-03T16:11:47.401Z,ns_1#cb-example-0000.cb-example.couchbase.svc:service_agent-index<0.26461.68>:service_agent:handle_info:231]Linked process <0.26456.68> died with reason {no_connection,
"index-service_api"}. Terminating
[ns_server:error,2019-04-03T16:11:47.401Z,ns_1#cb-example-0000.cb-example.couchbase.svc:service_agent-index<0.26461.68>:service_agent:terminate:260]Terminating abnormally
[ns_server:error,2019-04-03T16:11:47.401Z,ns_1#cb-example-0000.cb-example.couchbase.svc:service_rebalancer-index<0.26515.68>:service_rebalancer:run_rebalance:82]Agent terminated during the rebalance: {'DOWN',#Ref<0.0.48.106235>,process,
<0.26461.68>,
{linked_process_died,<0.26456.68>,
{no_connection,"index-service_api"}}}
[ns_server:error,2019-04-03T16:11:47.402Z,ns_1#cb-example-0000.cb-example.couchbase.svc:service_agent-index<0.27939.68>:service_agent:handle_call:182]Got rebalance-only call {if_rebalance,<0.26515.68>,unset_rebalancer} that doesn't match rebalancer pid undefined
[ns_server:error,2019-04-03T16:11:47.402Z,ns_1#cb-example-0000.cb-example.couchbase.svc:service_rebalancer-index<0.26515.68>:service_agent:process_bad_results:810]Service call unset_rebalancer (service index) failed on some nodes:
[{'ns_1#cb-example-0000.cb-example.couchbase.svc',nack}]
[ns_server:error,2019-04-03T16:11:47.402Z,ns_1#cb-example-0000.cb-example.couchbase.svc:cleanup_process<0.26517.68>:service_janitor:maybe_init_topology_aware_service:87]Initial rebalance for `index` failed: {error,
{initial_rebalance_failed,index,
{linked_process_died,<0.26456.68>,
{no_connection,
"index-service_api"}}}}
And this is from the current portion of the couchbase-operator log:
I [2019-04-03T16:15:13.959Z] "GET /poolsHTTP/1.1" 404 NR 0 0 0 - "-" "Go-http-client/1.1" "cc976505-818a-4930-9fc8-8bdcb047185d" "cb-example-0000.cb-example.couchbase.svc:8091" "-" - - 10.36.8.13:8091 10.36.9.12:59280
I [2019-04-03T16:15:13.963Z] "GET /poolsHTTP/1.1" 404 NR 0 0 0 - "-" "Go-http-client/1.1" "bfd981b2-9356-4132-a7f8-2a6c0d8ba15f" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.9.12:57624
I [2019-04-03T16:15:14.939Z] - 119 135 0 "127.0.0.1:8080" inbound|8080||mgmtCluster 127.0.0.1:37568 10.36.9.12:8080 10.36.9.1:44810
I [2019-04-03T16:15:17.939Z] - 119 135 0 "127.0.0.1:8080" inbound|8080||mgmtCluster 127.0.0.1:37574 10.36.9.12:8080 10.36.9.1:44816
I [2019-04-03T16:15:18.959Z] "GET /poolsHTTP/1.1" 404 NR 0 0 0 - "-" "Go-http-client/1.1" "997c061a-d5d2-425d-b123-bf76073d148a" "cb-example-0000.cb-example.couchbase.svc:8091" "-" - - 10.36.8.13:8091 10.36.9.12:59298
I [2019-04-03T16:15:18.962Z] "GET /poolsHTTP/1.1" 404 NR 0 0 0 - "-" "Go-http-client/1.1" "b03def85-726f-4107-8b27-9fc8b5bddea7" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.9.12:57642
I [2019-04-03T16:15:20.939Z] - 119 135 0 "127.0.0.1:8080" inbound|8080||mgmtCluster 127.0.0.1:37586 10.36.9.12:8080 10.36.9.1:44828
E time="2019-04-03T16:15:26Z" level=warning msg="cluster status: failed with error [Get http://cb-example-0000.cb-example.couchbase.svc:8091/pools/default: uuid check: unexpected status code '404 Not Found' from cb-example-0000.cb-example.couchbase.svc:8091], [Get http://cb-example-0003.cb-example.couchbase.svc:8091/pools/default: uuid check: unexpected status code '404 Not Found' from cb-example-0003.cb-example.couchbase.svc:8091] ...retrying" cluster-name=cb-example module=cluster
E time="2019-04-03T16:15:31Z" level=warning msg="cluster status: failed with error [Get http://cb-example-0000.cb-example.couchbase.svc:8091/pools/default: uuid check: unexpected status code '404 Not Found' from cb-example-0000.cb-example.couchbase.svc:8091], [Get http://cb-example-0003.cb-example.couchbase.svc:8091/pools/default: uuid check: unexpected status code '404 Not Found' from cb-example-0003.cb-example.couchbase.svc:8091] ...retrying" cluster-name=cb-example module=cluster
I [2019-04-03T16:15:23.939Z] - 119 135 0 "127.0.0.1:8080" inbound|8080||mgmtCluster 127.0.0.1:37592 10.36.9.12:8080 10.36.9.1:44834
I [2019-04-03T16:15:26.939Z] - 119 135 0 "127.0.0.1:8080" inbound|8080||mgmtCluster 127.0.0.1:37604 10.36.9.12:8080 10.36.9.1:44846
I [2019-04-03T16:15:26.987Z] "GET /poolsHTTP/1.1" 404 NR 0 0 0 - "-" "Go-http-client/1.1" "c20d057e-646d-4eb3-8931-a220126c27d5" "cb-example-0000.cb-example.couchbase.svc:8091" "-" - - 10.36.8.13:8091 10.36.9.12:59326
I [2019-04-03T16:15:26.991Z] "GET /poolsHTTP/1.1" 404 NR 0 0 0 - "-" "Go-http-client/1.1" "c46e9753-1249-4b8c-8fc9-889e74a0d70b" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.9.12:57670
I [2019-04-03T16:15:29.939Z] - 119 135 1 "127.0.0.1:8080" inbound|8080||mgmtCluster 127.0.0.1:37616 10.36.9.12:8080 10.36.9.1:44858
I [2019-04-03T16:15:31.986Z] "GET /poolsHTTP/1.1" 404 NR 0 0 0 - "-" "Go-http-client/1.1" "30c8522f-5797-488c-b234-d1c5a43d9826" "cb-example-0000.cb-example.couchbase.svc:8091" "-" - - 10.36.8.13:8091 10.36.9.12:59338
I [2019-04-03T16:15:31.990Z] "GET /poolsHTTP/1.1" 404 NR 0 0 0 - "-" "Go-http-client/1.1" "a4f74a5e-c026-46b4-b9e3-8b20f65477b4" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.9.12:57682
E time="2019-04-03T16:15:36Z" level=warning msg="cluster status: failed with error [Get http://cb-example-0000.cb-example.couchbase.svc:8091/pools/default: uuid check: unexpected status code '404 Not Found' from cb-example-0000.cb-example.couchbase.svc:8091], [Get http://cb-example-0003.cb-example.couchbase.svc:8091/pools/default: uuid check: unexpected status code '404 Not Found' from cb-example-0003.cb-example.couchbase.svc:8091] ...retrying" cluster-name=cb-example module=cluster
E time="2019-04-03T16:15:41Z" level=warning msg="cluster status: failed with error [Get http://cb-example-0000.cb-example.couchbase.svc:8091/pools/default: uuid check: unexpected status code '404 Not Found' from cb-example-0000.cb-example.couchbase.svc:8091], [Get http://cb-example-0003.cb-example.couchbase.svc:8091/pools/default: uuid check: unexpected status code '404 Not Found' from cb-example-0003.cb-example.couchbase.svc:8091] ...retrying" cluster-name=cb-example module=cluster
I [2019-04-03T16:15:32.939Z] - 119 135 0 "127.0.0.1:8080" inbound|8080||mgmtCluster 127.0.0.1:37624 10.36.9.12:8080 10.36.9.1:44866

How to count requests per hour?

I have access log file with data only per 1 day like:
10.2.21.120 "-" - [26/Jan/2013:19:15:11 +0000] "GET /server/ad?uid=abc&type=PST HTTP/1.1" 202 14 10
10.2.21.120 "-" - [26/Jan/2013:19:17:22 +0000] "GET /server/ad?uid=abc&type=PST HTTP/1.1" 204 14 9
10.2.22.130 "-" - [26/Jan/2013:19:27:53 +0000] "GET /server/ad?uid=abc&type=PST HTTP/1.1" 200 14 8
I am using the following command:
awk '$9 == 200 { s++ } END { print s / NR * 100; }' access.log
This awk may help you
$ awk -F[:\ ] '{count[$5]++}; $12 == 200 { hour[$5]++} END { for (i in hour) print i, hour[i]/count[i]*100 }' input
Test
$ cat input
10.1.20.123 "1.1.1.1" - [15/Oct/2014:12:14:17 +0000] "POST /server/ad?uid=abc&type=PST HTTP/1.1" 200 3014 10
10.1.20.123 "1.1.1.1" - [15/Oct/2014:12:14:17 +0000] "POST /server/ad?uid=abc&type=PST HTTP/1.1" 100 3014 10
10.1.20.123 "1.1.1.1" - [15/Oct/2014:13:14:26 +0000] "POST /server/ad?uid=abc&type=PST HTTP/1.1" 200 3014 9
10.1.20.123 "1.1.1.1" - [15/Oct/2014:13:24:55 +0000] "POST /server/ad?uid=abc&type=PST HTTP/1.1" 200 3014 8
$ awk -F[:\ ] '{count[$5]++}; $12 == 200 { hour[$5]++} END { for (i in hour) print i, hour[i]/count[i]*100 }' input
12 50
13 100
What it does
{count[$5]++} array count stores the number of occurence of each hour from the log file.
$12 == 200 { hour[$5]++} Now it the log is success, that is $12 == 200 then the corresponding value in hour array is incremented.
So count[13] will contain total enteries from hour 13 where as hour[13] would contain count of succuessfull entries
END { for (i in hour) print i, hour[i]/count[i]*100 } prints the hour, percentage
I would use something like this:
awk -F"[ :]" '
{tot[$5]++; if ($(NF-2)==200) succ[$5]++}
END {for (i in tot) printf "%d %d/%d %.2f%\n", i, succ[i], tot[i], succ[i]/tot[i]*100}' file
This defines : or space as field separators. Hence, the hour is stored in the 5th field and the return code in the one before the penultimate (NF-2) - we could also use the 12th as you do, but this allows support for longest input in the log.
Then, it keeps track of the amount of lines in the tot[] array and the successful ones in succ[] array.
Finally it prints the result
Here we have some more data with different hours and return codes:
$ cat a
10.1.20.123 "1.1.1.1" - [15/Oct/2014:13:14:17 +0000] "POST /server/ad?uid=abc&type=PST HTTP/1.1" 200 3014 10
10.1.20.123 "1.1.1.1" - [15/Oct/2014:13:14:26 +0000] "POST /server/ad?uid=abc&type=PST HTTP/1.1" 200 3014 9
10.1.20.123 "1.1.1.1" - [15/Oct/2014:13:24:55 +0000] "POST /server/ad?uid=abc&type=PST HTTP/1.1" 100 3014 8
10.1.20.123 "1.1.1.1" - [15/Oct/2014:17:24:55 +0000] "POST /server/ad?uid=abc&type=PST HTTP/1.1" 200 3014 8
10.1.20.123 "1.1.1.1" - [15/Oct/2014:17:24:55 +0000] "POST /server/ad?uid=abc&type=PST HTTP/1.1" 404 3014 8
Let's run the script:
$ awk -F"[ :]" '{tot[$5]++; if ($(NF-2)==200) succ[$5]++} END {for (i in tot) printf "%d %d/%d %.2f%\n", i, succ[i], tot[i], succ[i]/tot[i]*100}' a
13 2/3 66.67%
17 1/2 50.00%
% cat success.awk
NF==11{
split($4,a,":") ; hour = a[2]
total[hour] += 1
if($9==200) success[hour] += 1
}
END{
for(h in total) print h, 100*(0+success[h])/total[h]"% ("0+success[h]"/"total[h]")"
}
% awk -f success.awk mylog
13 66.6667% (2/3)
17 50% (1/2)
%