I found some strange call in my website access.log
This call come from where ? And How can i block it ?
> 40.77.167.156 - - [06/Jan/2019:14:18:16 +0100] "GET /nouvelles/news-k-1/17451-fight-card-glory-world-series-avec-semmy-schilt-vs-brice-guidon
> HTTP/1.1" 404 4060 "-" "Mozilla/5.0 (compatible; bingbot/2.0;
> +http://www.bing.com/bingbot.htm)
Related
I get this error when trying to get ALB logs:
root#b75651fde30e:/apps/tekton/deployment# kubectl logs -f ingress/tekton-dashboard-alb-dev
error: cannot get the logs from *v1.Ingress: selector for *v1.Ingress not implemented
The load balancer YAML:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: tekton-dashboard-alb-dev
namespace: tekton-pipelines
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/load-balancer-name: tekton-dashboard-alb-dev
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/backend-protocol: HTTP
alb.ingress.kubernetes.io/tags: "Cost=SwiftALK,VantaOnwer=foo#bar.com,VantaNonProd=true,VantaDescription=ALB Ingress for Tekton Dashboard,VantaContainsUserData=false,VantaUserDataStored=None"
alb.ingress.kubernetes.io/security-groups: sg-034ca9846b81fd721
kubectl.kubernetes.io/last-applied-configuration: ""
spec:
defaultBackend:
service:
name: tekton-dashboard
port:
number: 9097
Note: sg-034ca9846b81fd721 restricts access to our VPN CIDRs
Ingress is up as revealed from:
root#b75651fde30e:/apps/tekton/deployment# kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
tekton-dashboard-alb-dev <none> * tekton-dashboard-alb-dev-81361211.us-east-1.elb.amazonaws.com 80 103m
root#b75651fde30e:/apps/tekton/deployment# kubectl describe ingress/tekton-dashboard-alb-dev
Name: tekton-dashboard-alb-dev
Namespace: tekton-pipelines
Address: tekton-dashboard-alb-dev-81361211.us-east-1.elb.amazonaws.com
Default backend: tekton-dashboard:9097 (172.18.5.248:9097)
Rules:
Host Path Backends
---- ---- --------
* * tekton-dashboard:9097 (172.18.5.248:9097)
Annotations: alb.ingress.kubernetes.io/backend-protocol: HTTP
alb.ingress.kubernetes.io/load-balancer-name: tekton-dashboard-alb-dev
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/security-groups: sg-034ca9846b81fd721
alb.ingress.kubernetes.io/tags:
Cost=SwiftALK,VantaOnwer=swiftalkdevteam#digite.com,VantaNonProd=true,VantaDescription=ALB Ingress for SwifTalk Web Microservices,VantaCon...
alb.ingress.kubernetes.io/target-type: ip
kubernetes.io/ingress.class: alb
Events: <none>
The error you received means that the logs for your object are not implemented. It looks like you're trying to get logs from the wrong place.
I am not able to reproduce your problem on AWS, but I tried to do it on GCP and the situation was very similar. You cannot get logs from ingress/tekton-dashboard-alb-dev, and this is normal bahaviour. If you want to get logs of your ALB, you have to find the appropriate pod and then extract the logs from it. Let me show you how I did it on GCP. The commands are the same, but the pod names will be different.
First I have executed:
kubectl get pods --all-namespaces
Output:
NAMESPACE NAME READY STATUS RESTARTS AGE
ingress-nginx ingress-nginx-controller-57cb5bf694-722ml 1/1 Running 0 18d
-----
and many other not related pods in other namespaces
You can find directly your pod with command:
kubectl get pods -n ingress-nginx
Output:
NAME READY STATUS RESTARTS AGE
ingress-nginx-controller-57cb5bf694-722ml 1/1 Running 0 18d
Now you can get logs from ingress controller by command:
kubectl logs -n ingress-nginx ingress-nginx-controller-57cb5bf694-722ml
in your situation:
kubectl logs -n <your namespace> <your ingress controller pod>
The output should be similar to this:
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: v0.46.0
Build: 6348dde672588d5495f70ec77257c230dc8da134
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.19.6
-------------------------------------------------------------------------------
I0923 05:26:20.053561 8 flags.go:208] "Watching for Ingress" class="nginx"
W0923 05:26:20.053753 8 flags.go:213] Ingresses with an empty class will also be processed by this Ingress controller
W0923 05:26:20.054185 8 client_config.go:614] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0923 05:26:20.054502 8 main.go:241] "Creating API client" host="https://10.16.0.1:443"
I0923 05:26:20.069482 8 main.go:285] "Running in Kubernetes cluster" major="1" minor="20+" git="v1.20.9-gke.1001" state="clean" commit="1fe18c314ed577f6047d2712a9d1c8e498e22381" platform="linux/amd64"
I0923 05:26:20.842645 8 main.go:105] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0923 05:26:20.846132 8 main.go:115] "Enabling new Ingress features available since Kubernetes v1.18"
W0923 05:26:20.849470 8 main.go:127] No IngressClass resource with name nginx found. Only annotation will be used.
I0923 05:26:20.866252 8 ssl.go:532] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0923 05:26:20.917594 8 nginx.go:254] "Starting NGINX Ingress controller"
I0923 05:26:20.942084 8 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"42dc476e-3c5c-4cc9-a6a4-266edecb2a4b", APIVersion:"v1", ResourceVersion:"5600", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I0923 05:26:22.118459 8 nginx.go:296] "Starting NGINX process"
I0923 05:26:22.118657 8 leaderelection.go:243] attempting to acquire leader lease ingress-nginx/ingress-controller-leader-nginx...
I0923 05:26:22.119481 8 nginx.go:316] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I0923 05:26:22.120266 8 controller.go:146] "Configuration changes detected, backend reload required"
I0923 05:26:22.126350 8 status.go:84] "New leader elected" identity="ingress-nginx-controller-57cb5bf694-8c9tn"
I0923 05:26:22.214194 8 controller.go:163] "Backend successfully reloaded"
I0923 05:26:22.214838 8 controller.go:174] "Initial sync, sleeping for 1 second"
I0923 05:26:22.215234 8 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-57cb5bf694-722ml", UID:"b9672f3c-ecdf-473e-80f5-529bbc5bc4e5", APIVersion:"v1", ResourceVersion:"59016530", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0923 05:27:00.596169 8 leaderelection.go:253] successfully acquired lease ingress-nginx/ingress-controller-leader-nginx
I0923 05:27:00.596305 8 status.go:84] "New leader elected" identity="ingress-nginx-controller-57cb5bf694-722ml"
157.230.143.29 - - [23/Sep/2021:08:28:25 +0000] "GET / HTTP/1.1" 400 248 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:54.0) Gecko/20100101 Firefox/70.0" 165 0.000 [] [] - - - - d47be1e37ea504aca93d59acc7d36a2b
157.230.143.29 - - [23/Sep/2021:08:28:26 +0000] "\x00\xFFK\x00\x00\x00\xE2\x00 \x00\x00\x00\x0E2O\xAAC\xE92g\xC2W'\x17+\x1D\xD9\xC1\xF3,kN\x17\x14" 400 150 "-" "-" 0 0.076 [] [] - - - - c497187f4945f8e9e7fa84d503198e85
157.230.143.29 - - [23/Sep/2021:08:28:26 +0000] "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00" 400 150 "-" "-" 0 0.138 [] [] - - - - 4067a2d34d0c1f2db7ffbfc143540c1a
167.71.216.70 - - [23/Sep/2021:12:02:23 +0000] "\x16\x03\x01\x01\xFC\x01\x00\x01\xF8\x03\x03\xDB\xBBo*K\xAE\x9A&\x8A\x9B)\x1B\xB8\xED3\xB7\xE16N\xEA\xFCS\x22\x14V\xF7}\xC8&ga\xDA\x00\x01<\xCC\x14\xCC\x13\xCC\x15\xC00\xC0,\xC0(\xC0$\xC0\x14\xC0" 400 150 "-" "-" 0 0.300 [] [] - - - - ff6908bb17b0da020331416773b928b5
167.71.216.70 - - [23/Sep/2021:12:02:23 +0000] "\x16\x03\x01\x01\xFC\x01\x00\x01\xF8\x03\x03a\xBF\xFB\xC1'\x03S\x83D\x5Cn$\xAB\xE1\xA6%\x93G-}\xD1C\xB2\xB0E\x8C\x8F\xA8q-\xF7$\x00\x01<\xCC\x14\xCC\x13\xCC\x15\xC00\xC0,\xC0(\xC0$\xC0\x14\xC0" 400 150 "-" "-" 0 0.307 [] [] - - - - fee3a478240e630e6983c60d1d510f52
66.240.205.34 - - [23/Sep/2021:12:04:11 +0000] "145.ll|'|'|SGFjS2VkX0Q0OTkwNjI3|'|'|WIN-JNAPIER0859|'|'|JNapier|'|'|19-02-01|'|'||'|'|Win 7 Professional SP1 x64|'|'|No|'|'|0.7d|'|'|..|'|'|AA==|'|'|112.inf|'|'|SGFjS2VkDQoxOTIuMTY4LjkyLjIyMjo1NTUyDQpEZXNrdG9wDQpjbGllbnRhLmV4ZQ0KRmFsc2UNCkZhbHNlDQpUcnVlDQpGYWxzZQ==12.act|'|'|AA==" 400 150 "-" "-" 0 0.086 [] [] - - - - 365d42d67e7378359b95c71a8d8ce983
147.182.148.98 - - [23/Sep/2021:12:04:17 +0000] "\x16\x03\x01\x01\xFC\x01\x00\x01\xF8\x03\x03\xABA\xF4\xD5\xB7\x95\x85[.v\xDB\xD1\x1B\x04\xE7\xB4\xB8\x92\x82\xEC\xCC\xDDr\xB7/\xBD\x93/\xD0f4\xB3\x00\x01<\xCC\x14\xCC\x13\xCC\x15\xC00\xC0,\xC0(\xC0$\xC0\x14\xC0" 400 150 "-" "-" 0 0.152 [] [] - - - - 858c2ad7535de95c84dd0899708a3801
164.90.203.66 - - [23/Sep/2021:12:08:19 +0000] "\x16\x03\x01\x01\xFC\x01\x00\x01\xF8\x03\x03\x93\x81+_\x95\xFA\xEAj\xA7\x80\x15 \x179\xD7\x92\xAE\xA9i+\x9D`\xA07:\xD2\x22\xB3\xC6\xF3\x22G\x00\x01<\xCC\x14\xCC\x13\xCC\x15\xC00\xC0,\xC0(\xC0$\xC0\x14\xC0" 400 150 "-" "-" 0 0.237 [] [] - - - - 799487dd8ec874532dcfa7dad1c02a27
164.90.203.66 - - [23/Sep/2021:12:08:20 +0000] "\x16\x03\x01\x01\xFC\x01\x00\x01\xF8\x03\x03\xB8\x22\xCB>1\xBEM\xD4\x92\x95\xEF\x1C0\xB5&\x1E[\xC5\xC8\x1E2\x07\x1C\x02\xA1<\xD2\xAA\x91F\x00\xC6\x00\x01<\xCC\x14\xCC\x13\xCC\x15\xC00\xC0,\xC0(\xC0$\xC0\x14\xC0" 400 150 "-" "-" 0 0.193 [] [] - - - - 4604513713d4b9fb5a7199b7980fa7f2
164.90.203.66 - - [23/Sep/2021:12:16:10 +0000] "\x16\x03\x01\x01\xFC\x01\x00\x01\xF8\x03\x03[\x16\x02\x94\x98\x17\xCA\xB5!\xC11#\x08\xD9\x89RE\x970\xC2\xDF\xFF\xEBh\xA0i\x9Ee%.\x07{\x00\x01<\xCC\x14\xCC\x13\xCC\x15\xC00\xC0,\xC0(\xC0$\xC0\x14\xC0" 400 150 "-" "-" 0 0.116 [] [] - - - - 23019f0886a1c30a78092753f6828e74
77.247.108.81 - - [23/Sep/2021:14:52:51 +0000] "GET /admin/config.php HTTP/1.1" 400 248 "-" "python-requests/2.26.0" 164 0.000 [] [] - - - - 04630dbf3d0ff4a4b7138dbc899080e5
209.141.48.211 - - [23/Sep/2021:16:17:46 +0000] "" 400 0 "-" "-" 0 0.057 [] [] - - - - 3c623b242909a99e18178ec10a814d7b
209.141.62.185 - - [23/Sep/2021:18:13:11 +0000] "GET /config/getuser?index=0 HTTP/1.1" 400 248 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:76.0) Gecko/20100101 Firefox/76.0" 353 0.000 [] [] - - - - 2640cf06912615a7600e814dc893884b
125.64.94.138 - - [23/Sep/2021:19:49:08 +0000] "GET / HTTP/1.0" 400 248 "-" "-" 18 0.000 [] [] - - - - b633636176888bc3b7f6230f691e0724
2021/09/23 19:49:20 [crit] 39#39: *424525 SSL_do_handshake() failed (SSL: error:141CF06C:SSL routines:tls_parse_ctos_key_share:bad key share) while SSL handshaking, client: 125.64.94.138, server: 0.0.0.0:443
125.64.94.138 - - [23/Sep/2021:19:49:21 +0000] "GET /favicon.ico HTTP/1.1" 400 650 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4 240.111 Safari/537.36" 197 0.000 [] [] - - - - ede08c8fb12e8ebaf3adcbd2b7ea5fd5
125.64.94.138 - - [23/Sep/2021:19:49:22 +0000] "GET /robots.txt HTTP/1.1" 400 650 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4 240.111 Safari/537.36" 196 0.000 [] [] - - - - fae50b56a11600abc84078106ba4b008
125.64.94.138 - - [23/Sep/2021:19:49:22 +0000] "GET /.well-known/security.txt HTTP/1.1" 400 650 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4 240.111 Safari/537.36" 210 0.000 [] [] - - - - ad82bcac7d7d6cd9aa2d044d80bb719d
87.251.75.145 - - [23/Sep/2021:21:29:10 +0000] "\x03\x00\x00/*\xE0\x00\x00\x00\x00\x00Cookie: mstshash=Administr" 400 150 "-" "-" 0 0.180 [] [] - - - - 8c2b62bcdf26ac1592202d0940fc30b8
167.71.102.181 - - [23/Sep/2021:21:54:58 +0000] "\x00\x0E8K\xA3\xAAe\xBCn\x14\x1B\x00\x00\x00\x00\x00" 400 150 "-" "-" 0 0.027 [] [] - - - - 65b8ee37a2c6bf8368843e4db3b90b2a
185.156.72.27 - - [23/Sep/2021:22:03:55 +0000] "\x03\x00\x00/*\xE0\x00\x00\x00\x00\x00Cookie: mstshash=Administr" 400 150 "-" "-" 0 0.139 [] [] - - - - 92c6ad2d71b961bf7de4e345ff69da10
185.156.72.27 - - [23/Sep/2021:22:03:55 +0000] "\x03\x00\x00/*\xE0\x00\x00\x00\x00\x00Cookie: mstshash=Administr" 400 150 "-" "-" 0 0.140 [] [] - - - - fe0424f8ecf9afc1d0154bbca2382d13
34.86.35.21 - - [23/Sep/2021:22:54:41 +0000] "\x16\x03\x01\x00\xE3\x01\x00\x00\xDF\x03\x03\x0F[\xA9\x18\x15\xD3#4\x7F\x7F\x98'\xA9(\x8F\xE7\xCCDd\xF9\xFF`\xE3\xCE\x9At\x05\x97\x05\xB1\xC3}\x00\x00h\xCC\x14\xCC\x13\xC0/\xC0+\xC00\xC0,\xC0\x11\xC0\x07\xC0'\xC0#\xC0\x13\xC0\x09\xC0(\xC0$\xC0\x14\xC0" 400 150 "-" "-" 0 2.039 [] [] - - - - c09d38bf2cd925dac4d9e5d5cb843ece
2021/09/24 02:41:15 [crit] 40#40: *627091 SSL_do_handshake() failed (SSL: error:141CF06C:SSL routines:tls_parse_ctos_key_share:bad key share) while SSL handshaking, client: 184.105.247.252, server: 0.0.0.0:443
61.219.11.151 - - [24/Sep/2021:03:40:51 +0000] "dN\x93\xB9\xE6\xBCl\xB6\x92\x84:\xD7\x03\xF1N\xB9\xC5;\x90\xC2\xC6\xBA\xE1I-\x22\xDDs\xBA\x1FgC:\xB1\xA7\x80+\x00\x00\x00\x00%\xFDK:\xAAW.|J\xB2\xB5\xF5'\xA5l\xD3V(\xB7\x01%(CsK8B\xCE\x9A\xD0z\xC7\x13\xAD" 400 150 "-" "-" 0 0.203 [] [] - - - - 190d00221eefc869b5938ab6380f835a
46.101.155.106 - - [24/Sep/2021:04:56:37 +0000] "HEAD / HTTP/1.0" 400 0 "-" "-" 17 0.000 [] [] - - - - e8c108201c37d7457e4578cf68feacf8
46.101.155.106 - - [24/Sep/2021:04:56:38 +0000] "GET /system_api.php HTTP/1.1" 400 650 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36" 255 0.000 [] [] - - - - b3032f9a9b3f4f367bdee6692daeb05c
46.101.155.106 - - [24/Sep/2021:04:56:39 +0000] "GET /c/version.js HTTP/1.1" 400 650 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36" 253 0.000 [] [] - - - - 9104ab72a0232caf6ff98da57d325144
46.101.155.106 - - [24/Sep/2021:04:56:40 +0000] "GET /streaming/clients_live.php HTTP/1.1" 400 650 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36" 267 0.000 [] [] - - - - 341cbb6cf424b348bf8b788f79373b8d
46.101.155.106 - - [24/Sep/2021:04:56:41 +0000] "GET /stalker_portal/c/version.js HTTP/1.1" 400 650 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36" 268 0.000 [] [] - - - - 9954fd805fa092595057dbf83511bd92
46.101.155.106 - - [24/Sep/2021:04:56:42 +0000] "GET /stream/live.php HTTP/1.1" 400 248 "-" "AlexaMediaPlayer/2.1.4676.0 (Linux;Android 5.1.1) ExoPlayerLib/1.5.9" 209 0.000 [] [] - - - - 3c9409419c1ec59dfc08c10cc3eb6eef
It is as the message says: there is no such thing as "logs" for an Ingress resource. Specifically, what it is telling you is that running kubectl -n tekton-pipelines get pods -l $(kubectl -n tekton-pipelines get -o jsonpath={.metadata.labels} ingress/tekton-dashboard-alb-dev) correctly produced no Pods from which one could gather logs
Perhaps you actually wanted the logs for the Pods behind service/tekton-dashboard but asking for the logs of an Ingress is nonsense
We have configured the istio 1.4.0 with demo profile on Kubernetes cluster 1.15.1. It was working as expected but after some time facing issues with the application which are connecting to backend servers like mongo DB. Application pod is going in crashloopbackup and if i disabled istio it works properly.
Upon checking the istio-proxy logs found lines stating http/1.1 DPE and mongo DB IP and port number
Below is the Istio-proxy logs (sidecar),
#
[2020-03-11T13:40:28.504Z] "- - HTTP/1.1" 0 DPE "-" "-" 0 0 0 - "-" "-" "-" "-" "-" - - <mongo IP>:27017 10.233.92.103:49412 - -
[2020-03-11T13:40:28.508Z] "- - HTTP/1.1" 0 DPE "-" "-" 0 0 0 - "-" "-" "-" "-" "-" - - <mongo IP>:27017 10.233.92.103:52062 - -
[2020-03-11T13:40:28.528Z] "- - HTTP/1.1" 0 DPE "-" "-" 0 0 0 - "-" "-" "-" "-" "-" - - <mongo IP>:27017 10.233.92.103:37182 - -
[2020-03-11T13:40:28.529Z] "- - HTTP/1.1" 0 DPE "-" "-" 0 0 0 - "-" "-" "-" "-" "-" - - <mongo IP>:27017 10.233.92.103:49428 - -
[2020-03-11T13:40:28.530Z] "- - HTTP/1.1" 0 DPE "-" "-" 0 0 0 - "-" "-" "-" "-" "-" - - 10.26.61.18:27017 10.233.92.103:52078 - -
[2020-03-11T13:40:28.569Z] "POST /intake/v2/events HTTP/1.1" 202 - "-" "-" 941 0 3 1 "-" "elasticapm-node/3.3.0 elastic-apm-http-client/9.3.0 node/10.12.0" "8954f0a1-709b-963c-a480-05b078955c89" "<apm>:8200" "10.26.61.45:8200" PassthroughCluster - <apm>:8200 10.233.92.103:49992 - -
[2020-03-11T13:40:28.486Z] "- - -" 0 - "-" "-" 47 3671 98 - "-" "-" "-" "-" "<redis>:6379" PassthroughCluster 10.233.92.103:37254 <redis>:6379 10.233.92.103:37252 - -
[2020-03-11T13:40:30.168Z] "- - -" 0 - "-" "-" 632 1212236 104 - "-" "-" "-" "-" "104.16.25.35:443" PassthroughCluster 10.233.92.103:60760 104.16.25.35:443 10.233.92.103:60758 - -```
#
and application logs giving error as below
{ err: 'socketHandler', trace: '', bin: undefined, parseState: { sizeOfMessage: 1347703880, bytesRead: undefined, stubBuffer: undefined } }
The issue has been resolved.
RCA :-
I have manually created service and endpoint of MongoDB with port name as http
After that when i checked the listeners in proxy-config via istioctl command i found an entry with address 0.0.0.0 and port 27017
ADDRESS PORT TYPE
0.0.0.0 27017 TCP
In json output, i interpret that its going into blackholecluster even if i set allow_any in passthroughcluster.
The output in istio-proxy always give me DPE error.
After understanding the issue i changed the name from http to http1 and it worked properly.
Now need to understand why the name http was creating so much issue
Might be this issue: https://github.com/kubernetes/enhancements/issues/753
Basically, about ordering of containers. At startup time if your application container required networking and sidecar container(envoy proxy) not fully started, then application will raise an networking error.
Reference: https://discuss.istio.io/t/k8s-istio-sidecar-injection-with-other-init-containers/845
I have 3 .txt files in the following form:
1.
109.169.248.247 - - [12/Dec/2015:18:25:11 +0100] "GET /administrator/ HTTP/1.1" 200 4263 "-" "Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0" "-"
2.
081116 203518 143 INFO dfs.DataNode$DataXceiver: Receiving block blk_-1608999687919862906 src: /10.250.19.102:54106 dest: /10.250.19.102:50010
3.
081116 203521 19 INFO dfs.FSNamesystem: BLOCK* ask 10.250.14.224:50010 to replicate blk_-1608999687919862906 to datanode(s) 10.251.215.16:50010 10.251.71.193:50010
How do i import them into tables? manually is not an option since the size of .txt is over 1 MB each.
I have access log file with data only per 1 day like:
10.2.21.120 "-" - [26/Jan/2013:19:15:11 +0000] "GET /server/ad?uid=abc&type=PST HTTP/1.1" 202 14 10
10.2.21.120 "-" - [26/Jan/2013:19:17:22 +0000] "GET /server/ad?uid=abc&type=PST HTTP/1.1" 204 14 9
10.2.22.130 "-" - [26/Jan/2013:19:27:53 +0000] "GET /server/ad?uid=abc&type=PST HTTP/1.1" 200 14 8
I am using the following command:
awk '$9 == 200 { s++ } END { print s / NR * 100; }' access.log
This awk may help you
$ awk -F[:\ ] '{count[$5]++}; $12 == 200 { hour[$5]++} END { for (i in hour) print i, hour[i]/count[i]*100 }' input
Test
$ cat input
10.1.20.123 "1.1.1.1" - [15/Oct/2014:12:14:17 +0000] "POST /server/ad?uid=abc&type=PST HTTP/1.1" 200 3014 10
10.1.20.123 "1.1.1.1" - [15/Oct/2014:12:14:17 +0000] "POST /server/ad?uid=abc&type=PST HTTP/1.1" 100 3014 10
10.1.20.123 "1.1.1.1" - [15/Oct/2014:13:14:26 +0000] "POST /server/ad?uid=abc&type=PST HTTP/1.1" 200 3014 9
10.1.20.123 "1.1.1.1" - [15/Oct/2014:13:24:55 +0000] "POST /server/ad?uid=abc&type=PST HTTP/1.1" 200 3014 8
$ awk -F[:\ ] '{count[$5]++}; $12 == 200 { hour[$5]++} END { for (i in hour) print i, hour[i]/count[i]*100 }' input
12 50
13 100
What it does
{count[$5]++} array count stores the number of occurence of each hour from the log file.
$12 == 200 { hour[$5]++} Now it the log is success, that is $12 == 200 then the corresponding value in hour array is incremented.
So count[13] will contain total enteries from hour 13 where as hour[13] would contain count of succuessfull entries
END { for (i in hour) print i, hour[i]/count[i]*100 } prints the hour, percentage
I would use something like this:
awk -F"[ :]" '
{tot[$5]++; if ($(NF-2)==200) succ[$5]++}
END {for (i in tot) printf "%d %d/%d %.2f%\n", i, succ[i], tot[i], succ[i]/tot[i]*100}' file
This defines : or space as field separators. Hence, the hour is stored in the 5th field and the return code in the one before the penultimate (NF-2) - we could also use the 12th as you do, but this allows support for longest input in the log.
Then, it keeps track of the amount of lines in the tot[] array and the successful ones in succ[] array.
Finally it prints the result
Here we have some more data with different hours and return codes:
$ cat a
10.1.20.123 "1.1.1.1" - [15/Oct/2014:13:14:17 +0000] "POST /server/ad?uid=abc&type=PST HTTP/1.1" 200 3014 10
10.1.20.123 "1.1.1.1" - [15/Oct/2014:13:14:26 +0000] "POST /server/ad?uid=abc&type=PST HTTP/1.1" 200 3014 9
10.1.20.123 "1.1.1.1" - [15/Oct/2014:13:24:55 +0000] "POST /server/ad?uid=abc&type=PST HTTP/1.1" 100 3014 8
10.1.20.123 "1.1.1.1" - [15/Oct/2014:17:24:55 +0000] "POST /server/ad?uid=abc&type=PST HTTP/1.1" 200 3014 8
10.1.20.123 "1.1.1.1" - [15/Oct/2014:17:24:55 +0000] "POST /server/ad?uid=abc&type=PST HTTP/1.1" 404 3014 8
Let's run the script:
$ awk -F"[ :]" '{tot[$5]++; if ($(NF-2)==200) succ[$5]++} END {for (i in tot) printf "%d %d/%d %.2f%\n", i, succ[i], tot[i], succ[i]/tot[i]*100}' a
13 2/3 66.67%
17 1/2 50.00%
% cat success.awk
NF==11{
split($4,a,":") ; hour = a[2]
total[hour] += 1
if($9==200) success[hour] += 1
}
END{
for(h in total) print h, 100*(0+success[h])/total[h]"% ("0+success[h]"/"total[h]")"
}
% awk -f success.awk mylog
13 66.6667% (2/3)
17 50% (1/2)
%
I am working on an assignment where i need to parse a log file and create a website based on said log file. one of the requirements is that i count the number of hits that happened on yesterdays, im lost when it comes to this ive attached my code and the log file im working with hoping that someone can offer some advice, thanks
#!/usr/bin/perl
use strict;
use warnings;
use Time::Piece;
use Time::Seconds qw(ONE_DAY);
my $yesterday = localtime() - ONE_DAY();
print $yesterday;
open(LOGFILE,"<", "access.log")or die"Could not open log file.";
my $yesterdayHits=0;
my $totalhits=0;
my $webPage='log.html';
open(WEBPAGE,">",$webPage);
print WEBPAGE ("<HEAD><TITLE>Access Counts</TITLE></HEAD>");
print WEBPAGE ("<BODY>");
print WEBPAGE ("<H1> today is: ",scalar(localtime), "</H1>");
print WEBPAGE ("<h3>Yesterday was $yesterday</h3>");
print WEBPAGE ("<TABLE BORDER=1 CELLPADDING=10 width='500px'>");
foreach my $line (<LOGFILE>) {
$totalhits++;
my $w = "(.+?)";
$line =~ m/^$w $w $w \[$w:$w $w\] "$w $w $w" $w $w/;
my $site = $1;
my $logName = $2;
my $fullName = $3;
my $date = $4;
my $time = $5;
my $gmt = $6;
my $req = $7;
my $file = $8;
my $proto = $9;
my $status = $10;
my $length = $11;
#if($line =~ m/$yesterday/){$yesterdayHits++}
print WEBPAGE ("<Tr><TD>$site</TD><TD>$line</TD></Tr>\n\n");
}
close(LOGFILE);
print WEBPAGE ("<h2>Total hits: $totalhits</h2>");
print WEBPAGE ("<h3>Hits Yesterday: $yesterdayHits</h3>");
print WEBPAGE ("</TABLE></P>");
print WEBPAGE ("</BODY></HTML>");
close(WEBPAGE);
Access log
66.249.65.107 - - [11/Nov/2012:19:33:01 -0400] "GET /support.html HTTP/1.1" 200 11179
111.111.111.111 - - [11/Nov/2012:19:33:01 -0400] "GET / HTTP/1.1" 200 10801
111.111.111.111 - - [08/Oct/2007:11:17:55 -0400] "GET /style.css HTTP/1.1" 200 3225
123.123.123.123 - - [26/Apr/2000:00:23:48 -0400] "GET /pics/wpaper.gif HTTP/1.0" 200 6248
123.123.123.123 - - [26/Apr/2000:00:23:40 -0400] "GET /asctortf/ HTTP/1.0" 200 8130
123.123.123.123 - - [26/Apr/2000:00:23:48 -0400] "GET /pics/5star2000.gif HTTP/1.0" 200 4005
123.123.123.123 - - [26/Apr/2000:00:23:50 -0400] "GET /pics/5star.gif HTTP/1.0" 200 1031
123.123.123.123 - - [26/Apr/2000:00:23:51 -0400] "GET /pics/a2hlogo.jpg HTTP/1.0" 200 4282
123.123.123.123 - - [26/Apr/2000:00:23:51 -0400] "GET /cgi-bin/newcount?jafsof3&width=4&font=digital&noshow HTTP/1.0" 200 36
172.16.130.42 - - [26/Apr/2000:00:00:12 -0400] "GET /contacts.html HTTP/1.0" 200 4595
10.0.1.3 - - [26/Apr/2000:00:17:19 -0400] "GET /news/news.html HTTP/1.0" 200 16716
129.21.109.81 - - [26/Apr/2000:00:16:12 -0400] "GET /download/windows/asctab31.zip HTTP/1.0" 200 1540096
192.168.198.92 - - [22/Dec/2002:23:08:37 -0400] "GET / HTTP/1.1" 200 6394
192.168.198.92 - - [22/Dec/2002:23:08:38 -0400] "GET /images/logo.gif HTTP/1.1" 200 807
192.168.72.177 - - [22/Dec/2002:23:32:14 -0400] "GET /news/sports.html HTTP/1.1" 200 3500
192.168.72.177 - - [22/Dec/2002:23:32:14 -0400] "GET /favicon.ico HTTP/1.1" 404 1997
192.168.72.177 - - [04/Nov/2012:23:32:15 -0400] "GET /style.css HTTP/1.1" 200 4138
192.168.72.177 - - [22/Dec/2002:23:32:16 -0400] "GET /js/ads.js HTTP/1.1" 200 10229
192.168.72.177 - - [22/Dec/2002:23:32:19 -0400] "GET /search.php HTTP/1.1" 400 1997
127.0.0.1 - - [10/Apr/2007:10:39:11 +0300] "GET / HTTP/1.1" 500 606
127.0.0.1 - - [10/Apr/2007:10:39:11 +0300] "GET /favicon.ico HTTP/1.1" 200 766
139.12.0.2 - - [10/Apr/2007:10:40:54 +0300] "GET / HTTP/1.1" 500 612
139.12.0.2 - - [10/Apr/2007:10:40:54 +0300] "GET /favicon.ico HTTP/1.1" 200 766
127.0.0.1 - - [10/Apr/2007:10:53:10 +0300] "GET / HTTP/1.1" 500 612
127.0.0.1 - - [10/Apr/2007:10:54:08 +0300] "GET / HTTP/1.0" 200 3700
127.0.0.1 - - [10/Apr/2007:10:54:08 +0300] "GET /style.css HTTP/1.1" 200 614
127.0.0.1 - - [10/Apr/2007:10:54:08 +0300] "GET /img/pti-round.jpg HTTP/1.1" 200 17524
127.0.0.1 - - [10/Apr/2007:10:54:21 +0300] "GET /unix_sysadmin.html HTTP/1.1" 200 3880
217.0.22.3 - - [04/Nov/2012:10:54:51 +0300] "GET / HTTP/1.1" 200 34
217.0.22.3 - - [10/Apr/2007:10:54:51 +0300] "GET /favicon.ico HTTP/1.1" 200 11514
217.0.22.3 - - [10/Apr/2007:10:54:53 +0300] "GET /cgi/pti.pl HTTP/1.1" 500 617
127.0.0.1 - - [10/Apr/2007:10:54:08 +0300] "GET / HTTP/0.9" 200 3700
217.0.22.3 - - [10/Apr/2007:10:58:27 +0300] "GET / HTTP/1.1" 200 3700
217.0.22.3 - - [10/Apr/2007:10:58:34 +0300] "GET /unix_sysadmin.html HTTP/1.1" 200 3880
217.0.22.3 - - [10/Apr/2007:10:58:45 +0300] "GET /talks/Fundamentals/read-excel-file.html HTTP/1.1" 404 311
use POSIX;
$yesterday = strftime("%d/%b/%Y",localtime(time()-86400));
$yesterday now contains yesterday's date in the logfile's format (e.g. "11/Nov/2012"). You can filter lines by checking $line =~ /$yesterday/;
http://perldoc.perl.org/POSIX.html
A basic approach is to modify the $yesterday variable (to be the same as logfile) like this :
$yesterday =~ s!\w+\s+(\w+)\s+(\d+)\s+\d{2}:\d{2}:\d{2}\s+(\d+)!$2/$1/$3!;
now you can de-comment and change the line
if ($date eq $yesterday) { $yesterdayHits++ }
to start counting.
Not-so-fast but quite precise version; uses UNIX timestamps:
#!/usr/bin/env perl
use strict;
use warnings 'all';
use HTTP::Date;
while (<>) {
my $stamp;
print if
m{\s+\[(.+?)\]\s+\"}x
and $stamp = str2time($1)
and $stamp > time - 86_400 * 2
and $stamp < time - 86_400;
}