I just tried to install Linkerd viz extension following official documentation but all the pods are in a crash loop.
linkerd viz install | kubectl apply -f -
Linkerd-getting-started
r proxy-admin
[ 29.797889s] INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}: linkerd_app_inbound::policy::authorize::http: Request denied server=proxy-admin tls=None(NoClientHello) client=50.50.55.177:47068
[ 29.797910s] INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:rescue{client.addr=50.50.55.177:47068}: linkerd_app_core::errors::respond: Request failed error=unauthorized connection on server proxy-admin
[ 29.817790s] INFO ThreadId(01) linkerd_proxy::signal: received SIGTERM, starting shutdown
The error appeared on Kubernetes cluster Server Version: v1.21.5-eks-bc4871b
The issue was the policy that come by default installation.
This authorizes unauthenticated requests from IPs in the clusterNetworks configuration. If the source IP (<public-ip-address-of-hel-k1>) is not in that list, these connections will be denied. To fix this, the authorization policy could be updated with the following:
spec:
client:
unauthenticated: true
networks:
- cidr: 0.0.0.0/0
The default policy missing the client part
networks:
- cidr: 0.0.0.0/0
To update the policy, get the server authorization
k get ServerAuthorization -n linkerd-viz
NAME SERVER
admin admin
grafana grafana
metrics-api metrics-api
proxy-admin proxy-admin
Now edit admin,grafana, proxy-admin and grafana and add the networks part.
k edit ServerAuthorization metrics-api
as after fixing this I was also getting errors for grafana which help me to fix by adding network part.
[ 32.278014s] INFO ThreadId(01) inbound:server{port=3000}:rescue{client.addr=50.50.53.140:44718}: linkerd_app_core::errors::respond: Request failed error=unauthorized connection on server grafana
[ 38.176927s] INFO ThreadId(01) inbound:server{port=3000}: linkerd_app_inbound::policy::authorize::http: Request denied server=grafana tls=None(NoClientHello) client=50.50.55.177:33170
All linkerd-viz pods in CrashLoopBackOff
Related
getting error curl: (56) response reading failed while trying to send email via smtp using curl. checked the isto-proxy logs of sidecar but don't see any error logs related to this host. Tried the solution mentioned in How to access external SMTP server from within Kubernetes cluster with Istio Service Mesh as well but didn't work.
service entry
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: smtp
spec:
addresses:
- 192.168.8.45/32
hosts:
- smtp.example.com"
location: MESH_EXTERNAL
ports:
- name: tcp-smtp
number: 2255
protocol: TCP
Most probably port number is causing the error and if not, try deleting the mesh policies
Also please validate based on below points:
1.If you recently updated istio try downgrading it.
2.Look again in Sidecar logs for any conflicts or try disabling it.
3.When it comes to curl 56 error packet transmission; limit could be the problem.
The curl requests from primary container are routed via sidecar when istio is enabled, the response from smtp server is being masqueraded by sidecar and returned to primary container, which was quite misleading.
upon disabling Istio and trying to do curl on smtp port curl request failed with error Failed to connect to smtp.example.com port 2255: Operation timed out. which was because firewall from cluster to smtp server port was not open.
while istio was enabled the curl response didn't give timeout error but gave curl: (56) response reading failed which mislead me to think that the response was coming from smtp server.
I have Istio1.6 running in my k8 cluster. In the cluster I have also deployed sharded mongodb cluster with istio-injection disabled.
And I have a different namespace for my app with istio-injection enabled. And from the pod if I try to connect to the mongo I get this connection reset by peer error:
root#mongo:/# mongo "mongodb://mongo-sharded-cluster-mongos-0.mongo-service.mongodb.svc.cluster.local:27017,mongo-sharded-cluster-mongos-1.mongo-service.mongodb.svc.cluster.local:27017/?ssl=false"
MongoDB shell version v4.2.8
connecting to: mongodb://mongo-sharded-cluster-mongos-0.mongo-service.mongodb.svc.cluster.local:27017,mongo-sharded-cluster-mongos-1.mongo-service.mongodb.svc.cluster.local:27017/?compressors=disabled&gssapiServiceName=mongodb&ssl=false
2020-06-18T19:59:14.342+0000 I NETWORK [js] DBClientConnection failed to receive message from mongo-sharded-cluster-mongos-0.mongo-service.mongodb.svc.cluster.local:27017 - HostUnreachable: Connection reset by peer
2020-06-18T19:59:14.358+0000 I NETWORK [js] DBClientConnection failed to receive message from mongo-sharded-cluster-mongos-1.mongo-service.mongodb.svc.cluster.local:27017 - HostUnreachable: Connection reset by peer
2020-06-18T19:59:14.358+0000 E QUERY [js] Error: network error while attempting to run command 'isMaster' on host 'mongo-sharded-cluster-mongos-1.mongo-service.mongodb.svc.cluster.local:27017' :
connect#src/mongo/shell/mongo.js:341:17
#(connect):2:6
2020-06-18T19:59:14.362+0000 F - [main] exception: connect failed
2020-06-18T19:59:14.362+0000 E - [main] exiting with code 1
But if I disable the istio-injection to my app(pod) then I can successfully connect and use mongo as expected.
Is there a work around for this, I would like to have istio-proxy injected to my app/pod and use mongodb?
Injecting Databases with istio is complicated.
I would start with checking your mtls, if it´s STRICT, I would change it to permissive and check if it works. It´s well described here.
You see requests still succeed, except for those from the client that doesn’t have proxy, sleep.legacy, to the server with a proxy, httpbin.foo or httpbin.bar. This is expected because mutual TLS is now strictly required, but the workload without sidecar cannot comply.
Is there a work around for this, I would like to have istio-proxy injected to my app/pod and use mongodb?
If changing mtls won´t work, then in istio You can set up database without injecting and then add it to istio registry using ServiceEntry object so it would be able to communicate with the rest of istio services.
To add your mongodb database to istio you can use ServiceEntry.
ServiceEntry enables adding additional entries into Istio’s internal service registry, so that auto-discovered services in the mesh can access/route to these manually specified services. A service entry describes the properties of a service (DNS name, VIPs, ports, protocols, endpoints). These services could be external to the mesh (e.g., web APIs) or mesh-internal services that are not part of the platform’s service registry (e.g., a set of VMs talking to services in Kubernetes). In addition, the endpoints of a service entry can also be dynamically selected by using the workloadSelector field. These endpoints can be VM workloads declared using the WorkloadEntry object or Kubernetes pods. The ability to select both pods and VMs under a single service allows for migration of services from VMs to Kubernetes without having to change the existing DNS names associated with the services.
Example of ServiceEntry
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: external-svc-mongocluster
spec:
hosts:
- mymongodb.somedomain # not used
addresses:
- 192.192.192.192/24 # VIPs
ports:
- number: 27018
name: mongodb
protocol: MONGO
location: MESH_INTERNAL
resolution: STATIC
endpoints:
- address: 2.2.2.2
- address: 3.3.3.3
If You have mtls enabled You will also need DestinationRule that will define how to communicate with the external service.
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: mtls-mongocluster
spec:
host: mymongodb.somedomain
trafficPolicy:
tls:
mode: MUTUAL
clientCertificate: /etc/certs/myclientcert.pem
privateKey: /etc/certs/client_private_key.pem
caCertificates: /etc/certs/rootcacerts.pem
Additionally take a look at this documentation
https://istiobyexample.dev/databases/
https://istio.io/latest/blog/2018/egress-mongo/
I have been unable to reliably allow my istio enabled Google Kubernetes Engine cluster to connect to Google Cloud Endpoints (service management API) via the extensible service proxy. When I deploy my Pods the proxy will always fail to startup causing the Pod to be restarted, and output the following error:
INFO:Fetching an access token from the metadata service
WARNING:Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fea4abece90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /computeMetadata/v1/instance/service-accounts/default/token
ERROR:Failed fetching metadata attribute: http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token
However after restarting, the proxy reports everything is fine, it was able to grab an access token and I am able to make requests to the Pod successfully:
INFO:Fetching an access token from the metadata service
INFO:Fetching the service config ID from the rollouts service
INFO:Fetching the service configuration from the service management service
INFO:Attribute zone: europe-west2-a
INFO:Attribute project_id: my-project
INFO:Attribute kube_env: KUBE_ENV
nginx: [warn] Using trusted CA certificates file: /etc/nginx/trusted-ca-certificates.crt
10.154.0.5 - - [23/May/2020:21:19:36 +0000] "GET /domains HTTP/1.1" 200 221 "-" "curl/7.58.0"
After about an hour, presumably because the access token has expired, the proxy logs indicate that it was again unable to fetch an access token and I can no longer make requests to my Pod.
2020/05/23 22:14:04 [error] 9#9: upstream timed out (110: Connection timed out)
2020/05/23 22:14:04[error]9#9: Failed to fetch service account token
2020/05/23 22:14:04[error]9#9: Fetch access token unexpected status: INTERNAL: Failed to fetch service account token
I have in place a ServiceEntry resource that should be allowing the proxy to make requests to the metadata server on the GKE node:
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: google-metadata-server
spec:
hosts:
- metadata.google.internal # GCE metadata server
addresses:
- 169.254.169.254 # GCE metadata server
location: MESH_EXTERNAL
ports:
- name: http
number: 80
protocol: HTTP
- name: https
number: 443
protocol: HTTPS
I have confirmed this is working by execing into one of the containers and running:
curl -H "Metadata-Flavor: Google" http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token
How can I prevent this behaviour and reliably have the proxy communicate with the Google Service Infrastructure APIs?
Although I am not entirely convinced this is the solution it appears that using a dedicated service account to generate access tokens within the extensible service proxy container prevents the behaviour reported above from happening, and I am able to reliably make requests to the proxy and upstream service, even after an hour.
The service account I am using has the following roles:
roles/cloudtrace.agent
roles/servicemanagement.serviceController
Assuming this is a stable solution to the problem I am much happier with this as an outcome because I am not 100% comfortable using the metadata server since it relies on the service account associated with the GKE node. This service account is often more powerful that it needs to be for ESP to do its job.
I will however be continuing to monitor this just in case the proxy upstream becomes unreachable again.
I have deployed metric server using :
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.6/components.yaml
Metric server pod is running but in logs I am getting error :
I0601 20:00:01.321004 1 log.go:172] http: TLS handshake error from 20.99.219.64:34903: EOF
I0601 20:00:01.321160 1 log.go:172] http: TLS handshake error from 20.99.219.64:22575: EOF
I0601 20:00:01.332318 1 log.go:172] http: TLS handshake error from 20.99.219.64:14603: EOF
I0601 20:00:01.333174 1 log.go:172] http: TLS handshake error from 20.99.219.64:22517: EOF
I0601 20:00:01.351649 1 log.go:172] http: TLS handshake error from 20.99.219.64:3598: EOF
IP : 20.99.219.64
This is not present in Cluster. I have checked using :
kubectl get all --all-namespaces -o wide | grep "20.99.219.64"
Nothing is coming as O/P.
I have using Calico and initialize the cluster with --pod-network-cidr=20.96.0.0/12
Also kubectl top nodes is not working, Getting error :
node#kubemaster:~/Desktop/dashboard$ kubectl top nodes
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)
During deployment of metrics-server remember to add following line in args section:
- args:
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
Also add the following line at the spec:spec level outside of the previous containers level:
hostNetwork: true
restartPolicy: Always
Remember to apply changes.
Metrics-server attempts to authorize itself using token authentication. Please ensure that you're running your kubelets with webhook token authentication turned on.
Speaking about TLS directly. TLS message is big packet and MTU in calico is wrong so change it according calico-project-mtu.
Execute command:
$ kubectl edit configmap calico-config -n kube-system and change the MTU value from 1500 to 1430.
Take a look: metrics-server-mtu.
I also met the this problem that I can't make metrics-server to work in my k8s cluster (kubectl version is 1.25.4). I follow the instructions above and solve the issue!
I downloaded the components.yaml file and only add the - --kubelet-insecure-tls in the args of deployment. Then I got the metrics-server work!
I've been trying to enabled token auth for HTTP REST API Server access from a remote client.
I installed my CoreOS/K8S cluster controller using this script: https://github.com/coreos/coreos-kubernetes/blob/master/multi-node/generic/controller-install.sh
My cluster works fine. This is a TLS installation so I need to configure any kubectl clients with the client certs to access the cluster.
I then tried to enable token auth via running:
echo `dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64 | tr -d "=+/" | dd bs=32 count=1 2>/dev/null`
this gives me a token. I then added the token to a token file on my controller containing a token and default user:
$> cat /etc/kubernetes/token
3XQ8W6IAourkXOLH2yfpbGFXftbH0vn,default,default
I then modified the /etc/kubernetes/manifests/kube-apiserver.yaml to add in:
- --token-auth-file=/etc/kubernetes/token
to the startup param list
I then reboot (not sure the best way to restart API Server by itself??)
At this point, kubectl from a remote server quits working(won't connect). I then look at docker ps on the controller and see the api server. I run docker logs container_id and get no output. If I look at other docker containers I see output like:
E0327 20:05:46.657679 1 reflector.go:188]
pkg/proxy/config/api.go:33: Failed to list *api.Endpoints:
Get http://127.0.0.1:8080/api/v1/endpoints?resourceVersion=0:
dial tcp 127.0.0.1:8080: getsockopt: connection refused
So it appears that my api-server.yaml config it preventing the API Server from starting properly....
Any suggestions on the proper way to configure API Server for bearer token REST auth?
It is possible to have both TLS configuration and Bearer Token Auth configured, right?
Thanks!
I think your kube-apiserver dies because it's can't find the /etc/kubernetes/token. That's because on your deployment the apiserver is a static pod therefore running in a container which in turn means it has a different root filesystem than that of the host.
Look into /etc/kubernetes/manifests/kube-apiserver.yaml and add a volume and a volumeMount like this (I have omitted the lines that do not need changing and don't help in locating the correct section):
kind: Pod
metadata:
name: kube-apiserver
spec:
containers:
- name: kube-apiserver
command:
- ...
- --token-auth-file=/etc/kubernetes/token
volumeMounts:
- mountPath: /etc/kubernetes/token
name: token-kubernetes
readOnly: true
volumes:
- hostPath:
path: /etc/kubernetes/token
name: token-kubernetes
One more note: the file you quoted as token should not end in . (dot) - maybe that was only a copy-paste mistake but check it anyway. The format is documented under static token file:
token,user,uid,"group1,group2,group3"
If your problem perists execute the command below and post the output:
journalctl -u kubelet | grep kube-apiserver