I set up Grafana to run in GKE (Kubernetes) with a service and default Ingress controller to open it to the internet. Everything is working without any issues.
After creating some dashboards I wanted to setup Slack alerting using the slack webhook. After filling out all the details I received a 502 bad gateway error.
I have setup a second service to open port 443(Default slack webhook port) and exposed it with kubectl expose deployment --type=NodePort --port=443 and have also tried --type=LoadBalancer with no luck.
I've also tried setting up a second Ingress service pointing the second service, but then I run into readinessProbe issues.
Anyone had the same issue and if so how was it resolved?
Network Policy was enabled on the cluster and there were cluster policies denying outgoing traffic. After setting up my network policies for the pods in my own namespace I was able to connect without any issues.
Related
I am having real trouble understanding how I am suppose to debug my current situation. I have followed the setup instructions from https://docs.substra.org/en/stable/contributing/getting-started.html#
There is a backend service which was created as a ClusterIP, and therefore can not be accessed from the host.
I created a load balancer for this purpose. using the command
kubectl expose deployment deployment_name --port=8000 --target-port=8000 \
--name=lb_service --type=LoadBalancer
However, the attempt to access the backend service failed when I use the LoadBalancer Ingress ip and NodePort port with a connection timeout. I like to see the relevant logs to check where the problem occurred. However, apparently kubectl logs service only shows logs for pods, whereas the load balancer, according to the kubectl expose command is attached to the deployment. Therefore, I am not able to see any logs related either to the load balancer service, or the deployment component.
When I looked at the pod which is supposed to be hosting the deployment, the log showed no error.
Can someone point out where do I look for logs that can debug this failed connectivity?
You probably need to look at the ingress logs, se this page from the documentation: https://kubernetes.github.io/ingress-nginx/troubleshooting/.
it is true that you can only get logs from pods. However, that is sufficient to see the relevant error messages.
I have Installed Ambassador Api gateway on AWS EKS cluster. It's working as expected.
Now I'd like to integrate Istio service mesh.
I'm following the steps given in the ambassador's official documentation.
https://www.getambassador.io/docs/edge-stack/latest/howtos/istio/#istio-integration.
But after Istio integration some ambassador pods are keep crashing.
At a time only 1 pod shows healthy out of 3.
Note: Istio side car are integrated successfully in all ambassador pods. and I have tried with Ambassador 2.1.1 & 2.1.2. But both has same issue. I'm not able to keep all ambassador pod healthy.
My EKS version is v1.19.13-eks
Below are the error:
time="2022-03-02 12:30:17.0687" level=error msg="Post \"http://localhost:8500/_internal/v0/watt?url=http%3A%2F%2Flocalhost%3A9696%2Fsnapshot\": dial tcp 127.0.0.1:8500: connect: connection refused" func=github.com/datawire/ambassador/v2/cmd/entrypoint.notifyWebhookUrl file="/go/cmd/entrypoint/notify.go:124" CMD=entrypoint PID=1 THREAD=/watcher
Please do let me know if the above documentation is not sufficient for Istio integration with Ambassador on AWS EKS
Edit 1: In further investigation I found the issue comes when I tried to integrate Istio with PeerAuthentication STRICT mode. There is no such issue with default (permissive) mode.
But another issue comes when enable the STRICT mode, and now it's failing to connect with redis service
After some investigation and testing I find out the way to integrate Istio with Ambassador with PeerAuthentication STRICT mode.
the fix :
update the REDIS_URL env variable with https
from:
REDIS_URL: ambassador-redis:6379
to
REDIS_URL: https://ambassador-redis:6379
I have a gateway running as a pod and accessing it via NodePort service on port 3XXXX on a server.
I am able to send traffic to this gateway.
But I am not able to forward traffic from this gateway pod to a service that is on a different and server(this service is not a Kubernetes service).
It is showing timeout.
I am also not able to ping it from indside of pod.
I have whitelisted both servers with each other.
Please help me.
Our on-premise Kubernetes/Kubespray cluster has suddenly stopped routing traffic between the nginx-ingress and node port services. All external requests to the ingress endpoint return a "504 - gateway timeout" error.
How do I diagnose what has broken?
I've confirmed that the containers/pods are running, the node application has started and if I exec into the pod then I can run a local curl command and get a response from the app.
I've checked the logs on the ingress pods and traffic is arriving and nginx is trying to forward the traffic on to the service endpoint/node port but it is reporting an error.
I've also tried to curl directly to the node via the node port but I get no response.
I've looked at the ipvs configuration and the settings look valid (e.g. there are rules for the node to forward traffic on the node port the service endpoint address/port)
We couldn't resolve this issue and, in the end, the only workaround was to uninstall and reinstall the cluster.
I was getting this because the nginx ingress controller pod was running out of memory, I just increased the memory for the pod and it worked.
I was facing a similar issue and the simple fix was to increase the values for the K8S_CPU_LIMIT and K8S_MEMORY_LIMIT for the application pods running on the cluster.
I have started with kubernetes and followed this link to get the response as they mentioned. I followed the exact steps but when I am trying to open the port I get the following error:
How to solve this issue? I have tried by adding the IP address and port in the Browser proxy.
Can anyone help me on this?
Here is the service image: my service image
List of pods: Kubectl Pods
List of kubectl deployments:Deployment List
I believe you're using the baremetal(simple laptop) to deploy your service.
If you have look at my-service it is in pending state and it is of type LoadBalancer. The type load balance is supported only for the cloud providers like aws,azure and google cloud. Hence you are not able to access anything.
I will suggest you to follow this tutorial here which allow you to deploy nginx as a pod and deploy a service around that and export that service as nodeport (without load balancer) to be able to access from outside.