Connection timedout when attempting to access any service in kubernetes - deployment

I've create a deployment and a service and deployed them using kubernetes, and when i tried to access them by curl, always i got a connection timed out error.
Here's my yml files:
Deployment.yml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
app: locations-service
name: locations-service
namespace: default
spec:
replicas: 2
selector:
matchLabels:
app: locations-service
template:
metadata:
labels:
app: locations-service
spec:
containers:
- image: dropwizard:latest
imagePullPolicy: Never # just for testing!
name: locations-service
ports:
- containerPort: 8080
protocol: TCP
name: app-port
- containerPort: 8081
protocol: TCP
name: admin-port
resources: {}
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
Service.yml
apiVersion: v1
kind: Service
metadata:
labels:
app: locations-service
name: locations-service
namespace: default
spec:
type: NodePort
externalTrafficPolicy: Cluster
ports:
- name: "8080"
port: 8080
targetPort: 8080
protocol: TCP
- name: "8081"
port: 8081
targetPort: 8081
protocol: TCP
selector:
app: locations-service
Also i tried to add ingress routes, and tried to hit them but still the same problem occur.
Note that the application is successfully deployed, and i can check the logs from k8s dashboard
Another example, i have the following svc
kubectl describe service webapp1-svc
Name: webapp1-svc
Namespace: default
Labels: app=webapp1
Annotations: <none>
Selector: app=webapp1
Type: NodePort
IP: 10.0.0.219
Port: <unset> 80/TCP
TargetPort: 80/TCP
NodePort: <unset> 30080/TCP
Endpoints: 172.17.0.4:80
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
and tried to access it using
curl -v 10.0.0.219:30080
* Rebuilt URL to: 10.0.0.219:30080/
* Trying 10.0.0.219...
* connect to 10.0.0.219 port 30080 failed: Connection timed out
* Failed to connect to 10.0.0.219 port 30080: Connection timed out
* Closing connection 0
curl: (7) Failed to connect to 10.0.0.219 port 30080: Connection timed out

Related

Service Endpoint not created although container port is online

I have a simple Service that connects to a port from a container inside a pod.
All pretty straight forward.
This was working too but out of nothing, the endpoint is not created for port 18080.
So I began to investigate and looked at this question but nothing that helped there.
The container is up, no errors/events, all green.
I can also call the request with the pods ip:18080 from an internal container, so the endpoint should be reachable for the service.
I can't see errors in:
journalctl -u snap.microk8s.daemon-*
I am using microk8s v1.20.
Where else can I debug this situation?
I am out of tools.
Service:
kind: Service
apiVersion: v1
metadata:
name: aedi-service
spec:
selector:
app: server
ports:
- name: aedi-host-ws #-port
port: 51056
protocol: TCP
targetPort: host-ws-port
- name: aedi-http
port: 18080
protocol: TCP
targetPort: fcs-http
Deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: server-deployment
labels:
app: server
spec:
replicas: 1
selector:
matchLabels:
app: server
template:
metadata:
labels:
app: server
srv: os-port-mapping
name: dns-service
spec:
hostname: fcs
containers:
- name: fcs
image: '{{$fcsImage}}'
imagePullPolicy: {{$pullPolicy}}
ports:
- containerPort: 18080
Service Description:
Name: aedi-service
Namespace: fcs-only
Labels: app.kubernetes.io/managed-by=Helm
Annotations: meta.helm.sh/release-name: fcs-only
meta.helm.sh/release-namespace: fcs-only
Selector: app=server
Type: ClusterIP
IP Families: <none>
IP: 10.152.183.247
IPs: 10.152.183.247
Port: aedi-host-ws 51056/TCP
TargetPort: host-ws-port/TCP
Endpoints: 10.1.116.70:51056
Port: aedi-http 18080/TCP
TargetPort: fcs-http/TCP
Endpoints:
Session Affinity: None
Events: <none>
Pod Info:
NAME READY STATUS RESTARTS AGE LABELS
server-deployment-76b5789754-q48xl 6/6 Running 0 23m app=server,name=dns-service,pod-template-hash=76b5789754,srv=os-port-mapping
kubectl get svc aedi-service -o wide:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
aedi-service ClusterIP 10.152.183.247 <none> 443/TCP,1884/TCP,51052/TCP,51051/TCP,51053/TCP,51056/TCP,18080/TCP,51055/TCP 34m app=server
Your service spec refer to a port named "fcs-http" but it was not declared in the deployment. Try:
apiVersion: apps/v1
kind: Deployment
metadata:
name: server-deployment
...
ports:
- containerPort: 18080
name: fcs-http # <-- add the name here
...
Wrong service configuration
- name: aedi-http
port: 18080 -----> which expose service, it has not related with container port.
protocol: TCP
targetPort: fcs-http -----> Here should be 18080, correspond to container port
If you still want to use name instead of port number, you should define name too in deployment yaml, like below:
containers:
- name: fcs
image: '{{$fcsImage}}'
imagePullPolicy: {{$pullPolicy}}
ports:
- containerPort: 18080
name: fcs-http

Access pod from another pod with kubernetes url

I have two pods created with deployment and service. my problem is as follows the pod "my-gateway" accesses the url "adm-contact" of "http://127.0.0.1:3000/adm-contact" which accesses another pod called "my-adm-contact" as can i make this work? I tried the following command: kubectl port-forward my-gateway-5b85498f7d-5rwnn 3000:3000 8879:8879 but it gives this error:
E0526 21:56:34.024296 12428 portforward.go:400] an error occurred forwarding 3000 -> 3000: error forwarding port 3000 to pod 2d5811c20c3762c6c249a991babb71a107c5dd6b080c3c6d61b4a275b5747815, uid : exit status 1: 2022/05/27 00:56:35 socat[2494] E connect(16, AF=2 127.0.0.1:3000, 16): Connection refused
Remembering that the images created with dockerfile are with EXPOSE 3000 8879
follow my yamls:
Deployment my-adm-contact:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-adm-contact
labels:
app: my-adm-contact
spec:
selector:
matchLabels:
run: my-adm-contact
template:
metadata:
labels:
run: my-adm-contact
spec:
containers:
- name: my-adm-contact
image: my-contact-adm
imagePullPolicy: Never
ports:
- containerPort: 8879
hostPort: 8879
name: admcontact8879
readinessProbe:
httpGet:
path: /adm-contact
port: 8879
initialDelaySeconds: 30
periodSeconds: 10
failureThreshold: 6
Sevice my-adm-contact:
apiVersion: v1
kind: Service
metadata:
name: my-adm-contact
labels:
run: my-adm-contact
spec:
selector:
app: my-adm-contact
ports:
- name: 8879-my-adm-contact
port: 8879
protocol: TCP
targetPort: 8879
type: LoadBalancer
status:
loadBalancer: {}
Deployment my-gateway:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-gateway
labels:
app: my-gateway
spec:
selector:
matchLabels:
run: my-gateway
template:
metadata:
labels:
run: my-gateway
spec:
containers:
- name: my-gateway
image: api-gateway
imagePullPolicy: Never
ports:
- containerPort: 3000
hostPort: 3000
name: home
#- containerPort: 8879
# hostPort: 8879
# name: adm
readinessProbe:
httpGet:
path: /adm-contact
port: 8879
path: /
port: 3000
initialDelaySeconds: 30
periodSeconds: 10
failureThreshold: 6
Service my-gateway:
apiVersion: v1
kind: Service
metadata:
name: my-gateway
labels:
run: my-gateway
spec:
selector:
app: my-gateway
ports:
- name: 3000-my-gateway
port: 3000
protocol: TCP
targetPort: 3000
- name: 8879-my-gateway
port: 8879
protocol: TCP
targetPort: 8879
type: LoadBalancer
status:
loadBalancer: {}
What k8s-cluster environment are you running this in? I ask because the service.type of LoadBalancer is a special kind: at pod initialisation your cloud provider's admission controller will spot this and add in a loadbalancer config See https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer
If you're not deploying this in a suitable cloud environment, your services won't do anything.
I had a quick look at your SO profile and - sorry if this is presumptious, I don't mean to be - it looks like you're relatively new to k8s. You shouldn't need to do any port-forwarding/kubectl proxying, and this should be a lot simpler than you might think.
When you create a service k8s will 'create' a DNS entry for you which points to the pod(s) specified by your selector.
I think you're trying to reach a setup where code running in my-gateway pod can connect to http://adm-contact on port 3000 and reach a listening service on the adm-contact pod. Is that correct?
If so, the outline solution is to expose tcp/3000 in the adm-contact pod, and create a service called adm-contact that has a selector for adm-contact pod.
This is a sample manifest I've just created which runs nginx and then creates a service for it, allowing any pod on the cluster to connect to it e.g. curl http://nginx-service.default.svc In this example I'm exposing port 80 because I didn't want to have to modify the nginx config, but the principle is the same.
apiVersion: v1
kind: Pod
metadata:
labels:
app: nginx
name: nginx
spec:
containers:
- image: nginx
imagePullPolicy: Always
name: nginx
ports:
- containerPort: 80
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
name: nginx-service
spec:
ports:
- port: 80
protocol: TCP
targetPort: 80
selector:
app: nginx
type: ClusterIP
The k8s docs on Services are pretty helpful if you want more https://kubernetes.io/docs/concepts/services-networking/connect-applications-service/
a service can be reached on it's own name from pods in it's namespace:
so a service foo in namespace bar can be reached at http://foo from a pod in namespace bar
from other namespaces that service is reachable at http://foo.bar.svc.cluster.local. Change out the servicename and namespace for your usecase.
k8s dns is explained here in the docs:
https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
I have taken the YAML you provided and assembled it here.
From another comment I see the URL you're trying to connect to is: http://gateway-service.default.svc.cluster.local:3000/my-adm-contact-service
The ability to resolve service names to pods only functions inside the cluster: coredns (a k8s pod) is the part which recognises when a service has been created and what IP(s) it's available at.
So another pod in the cluster e.g. one created by kubectl run bb --image=busybox -it -- sh would be able to resolve the command ping gateway-service, but pinging gateway-service from your desktop will fail because they're not both seeing the same DNS.
The api-gateway container will be able to make a connect to my-adm-contact-service on ports 3000 or 8879, and the my-adm-contact container will equally be able to connect to gateway-service on port 3000 - but only when those containers are running inside the cluster.
I think you're trying to access this from outside the cluster, so now the port/service types are correct you could re-try a kubectl port-forward svc/gateway-service 3000:3000 This will let you connect to 127.0.0.1:3000 and the traffic will be routed to port 3000 on the api-gateway container.
If you need to proxy to the other my-adm-contact-service then you'll have to issue similar kubectl commands in other shells, one per service:port combination. For completeness, if you wanted to route traffic from your local machine to all three container/port sets, you'd run:
# format kubectl port-forward svc/name src:dest (both TCP)
kubectl port-forward svc/gateway-service 3000:3000
kubectl port-forward svc/my-adm-contact-service 8879:8879
kubectl port-forward svc/my-adm-contact-service 3001:3000 #NOTE the changed local port, because localhost:3000 is already used
You will need a new shell for each kubectl, or run it as a background job.
apiVersion: v1
kind: Pod
metadata:
name: my-adm-contact
labels:
app: my-adm-contact
spec:
containers:
- image: my-contact-adm
imagePullPolicy: Never
name: my-adm-contact
ports:
- containerPort: 8879
protocol: TCP
- containerPort: 3000
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
name: my-adm-contact-service
spec:
ports:
- port: 8879
protocol: TCP
targetPort: 8879
name: adm8879
- port: 3000
protocol: TCP
targetPort: 3000
name: adm3000
selector:
app: my-adm-contact
type: ClusterIP
---
apiVersion: v1
kind: Pod
metadata:
name: my-gateway
labels:
app: my-gateway
spec:
containers:
- image: api-gateway
imagePullPolicy: Never
name: my-gateway
ports:
- containerPort: 3000
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
name: gateway-service
spec:
ports:
- port: 3000
protocol: TCP
targetPort: 3000
selector:
app: my-gateway
type: ClusterIP

Connecting Frontend API application to Backend database

Can someone please help me complete this scenario as I have everything running successfully but with no end result as expected. I have a PostgreSQL database and Redis (back-end) Pods along with a front-end API application that I will be using to send API requests. Unfortunately, after setting up everything and making sure they are all running on the Kubernetes dashboard, when sending an API request there is nothing happening as if there is no connection between my services. Before desperately posting here I did some research to find a solution to this problem, I tried this tutorial of using a ingress-service to connect my services but did not work. I also came across this tutorial that will connect my back-end service to my front-end service using some sort of upstream configuration file but it did not do the trick.
Here is my YAML configuration files if anyone is interested:
apiVersion: apps/v1
kind: Deployment
metadata:
name: dictionary-project
labels:
app: net
spec:
replicas: 1
selector:
matchLabels:
app: dictionary
tier: frontend
template:
metadata:
labels:
app: dictionary
tier: frontend
spec:
hostNetwork: true
containers:
- name: dictionaryapi
image: amin/dictionary_server_api:latest
ports:
- containerPort: 400
apiVersion: v1
kind: Service
metadata:
name: dictionary-service
labels:
app: net
spec:
selector:
app: dictionary
tier: frontend
type: NodePort
ports:
- nodePort: 31003
port: 67
targetPort: 400
protocol: TCP
apiVersion: v1
kind: Service
metadata:
name: postgrebackendservice
labels:
run: backend
spec:
selector:
app: postgres
tier: backend
type: ClusterIP
ports:
- port: 5432
targetPort: 5432
name: postgresdb
apiVersion: v1
kind: Service
metadata:
name: reddisbackendservice
labels:
run: backend
spec:
selector:
app: reddis
tier: backend
type: ClusterIP
ports:
- port: 6379
targetPort: 6379
name: client
- port: 16379
targetPort: 16379
name: gossip
Output of all services:
Name: dictionary-service
Namespace: default
Labels: app=net
Annotations: <none>
Selector: app=dictionary,tier=frontend
Type: NodePort
IP: 10.126.146.18
Port: <unset> 67/TCP
TargetPort: 400/TCP
NodePort: <unset> 31003/TCP
Endpoints: 192.168.x.x:400
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
Name: postgrebackendservice
Namespace: default
Labels: run=backend
Annotations: Selector: app=postgres,tier=backend
Type: ClusterIP
IP: 10.182.374.13
Port: postgresdb 5432/TCP
TargetPort: 5432/TCP
Endpoints: 192.168.x.x:5432
Session Affinity: None
Events: <none>
Name: reddisbackendservice
Namespace: default
Labels: run=backend
Annotations: Selector: app=reddis,tier=backend
Type: ClusterIP
IP: 10.182.0.60
Port: client 6379/TCP
TargetPort: 6379/TCP
Endpoints: 192.168.x.x:6379
Port: gossip 16379/TCP
TargetPort: 16379/TCP
Endpoints: 192.168.x.x:16379
Session Affinity: None
Events: <none>
I am testing my front-end API application on a web browser by sending a http://{workernodeIP}:31003/swagger but the page is not loading due to no connection to the server.
Cluster information:
Kubernetes version: v1.18.6
Environment being used: bare-metal, 1 VM Master Node and 1 VM Worker Node
Installation method: kubeadm
Host OS: Ubuntu 18.04
CNI and version: calico
CRI and version: Docker 19.03.6
From the docs here
containerPort:
List of ports to expose from the container. Exposing a port here gives
the system additional information about the network connections a
container uses, but is primarily informational. Not specifying a port
here DOES NOT prevent that port from being exposed. Any port which is
listening on the default "0.0.0.0" address inside a container will be
accessible from the network. Cannot be updated.
As you can see containerPort is information and does not make the app listen on that port. For really making the app listen on port 400 you need change port in code.

One Deployment is reported with unhealthy upstream

Using Istio with Kubernetes and facing an issue with a simple routerule splitting traffic between two Deployments. Only one of the Deployment (howdy) is returning results correctly. The other Deployment (hello) is reporting "no healthy upstream".
Here is the k8s manifest:
apiVersion: v1
kind: Service
metadata:
name: greeting
labels:
name: greeting
spec:
selector:
app: greeting
ports:
- name: http
protocol: TCP
port: 8081
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: greeting-hello
labels:
name: greeting-hello
spec:
replicas: 1
template:
metadata:
labels:
app: greeting
greeting: hello
spec:
containers:
- name: greeting
image: arungupta/greeting:hello
imagePullPolicy: Always
ports:
- containerPort: 8081
readinessProbe:
httpGet:
path: /resources/greeting
port: 8081
initialDelaySeconds: 50
periodSeconds: 5
--- apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: greeting-howdy
labels:
name: greeting-howdy
spec:
replicas: 1
template:
metadata:
labels:
app: greeting
greeting: howdy
spec:
containers:
- name: greeting
image: arungupta/greeting:howdy
imagePullPolicy: Always
ports:
- containerPort: 8081
readinessProbe:
httpGet:
path: /resources/greeting
port: 8081
initialDelaySeconds: 50
periodSeconds: 5
And the route is:
apiVersion: config.istio.io/v1alpha2
kind: RouteRule
metadata:
name: greeting-50-50
spec:
destination:
name: greeting
route:
- labels:
app: greeting
greeting: hello
weight: 100
- labels:
app: greeting
greeting: howdy
weight: 0
Any idea?
This is also tracked at https://github.com/aws-samples/aws-microservices-deploy-options/issues/239
I've reproduced your deployment issue and found the following errors during the process:
Warning Unhealthy 48m (x7 over 49m) kubelet, gke-cluster-1-default-pool-e44042f6-kndh Readiness probe failed: Get http://10.0.2.30:8081/resources/greeting: dial tcp 10.0.2.30:8081: getsockopt: connection refused
I would recommend you to check if docker containers in this particular configuration are
exposing endpoints on the specified (8081) port. I also found that pods are not registered in any
of the endpoints:
$ kubectl describe service
[...]
IP: 10.3.247.97
Port: http 8081/TCP
TargetPort: 8081/TCP
Endpoints:
Session Affinity: None
Events: <none>
The problem is that the application did not bind to the port which was set as a container port,
and the application did not receive connection.

How to expose multiple port using a load balancer services in Kubernetes

I have created a cluster using the google cloud platform (container engine) and deployed a pod using the following YAML file:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: deployment-name
spec:
replicas: 1
template:
metadata:
name: pod-name
labels:
app: app-label
spec:
containers:
- name: container-name
image: gcr.io/project-id/image-name
resources:
requests:
cpu: 1
ports:
- name: port80
containerPort: 80
- name: port443
containerPort: 443
- name: port6001
containerPort: 6001
Then I want to create a service that enables the pod to listen on all these ports. I know that the following YAML file works to create a service that listens on one port:
apiVersion: v1
kind: Service
metadata:
name: service-name
spec:
ports:
- port: 80
targetPort: 80
selector:
app: app-label
type: LoadBalancer
However when I want the pod to listen on multiple ports like this, it doesn't work:
apiVersion: v1
kind: Service
metadata:
name: service-name
spec:
ports:
- port: 80
targetPort: 80
- port: 443
targetPort: 443
- port: 6001
targetPort: 6001
selector:
app: app-label
type: LoadBalancer
How can I make my pod listen to multiple ports?
You have two options:
You could have multiple services, one for each port. As you pointed out, each service will end up with a different IP address
You could have a single service with multiple ports. In this particular case, you must give all ports a name.
In your case, the service becomes:
apiVersion: v1
kind: Service
metadata:
name: service-name
spec:
ports:
- name: http
port: 80
targetPort: 80
- name: https
port: 443
targetPort: 443
- name: something
port: 6001
targetPort: 6001
selector:
app: app-label
type: LoadBalancer
This is necessary so that endpoints can be disambiguated.