I'm learning kubernetes via a LinkedIn learning course. A tutorial I'm doing runs this hello world application via kubectl and minikube. Everything appears in working order, but I cannot interact with the application using minikube service helloworld. The request keeps timing out.
The tutorial first asks to create a deployment using the command kubectl create -f helloworld.yaml then to expose the service via command kubectl expose deployment helloworld --type=NodePort and then it says interact with the app by doing minikube service helloworld. The diagnostics after create and expose show that everything on my end matches the tutorial's setup, but the last step fails for me whereas it launches the browser and shows the hello world app in the tutorial demo.
How would I go about debugging this error as an absolute beginner?
EDIT:
When I run kubectl describe services, I get the following output
$ kubectl describe services
Name: helloworld
Namespace: default
Labels: <none>
Annotations: <none>
Selector: app=helloworld
Type: NodePort
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.96.6.203
IPs: 10.96.6.203
Port: <unset> 80/TCP
TargetPort: 80/TCP
NodePort: <unset> 30433/TCP
Endpoints: 172.17.0.2:80
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
When I check the port 30433, by doing nc -zv <hostname> 30433, I get an error:
nc: connectx to <hostname> port 30433 (tcp) failed: Connection refused
nc: connectx to <hostname> port 30433 (tcp) failed: Network is unreachable
You can try to access your application with the help of this shortcut - it is used to fetch minikube IP and a service’s NodePort:
minikube service --url helloworld
The output of the command will display Kubernetes service URL in CLI, instead of trying to launch it in your default browser with minikube service helloworld command. Using this URL, you will be able to access the exposed service in the browser.
In general, you can check the list of all available services in your minikube cluster and their URLs by using minikube service list command.
Related
I'm new to K8s and am currently using Minikube to play around with the platform. How do I configure a public (i.e. outside the cluster) port for the service? I followed the nginx example, and K8s service tutorials. In my case, I created the service like so:
kubectl expose deployment/mysrv --type=NodePort --port=1234
The service's port is 1234 for anyone trying to access it from INSIDE the cluster. The minikube tutorials say I need to access the service directly through it's random nodePort, which works for manual testing purposes:
kubectl describe service mysrv | grep NodePort
...
NodePort: <unset> 32387/TCP
# curl "http://`minikube ip`:32387/"
But I don't understand how, in a real cluster, the service could have a fixed world-accessible port. The nginx examples describe something about using the LoadBalancer service kind, but they don't even specify ports there...
Any ideas how to fix the external port for the entire service?
The minikube tutorials say I need to access the service directly through it's random nodePort, which works for manual testing purposes:
When you create service object of type NodePort with a $ kubectl expose command you cannot choose your NodePort port. To choose a NodePort port you will need to create a YAML definition of it.
You can manually specify the port in service object of type Nodeport with below example:
apiVersion: v1
kind: Service
metadata:
name: example-nodeport
spec:
type: NodePort
selector:
app: hello # selector for deployment
ports:
- name: example-port
protocol: TCP
port: 1234 # CLUSTERIP PORT
targetPort: 50001 # POD PORT WHICH APPLICATION IS RUNNING ON
nodePort: 32222 # HERE!
You can apply above YAML definition by invoking command:
$ kubectl apply -f FILE_NAME.yaml
Above service object will be created only if nodePort port is available to use.
But I don't understand how, in a real cluster, the service could not have a fixed world-accessible port.
In clusters managed by cloud providers (for example GKE) you can use a service object of type LoadBalancer which will have a fixed external IP and fixed port.
Clusters that have nodes with public IP's can use service object of type NodePort to direct traffic into the cluster.
In minikube environment you can use a service object of type LoadBalancer but it will have some caveats described in last paragraph.
A little bit of explanation:
NodePort
Nodeport is exposing the service on each node IP at a static port. It allows external traffic to enter with the NodePort port. This port will be automatically assigned from range of 30000 to 32767.
You can change the default NodePort port range by following this manual.
You can check what is exactly happening when creating a service object of type NodePort by looking on this answer.
Imagine that:
Your nodes have IP's:
192.168.0.100
192.168.0.101
192.168.0.102
Your pods respond on port 50001 with hello and they have IP's:
10.244.1.10
10.244.1.11
10.244.1.12
Your Services are:
NodePort (port 32222) with:
ClusterIP:
IP: 10.96.0.100
port:7654
targetPort:50001
A word about targetPort. It's a definition for port on the pod that is for example a web server.
According to above example you will get hello response with:
NodeIP:NodePort (all the pods could respond with hello):
192.168.0.100:32222
192.168.0.101:32222
192.168.0.102:32222
ClusterIP:port (all the pods could respond with hello):
10.0.96.100:7654
PodIP:targetPort (only the pod that request is sent to can respond with hello)
10.244.1.10:50001
10.244.1.11:50001
10.244.1.12:50001
You can check access with curl command as below:
$ curl http://NODE_IP:NODEPORT
In the example you mentioned:
$ kubectl expose deployment/mysrv --type=NodePort --port=1234
What will happen:
It will assign a random port from range of 30000 to 32767 on your minikube instance directing traffic entering this port to pods.
Additionally it will create a ClusterIP with port of 1234
In the example above there was no parameter targetPort. If targetPort is not provided it will be the same as port in the command.
Traffic entering a NodePort will be routed directly to pods and will not go to the ClusterIP.
From the minikube perspective a NodePort will be a port on your minikube instance. It's IP address will be dependent on the hypervisor used. Exposing it outside your local machine will be heavily dependent on operating system.
LoadBalancer
There is a difference between a service object of type LoadBalancer(1) and an external LoadBalancer(2):
Service object of type LoadBalancer(1) allows to expose a service externally using a cloud provider’s LoadBalancer(2). It's a service within Kubernetes environment that through service controller can schedule a creation of external LoadBalancer(2).
External LoadBalancer(2) is a load balancer provided by cloud provider. It will operate at Layer 4.
Example definition of service of type LoadBalancer(1):
apiVersion: v1
kind: Service
metadata:
name: example-loadbalancer
spec:
type: LoadBalancer
selector:
app: hello
ports:
- port: 1234 # LOADBALANCER PORT
targetPort: 50001 # POD PORT WHICH APPLICATION IS RUNNING ON
nodePort: 32222 # PORT ON THE NODE
Applying above YAML will create a service of type LoadBalancer(1)
Take a specific look at:
ports:
- port: 1234 # LOADBALANCER PORT
This definition will simultaneously:
specify external LoadBalancer(2) port as 1234
specify ClusterIP port as 1234
Imagine that:
Your external LoadBalancer(2) have:
ExternalIP: 34.88.255.5
port:7654
Your nodes have IP's:
192.168.0.100
192.168.0.101
192.168.0.102
Your pods respond on port 50001 with hello and they have IP's:
10.244.1.10
10.244.1.11
10.244.1.12
Your Services are:
NodePort (port 32222) with:
ClusterIP:
IP: 10.96.0.100
port:7654
targetPort:50001
According to above example you will get hello response with:
ExternalIP:port (all the pods could respond with hello):
34.88.255.5:7654
NodeIP:NodePort (all the pods could respond with hello):
192.168.0.100:32222
192.168.0.101:32222
192.168.0.102:32222
ClusterIP:port (all the pods could respond with hello):
10.0.96.100:7654
PodIP:targetPort (only the pod that request is sent to can respond with hello)
10.244.1.10:50001
10.244.1.11:50001
10.244.1.12:50001
ExternalIP can be checked with command: $ kubectl get services
Flow of the traffic:
Client -> LoadBalancer:port(2) -> NodeIP:NodePort -> Pod:targetPort
Minikube: LoadBalancer
Note: This feature is only available for cloud providers or environments which support external load balancers.
-- Kubernetes.io: Create external LoadBalancer
On cloud providers that support load balancers, an external IP address would be provisioned to access the Service. On Minikube, the LoadBalancer type makes the Service accessible through the minikube service command.
-- Kubernetes.io: Hello minikube
Minikube can create service object of type LoadBalancer(1) but it will not create an external LoadBalancer(2).
The ExternalIP in command $ kubectl get services will have pending status.
To address that there is no external LoadBalancer(2) you can invoke $ minikube tunnel which will create a route from host to minikube environment to access the CIDR of ClusterIP directly.
There is a small mistake in Dawid Kruk’s answer,
Traffic entering a NodePort will be routed directly to pods and will
not go to the ClusterIP.
But as k8s documented here:
NodePort: Exposes the Service on each Node's IP at a static port (the
NodePort). A ClusterIP Service, to which the NodePort Service
routes, is automatically created. You'll be able to contact the
NodePort Service, from outside the cluster, by requesting
:.
Traffic entering a NodePort did go to ClusterIP.
I have a local kubernetes cluster on my local docker desktop.
This is how my kubernetes service looks like when I do a kubectl describe service
Name: helloworldsvc
Namespace: test
Labels: app=helloworldsvc
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"helloworldsvc"},"name":"helloworldsvc","namespace":"test...
Selector: app=helloworldapp
Type: ClusterIP
IP: 10.108.182.240
Port: http 9111/TCP
TargetPort: 80/TCP
Endpoints: 10.1.0.28:80
Session Affinity: None
Events: <none>
This service is pointing to a deployment with a web app.
My question how to I find the url for this service?
I already tried http://localhost:9111/ and that did not work.
I verified that the pod that this service points to is up and running.
URL of service is in the below format:
<service-name>.<namespace>.svc.cluster.local:<service-port>
In your case it is:
helloworldsvc.test.svc.cluster.local:9111
Get the service name: kubectl get service -n test
URL to a kubernetes service is service-name.namespace.svc.cluster.local:service-port where cluster.local is the kubernetes cluster name.
To get the cluster name: kubectl config get-contexts | awk {'print $2'}
URL to service in your case will be helloworldsvc.test.svc.cluster.local:9111
The way you are trying to do won't work as to make it available on your localhost you need to make the service available at nodeport or using port-forward or using kubectl proxy.
However, if you want dont want a node port and to check if inside the container everything works fine then follow these steps to get inside the container if it has a shell.
kubectl exec -it container-name -n its-namespace-name sh
then do a
curl localhost:80 or curl helloworldsvc.test.svc.cluster.local:9111 or curl 10.1.0.28:80
but both curl commands will work only inside Kubernetes pod and not on your localhost machine.
To access on your host machine kubectl port-forward svc/helloworldsvc 80:9111 -n test
The service you have created is of type ClusterIP which is only accessible from inside the cluster. You have two ways to access it from your desktop:
Create a nodeport type service and then access it via nodeip:nodeport
Use Kubectl port forward and then access it via localhost:forwardedport
The following url variations worked for me when in the same cluster and on the same namespace (namespace: default; though all but first should still work when services are on different namespaces):
http://helloworldsvc
http://helloworldsvc.default
http://helloworldsvc.default.svc
http://helloworldsvc.default.svc.cluster.local
http://helloworldsvc.default.svc.cluster.local:80
//
using HttpClient client = new();
string result = await client.GetStringAsync(url);
Notes:
I happen to be calling to and from an ASP.NET 6 application using HttpClient
That client I think just sets port to 80 by default, so no 80 port needs to be explicitly set to work. But I did verify for all of these it can be added or removed from the url
http only (not https, unless you configured it specially)
namespace can only be omitted in the first case (i.e. when domain / 'authority' is just the service name alone). So helloworldsvc.svc.cluster.local:80 fails with exception "Name or service not known (helloworldsvc.svc.cluster.local:80)"
If you are working with minikube , you can run the code below
minikube service --all
for specific service
minikube service service-name --url
Here is another way to get the URL of service
Enter one pod through kubectl exec
kubectl exec -it podName -n namespace -- /bin/sh
Then execute nslookup IP of service such as 172.20.2.213 in the pod
/ # nslookup 172.20.2.213
nslookup: can't resolve '(null)': Name does not resolve
Name: 172.20.2.213
Address 1: 172.20.2.213 172-20-2-213.servicename.namespace.svc.cluster.local
Or execute nslookup IP of serviceName in the pod
/ # nslookup servicename
nslookup: can't resolve '(null)': Name does not resolve
Name: 172.20.2.213
Address 1: 172.20.2.213 172-20-2-213.servicename.namespace.svc.cluster.local
Now the service URL is servicename.namespace.svc.cluster.local attached with the service port after removing IP for the output of nslookup.
Install a latest OpenShift CodeReady Container on CentOS VM, and then run a TCP server app written by Java on OpenShift. The TCP Server is listening on port 7777.
Run app and expose it as a service with NodePort, seems that everything runs well. The pod port is 7777, and the service port is 31777.
$ oc get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
tcpserver-57c9b44748-k9dxg 1/1 Running 0 113m 10.128.0.229 crc-2n9vw-master-0 <none> <none>
$ oc get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
tcpserver-ingres NodePort 172.30.149.98 <none> 7777:31777/TCP 18m
Then get node IP, the command shows as 192.168.130.11, I can ping this ip on my VM successfully.
$ oc get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
crc-2n9vw-master-0 Ready master,worker 26d v1.14.6+6ac6aa4b0 192.168.130.11 <none> Red Hat Enterprise Linux CoreOS 42.81.20191119.1 (Ootpa) 4.18.0-147.0.3.el8_1.x86_64 cri-o://1.14.11-0.24.dev.rhaos4.2.gitc41de67.el8
Now, run a client app which is located in my VM, because I can ping OpenShift Node IP, so I think I can run the client app successfully. The result is that connection time out, my client fails to connect server running on OpenShift.
Please give your advice how to troubleshoot the issue, or any ideas for the issue.
I understood your problem. As per what you described, I can see your Node port is 31777.
The best way to debug this problem is going step by step.
Step 1:
Check if you are able to access your app server using your pod IP and port i.e curl 10.128.0.229:7777/endpoint from one of your nodes within your cluster. This helps you with checking if pod is working or not. Even though kubectl describe pod gives you everything.
Step 2:
After that, on the Node which the pod is deployed i.e 192.168.130.11 on this try to access your app server using curl localhost:31777/endpoint. If this works, Nodeport is accessible i.e your service is working fine without any issues.
Step 3:
After that, try to connect to your node using curl 192.168.130.11:31777/endpoint from the vm running your client server. Just to let you know, 192. is class A private ip, so I am assuming your client is within the same network and able to talk to 192.169.130.11:31777 Or make sure you open your the respective 31777 port of 192.169.130.11 to the vm ip that has client server.
This is a small process of debugging the issue with service and pod. But the best is to use the ingress and an ingress controller, which will help you to talk to your app server with a url instead of ip address and port numbers. However, even with ingress and ingress controller the best way to debug all the parts are working as expected is following these steps.
Please feel free to let me know for any issues.
Thanks prompt answer.
Regarding Step 1,
I don't know where I could run "curl 10.128.0.229:7777/endpoint" inside cluster, but I check the status of pod via going to inside pod, port 777 is listening as expected.
$ oc rsh tcpserver-57c9b44748-k9dxg
sh-4.2$ netstat -nap | grep 7777
tcp6 0 0 127.0.0.1:7777 :::* LISTEN 1/java
Regarding Step 2,
run command "curl localhost:31777/endpoint" on Node where pod is deployed, it failed.
$ curl localhost:31777/endpoint
curl: (7) Failed to connect to localhost port 31777: Connection refused
That means, it seems that 31777 is not opened by OpenShift.
Do you have any ideas how to check why 31777 is not opened by OpenShift.
More information about service definition:
apiVersion: v1
kind: Service
metadata:
name: tcpserver-ingress
labels:
app: tcpserver
spec:
selector:
app: tcpserver
type: NodePort
ports:
- protocol: TCP
port: 7777
targetPort: 7777
nodePort: 31777
Service status:
$ oc describe svc tcpserver-ingress
Name: tcpserver-ingress
Namespace: myproject
Labels: app=tcpserver
Annotations: <none>
Selector: app=tcpserver
Type: NodePort
IP: 172.30.149.98
Port: <unset> 7777/TCP
TargetPort: 7777/TCP
NodePort: <unset> 31777/TCP
Endpoints: 10.128.0.229:7777
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
I'm deploying a spring boot app in minikube that connects to a database running on the host. Where do I find the IP address that the app can use to get back to the host? For docker I can use ifconfig and get the IP address from the docker0 entry. ifconfig shows another device with IP address 172.18.0.1. Would that be how my app would get back to the host?
I think I understood you correctly and this is what you are asking for.
Minikube is started as a VM on your machine. You need to know the IP which Minikube starts with. This can be done with minikube status or minikube ip, output might look like:
$ minikube status
minikube: Running
cluster: Running
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.1
This will only provide you the IP address of Minikube not your application.
In order to connect to your app from outside the Minikube you need to expose it as a Service.
Example of a Service might look like this:
apiVersion: v1
kind: Service
metadata:
name: webapp
spec:
type: NodePort
ports:
- nodePort: 31317
port: 8080
protocol: TCP
targetPort: 8080
selector:
app: webapp
You can see results:
$ kubectl get services -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
postgres ClusterIP 10.0.0.140 <none> 5432/TCP 32m app=postgres
webapp NodePort 10.0.0.235 <none> 8080:31317/TCP 2s app=webapp
You will be able to connect to the webapp from inside the Cluster using 10.0.0.235:8080 of from outside the Cluster using Minikube IP and port 31317.
I also recommend going through Hello Minikube tutorial.
It was the 172.18.0.1 IP address. I passed it to the Spring app running in minikube with a configmap like this:
kubectl create configmap springdatasourceurl --from-literal=SPRING_DATASOURCE_URL=jdbc:postgresql://172.18.0.1:5432/bookservice
The app also needed SPRING_DATASOURCE_DRIVER_CLASS_NAME to be set in a configmap and that credentials SPRING_DATASOURCE_PASSWORD and SPRING_DATASOURCE_USERNAME be set as secrets.
More information on configmap and secret are here.
I installed minikube on my mac and created deployment and a service for my nodejs app. I tested that everything is working by getting the URL of my service using the following command:
minikube service my-nodejs-app --url
and then I run this URL in the browser and got results. The problem is when i tried to access the same URL from another machine inside the same network it didn't worked.
my service.yml file is:
apiVersion: v1
kind: Service
metadata:
name: my-nodejs-app
spec:
type: NodePort
ports:
- port: 80
targetPort: 1337
protocol: TCP
name: app-server
selector:
app: my-nodejs-app
I tried to use port forwarding to forward my pod port to my localhost and it works only on the same machine who host the cluster and when I try to access from another machine on the same network (via the IP address of the machine where the cluster deployed) I still get page not found.
You can use "port forward a service". Assuming:
Your local machine IP: 166.6.6.6 (which hold minikube)
Your minikube IP: 192.168.99.100 (check the real IP with command $minikube ip)
The nodePort of your service 'my-nodejs-app': 31000 (check the real
nodePort with command: $kubectl get service)
In order to access your service from remote, you can forward a port (like 31000, recommend the same port with nodePort) to your service through the following command in your local machine:
ssh -i ~/.minikube/machines/minikube/id_rsa docker#$(minikube ip) -L \*:31000:0.0.0.0:31000
Then you can access your service through URL: http://166.6.6.6:31000, which will be forwarded to your service URL http://192.168.99.100:31000
Thx: https://github.com/kubernetes/minikube/issues/877
Probably a bit late, but if anyone still having this issue-
Check the list of services and the one you want to expose if it is present
kubectl get svc -n {namespace_name}
Change the type to NodePort if it is of cluster IP type.
kubectl patch svc {service_name} -n {namespace_name} --type='json' -p '[{"op":"replace","path":"/spec/type","value":"NodePort"}]'
Expose the above Node Port available to your local machine now for other machines on same network:
service_port=$(minikube service {service_name} -n {namespace_name} --url | cut -d':' -f3)
ssh -i ~/.minikube/machines/minikube/id_rsa docker#$(minikube ip) -NL \*:${service_port}:0.0.0.0:${service_port}
Now you can access the above service from other machines on same network by just hitting the link-
{your_hostname}:{node_port}
Sounds like reaching it from another machine compares to exposing a ssevice to the web.
In that case you need to look into spec/type:LoadBalancer (http://kubernetes.io/docs/user-guide/load-balancer/)
That said, with minikube i'd stick to a single machine and development only tests
If I understand your problem correctly:
Your machine's IP: 192.168.1.4
Your minikube IP: 192.168.99.100
Accessing your service from a browser on your machine: http://192.168.99.100:30080
Now, let's say you're on another machine, say192.168.1.5, and you want to access this service.
The problem is that you need to map your machine's port to minikube's 30080 because minikube is a VM running on your machine (which cannot be accessed from outside your machine).
So you can try: Virtualbox "port forward" from Guest to Host.
Another alternative is to forward a port from your localhost to a pod directly (not the k8s svc unfortunately) by using kubectl port-forward.
You have not specified nodePort in ports.
Add below configuration in port
nodePort: 30000
You can access your service at http://[IP address]:30000