I am trying to run a local cluster on Mac with M1 chip using Minikube (Docker driver). I enabled ingress addon in Minikube, I have a separate terminal in which I'm running minikube tunnel and I enabled Minikube dashboard, which I want to expose using Ingress.
This is my configuration file:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: dashboard-ingress
namespace: kubernetes-dashboard
spec:
rules:
- host: dashboard.com
http:
paths:
- backend:
service:
name: kubernetes-dashboard
port:
number: 80
pathType: Prefix
path: /
I also put "dashboard.com" in my /etc/hosts file and it's actually resolving to the right IP, but it's not responding when I put "http://dashboard.com" in a browser or when I try to ping it and I always receive a timeout.
NOTE: when I run minikube tunnel I get
❗ The service/ingress dashboard-ingress requires privileged ports to be exposed: [80 443]
🔑 sudo permission will be asked for it.
I insert my sudo password and then nothing gets printed afterwards. Not sure if this is is an issue or the expected behavior.
What am I doing wrong?
I had the same behavior, and apparently what's needed for minikube tunnel to work is to map "127.0.0.1" in /etc/hosts, instead of the output from minikube ip or the ingress description.
that fixed it for me
had a similar issue on mac m1, initialy tried addon ingress-dns but then realised while it can be enabled its not currently working or supported using the docker driver https://github.com/kubernetes/minikube/issues/7332#issuecomment-608133325
some other mac intel users have got it working using hyperkit driver but thats not available for mac m1 yet
my answer for now is to use minikube tunnel https://minikube.sigs.k8s.io/docs/handbook/accessing/ and add entry to /etc/hosts for ingress, also have to pass in the cluster name using the -p parameter eg: minikube tunnel --cleanup -p <CLUSTER_NAME>
Related
How do I expose an ingress when running kubernetes with minikube in windows 10?
I have enabled the minikube ingress add on.
My ingress is running here...
NAME CLASS HOSTS ADDRESS PORTS AGE
helmtest-ingress nginx helmtest.info 192.168.49.2 80 37m
I have added my hosts entry...
192.168.49.2 helmtest.info
I just get nothing when attempting to browse or ping either 192.168.49.2 or helmtest.info
My ingress looks like the following
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: helmtest-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
rules:
- host: helmtest.info
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: helmtest-service
port:
number: 80
My service looks like the following...
apiVersion: v1
kind: Service
metadata:
name: helmtest-service
labels:
app: helmtest-service
spec:
type: ClusterIP
selector:
app: helmtest
ports:
- port: 80
targetPort: 80
protocol: TCP
I can access my service successfully in the browser after running minikube service helmtest-service --url
If I run minikube tunnel it just hangs here....
minikube tunnel
❗ Access to ports below 1024 may fail on Windows with OpenSSH clients older than v8.1. For more information, see: https://minikube.sigs.k8s.io/docs/handbook/accessing/#access-to-ports-1024-on-windows-requires-root-permission
🏃 Starting tunnel for service helmtest-ingress.
Where am I going wrong here?
OP didn't provide further information so I will provide answer based on the current information.
You can run Ingress on Minikube using the $ minikube addons enable ingress command. However, ingress has more addons, like Ingress DNS using minikube addons enabled ingress-dns. In Minikube documentation you can find more details about this addon and when you should use it.
Minikube has quite a well described section about tunnel. Quite important fact about the tunnel is that it must be run in a separate terminal window to keep the LoadBalancer running.
Services of type LoadBalancer can be exposed via the minikube tunnel command. It must be run in a separate terminal window to keep the LoadBalancer running. Ctrl-C in the terminal can be used to terminate the process at which time the network routes will be cleaned up.
This part is described in Accessing apps documentation.
As OP mention
I can access my service successfully in the browser after running minikube service helmtest-service --url
If I run minikube tunnel it just hangs here....
Possible Solution
You might use the old version of SSH, update it.
You are using ports <1024. This situation it's described in this known issue part. Try to use higher port like 5000 like in this example
It might look like it just hangs, but you need a separate terminal window. Maybe it works correctly but you have to use another terminal
Useful links
How do I expose ingress to my local machine? (minikube on windows)
Cannot export a IP in minikube and haproxy loadBalancer - using minikube tunnel
It might be the host file missing minikube ip address with your host name. If ingress cannot resolve the hostname you set in yaml file it just stays in the schedule to sync phase
Similar answer
I am trying to run the bookinfo example on my local with wsl2 and docker desk. I am having issues when trying to access the productpage service via the gateway as I got the connection refused. I am not sure whether I missed anything. Here is what I have done after googled a lot on the internet
Deployed all services from bookinfo example and all up running, I can curl productpage from other service using kubectl exec
Deployed bookinfo-gateway using the file from the example without any change under the default namespace
Name: bookinfo-gateway
Namespace: default
Labels: <none>
Annotations: <none>
API Version: networking.istio.io/v1beta1
Kind: Gateway
Metadata:
Creation Timestamp: 2021-06-06T20:47:18Z
Generation: 1
Managed Fields:
API Version: networking.istio.io/v1alpha3
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:selector:
.:
f:istio:
f:servers:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2021-06-06T20:47:18Z
Resource Version: 2053564
Self Link: /apis/networking.istio.io/v1beta1/namespaces/default/gateways/bookinfo-gateway
UID: aa390a1d-2e34-4599-a1ec-50ad7aa9bdc6
Spec:
Selector:
Istio: ingressgateway
Servers:
Hosts:
*
Port:
Name: http
Number: 80
Protocol: HTTP
Events: <none>
The istio-ingressgateway can expose to the outside via localhost (not sure how this can be configured as it is deployed during istio installation) on 80, which I as understand will be used by bookinfo-gateway
kubectl get svc istio-ingressgateway -n istio-system
following Determining the ingress IP and ports section in the instruction.
My INGRESS_HOST=127.0.0.1 and INGRESS_PORT is 80
curl -v -s http://127.0.0.1:80/productpage | grep -o ".*"
* Trying 127.0.0.1:80...
* TCP_NODELAY set
* connect to 127.0.0.1 port 80 failed: Connection refused
* Failed to connect to 127.0.0.1 port 80: Connection refused
* Closing connection 0
trying this http://127.0.0.1/productpage on browser, return 404. Does this 404 mean the gateway is kind of up but virtual service is not working??
further question if it is relevant. I am a bit confusing how wsl2 works now. It looks like localhost on windows browser and wsl2 terminal are not the same thing, though I know there is kind of forwarding from windows to wsl2 server (which I can get its IP from /etc/resolv.conf). if it is the same, why one return connection refused and the other return 404
On windows I have tried to disable IIS or anything running on port 80 (net stop http). Somehow, I still can see something listen to port 80
netstat -aon | findstr :80
TCP 0.0.0.0:80 0.0.0.0:0 LISTENING 4
tasklist /svc /FI "PID eq 4"
Image Name PID Services
========================= ======== ============================================
System 4 N/A
I am wondering whether this is what causes the difference in point 7? As windows is running on another http server on port 80?
I know this a lot of questions asked. I believe many of us that new to istio and wsl2 may have similar questions. Hopefully, this helps others as well. Please advise.
There seems to be a problem with WSL2 itself, probably connected with Local sites running in WSL2 not accessible in browser #5298.
You can work around that by issuing
ip addr show
in your WSL distro, and replacing 127.0.0.1/localhost with eth0 address. In my case it is 172.21.29.254 - so URL is http://172.21.29.254/productpage
This workaround worked for me.
I managed to get this working: This is what I did.
Shell into the distro (mine was Ubuntu 20.04 LTS)
Run:
sudo apt-get -y install socat
sudo apt update
sudo apt upgrade
exit
The above will add socat (which I was getting errors with when looking at the istio logs - connection refused) and update the distro to the latest updates (and upgrade them)
Now you have to run a port-forward to be able to host localhost: to hit the istio gateway with:
kubectl port-forward svc/istio-ingressgateway 8080:80 -n istio-system
If you are already using 8080, just remove it from the command, just use :80 and the port forward will select a free port.
Now go to
http://localhost:8080/productpage
You should hit the page and the port-forward should output
Handling connection for 8080
Hope that helps...
Good thing is now I don't have to use Hyper-V or another cluster installer like minikube/microk8s and use the built-in kubernetes in docker desktop and... my laptop doesn't seem under load for what I'm doing too.
I'm trying to do a straight up thing that I would think is simple. I need to have https://localhost:44301, https://localhost:5002, https://localhost:5003 to be listened to in my k8s environment in docker desktop, and be proxied using a pfx file/password that I specify and have it forward by the port to pods listening on specific addresses (could be port 80, doesn't matter)
The documentation is mind numbingly complex for what looks like it should be straight forward. I can get the pods running, I can use kubectl port-forward and they work fine, but I can't figure out how to get ingress working with ha-proxy or nginx or anything else in a way that makes any sense.
Can someone do an ELI5 telling me how to turn this on? I'm on Windows 10 2004 with WSL2 and Docker experimental so I should have access to the ingress stuff they reference in the docs and make clear as mud.
Thanks!
As discussed in the comments this is a community wiki answer:
I have managed to create Ingress resource in Kubernetes on Docker in Windows.
Steps to reproduce:
Enable Hyper-V
Install Docker for Windows and enable Kubernetes
Connect kubectl
Enable Ingress
Create deployment
Create service
Create ingress resource
Add host into local hosts file
Test
Enable Hyper-V
From Powershell with administrator access run below command:
Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V -All
System could ask you to reboot your machine.
Install Docker for Windows and enable Kubernetes
Install Docker application with all the default options and enable Kubernetes
Connect kubectl
Install kubectl .
Enable Ingress
Run this commands:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud-generic.yaml
Edit: Make sure no other service is using port 80
Restart your machine. From a cmd prompt running as admin, do:
net stop http
Stop the listed services using services.msc
Use: netstat -a -n -o -b and check for other processes listening on port 80.
Create deployment
Below is simple deployment with pods that will reply to requests:
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello
spec:
selector:
matchLabels:
app: hello
version: 2.0.0
replicas: 3
template:
metadata:
labels:
app: hello
version: 2.0.0
spec:
containers:
- name: hello
image: "gcr.io/google-samples/hello-app:2.0"
env:
- name: "PORT"
value: "50001"
Apply it by running command:
$ kubectl apply -f file_name.yaml
Create service
For pods to be able for you to communicate with them you need to create a service.
Example below:
apiVersion: v1
kind: Service
metadata:
name: hello-service
spec:
type: NodePort
selector:
app: hello
version: 2.0.0
ports:
- name: http
protocol: TCP
port: 80
targetPort: 50001
Apply this service definition by running command:
$ kubectl apply -f file_name.yaml
Create Ingress resource
Below is simple Ingress resource using service created above:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: hello-ingress
spec:
rules:
- host: kubernetes.docker.internal
http:
paths:
- path: /
backend:
serviceName: hello-service
servicePort: http
Take a look at:
spec:
rules:
- host: hello-test.internal
hello-test.internal will be used as the hostname to connect to your pods.
Apply your Ingress resource by invoking command:
$ kubectl apply -f file_name.yaml
Add host into local hosts file
I found this Github link that will allow you to connect to your Ingress resource by hostname.
To achieve that add a line 127.0.0.1 hello-test.internal to your C:\Windows\System32\drivers\etc\hosts file and save it.
You will need Administrator privileges to do that.
Edit: The newest version of Docker Desktop for Windows already adds a hosts file entry:
127.0.0.1 kubernetes.docker.internal
Test
Display the information about Ingress resources by invoking command:
kubectl get ingress
It should show:
NAME HOSTS ADDRESS PORTS AGE
hello-ingress hello-test.internal localhost 80 6m2s
Now you can access your Ingress resource by opening your web browser and typing
http://kubernetes.docker.internal/
The browser should output:
Hello, world!
Version: 2.0.0
Hostname: hello-84d554cbdf-2lr76
Hostname: hello-84d554cbdf-2lr76 is the name of the pod that replied.
If this solution is not working please check connections with the command:
netstat -a -n -o
(with Administrator privileges) if something is not using port 80.
I'm trying to get an ingress controller working in Minikube and am following the steps in the K8s documentation here, but am seeing a different result in that the IP address for the ingress controller is different than that for Minikube (the example seems to indicate they should be the same):
$ kubectl get ingress
NAME HOSTS ADDRESS PORTS AGE
example-ingress hello-world.info 10.0.2.15 80 12m
$ minikube ip
192.168.99.101
When I try to connect to the Minikube IP address (using the address directly vs. adding it to my local hosts file), I'm getting a "Not found" response from NGINX:
$ curl http://`minikube ip`/
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>openresty/1.15.8.1</center>
</body>
</html>
When I try to connect to the IP address associated with the ingress controller, it just hangs.
Should I expect the addresses to be the same as the K8s doc indicates?
Some additional information:
$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
minikube Ready master 2d23h v1.16.0 10.0.2.15 <none> Buildroot 2018.05.3 4.15.0 docker://18.9.9
$ kubectl get ingresses example-ingress -o yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"networking.k8s.io/v1beta1","kind":"Ingress","metadata":{"annotations":{"nginx.ingress.kubernetes.io/rewrite-target":"/$1"},"name":"example-ingress","namespace":"default"},"spec":{"rules":[{"host":"hello-world.info","http":{"paths":[{"backend":{"serviceName":"web","servicePort":8080},"path":"/"}]}}]}}
nginx.ingress.kubernetes.io/rewrite-target: /$1
creationTimestamp: "2019-10-28T15:36:57Z"
generation: 1
name: example-ingress
namespace: default
resourceVersion: "25609"
selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/example-ingress
uid: 5e96c378-fbb1-4e8f-9738-3693cbce7d9b
spec:
rules:
- host: hello-world.info
http:
paths:
- backend:
serviceName: web
servicePort: 8080
path: /
status:
loadBalancer:
ingress:
- ip: 10.0.2.15
Here’s what worked for me:
minikube start
minikube addons enable ingress
minikube addons enable ingress-dns
Wait until you see the ingress-nginx-controller-XXXX is up and running using Kubectl get pods -n ingress-nginx
Create an ingress using the K8s example yaml file
Update the service section to point to the NodePort Service that you already created
Append
127.0.0.1 hello-world.info
to your /etc/hosts file on MacOS (NOTE:
Do NOT use the Minikube IP)
Run minikube tunnel ( Keep the window open. After you entered the password there will be no more messages, and the cursor just blinks)
Hit the hello-world.info ( or whatever host you configured in the yaml file) in a browser and it should work
I've reproduced your scenario in a Linux environment (on GCP) and I also have different IPs:
user#bf:~$ minikube ip
192.168.39.144
user#bf:~$ kubectl get ingresses
NAME HOSTS ADDRESS PORTS AGE
example-ingress * 192.168.122.173 80 30m
Your problem is not related to the fact you have different IPs. The guide instructs us to create an ingress with the following rule:
spec:
rules:
- host: hello-world.info
This rule is telling the ingress service that a DNS record with hello-world.info name is expected.
If you follow the guide a bit further, it instructs you to create an entry on your hosts file pointing to your ingress IP or Minikube IP.
Note: If you are running Minikube locally, use minikube ip to get the external IP. The IP address displayed within the ingress list
will be the internal IP.
Source: Set up Ingress on Minikube with the NGINX Ingress Controller
(if you want to curl the IP instead of DNS name, you need to remove the host rule from your ingress)
It should look like this:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: example-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- http:
paths:
- path: /
backend:
serviceName: web
servicePort: 8080
Apply your changes:
user#bf:~$ kubectl apply -f example-ingress.yaml
And curl the IP using -Lk options to surpass problems related to secure connections.
user#bf:~$ curl -Lk 192.168.39.144
Hello, world!
Version: 1.0.0
Hostname: web-9bbd7b488-l5gc9
In addition to the accepted answer, minikube now has a tunnel command which allows you generate external ip addresses for your services which can be accessed directly on your host machine without using the general minikube ip.
Run minikube tunnel in a separate terminal. This runs in the foreground as a daemon.
In a different terminal, execute your kubectl apply -f <file_name> command to deploy your desired service. It should generate an ip address for you that is routed directly to your service and available on port 80 on that address.
More here on the minikube documentation: https://minikube.sigs.k8s.io/docs/tasks/loadbalancer/
I got Minikube on Windows 11 to work for me
minikube start --vm-driver=hyperv
Install minikube Ingress Controller
minikube addons enable ingress
minikube addons enable ingress-dns
Deploy Helm Chart
helm install ...
Get Kubernetes IP Address
nslookup <host-found-in-ingress> $(minikube ip)
Add to etc/host
<minikube-ip> <domain-url>
Live!
curl <domain-url>
I have a timeout problem with my site hosted on Kubernetes cluster provided by DigitalOcean.
u#macbook$ curl -L fork.example.com
curl: (7) Failed to connect to fork.example.com port 80: Operation timed out
I have tried everything listed on the Debug Services page. I use a k8s service named df-stats-site.
u#pod$ nslookup df-stats-site
Server: 10.245.0.10
Address: 10.245.0.10#53
Name: df-stats-site.deepfork.svc.cluster.local
Address: 10.245.16.96
It gives the same output when I do it from node:
u#node$ nslookup df-stats-site.deepfork.svc.cluster.local 10.245.0.10
Server: 10.245.0.10
Address: 10.245.0.10#53
Name: df-stats-site.deepfork.svc.cluster.local
Address: 10.245.16.96
With the help of Does the Service work by IP? part of the page, I tried the following command and got the expected output.
u#node$ curl 10.245.16.96
*correct response*
Which should mean that everything is fine with DNS and service. I confirmed that kube-proxy is running with the following command:
u#node$ ps auxw | grep kube-proxy
root 4194 0.4 0.1 101864 17696 ? Sl Jul04 13:56 /hyperkube proxy --config=...
But I have something wrong with iptables rules:
u#node$ iptables-save | grep df-stats-site
(unfortunately, I was not able to copy the output from node, see the screenshot below)
It is recommended to restart kube-proxy with with the -v flag set to 4, but I don't know how to do it with DigitalOcean provided cluster.
That's the configuration I use:
apiVersion: v1
kind: Service
metadata:
name: df-stats-site
spec:
ports:
- port: 80
targetPort: 8002
selector:
app: df-stats-site
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: df-stats-site
annotations:
kubernetes.io/ingress.class: nginx
certmanager.k8s.io/cluster-issuer: letsencrypt-prod
spec:
tls:
- hosts:
- fork.example.com
secretName: letsencrypt-prod
rules:
- host: fork.example.com
http:
paths:
- backend:
serviceName: df-stats-site
servicePort: 80
Also, I have a NGINX Ingress Controller set up with the help of this answer.
I must note that it worked fine before. I'm not sure what caused this, but restarting the cluster would be great, though I don't know how to do it without removing all the resources.
The solution for me was to add HTTP and HTTPS inbound rules in the Firewall (these are missing by default).
For DigitalOcean provided Kubernetes cluster, you can open it at https://cloud.digitalocean.com/networking/firewalls/.
UPDATE: Make sure to create a new firewall record rather than editing an existing one. Otherwise, your rules will be automatically removed in a couple of hours/days, because DigitalOcean k8s persists the set of rules in the firewall.
ClusterIP services are only accessible from within the cluster. If you want to access it from outside the cluster, it needs to be configured as NodePort or LoadBalancer.
If you are just trying to test something locally, you can use kubectl port-forward to forward a port on your local machine to a ClusterIP service on a remote cluster. Here's an example of creating a deployment from an image, exposing it as a ClusterIP service, then accessing it via kubectl port-forward:
$ kubectl run --image=rancher/hello-world hello-world --replicas 2
$ kubectl expose deployment hello-world --type=ClusterIP --port=8080 --target-port=80
$ kubectl port-forward svc/hello-world 8080:8080
This service is now accessible from my local computer at http://127.0.0.1:8080