Remote access Zero to JupyterHub over Ethernet with ingress in Kubernetes - kubernetes

Context
I installed Kubernetes on a bare-metal server (4 nodes) and deployed Zero to JupyterHub to it.
This works fine; I can correctly access the hub from the master-node.
Now I want to access the Hub on the server from an external computer via Ethernet. Therefore, I followed the official instructions and installed MetalLB in order to provide an external IP for my proxy-public-service (which correctly sets).
Additionally, I installed the nginx-ingress-controller in order to be able to do an ingress, which also successfully gets an external IP (little hint: Use the Helm-chart; I couldn't get the service running when applying the other recommended steps).
Since I had a little trouble figuring out how to do this ingress, here is an example:
kubectl apply -f ingress.yaml --namespace jhub
#ingress.yaml:
#apiVersion: networking.k8s.io/v1beta1
#kind: Ingress
#metadata:
# name: jupyterhub-ingress
# annotations:
# nginx.ingress.kubernetes.io/rewrite-target: /$1
#spec:
# rules:
# - host: jupyterhub.cluster
# http:
# paths:
# - path: /
# backend:
# serviceName: proxy-public
# servicePort: 80
Anyhow, I cannot open the external IP proxy-public provides (meaning I'm inserting the external IP in my browser).
Question
How can I remotely access my JupyterHub over the external IP; what am I missing?

I missed that this can be achieved in the same way as with the Kubernetes-Dashboard: You have to establish an open ssh-connection (hence, open a tunnel -> tunneling) from the external computer.
Of course this is not the "exernal" access I had in mind, but a working and fast solution for my test-environment (and maybe yours).
How to establish this ssh-connect
First, get the external IP-address of your proxy-public:
$: kubectl get services --namespace jhub
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hub ClusterIP 10.99.241.72 <none> 8081/TCP 95m
proxy-api ClusterIP 10.107.175.27 <none> 8001/TCP 95m
proxy-public LoadBalancer 10.102.171.162 192.168.1.240 80:31976/TCP,443:32568/TCP 95m
Note: The range of the external IP was defined in my layer2 in my MetalLB-config.
Using this information (and assuming you're on Linux), open a terminal and use the following command:
$ ssh pi#10.10.10.2 -L 8000:192.168.1.240:80
# -L opens a localhost-connection
# pi#10.10.10.2 logs me into my second node with user pi
Note1: That localhost:8000 is configured as targetPort for proxy-public with http can also be seen when you describe the service and take a look at the specs respectively ports (you can also get the settings for https there):
kind: Service
apiVersion: v1
metadata:
name: proxy-public
namespace: jhub
...
spec:
ports:
- name: http
protocol: TCP
port: 80
targetPort: 8000
nodePort: 31976
- name: https
...
Finally, type http://localhost:8000/ into your browser - et voila, you get to your JupyterHub login-page!

Related

minikube ingress not resolving in windows

How do I expose an ingress when running kubernetes with minikube in windows 10?
I have enabled the minikube ingress add on.
My ingress is running here...
NAME CLASS HOSTS ADDRESS PORTS AGE
helmtest-ingress nginx helmtest.info 192.168.49.2 80 37m
I have added my hosts entry...
192.168.49.2 helmtest.info
I just get nothing when attempting to browse or ping either 192.168.49.2 or helmtest.info
My ingress looks like the following
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: helmtest-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
rules:
- host: helmtest.info
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: helmtest-service
port:
number: 80
My service looks like the following...
apiVersion: v1
kind: Service
metadata:
name: helmtest-service
labels:
app: helmtest-service
spec:
type: ClusterIP
selector:
app: helmtest
ports:
- port: 80
targetPort: 80
protocol: TCP
I can access my service successfully in the browser after running minikube service helmtest-service --url
If I run minikube tunnel it just hangs here....
minikube tunnel
❗ Access to ports below 1024 may fail on Windows with OpenSSH clients older than v8.1. For more information, see: https://minikube.sigs.k8s.io/docs/handbook/accessing/#access-to-ports-1024-on-windows-requires-root-permission
🏃 Starting tunnel for service helmtest-ingress.
Where am I going wrong here?
OP didn't provide further information so I will provide answer based on the current information.
You can run Ingress on Minikube using the $ minikube addons enable ingress command. However, ingress has more addons, like Ingress DNS using minikube addons enabled ingress-dns. In Minikube documentation you can find more details about this addon and when you should use it.
Minikube has quite a well described section about tunnel. Quite important fact about the tunnel is that it must be run in a separate terminal window to keep the LoadBalancer running.
Services of type LoadBalancer can be exposed via the minikube tunnel command. It must be run in a separate terminal window to keep the LoadBalancer running. Ctrl-C in the terminal can be used to terminate the process at which time the network routes will be cleaned up.
This part is described in Accessing apps documentation.
As OP mention
I can access my service successfully in the browser after running minikube service helmtest-service --url
If I run minikube tunnel it just hangs here....
Possible Solution
You might use the old version of SSH, update it.
You are using ports <1024. This situation it's described in this known issue part. Try to use higher port like 5000 like in this example
It might look like it just hangs, but you need a separate terminal window. Maybe it works correctly but you have to use another terminal
Useful links
How do I expose ingress to my local machine? (minikube on windows)
Cannot export a IP in minikube and haproxy loadBalancer - using minikube tunnel
It might be the host file missing minikube ip address with your host name. If ingress cannot resolve the hostname you set in yaml file it just stays in the schedule to sync phase
Similar answer

Treafik Let's encrypt simplest example on GKE

I am trying to work on simplest possible example of implementing let's encrypt with Traefik on GKE using this article. I have made some changes to suit my requirement but I am unable to get the ACME certificate.
What I have done so far
Run the following command and create all the resource objects except ingress-route
$ kubectl apply -f 00-resource-crd-definition.yml,05-traefik-rbac.yml,10-service-account.yaml,15-traefik-deployment.yaml,20-traefik-service.yaml,25-whoami-deployment.yaml,30-whoami-service.yaml
customresourcedefinition.apiextensions.k8s.io/ingressroutes.traefik.containo.us created
customresourcedefinition.apiextensions.k8s.io/middlewares.traefik.containo.us created
customresourcedefinition.apiextensions.k8s.io/ingressroutetcps.traefik.containo.us created
customresourcedefinition.apiextensions.k8s.io/tlsoptions.traefik.containo.us created
customresourcedefinition.apiextensions.k8s.io/traefikservices.traefik.containo.us created
clusterrole.rbac.authorization.k8s.io/traefik-ingress-controller created
clusterrolebinding.rbac.authorization.k8s.io/traefik-ingress-controller created
serviceaccount/traefik-ingress-controller created
deployment.apps/traefik created
service/traefik created
deployment.apps/whoami created
service/whoami created
Get the IP of the Traefik Service exposed as Load Balancer
$ kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.109.0.1 <none> 443/TCP 6h16m
traefik LoadBalancer 10.109.15.230 34.69.16.102 80:32318/TCP,443:32634/TCP,8080:32741/TCP 70s
whoami ClusterIP 10.109.14.91 <none> 80/TCP 70s
Create a DNS record for this IP
$ nslookup k8sacmetest.gotdns.ch
Server: 192.168.1.1
Address: 192.168.1.1#53
Non-authoritative answer:
Name: k8sacmetest.gotdns.ch
Address: 34.69.16.102
Create the resource ingress-route
$ kubectl apply -f 35-ingress-route.yaml
ingressroute.traefik.containo.us/simpleingressroute created
ingressroute.traefik.containo.us/ingressroutetls created
Logs of traefik
time="2020-04-25T20:10:31Z" level=info msg="Configuration loaded from flags."
time="2020-04-25T20:10:32Z" level=error msg="subset not found for default/whoami" providerName=kubernetescrd ingress=simpleingressroute namespace=default
time="2020-04-25T20:10:32Z" level=error msg="subset not found for default/whoami" providerName=kubernetescrd ingress=ingressroutetls namespace=default
time="2020-04-25T20:10:52Z" level=error msg="Unable to obtain ACME certificate for domains \"k8sacmetest.gotdns.ch\": unable to generate a certificate for the domains [k8sacmetest.gotdns.ch]: acme: Error -> One or more domains had a problem:\n[k8sacmetest.gotdns.ch] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Timeout during connect (likely firewall problem), url: \n" routerName=default-ingressroutetls-08dd2bb9eecaa72a6606#kubernetescrd rule="Host(`k8sacmetest.gotdns.ch`) && PathPrefix(`/tls`)" providerName=default.acme
What i have acheived
Traefik Dashboard
link
Whoami with notls
link
NOT ABLE TO GET THE ACME CERTIFICATE USING FOR TLS WHOAMI
my-pain
INFRA Details
I am using Google Kubernetes Cluster (the one being talked about here -cloud.google.com/kubernetes-engine, click on Go to Console).
Traefik version is 2.2.
And I am using "CloudShell" to access the cluster".
ASK:
1) Where am i going wrong to get the TLS Certificate?
2) If its firewall issue how to resolve?
3) If you have any other better example for Treafik Let's encrypt simplest example on GKE please let me know
Just run sudo before kubectl port-forward command. You are trying to bind to privileged ports, so you need more permissions.
It is not the simplest example for GKE, because you could use GKE LoadBalnacer instead of kubectl port-forward.
Try with this:
apiVersion: v1
kind: Service
metadata:
name: traefik
spec:
ports:
- protocol: TCP
name: web
port: 80
targetPort: web
- protocol: TCP
name: websecure
port: 443
targetPort: websecure
selector:
app: traefik
type: LoadBalancer
Then you can find your new IP with kubectl get svc in EXTERNAL-IP column, add proper DNS record for your domain and you should be fine.

Kubernetes: Handle connections with multiple LoadBalancer in cluster via traefik ingress controller

It might be hard to explain so sorry if ı can not explain correctly.
In our k8s cluster we have two OpenStack-Load Balancer because we would like to expose our application through ingress which has to be internet facing. In same cluster we also deployed pgadmin4 which has to be intranet facing.(only reachable from internal network.)
So in front of these OpenStack-LB, we have also f5 Load Balancer which handle https connection,ssl .. and also logic to expose via intranet or internet.
MyApp is internet facing and needs to reachable with host.internet.net
PgAdmin4 is intranet and needs to reachable via host.intranet.net/pgadmin4
So the issue is, when I try to expose my application through ingress using host.internet.net it won't works and ı received below error cause probably it can not able to communicate with correct openStack-LB. When ı tried to expose via openStack-lb IP everything works properly.
{"level":"error","msg":"Service not found for
dev/oneapihub-ui-dev","time":"2020-03-26T05:20:05Z"}
{"level":"error","msg":"endpoints not found for
dev/oneapihub-ui-dev","time":"2020-03-26T05:20:05Z"}
And the question is , how can I handle this issue via ingress controller? Should I intall another traefik ingress controller?
capel0068340585:~ semural$ kubectl get ingress -n ingress
NAME HOSTS ADDRESS PORTS AGE
ingress-traefik-dashboard * 80 21d
kubectl get tenantSpec -o yaml
loadBalancers:
- ip: <IP1>
name: LBaaS2
ports:
- extPort: 80
name: "80"
nodePort: 30001
- ip: <IP2>
name: LBaaS1
ports:
- extPort: 80
name: "80"
nodePort: 30000
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/oneapihub-ui-dev ClusterIP 10.254.173.130 <none> 80/TCP 15m
NAME ENDPOINTS AGE
endpoints/oneapihub-ui-dev 10.6.24.136:3000 15m
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: traefik
hosts:
- host: host.internet.net -> example
paths: [/]
tls: []
ingress:
enabled: ingress
annotations:
kubernetes.io/ingress.class: traefik
hosts:
- host: host.intranet.net
paths:
- /pgadmin4
You error state "Service not found for dev/oneapihub-ui-dev", which means traefik is trying to connect to a Service in the dev namespace called "oneapihub-ui-dev" which it cannot find.
You need to make sure that both the Service exists and that it has endpoints. You can check if the Service exists with kubectl -n dev get service oneapihub-ui-dev. If it exists, check if it has endpoints with kubectl -n dev get ep oneapihub-ui-dev.
EDIT: If the Service exists and has Endpoints, than you may want to look into the RBAC permissions of traefik to see if it has enough permissions to look in the dev namespace and if you do not deploy any NetworkPolicies on the dev namespace that prevent the ingress namespace from connecting.
I solved this issue using via using labelSelector for traefik.. The the services that I'd expose only for internal networking has a label such as traffic-type=internal.. You could also provide a namespace for RBAC permissions.
kubernetes:
namespaces:
- default
- database
- monitoring
- logging
- ingress
labelSelector: "traffic-type=internal"

Minikube with ingress example not working

I'm trying to get an ingress controller working in Minikube and am following the steps in the K8s documentation here, but am seeing a different result in that the IP address for the ingress controller is different than that for Minikube (the example seems to indicate they should be the same):
$ kubectl get ingress
NAME HOSTS ADDRESS PORTS AGE
example-ingress hello-world.info 10.0.2.15 80 12m
$ minikube ip
192.168.99.101
When I try to connect to the Minikube IP address (using the address directly vs. adding it to my local hosts file), I'm getting a "Not found" response from NGINX:
$ curl http://`minikube ip`/
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>openresty/1.15.8.1</center>
</body>
</html>
When I try to connect to the IP address associated with the ingress controller, it just hangs.
Should I expect the addresses to be the same as the K8s doc indicates?
Some additional information:
$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
minikube Ready master 2d23h v1.16.0 10.0.2.15 <none> Buildroot 2018.05.3 4.15.0 docker://18.9.9
$ kubectl get ingresses example-ingress -o yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"networking.k8s.io/v1beta1","kind":"Ingress","metadata":{"annotations":{"nginx.ingress.kubernetes.io/rewrite-target":"/$1"},"name":"example-ingress","namespace":"default"},"spec":{"rules":[{"host":"hello-world.info","http":{"paths":[{"backend":{"serviceName":"web","servicePort":8080},"path":"/"}]}}]}}
nginx.ingress.kubernetes.io/rewrite-target: /$1
creationTimestamp: "2019-10-28T15:36:57Z"
generation: 1
name: example-ingress
namespace: default
resourceVersion: "25609"
selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/example-ingress
uid: 5e96c378-fbb1-4e8f-9738-3693cbce7d9b
spec:
rules:
- host: hello-world.info
http:
paths:
- backend:
serviceName: web
servicePort: 8080
path: /
status:
loadBalancer:
ingress:
- ip: 10.0.2.15
Here’s what worked for me:
minikube start
minikube addons enable ingress
minikube addons enable ingress-dns
Wait until you see the ingress-nginx-controller-XXXX is up and running using Kubectl get pods -n ingress-nginx
Create an ingress using the K8s example yaml file
Update the service section to point to the NodePort Service that you already created
Append
127.0.0.1 hello-world.info
to your /etc/hosts file on MacOS (NOTE:
Do NOT use the Minikube IP)
Run minikube tunnel ( Keep the window open. After you entered the password there will be no more messages, and the cursor just blinks)
Hit the hello-world.info ( or whatever host you configured in the yaml file) in a browser and it should work
I've reproduced your scenario in a Linux environment (on GCP) and I also have different IPs:
user#bf:~$ minikube ip
192.168.39.144
user#bf:~$ kubectl get ingresses
NAME HOSTS ADDRESS PORTS AGE
example-ingress * 192.168.122.173 80 30m
Your problem is not related to the fact you have different IPs. The guide instructs us to create an ingress with the following rule:
spec:
rules:
- host: hello-world.info
This rule is telling the ingress service that a DNS record with hello-world.info name is expected.
If you follow the guide a bit further, it instructs you to create an entry on your hosts file pointing to your ingress IP or Minikube IP.
Note: If you are running Minikube locally, use minikube ip to get the external IP. The IP address displayed within the ingress list
will be the internal IP.
Source: Set up Ingress on Minikube with the NGINX Ingress Controller
(if you want to curl the IP instead of DNS name, you need to remove the host rule from your ingress)
It should look like this:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: example-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- http:
paths:
- path: /
backend:
serviceName: web
servicePort: 8080
Apply your changes:
user#bf:~$ kubectl apply -f example-ingress.yaml
And curl the IP using -Lk options to surpass problems related to secure connections.
user#bf:~$ curl -Lk 192.168.39.144
Hello, world!
Version: 1.0.0
Hostname: web-9bbd7b488-l5gc9
In addition to the accepted answer, minikube now has a tunnel command which allows you generate external ip addresses for your services which can be accessed directly on your host machine without using the general minikube ip.
Run minikube tunnel in a separate terminal. This runs in the foreground as a daemon.
In a different terminal, execute your kubectl apply -f <file_name> command to deploy your desired service. It should generate an ip address for you that is routed directly to your service and available on port 80 on that address.
More here on the minikube documentation: https://minikube.sigs.k8s.io/docs/tasks/loadbalancer/
I got Minikube on Windows 11 to work for me
minikube start --vm-driver=hyperv
Install minikube Ingress Controller
minikube addons enable ingress
minikube addons enable ingress-dns
Deploy Helm Chart
helm install ...
Get Kubernetes IP Address
nslookup <host-found-in-ingress> $(minikube ip)
Add to etc/host
<minikube-ip> <domain-url>
Live!
curl <domain-url>

Kubernetes service is reachable from node but not from my machine

I have a timeout problem with my site hosted on Kubernetes cluster provided by DigitalOcean.
u#macbook$ curl -L fork.example.com
curl: (7) Failed to connect to fork.example.com port 80: Operation timed out
I have tried everything listed on the Debug Services page. I use a k8s service named df-stats-site.
u#pod$ nslookup df-stats-site
Server: 10.245.0.10
Address: 10.245.0.10#53
Name: df-stats-site.deepfork.svc.cluster.local
Address: 10.245.16.96
It gives the same output when I do it from node:
u#node$ nslookup df-stats-site.deepfork.svc.cluster.local 10.245.0.10
Server: 10.245.0.10
Address: 10.245.0.10#53
Name: df-stats-site.deepfork.svc.cluster.local
Address: 10.245.16.96
With the help of Does the Service work by IP? part of the page, I tried the following command and got the expected output.
u#node$ curl 10.245.16.96
*correct response*
Which should mean that everything is fine with DNS and service. I confirmed that kube-proxy is running with the following command:
u#node$ ps auxw | grep kube-proxy
root 4194 0.4 0.1 101864 17696 ? Sl Jul04 13:56 /hyperkube proxy --config=...
But I have something wrong with iptables rules:
u#node$ iptables-save | grep df-stats-site
(unfortunately, I was not able to copy the output from node, see the screenshot below)
It is recommended to restart kube-proxy with with the -v flag set to 4, but I don't know how to do it with DigitalOcean provided cluster.
That's the configuration I use:
apiVersion: v1
kind: Service
metadata:
name: df-stats-site
spec:
ports:
- port: 80
targetPort: 8002
selector:
app: df-stats-site
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: df-stats-site
annotations:
kubernetes.io/ingress.class: nginx
certmanager.k8s.io/cluster-issuer: letsencrypt-prod
spec:
tls:
- hosts:
- fork.example.com
secretName: letsencrypt-prod
rules:
- host: fork.example.com
http:
paths:
- backend:
serviceName: df-stats-site
servicePort: 80
Also, I have a NGINX Ingress Controller set up with the help of this answer.
I must note that it worked fine before. I'm not sure what caused this, but restarting the cluster would be great, though I don't know how to do it without removing all the resources.
The solution for me was to add HTTP and HTTPS inbound rules in the Firewall (these are missing by default).
For DigitalOcean provided Kubernetes cluster, you can open it at https://cloud.digitalocean.com/networking/firewalls/.
UPDATE: Make sure to create a new firewall record rather than editing an existing one. Otherwise, your rules will be automatically removed in a couple of hours/days, because DigitalOcean k8s persists the set of rules in the firewall.
ClusterIP services are only accessible from within the cluster. If you want to access it from outside the cluster, it needs to be configured as NodePort or LoadBalancer.
If you are just trying to test something locally, you can use kubectl port-forward to forward a port on your local machine to a ClusterIP service on a remote cluster. Here's an example of creating a deployment from an image, exposing it as a ClusterIP service, then accessing it via kubectl port-forward:
$ kubectl run --image=rancher/hello-world hello-world --replicas 2
$ kubectl expose deployment hello-world --type=ClusterIP --port=8080 --target-port=80
$ kubectl port-forward svc/hello-world 8080:8080
This service is now accessible from my local computer at http://127.0.0.1:8080