Zeppelin Notebooks and Interpreters Missing Behind Kubernetes Ingress - kubernetes

I am running into an issue where the interpreters are missing after enabling the Kubernetes ingress to reach my Zeppelin container.
I'm running Zeppelin 0.8.2 on Kubernetes, and everything works correctly when it is not behind an ingress.
I am able to choose a default interpreter in the notebook creation popup, and run paragraphs without issue.
When an ingress is setup in Kubernetes, previously created notebooks and interpreter selection during new notebook creation are missing.
I am still able to reach the interpreter settings page, and it appears they are all still there.
Required change made in zeppelin-site.xml to run without the ingress.
<property>
<name>zeppelin.server.addr</name>
<value>0.0.0.0</value>
<description>Server binding address</description>
</property>
By default, was 127.0.0.0
Changes made in shiro.ini based on this forum discussion:
https://community.cloudera.com/t5/Support-Questions/How-do-I-recover-missing-Zeppelin-Interpreter-tab-and-its/m-p/160680
sessionManager = org.apache.shiro.web.session.mgt.DefaultWebSessionManager
cacheManager = org.apache.shiro.cache.MemoryConstrainedCacheManager
securityManager.cacheManager = $cacheManager
securityManager.sessionManager = $sessionManager
securityManager.sessionManager.globalSessionTimeout = 86400000
Ingress used below with variable names changed
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
ingress.kubernetes.io/ssl-redirect: "true"
name: zeppelin-name
namespace: namespace_here
spec:
rules:
- host: zeppelin.zeppelin-services.my-host.org
http:
paths:
- backend:
serviceName: zeppelin-name
servicePort: 8080
path: /
tls:
- hosts:
- zeppelin.zeppelin-services.my-host.org
secretName: secret-name
status:
loadBalancer: {}
What am I missing for Zeppelin to work correctly behind the Kubernetes Ingress, to allow me to see previously created notebooks, and interpreters present in the dropdown menu when creating new notebooks?

Coworker found the answer.
It turns out the information was not reaching the zeppelin page.
On inspection, there was a 405 error:
'wss://zeppelin.zeppelin-services.my-host.org/ws' failed: Error during WebSocket handshake: Unexpected response code: 405
He checked the logs on the nginx-ingress pod and found similar failure messages.
Issue was in the ingress yaml used and required an addition to the annotation
metadata:
annotations:
nginx.org/websocket-services: zeppelin-name

Related

Access a k8s service via Ingress-exposed browser app

I have an API and a GUI application.
The API is a simple Python FastAPI app with one route. The GUI is a React app that is served via nginx.
So I have a working local cluster with those two apps. Each has a service. GUI additionally has an Ingress. Using k3d, I am able to get to the GUI app, because I start the cluster with: k3d cluster create -p "8081:80#loadbalancer", so I can go to localhost:8081 and I see it. The Ingress config:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: foo-gui-ingress
annotations:
external-dns/manage-entries: "true"
external-dns.alpha.kubernetes.io/ttl: "300"
spec:
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: foo-gui-service
port:
number: 80
Service configuration are pretty much the same, just different names:
apiVersion: v1
kind: Service
metadata:
  name: foo-gui-service
  labels:
    app.kubernetes.io/name: foo-gui
    app.kubernetes.io/component: server
    app.kubernetes.io/version: {{ .Chart.AppVersion }}
    app.kubernetes.io/part-of: foo-gui
spec:
  selector:
    app: foo-gui
    component: foo-gui
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
GUI makes calls to the API. The API Service is called foo-api-service, so I want to make calls to http://foo-api-service:80/<route>. It works in a temporary pod that I install for debugging network issues (I can curl the API). It even works when I attach to the GUI Pod.
The problem is: it doesn't work on the GUI. I receive the following error in the dev console in the browser: POST http://foo-api-service/test net::ERR_NAME_NOT_RESOLVED. It looks like the http://foo-api-service is being resolved in my networking, not in the cluster's.
My question is: Is there a way to overcome that? To have http://foo-api-service calls to resolve this name inside the local cluster? Or should I provide Ingress for the API as well and use the domain name from API's Ingress as a URL in the GUI app?

How can I acces the Kubernetes Dashboard on a remote Machine

I am new at kubernetes and I am trying to setup a clsuter on a remote server.
For this I use microk8s and a server from hetzner-cloud (https://www.hetzner.com/de/cloud).
I entered the server with ssh and followd the microk8s installation instructions for linux(https://microk8s.io/). That all seemed to work fine. My problem is now that I have not found a way to access the kubernetes-dashboard.
I've tryed the workaround with NodePort and microk8s kubectl proxy --disable-filter = true but it does not work and is not recommended for security reasons. With the disable Filter Method it is possible to access the login page but it does not respond.
I've also tryed to access the dashbourd from outside using a ssh tunnel and this tutorial: How can I remotely access kubernetes dashboard with token
The tunnel seems to works fine, but I still cannot access the port.
Now I have two main questions:
1: How do you usualy use kubernetes, if kubernetes does not wan you to access the dashboard from outside. Because don't you run your services usualy on a rented server that is not in you living room? What's the point I simply do not get?
2: How can I access the Dashborad?
I would be really happy, if anybody could help me. I am already struggling with this problem since a month now. :)
best greetings,
mamo
In order to access the K8s services from outside using HTTP, you should configure and use ingress controller.
after ingress is running, you will be able to specify a "path" or route and a port and name that points to your service.
Once this is done, you should be able to access the dashboard
Sample configuration (reference)
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: dashboard-google
namespace: kube-system
annotations:
nginx.ingress.kubernetes.io/secure-backends: "true"
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
spec:
tls:
- hosts:
- kube.mydomain.com
secretName: tls-secret
rules:
- host: kube.mydomain.com
http:
paths:
- path: /
backend:
serviceName: kubernetes-dashboard
servicePort: 443

Nginx Ingress returns 502 Bad Gateway on Kubernetes

I have a Kubernetes cluster deployed on AWS (EKS). I deployed the cluster using the “eksctl” command line tool. I’m trying to deploy a Dash python app on the cluster without success. The default port for Dash is 8050. For the deployment I used the following resources:
pod
service (ClusterIP type)
ingress
You can check the resource configuration files below:
pod-configuration-file.yml
kind: Pod
apiVersion: v1
metadata:
name: dashboard-app
labels:
app: dashboard
spec:
containers:
- name: dashboard
image: my_image_from_ecr
ports:
- containerPort: 8050
service-configuration-file.yml
kind: Service
apiVersion: v1
metadata:
name: dashboard-service
spec:
selector:
app: dashboard
ports:
- port: 8050 # exposed port
targetPort: 8050
ingress-configuration-file.yml (host based routing)
kind: Ingress
metadata:
name: dashboard-ingress
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: dashboard.my_domain
http:
paths:
- backend:
serviceName: dashboard-service
servicePort: 8050
path: /
I followed the steps below:
kubectl apply -f pod-configuration-file.yml
kubectl apply -f service-configuration-file.yml
kubectl apply -f ingress-confguration-file.yml
I also noticed that the pod deployment works as expected:
kubectl logs my_pod:
and the output is:
Dash is running on http://127.0.0.1:8050/
Warning: This is a development server. Do not use app.run_server
in production, use a production WSGI server like gunicorn instead.
* Serving Flask app "annotation_analysis" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: on
You can see from the ingress configuration file that I want to do host based routing using my domain. For this to work, I have also deployed an nginx-ingress. I have also created an “A” record set using Route53
that maps the “dashboard.my_domain” to the nginx-ingress:
kubectl get ingress
and the output is:
NAME HOSTS ADDRESS. PORTS. AGE
dashboard-ingress dashboard.my_domain nginx-ingress.elb.aws-region.amazonaws.com 80 93s
Moreover,
kubectl describe ingress dashboard-ingress
and the output is:
Name: dashboard-ingress
Namespace: default
Address: nginx-ingress.elb.aws-region.amazonaws.com
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host Path Backends
---- ---- --------
host.my-domain
/ dashboard-service:8050 (192.168.36.42:8050)
Annotations:
nginx.ingress.kubernetes.io/force-ssl-redirect: false
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/ssl-redirect: false
Events: <none>
Unfortunately, when I try to access the Dash app on the browser, I get a
502 Bad Gateway error from the nginx. Could you please help me because my Kubernetes knowledge is limited.
Thanks in advance.
It had nothing to do with Kubernetes or AWS settings. I had to change my python Dash code from:
if __name__ == "__main__":
app.run_server(debug=True)
to:
if __name__ == "__main__":
app.run_server(host='0.0.0.0',debug=True).
The addition of host='0.0.0.0' did the trick!
I think you'll need to check whether any other service is exposed at path / on the same host.
Secondly, try removing rewrite-target annotation. Also can you please update your question with output of kubectl describe ingress <ingress_Name>
I would also suggest you to use backend-protocol annotation with value as HTTP (default value, you can avoid using this if dashboard application is not SSL Configured, and only this application is served at the said host.) But, you may need to add this if multiple applications are served at this host, and create one Ingress with backend-protocol: HTTP for non SSL services, and another with backend-protocol: HTTPS to serve traffic to SSL enabled services.
For more information on backend-protocol annotation, kindly refer this link.
I have often faced this issue in my Ingress Setup and these steps have helped me resolve it.

Define a fallback service for an isomorphic JavaScript app

I have an isomorphic JavaScript app that uses Vue's SSR plugin running on K8s. This app can either be rendered server-side by my Express server with Node, or it can be served straight to the client as with Nginx and rendered in the browser. It works pretty flawlessly either way.
Running it in Express with SSR is a much higher resource use however, and Express is more complicated and prone to fail if I misconfigure something. Serving it with Nginx to be rendered client side on the other hand is dead simple, and barely uses any resources in my cluster.
What I want to do is have a few replicas of a pod running my Express server that's performing the SSR, but if for some reason these pods go down, I want a fallback service on the ingress that will serve from a backup pod with just Nginx serving the client-renderable code.
Setting up the pods is easy enough, but how can I tell an ingress to serve from a different service then normal if the normal service is unreachable and/or responding too slowly to requests?
The easiest way to setup NGINX Ingress to meet your needs is by using the default-backend annotation.
This annotation is of the form
nginx.ingress.kubernetes.io/default-backend: <svc name> to specify
a custom default backend. This <svc name> is a reference to a
service inside of the same namespace in which you are applying this
annotation. This annotation overrides the global default backend.
This service will be handle the response when the service in the
Ingress rule does not have active endpoints. It will also handle the
error responses if both this annotation and the custom-http-errors
annotation is set.
Example:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-app-ingress
namespace: default
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: "/"
nginx.ingress.kubernetes.io/custom-http-errors: '404'
nginx.ingress.kubernetes.io/default-backend: default-http-backend
spec:
rules:
- host: myapp.mydomain.com
http:
paths:
- path: "/"
backend:
serviceName: custom-http-backend
servicePort: 80
In this example NGINX is serving custom-http-backend as primary resource and if this service fails, it will redirect the end-user to default-http-backend.
You can find more details on this example here.

How do I make my admin ui of cockroachdb publicly available via traefik ingress controller on kubernetes?

Kubernetes dedicated cockroachdb node - accessing admin ui via traefik ingress controller fails - page isn't redirecting properly
I have a dedicated kubernetes node running cockroachdb. The pods get scheduled and everything is setup. I want to access the admin UI from a subdomain like so: cockroachdb.hostname.com. I have done this with traefik dashboard and ceph dashboard so I know my ingress setup is working. I even have cert-manager running to have https enabled. I get the error from the browser that the page is not redirecting properly.
Do I have to specify the host name somewhere special?
I have tried adding this with no success: --http-host cockroachdb.hostname.com
This dedicated node has its own public ip which is not mapped to hostname.com. I think I need to change a setting in cockroachdb, but I don't know which because I am new to it.
Does anyone know how to publish admin UI via an ingress?
EDIT01: Added ingress and service config files
Ingress:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: cockroachdb-public
annotations:
kubernetes.io/ingress.class: traefik
traefik.frontend.rule.type: PathPrefixStrip
certmanager.k8s.io/issuer: "letsencrypt-prod"
certmanager.k8s.io/acme-challenge-type: http01
ingress.kubernetes.io/ssl-redirect: "true"
ingress.kubernetes.io/ssl-temporary-redirect: "true"
ingress.kubernetes.io/ssl-host: "cockroachdb.hostname.com"
traefik.frontend.rule: "Host:cockroachdb.hostname.com,www.cockroachdb.hostname.com"
traefik.frontend.redirect.regex: "^https://www.cockroachdb.hostname.com(.*)"
traefik.frontend.redirect.replacement: "https://cockroachdb.hostname.com/$1"
spec:
rules:
- host: cockroachdb.hostname.com
http:
paths:
- path: /
backend:
serviceName: cockroachdb-public
servicePort: http
- host: www.cockroachdb.hostname.com
http:
paths:
- path: /
backend:
serviceName: cockroachdb-public
servicePort: http
tls:
- hosts:
- cockroachdb.hostname.com
- www.cockroachdb.hostname.com
secretName: cockroachdb-secret
Serice:
apiVersion: v1
kind: Service
metadata:
# This service is meant to be used by clients of the database. It exposes a ClusterIP that will
# automatically load balance connections to the different database pods.
name: cockroachdb-public
labels:
app: cockroachdb
spec:
ports:
# The main port, served by gRPC, serves Postgres-flavor SQL, internode
# traffic and the cli.
- port: 26257
targetPort: 26257
name: grpc
# The secondary port serves the UI as well as health and debug endpoints.
- port: 8080
targetPort: 8080
name: http
selector:
app: cockroachdb
EDIT02:
I can access the Admin UI page now but only by going over the external ip address of the server with port 8080. I think I need to tell my server that its ip address is mapped to the correct sub domain?
EDIT03:
On both scheduled traefik-ingress pods the following logs are created:
time="2019-04-29T04:31:42Z" level=error msg="Service not found for default/cockroachdb-public"
Your referencing looks good on the ingress side. You are using quite a few redirects, unless you really know what each one is accomplishing, don't use them, you might end up in an infinite loop of redirects.
You can take a look at the following logs and methods to debug:
Run kubectl logs <traefik pod> and see the last batch of logs.
Run kubectl get service, and from what I hear, this is likely your main issue. Make sure your service exists in the default namespace.
Run kubectl port-forward svc/cockroachdb-public 8080:8080 and try connecting to it through localhost:8080 and see terminal for potential error messages.
Run kubectl describe ingress cockroachdb-public and look at the events, this should give you something to work with.
Try accessing the service from another pod you have running ping cockroachdb-public.default.svc.cluster.local and see if it resolves the IP address.
Take a look at your clusterrolebindings and serviceaccount, it might be limited and not have permission to list services in the default namespace: kubectl create clusterrolebinding default-admin --clusterrole cluster-admin --serviceaccount=default:default