Exposing UDP and TCP ports for sftp server using Ingress in GKE - kubernetes

I am trying to set up a multi-cluster deployment in which there are multiple clusters and one ingress is load balancing the requests between them.
HTTP services work well with the set-up the problem here is the sftp server.
Referring to this answer and this documentation I am trying to access port 22 of the sftp service.
Deployment of sftp is being exposed on port 22. Below is the manifest:
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: sftp
labels:
environment: production
app: sftp
spec:
replicas: 1
minReadySeconds: 10
template:
metadata:
labels:
environment: production
app: sftp
annotations:
container.apparmor.security.beta.kubernetes.io/sftp: runtime/default
spec:
containers:
- name: sftp
image: atmoz/sftp:alpine
imagePullPolicy: Always
args: ["user:1001:100:upload"]
ports:
- containerPort: 22
securityContext:
capabilities:
add: ["SYS_ADMIN"]
resources: {}
Here is the simple manifest for the sftp-service using NodePort service:
apiVersion: v1
kind: Service
metadata:
labels:
environment: production
name: sftp-service
spec:
type: NodePort
ports:
- name: sftp-port
targetPort: 9000
port: 9000
nodePort: 30063
protocol: TCP
selector:
app: sftp
ConfigMap create to referring to the above mentioned documentation and answer looks like below:
apiVersion: v1
kind: ConfigMap
metadata:
name: sftp-service
data:
9000: "default/sftp-service:22"
And finally the ingress manifest is something like below:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-foo
annotations:
kubernetes.io/ingress.global-static-ip-name: static-ip
kubernetes.io/ingress.class: gce-multi-cluster
spec:
backend:
serviceName: http-service-zone-printer
servicePort: 80
rules:
- http:
paths:
- path: /sftp
backend:
serviceName: sftp-service
servicePort: 22
template:
spec:
containers:
- name: proxy-port
args:
- "--tcp-services-configmap=default/sftp-service"
I feel, I have not understood the way to expose the TCP/UDP port for sftp server on kubernetes using ingress. What am I doing wrong here?
Is there any other way to simple setup an sftp using ingress and NodePort service in a multicluster deployment?
Here is the official document I am referring to do the set-up.

looks like this isn't supported with ingress which is the reason that this issue exist
A possible solution could be to use nodeport for sftp as described in this document

You need to run an HTTP server.
You can run an HTTP server that exposes the same files maybe with a side container in the same pod

Related

Localhost kubernetes ingress not exposing services to local machine

I'm running kuberenetes in localhost, the pod is running and I can access the services when I port forwarding:
kubectl port-forward svc/my-service 8080:8080
I can get/post etc. the services in localhost.
I'm trying to use it with ingress to access it, here is the yml file:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-ingress
spec:
ingressClassName: nginx
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: my-service
port:
number: 8080
I've also installed the ingress controller. But it isn't working as expected. Anything wrong with this?
EDIT: the service that Im trying to connect with ingress:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-service
labels:
app: my-service
spec:
replicas: 1
selector:
matchLabels:
app: my-service
template:
metadata:
labels: my-service
app: my-service
spec:
containers:
- image: test/my-service:0.0.1-SNAPSHOT
name: my-service
ports:
- containerPort:8080
... other spring boot override properties
---
apiVersion: v1
kind: Service
metadata:
name: my-service
labels:
app: my-service
spec:
type: ClusterIP
selector:
app: my-service
ports:
- name: 8080-8080
port: 8080
protocol: TCP
targetPort: 8080
service is working by itself though
EDIT:
It worked when I used https instead of http
Is ingress resource in the same namespace as the service? Can you share the manifest of service? Also, what do logs of nginx ingress-controller show and what sort of error do you face when hitting the endpoint in the browser?
Ingress's YAML file looks OK to me BTW.
I was being stupid. It worked when I used https instead of http

GKE Ingress with Multiple Backend Services returns 404

I'm trying to create a GKE Ingress that points to two different backend services based on path. I've seen a few posts explaining this is only possible with an nginx Ingress because gke ingress doesn't support rewrite-target. However, this Google documentation, GKE Ingresss - Multiple backend services, seems to imply otherwise. I've followed the steps in the docs but haven't had any success. Only the service that is available on the path prefix of / is returned. Any other path prefix, like /v2, returns a 404 Not found.
Details of my setup are below. Is there an obvious error here -- is the Google documentation incorrect and this is only possible using nginx ingress?
-- Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: app-ingress
annotations:
kubernetes.io/ingress.global-static-ip-name: app-static-ip
networking.gke.io/managed-certificates: app-managed-cert
spec:
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: api-service
port:
number: 80
- path: /v2
pathType: Prefix
backend:
service:
name: api-2-service
port:
number: 8080
-- Service 1
apiVersion: v1
kind: Service
metadata:
name: api-service
labels:
app: api
spec:
type: NodePort
selector:
app: api
ports:
- port: 80
targetPort: 5000
-- Service 2
apiVersion: v1
kind: Service
metadata:
name: api-2-service
labels:
app: api-2
spec:
type: NodePort
selector:
app: api-2
ports:
- port: 8080
targetPort: 5000
GCP Ingress supports multiple paths. This is also well described in Setting up HTTP(S) Load Balancing with Ingress. For my test I've used both Hello-world v1 and v2.
There are 3 possible issues.
Issue is with container ports opened. You can check it using netstat:
$ kk exec -ti first-55bb869fb8-76nvq -c container -- bin/sh
/ # netstat -plnt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 :::8080 :::* LISTEN 1/hello-app
Issue might be also caused by the Firewall configuration. Make sure you have proper settings. (In general, in the new cluster I didn't need to add anything but if you have more stuff and have specific Firewall configurations it might block).
Misconfiguration between port, containerPort and targetPort.
Below my example:
1st deployment with
apiVersion: apps/v1
kind: Deployment
metadata:
name: first
labels:
app: api
spec:
selector:
matchLabels:
app: api
template:
metadata:
labels:
app: api
spec:
containers:
- name: container
image: gcr.io/google-samples/hello-app:1.0
ports:
- containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: api-service
labels:
app: api
spec:
type: NodePort
selector:
app: api
ports:
- port: 5000
targetPort: 8080
2nd deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: second
labels:
app: api-2
spec:
selector:
matchLabels:
app: api-2
template:
metadata:
labels:
app: api-2
spec:
containers:
- name: container
image: gcr.io/google-samples/hello-app:2.0
ports:
- containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: api-2-service
labels:
app: api-2
spec:
type: NodePort
selector:
app: api-2
ports:
- port: 6000
targetPort: 8080
Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: app-ingress
spec:
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: api-service
port:
number: 5000
- path: /v2
pathType: Prefix
backend:
service:
name: api-2-service
port:
number: 6000
Outputs:
$ curl 35.190.XX.249
Hello, world!
Version: 1.0.0
Hostname: first-55bb869fb8-76nvq
$ curl 35.190.XX.249/v2
Hello, world!
Version: 2.0.0
Hostname: second-d7d87c6d8-zv9jr
Please keep in mind that you can also use Nginx Ingress on GKE by adding specific annotation.
kubernetes.io/ingress.class: "nginx"
Main reason why people use nginx ingress on GKE is using rewrite annotation and possibility to use ClusterIP or NodePort as serviceType, where GCP ingress allows only NodePort serviceType.
Additional information you can find in GKE Ingress for HTTP(S) Load Balancing

SFTP server is not accessible when deployed to Kubernetes (GKE)

SFTP server is not accessible when exposed using a NodePort service and an Kubernetes Ingress. However, if the same deployment is exposed using a Service of type LoadBalancer it works fine.
Below is the deployment file for SFTP in GKE using atmoz/sftp Dockerfile.
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: sftp
labels:
environment: production
app: sftp
spec:
replicas: 1
minReadySeconds: 10
template:
metadata:
labels:
environment: production
app: sftp
annotations:
container.apparmor.security.beta.kubernetes.io/sftp: runtime/default
spec:
containers:
- name: sftp
image: atmoz/sftp:alpine
imagePullPolicy: Always
args: ["user:pass:1001:100:upload"]
ports:
- containerPort: 22
securityContext:
capabilities:
add: ["SYS_ADMIN"]
resources: {}
If I expose this deployment normally using a Kubernetes Service of type LoadBalancer like below:
apiVersion: v1
kind: Service
metadata:
labels:
environment: production
name: sftp-service
spec:
type: LoadBalancer
ports:
- name: sftp-port
port: 22
protocol: TCP
targetPort: 22
selector:
app: sftp
Above Service gets an external IP which I can simply use in the command sftp xxx.xx.xx.xxx command to get access using the pass password.
However, I try to expose the same deployment using GKE Ingress it does not work. Below is the manifest for the ingress:
# First I create a NodePort service to expose the deployment internally
---
apiVersion: v1
kind: Service
metadata:
labels:
environment: production
name: sftp-service
spec:
type: NodePort
ports:
- name: sftp-port
port: 22
protocol: TCP
targetPort: 22
nodePort: 30063
selector:
app: sftp
# Ingress service has SFTP service as it's default backend
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: basic-ingress-2
spec:
backend:
serviceName: sftp-service
servicePort: 22
rules:
- http:
paths:
# "http-service-sample" is a service exposing a simple hello-word app deployment
- path: /sample
backend:
serviceName: http-service-sample
servicePort: 80
After an external IP is assigned to the Ingress (I know it takes a few minutes to completely set up) and xxx.xx.xx.xxx/sample starts working but sftp -P 80 xxx.xx.xx.xxx doesn't work.
Below is the error I get from the server:
ssh_exchange_identification: Connection closed by remote host
Connection closed
What am I doing wrong in the above set-up? Why does LoadBalancer service is able to allow access to SFTP service, while Ingress fails?
That's currently not fully supported to route in Kubernetes Ingress any other traffic than HTTP/HTTPS protocols (see docs).
You can try to make some workaround as described there: Kubernetes: Routing non HTTP Request via Ingress to Container

Bad Gateway with traefik Ingress

I'm using minikube with traefik ingress to create a sticky sessions.
So i have done the deploy of traefik that user guide kubernetes provides me. https://docs.traefik.io/user-guide/kubernetes/
I deploy traefik using DaemonSet. Cause it's a small project and is my first time using kubernetes and docker.
This is my ingress yaml file
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: cp-pluggin
annotations:
kubernetes.io/ingress.class: traefik
spec:
rules:
- host: cppluggins.minikube
http:
paths:
- path: /
backend:
serviceName: cp-pluggin
servicePort: 80
My service yaml file
apiVersion: v1
kind: Service
metadata:
name: cp-pluggin
annotations:
traefik.ingress.kubernetes.io/affinity: "true"
traefik.ingress.kubernetes.io/session-cookie-name: "sticky"
spec:
type: NodePort
ports:
- port: 80
targetPort: 8080
protocol: TCP
name: http
selector:
app: cp-pluggin-app
Finally, my deployment yaml file
apiVersion: apps/v1
kind: Deployment
metadata:
name: cp-pluggin-app
labels:
app: cp-pluggin-app
spec:
replicas: 3
selector:
matchLabels:
app: cp-pluggin-app
template:
metadata:
labels:
app: cp-pluggin-app
spec:
containers:
- name: cp-pluggin-app
image: essoca/ubuntu-tornado
ports:
- containerPort: 8080
I expected
Hello world from: [ipserver]
But i get a
bad gateway
I assume you are using Traefik 2.0, the latest version as of now. There are quite some changes in this version, i.e. the annotations are not used anymore. Besides that, I think the code that you posted is missing a big part of the required changes.
Also, it's not very useful to use a DaemonSet because you are using minikube and that's always one node. Using a Deployment will at least allow you to play with the scale up/down functionality of Kubernetes.
I wrote this article that might be useful for you Traefik 2 as Ingress Controller

How to set up https on kubernetes bare metal using traefik ingress controller

I'm running a kubernetes cluster which consists of three nodes and brilliantly works, but it's time to make my web application secure, so I deployed an ingress controller(traefik). But I was unable to find instructions for setting up https on it. I know most of things I will have to do, like setting up a "secret"(container with certs) etc. but I was wondering how to configure my ingress controller and all files related to it so I would be able to use secure connection
I have already configured ingress controller and created some frontends and backends. Also I configured nginx server(It's actually a web application I'm running) to work on 443 port
My web application deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
strategy:
type: Recreate
selector:
matchLabels:
app: nginx
replicas: 3 # tells deployment to run 3 pods matching the template
template: # create pods using pod definition in this template
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: ilchub/my-nginx
ports:
- containerPort: 443
tolerations:
- key: "primary"
operator: Equal
value: "true"
effect: "NoSchedule"
Traefik ingress controller deployment code
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: traefik-ingress
namespace: kube-system
labels:
k8s-app: traefik-ingress-lb
spec:
replicas: 1
selector:
matchLabels:
k8s-app: traefik-ingress-lb
template:
metadata:
labels:
k8s-app: traefik-ingress-lb
name: traefik-ingress-lb
spec:
serviceAccountName: traefik-ingress
terminationGracePeriodSeconds: 60
containers:
- image: traefik
name: traefik-ingress-lb
ports:
- name: http
containerPort: 80
- name: https
containerPort: secure
- name: admin
containerPort: 8080
args:
- --api
- --kubernetes
- --logLevel=INFO
Ingress for traefik dashboard
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: traefik-web-ui
namespace: kube-system
spec:
rules:
- host: cluster.aws.ctrlok.dev
http:
paths:
- path: /
backend:
serviceName: traefik-web-ui
servicePort: web
External expose related config
kind: Service
apiVersion: v1
metadata:
name: traefik-ingress-service
namespace: kube-system
spec:
selector:
k8s-app: traefik-ingress-lb
ports:
- protocol: TCP
port: 80
nodePort: 30036
name: web
- protocol: TCP
port: 443
nodePort: 30035
name: secure
- protocol: TCP
port: 8080
nodePort: 30034
name: admin
type: NodePort
What I want to do is securing my application which is already running. Final result has to be a webpage running over https
Actually you have 3 ways to configure Traefik to use https to communicate with backend pods:
If the service port defined in the ingress spec is 443 (note that you can still use targetPort to use a different port on your pod).
If the service port defined in the ingress spec has a name that starts with https (such as https-api, https-web or just https).
If the ingress spec includes the annotation ingress.kubernetes.io/protocol: https.
If either of those configuration options exist, then the backend communication protocol is assumed to be TLS, and will connect via TLS automatically.
Also additional authentication annotations should be added to the Ingress object, like:
ingress.kubernetes.io/auth-tls-secret: secret
And of course, add a TLS Certificate to the Ingress