Application webhook endpoints are blocked on GKE (Kubernetes) - kubernetes

We are implementing kubernetes on GKE.
I got stuck with the following.
The staging application on is hosted on GKE 1.19.7-gke.1500.
We have an application where we have to set up webhooks with applications. When setting up the webhook I get the same error. Namely that the url is not available or does not exist.
The application on kubernetes has a valid certificate. The application works locally (minikube and skaffold) when we use ngrok.
In addition, it is possible to perform an oauth with a third party software. But we cannot activate the webhook because our webhook endpoint is not found.
I looked at the cert-manager documentation, which states that there are known issues between Cert-manager and GKE. They say that this has to do with the settings in GKE, I opened some firewall myself, but without result. https://cert-manager.io/docs/installation/compatibility/
In addition, I have looked on the internet to find out if this problem can be fixed. But I find a lot of information about webhook to be set up for Kubernetes itself but not the application that runs on it.
I think it has to do with a port incorrectly GKE configured?
See our Pod Service setting:
---
apiVersion: v1
kind: Service
metadata:
name: backend
spec:
selector:
app: back-end
ports:
- name: back-end
protocol: TCP
port: 5000
targetPort: 5000
Our Ingress setting:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-service
annotations:
kubernetes.io/ingress.global-static-ip-name: "**.**.***.***"
kubernetes.io/ingress.class: 'nginx'
nginx.ingress.kubernetes.io/use-regex: 'true'
cert-manager.io/issuer: '****'
kubernetes.io/tls-acme: "true"
spec:
tls:
- hosts:
- test.xxxxxx.com
secretName: xxxxx
rules:
- host: test.xxxxx.com
http:
paths:
- path: /api/?(.*)
backend:
serviceName: backend
servicePort: 5000
- path: /?(.*)
backend:
serviceName: client
servicePort: 3000
Updated 03/15/2021
For example we want to set up a webhook for the following endpoint:
"https://test.xxxx.nl/api/v1/webhook/person".
We make a request to PipeDrive to set up a webhook, but then PipeDrive gives the following error: "errors: { subscription_url: [ 'invalid or non-reachable URL' ] } }"
response: [cmd]: kubectl get ep backend
NAME: backend
ENDPOINTS: 10.0.0.21:5000, 10.0.0.22:5000
AGE: 81d

Related

Access https backend with ingress and traefik

I am trying to use 'Kubernetes Ingress with Traefik, CertManager, LetsEncrypt and HAProxy' for certificates management.
What I want to do is use certificates in my application which deployed on kubernetes.
My application contains following services:
my-app1 NodePort 10.43.204.206 16686:31149/TCP
my-app2 NodePort 10.43.204.208 2746:30972/TCP
So for my-app1 without certificates I was accessing it as http://{IP}:31149/app1. And with certificates I am now accessing it as https://my-dns.com/app1. For this I am using this link. I created following ingress resource:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: prod-ingress
namespace: my-ns
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
kubernetes.io/ingress.class: traefik
traefik.ingress.kubernetes.io/router.tls: "true"
spec:
tls:
- hosts:
- "my-dns.com"
secretName: prod-cert
rules:
- host: "my-dns.com"
http:
paths:
- path: /app1
pathType: Prefix
backend:
service:
name: my-app1
port:
number: 16686
But for my-app2 without certificates I was accessing it as https://{IP}:30972/app2. So as I am already using https for my-app2 but I want to also use certificates for this service.
Any idea how to do this?
So the issue was, I was deploying my application with self signed certificates. So because of this I am getting issue while accessing my dashboard.
So now I just disabled self signed certificates in my application. And now I am able
to access dashboard with domain name https://my-dns.com.

K8s ingress socket.io 503

I have a Nodejs with socket.io app running on EKS with a ELB. I have configured an ingress with the following yaml.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/websocket-services: ws-gateway
nginx.org/websocket-services: ws-gateway
name: web-socket
namespace: default
spec:
defaultBackend:
service:
name: ws-gateway
port:
number: 9090
rules:
- host: ws.example.com
http:
paths:
- backend:
service:
name: ws-gateway
port:
number: 9090
path: /
pathType: Prefix
tls:
- hosts:
- ws.example.com
secretName: ws-tls
However, when testing this I am receiving 503 Service Temporarily Unavailable. The domain name and service name and port are correct. The request is not reaching the app either.
Is there something that I am missing in the configuration?
The issue was on the service side. The containers were restarting unexpectedly because of connection lost to some other service, which cause a 503 every time I tried to access it.

Enable App Gateway ingress for Azure Kubernetes on 443 (https)

I am new to AKS and trying to set up the cluster and expose it via an app gateway ingress controller. While I was able to set up the cluster using az commands and was able to deploy and hit it using HTTP. I am having some challenges in enabling HTTPS over 443 in-app gateway ingress and looking to get some help.
Below is our workflow and I am trying to setup app gateway listener on port 443
Below is the k8 we used for enabling the ingress. If I apply is without ssl cert it woks but if I give ssl cert I get a 502 bad gateway.
Cert is uploaded to KV and Cluster has KV add-on installed. But I am not sure how to attach this specific kv to cluster and whether the cert should be uploaded to gateway or Kubernetes.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: frontend-web-ingress
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/appgw-ssl-certificate: workspace-dev-cluster-cert
appgw.ingress.kubernetes.io/cookie-based-affinity: "true"
appgw.ingress.kubernetes.io/request-timeout: "90"
appgw.ingress.kubernetes.io/backend-path-prefix: "/"
spec:
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: frontend-svc
port:
number: 80
This link can help you with KV add-on certificate on App GW: https://azure.github.io/application-gateway-kubernetes-ingress/features/appgw-ssl-certificate/
I use different configuration to set certs on Appgw.
I'm getting certificates via the akv2k8s tool. This creates secrets on k8s cluster.
Then I use those certs in the ingress configuration. Please check tls definition under spec.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: frontend-web-ingress
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/appgw-ssl-certificate: workspace-dev-cluster-cert
appgw.ingress.kubernetes.io/cookie-based-affinity: "true"
appgw.ingress.kubernetes.io/request-timeout: "90"
appgw.ingress.kubernetes.io/backend-path-prefix: "/"
spec:
tls:
- hosts:
- yourdomain.com
secretName: your-tls-secret-name
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: frontend-svc
port:
number: 80

GKE Ingress with container-native load balancing does not detect health check (Invalid value for field 'resource.httpHealthCheck')

I am running a cluster on Google Kubernetes Engine and I am currently trying to switch from using an Ingress with external load balancing (and NodePort services) to an ingress with container-native load balancing (and ClusterIP services) following this documentation: Container native load balancing
To communicate with my services I am using the following ingress configuration that used to work just fine when using NodePort services instead of ClusterIP:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: mw-ingress
annotations:
kubernetes.io/ingress.global-static-ip-name: mw-cluster-ip
networking.gke.io/managed-certificates: mw-certificate
kubernetes.io/ingress.allow-http: "false"
spec:
rules:
- http:
paths:
- path: /*
backend:
serviceName: billing-frontend-service
servicePort: 80
- path: /auth/api/*
backend:
serviceName: auth-service
servicePort: 8083
Now following the documentation, instead of using a readinessProbe as a part of the container deployment as a health check I switched to using ClusterIP services in combination with BackendConfig instead. For each deployment I am using a service like this:
apiVersion: v1
kind: Service
metadata:
labels:
app: auth
name: auth-service
namespace: default
annotations:
cloud.google.com/backend-config: '{"default": "auth-hc-config"}'
spec:
type: ClusterIP
selector:
app: auth
ports:
- port: 8083
protocol: TCP
targetPort: 8083
And a Backend config:
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
name: auth-hc-config
spec:
healthCheck:
checkIntervalSec: 10
port: 8083
type: http
requestPath: /auth/health
As a reference, this is what the readinessProbe used to look like before:
readinessProbe:
failureThreshold: 3
httpGet:
path: /auth/health
port: 8083
scheme: HTTP
periodSeconds: 10
Now to the actual problem. I deploy the containers and services first and they seem to startup just fine. The ingress however does not seem to pick up the health checks properly and shows this in the Cloud console:
Error during sync: error running backend syncing routine: error ensuring health check: googleapi: Error 400: Invalid value for field 'resource.httpHealthCheck': ''. HTTP healthCheck missing., invalid
The cluster as well as the node pool are running GKE version 1.17.6-gke.11 so the annotation cloud.google.com/neg: '{"ingress": true}' is not necessary. I have checked and the service is annotated correctly:
Annotations: cloud.google.com/backend-config: {"default": "auth-hc-config"}
cloud.google.com/neg: {"ingress":true}
cloud.google.com/neg-status: {"network_endpoint_groups":{"8083":"k8s1-2078beeb-default-auth-service-8083-16a14039"},"zones":["europe-west3-b"]}
I have already tried to re-create the cluster and the node-pool with no effect. Any ideas on how to resolve this? Am I missing an additional health check somewhere?
I found my issue. Apparently the BackendConfig's type attribute is case-sensitive. Once I changed it from http to HTTP it worked after I recreated the ingress.

How can I generate External IP when creating an ingress that uses nginx controller in kubernetes

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: helloworld-rules
spec:
rules:
- host: helloworld-v1.example.com
http:
paths:
- path: /
backend:
serviceName: helloworld-v1
servicePort: 80
- host: helloworld-v2.example.com
http:
paths:
- path: /
backend:
serviceName: helloworld-v2
servicePort: 80
I'm making kubernetes cluster and I will apply that cloudPlatform Isolated(not aws or google).
When creating an ingress for service I can choose host url but that is not exist anywhere(that address is not registrated something like DNS server) So I can't access that url. Visiting this IP just gives a 404.
how can I get or configure URL that can access external browser
:(...
It depends on how you configure your nginx controller.
You should have a Service configured which is the entry point when accessing from outside see the docs https://kubernetes.io/docs/concepts/services-networking/ingress/#what-is-ingress.
So basically you have a Service that points to the ingress controller and this will redirect the traffic to your pods based on Ingress Objects.
Ingress -> Services -> Pods
Since you don't run on aws or google You would have to use externalIp or NodePort and configure the service accordingly
kind: Service
apiVersion: v1
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app: ingress-nginx
spec:
selector:
app: ingress-nginx
ports:
- name: http
port: 80
targetPort: http
- name: https
port: 443
targetPort: http
externalIPs:
- 80.11.12.10
And DNS needs to be managed with whatever you have for your domains in order to resolve, or for locally testing you can just edit your /etc/hostnames
Basically in AWS or Google you just create a service with type: LoadBalancer and point your dns records to the balancer address (CNAME for aws and the IP for google)