Trying to set up an ingress with tls and open to some IPs only on GKE - kubernetes

Question

I'm having trouble setting up an ingress open only to some specific IPs, checked docs, tried a lot of stuff and an IP out of the source keep accessing. that's a Zabbix web interface on an alpine with nginx, set up a service on node-port 80 then used an ingress to set up a loadbalancer on GCP, it's all working, the web interface is working fine, but how can I make it accessible only to desired IPs?
my firewall rules are ok and it's only accessible through load balancer IP
Also, I have a specific namespace for this deploy.
Cluster version 1.11.5-gke.5
EDIT i'm using GKE standard ingress GLBC
My template is config as follow can someone help enlighten me on what is missing:
apiVersion: v1
kind: ReplicationController
metadata:
name: zabbix-web
namespace: zabbix-prod
labels:
app: zabbix
tier: frontend
spec:
replicas: 1
template:
metadata:
labels:
name: zabbix-web
app: zabbix
spec:
volumes:
- name: cloudsql-instance-credentials
secret:
defaultMode: 420
secretName: cloudsql-instance-credentials
containers:
- command:
- /cloud_sql_proxy
- -instances=<conection>
- -credential_file=/secrets/cloudsql/credentials.json
image: gcr.io/cloudsql-docker/gce-proxy:1.11
imagePullPolicy: IfNotPresent
name: cloudsql-proxy
resources: {}
securityContext:
allowPrivilegeEscalation: false
runAsUser: 2
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /secrets/cloudsql
name: credentials
readOnly: true
- name: zabbix-web
image: zabbix/zabbix-web-nginx-mysql:alpine-3.2-latest
ports:
- containerPort: 80
env:
- name: MYSQL_USER
valueFrom:
secretKeyRef:
key: <user>
name: <user>
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
key: <pass>
name: <pass>
- name: DB_SERVER_HOST
value: 127.0.0.1
- name: MYSQL_DATABASE
value: <db>
- name: ZBX_SERVER_HOST
value: <db>
readinessProbe:
failureThreshold: 3
httpGet:
path: /index.php
port: 80
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 30
---
apiVersion: v1
kind: Service
metadata:
name: "zabbix-web-service"
namespace: "zabbix-prod"
labels:
app: zabbix
spec:
ports:
- port: 80
targetPort: 80
selector:
name: "zabbix-web"
type: "NodePort"
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: zabbix-web-ingress
namespace: zabbix-prod
annotations:
ingress.kubernetes.io/service.spec.externalTrafficPolicy: local
ingress.kubernetes.io/whitelist-source-range: <xxx.xxx.xxx.xxx/32>
spec:
tls:
- secretName: <tls-cert>
backend:
serviceName: zabbix-web-service
servicePort: 80

Answer 1

You can whitelist IPs by configuring Ingress and Cloud Armour:
Switch to project:
gcloud config set project $PROJECT
Create a policy:
gcloud compute security-policies create $POLICY_NAME --description "whitelisting"
Change default policy to deny:
gcloud compute security-policies rules update 2147483647 --action=deny-403 \
--security-policy $POLICY_NAME
On lower priority than the default whitelist all IPs you want to whitelist:
gcloud compute security-policies rules create 2 \
--action allow \
--security-policy $POLICY_NAME \
--description "allow friends" \
--src-ip-ranges "93.184.17.0/24,151.101.1.69/32"
With a maximum of ten per range.
Note you need valid CIDR ranges, for that you can use CIDR to IP Range -> IP Range to CIDR.
View the policy as follows:
gcloud compute security-policies describe $POLICY_NAME
To throw away an entry:
gcloud compute security-policies rules delete $PRIORITY --security-policy $POLICY_NAME
or the full policy:
gcloud compute security-policies delete $POLICY_NAME
Create a BackendConfig for the policy:
# File backendconfig.yaml:
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
namespace: <namespace>
name: <name>
spec:
securityPolicy:
name: $POLICY_NAME
$ kubectl apply -f backendconfig.yaml
backendconfig.cloud.google.com/backendconfig-name created
Add the BackendConfig to the Service:
metadata:
namespace: <namespace>
name: <service-name>
labels:
app: my-app
annotations:
cloud.google.com/backend-config: '{"ports": {"80":"backendconfig-name"}}'
spec:
type: NodePort
selector:
app: hello-app
ports:
- port: 80
protocol: TCP
targetPort: 8080
Use the right selectors and point the receiving port of the Service to the BackendConfig created earlier.
Now Cloud Armour will add the policy to the GKE service.
Visible in https://console.cloud.google.com/net-security/securitypolicies (after selecting $PROJECT).

Answer 2

AFAIK, you can't restrict IP addresses through GLBC or on GCP L7 Load Balancer itself. Note that GLBC is also a work in progress as of this writing.
ingress.kubernetes.io/whitelist-source-range works great but when you are using something like an nginx ingress controller because nginx itself can restrict IP addresses.
The general way to restrict/whitelist IP addresses is using VPC Firewall Rules (which seems like you are doing already). Essentially you can restrict/whitelist the IP addresses to the network where your K8s nodes are running on.

Answer 3

One of the best options to accomplish your goal is using firewall rules since you can't restrict IP addresses through the Global LB or on GCP L7 LB itself. However, another option if you are using Ingress on your Kubernetes cluster, it is possible to restrict access to your application based on dedicated IP addresses.
One possible use case would be that you have a development setup and don’t want to make all the fancy new features available to everyone, especially competitors. In such cases, IP whitelisting to restrict access can be used.
This can be done with specifying the allowed client IP source ranges through the ingress.kubernetes.io/whitelist-source-range annotation.
The value is a comma separated list of CIDR block.
For example:
10.0.0.0/24, 1.1.1.1/32.
Please get more information here.

Answer 4

For anyone who stumbles on this question via Google like I did, there is now a solution. You can implement this via a BackendConfig from the cloud.google.com Kubernetes API in conjunction with a GCE CloudArmor policy.
https://cloud.google.com/kubernetes-engine/docs/how-to/cloud-armor-backendconfig