I have an EKS cluster with an nginx deployment on namespace gitlab-managed-apps. Exposing the application to the public from ALB ingress. I'm trying to block a specific Public IP (ex: x.x.x.x/32) from accessing the webpage. I tried Calico and K8s network policies. Nothing worked for me. I created this Calico policy with my limited knowledge of Network policies, but it blocks everything from accessing the nginx app, not just x.x.x.x/32 external IP. Showing everyone 504 Gateway timeout from ALB
apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
name: ingress-external
namespace: gitlab-managed-apps
spec:
selector:
app == 'nginx'
types:
- Ingress
ingress:
- action: Deny
source:
nets:
- x.x.x.x/32
Try this:
apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
name: ingress-external
namespace: gitlab-managed-apps
spec:
selector:
app == 'nginx'
types:
- Ingress
ingress:
- action: Deny
source:
nets:
- x.x.x.x/32
- action: Allow
calico docs suggests:
If one or more network policies apply to a pod containing ingress rules, then only the ingress traffic specifically allowed by those policies is allowed.
So this means that any traffic is denied by default and only allowed if you explicitly allow it. This is why adding additional rule action: Allow should allow all other traffic that was not matched by the previous rule.
Also remember what docs mention about rules:
A single rule matches a set of packets and applies some action to them. When multiple rules are specified, they are executed in order.
So default Allow rule has to follow the Deny rule for the specific IP, not the other way around.
Related
I have a mosquitto broker running on a pod, this server is exposed as a service as both DNS and IP address.
But this service is accessible by any pod in the cluster.
I want to restrict access to this service such that pods trying to connect to this DNS or IP address should only be able to if the pods have certain name/metadata.
One solution I guess will be to use namespaces? What other solution is there?
The UseCase you are describing is exactly what NetworkPolicies are here for.
Basically you define selector for pods which the network traffic should be restricted (i.e. your mosquito broker) and what specifica pods need to have in order to be allowed to reach it. For example a label "broker-access: true" or whatever seems to be suitable for you.
an example network policy could look like this:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: broker-policy
namespace: default
spec:
podSelector:
matchLabels:
role: message-broker
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
broker-access: true
ports:
- protocol: TCP
port: 6379
this network policy would be applied to every pod with label role=message-broker.
and it would restrict all incoming traffic except for traffic from pods with label broker-acces=true on port 6379.
Hope this helps and gives you a bit of a skaffold for your NetworkPolicy
I was doing some research about ingress and it seems I have to create a new ingress resource for each namespace. Is that correct?
I just created 2 separate ingress resources in different namespaces in my GKE cluster but it seems to use the same LB in(which is great for cost) but I would think it is possible to have clashes then. (when using same path). I just tried it and the first one I've created is still working on the path, the other newer one on the same path is just not working.
Can someone explain me the correct setup for ingress?
As Kubernetes works, ingress controller won't pass a packet to a service that is in a different namespace from the ingress resource. So, if you create an ingress resource in the default namespace, all your services must be in the default namespace as well.
This is something that won't change. EVER. There has been a feature request years ago, and kubernetes team announced that it's not going to happen. It introduces a security hole when ingress controller is being able to transpass a namespace.
Now, what we do in these situations is actually pretty neat. You will have to do the following:
Say you have 2 services in the namespaces you need. e.g. service1.foo and service2.bar.
create 2 headless services without selectors and 2 Endpoint objects pointing to the IP addresses of the services service1.foo and service2.bar, in the same namespace as the ingress resource. The headless service without selectors will force kube-dns (or coreDNS) to search for either ExternalName type service or an Endpoint object. Now, the only requirement here is that your headless service and the Endpoint object must have the same name.
Create your ingress resource pointing to the headless services.
It should look like this (for 1 service):
Say the IP address of service1.foo is 10.10.10.10. Your headless service and the Endpoint object would be:
apiVersion: v1
kind: Service
metadata:
name: bait-svc
spec:
clusterIP: None
ports:
- name: http
port: 80
targetPort: 80
---
apiVersion: v1
kind: Endpoints
metadata:
name: bait-svc
subsets:
- addresses:
- ip: 10.10.10.10
ports:
- port: 80
protocol: TCP
and Ingress resource:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
tls:
- secretName: ssl-certs
rules:
- host: site1.training.com
http:
paths:
- path: /
backend:
serviceName: bait-svc
servicePort: 80
So, the Ingress points to the bait-svc, and bait-svc points to service1.foo. And you will do this for each service.
UPDATE
I am thinking now, it might not work with GKE Ingress Controller, as on GKE you need a NodePort type service for the HTTP load balancer to reach the service. As you can see, in my example I've got nginx Ingress Controller.
Independently if it works or not, I would recommend using some other Ingress Controller. It's not that GKE IC is not good. It is quite robust, but almost always you end up hitting some limitation. Other ICs are more flexible.
The behavior of conflicting Ingress routes is undefined and implementation dependent. In most cases it’s just last writer wins.
I've recently started working with Kubernetes clusters. The flow of network calls for a given Kubernetes service in our cluster is something like the following:
External Non-K8S Load Balancer -> Ingress Controller -> Ingress Resource -> Service -> Pod
For a given service, there are two replicas. By looking at the logs of the containers in the replicas, I can see that calls are being routed to different pods. As far as I can see, we haven't explicitly set up any load-balancing policies anywhere for our services in Kubernetes.
I've got a few questions:
1) Is there a default load-balancing policy for K8S? I've read about kube-proxy and random routing. It definitely doesn't appear to be round-robin.
2) Is there an obvious way to specify load balancing rules in the Ingress resources themselves? On a per-service basis?
Looking at one of our Ingress resources, I can see that the 'loadBalancer' property is empty:
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
ingress.kubernetes.io/rewrite-target: /
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"ingress.kubernetes.io/rewrite-target":"/","nginx.ingress.kubernetes.io/rewrite-target":"/"},"name":"example-service-ingress","namespace":"member"},"spec":{"rules":[{"host":"example-service.x.x.x.example.com","http":{"paths":[{"backend":{"serviceName":"example-service-service","servicePort":8080},"path":""}]}}]}}
nginx.ingress.kubernetes.io/rewrite-target: /
creationTimestamp: "2019-02-13T17:49:29Z"
generation: 1
name: example-service-ingress
namespace: x
resourceVersion: "59178"
selfLink: /apis/extensions/v1beta1/namespaces/x/ingresses/example-service-ingress
uid: b61decda-2fb7-11e9-935b-02e6ca1a54ae
spec:
rules:
- host: example-service.x.x.x.example.com
http:
paths:
- backend:
serviceName: example-service-service
servicePort: 8080
status:
loadBalancer:
ingress:
- {}
I should specify - we're using an on-prem Kubernetes cluster, rather than on the cloud.
Cheers!
The "internal load balancing" between Pods of a Service has already been covered in this question from a few days ago.
Ingress isn't really doing anything special (unless you've been hacking in the NGINX config it uses) - it will use the same Service rules as in the linked question.
If you want or need fine-grained control of how pods are routed to within a service, it is possible to extend Kubernetes' features - I recommend you look into the traffic management features of Istio, as one of its features is to be able to dynamically control how much traffic different pods in a service receive.
I see two options that can be used with k8s:
Use istio's traffic management and create a DestinationRule. It currently supports three load balancing modes:
Round robin
Random
Weighted least request
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
...
spec:
...
subsets:
- name: test
...
trafficPolicy:
loadBalancer:
simple: ROUND_ROBIN
Use lb_type in envoy proxy with ambassador on k8s. More info about ambassador is in https://www.getambassador.io.
I am a newbie to Kubernetes and trying to learn calico networking.
I am following this documentation (https://docs.aws.amazon.com/eks/latest/userguide/calico.html)
and I tried to create a networkpolicy for the traffic to flow between backend to client :
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
namespace: stars
name: backend-client
spec:
podSelector:
matchLabels:
role : client
ingress:
- from:
- namespaceSelector:
matchLabels:
role: backend
ports:
- protocol: TCP
port: 9000
I finished all the 10 steps in the documentation, and i tried to test by creating a policy that would send traffic from the backend to the client with the above policy.
When i applied the policy there was no error , but i don't see the traffic/connection between the two.
Please let me know what is wrong.
Creating NetworkPolicy alone will not help in ensuring that the NetworkPolicy is enforced. We should configure the network plugin like Calico which is integrated with Kubernetes and executes the necessary operations to achieve the intent of the given Network Policy
https://kubernetes.io/docs/concepts/services-networking/network-policies/ says
"Network policies are implemented by the network plugin, so you must be using a networking solution which supports NetworkPolicy - simply creating the resource without a controller to implement it will have no effect."
I believe you need to put your policy in the client namespace instead of the stars namespace. I don't believe there are any pods with role: client in the stars namespace. A pod selector like you've specified only applies to pods in the namespace the policy is in.
While I don't think it is as direct as it could be the Kubernetes Network Policy docs do mention that a NetworkPolicy applies in the given namespace. I suggest you check them out if you haven't already.
I hope that helps.
I have a Kubernetes cluster running on Google Kubernetes Engine (GKE) with network policy support enabled.
I created an nginx deployment and load balancer for it:
kubectl run nginx --image=nginx
kubectl expose deployment nginx --port=80 --type=LoadBalancer
Then I created this network policy to make sure other pods in the cluster won't be able to connect to it anymore:
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: access-nginx
spec:
podSelector:
matchLabels:
run: nginx
ingress:
- from:
- namespaceSelector:
matchLabels:
name: kube-system
ports:
- protocol: TCP
port: 80
Now other pods in my cluster can't reach it (as intended):
kubectl run busybox --rm -ti --image=busybox /bin/sh
If you don't see a command prompt, try pressing enter.
/ # wget --spider --timeout=1 nginx
Connecting to nginx (10.63.254.50:80)
wget: download timed out
However, it surprised me that using my external browser I also can't connect anymore to it through the load balancer:
open http://$(kubectl get svc nginx --output=jsonpath={.status.loadBalancer.ingress[0].ip})
If I delete the policy it starts to work again.
So, my question is: how do I block other pods from reaching nginx, but keep access through the load balancer open?
I talked about this in my Network Policy recipes repository.
"Allowing EXTERNAL load balancers while DENYING local traffic" is not a use case that makes sense, therefore it's not possible to using network policy.
For Service type=LoadBalancer and Ingress resources to work, you must allow ALL traffic to the pods selected by these resources.
If you REALLY want you can use the from.ipBlock.cidr and from.ipBlock.cidr.except resources to allow traffic from 0.0.0.0/0 (all IPv4) and then excluding 10.0.0.0/8 (or whatever private IP range GKE uses).
I recently had to do something similar. I needed a policy that didn't allow pods from other namespaces to talk to prod, but did allow the LoadBalancer services to reach pods in prod. Here's what worked (based on Ahmet's post):
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: isolate-prod
namespace: prod
spec:
podSelector: {}
ingress:
- from:
- podSelector: {}
- from:
- ipBlock:
cidr: '0.0.0.0/0'
except: ['10.0.0.0/8']
I'd like to share solution that builds on the excellent answers of #ahmetb-google and #tammer-saleh
The situation: 1 cluster, 4 namespaces, a public HTTPS-terminating Ingress for 3 of the namespaces that allows specific traffic inbound and routes it appropriately.
Goal: Block all inter-namespace traffic, allow only public traffic coming in via the Ingress.
Problem: When deploying a "deny from other namespaces" rule, that it also denied traffic from my Ingress so the pods were not accessible from the outside.
Solution:
I created an additional policy to allow only port 80 traffic targetting pods with the label role=web. It uses the allow/except trick to still block traffic from other namespaces but allow it from the public ingresses.
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: allow-http-from-ingress
spec:
podSelector:
matchLabels:
role: web
ingress:
- from:
- ipBlock:
cidr: '0.0.0.0/0'
except: ['10.0.0.0/8']
ports:
- port: 80
With this deployed, traffic still flows from the public, via the Ingress, to the web-serving pods. All inter-namespace traffic is blocked, including HTTP.
This is a useful setup when for example you're using namespaces for different deployment stages (production, testing, edge, etc) and you have private HTTP APIs that you would not want to accidentally hit cross-stage.