I am using calico 2.6 with docker 17.09.0-ce and it is working fine. I only have 2 questions that I cannot find an answer to.
1) How do I turn off the calico/felix iptables rules completely? I was thinking this might work with the CALICO_LIBNETWORK_CREATE_PROFILES=false environment variable, but the iptable rules are still set. Or alternatively how do I set the system to "allow all".
2) If the answer to 1) is that this is not possible then how do I allow any host in the cluster to talk to (e.g. ping or curl) any workloadEndpoint? I can talk to the workloads from the host where the workload is running on, but not from any other host in the cluster. I've tried by applying a profile with "egress: action: allow" and "ingress: action: allow", but it still blocks the traffic from other hosts. I verified that it is really due to the iptables rules that calico generates by tracing them and also by disabling them. Then it works. I used the following script after stopping the calico/node container: https://github.com/projectcalico/calico/blob/master/hack/remove-calico-policy/remove-calico-policy.sh
By the way, the FELIX_DEFAULTENDPOINTTOHOSTACTION=ACCEPT flag also does not seem to work.
I just got a reply via the slack channel: https://calicousers.slack.com/archives/C0BCA44LA/p1512125998000122
The answer is to use a policy file like the below:
apiVersion: v1
kind: policy
metadata:
name: allow-all
spec:
selector: "all()"
types:
- ingress
- egress
ingress:
- action: allow
egress:
- action: allow
Related
I have just set up a kubernetes cluster on bare metal using kubeadm, Flannel and MetalLB. Next step for me is to install ArgoCD.
I installed the ArgoCD yaml from the "Getting Started" page and logged in.
When adding my Git repositories ArgoCD gives me very weird error messages:
The error message seems to suggest that ArgoCD for some reason is resolving github.com to my public IP address (I am not exposing SSH, therefore connection refused).
I can not find any reason why it would do this. When using https:// instead of SSH I get the same result, but on port 443.
I have put a dummy pod in the same namespace as ArgoCD and made some DNS queries. These queries resolved correctly.
What makes ArgoCD think that github.com resolves to my public IP address?
EDIT:
I have also checked for network policies in the argocd namespace and found no policy that was restricting egress.
I have had this working on clusters in the same network previously and have not changed my router firewall since then.
I solved my problem!
My /etc/resolv.conf had two lines that caused trouble:
domain <my domain>
search <my domain>
These lines were put there as a step in the installation of my host machine's OS that I did not realize would affect me in this way. After removing these lines, everything is now working perfectly.
Multiple people told me to check resolv.conf, but I didn't realize what these two lines did until now.
That looks like argoproj/argo-cd issue 1510, where the initial diagnostic was that the cluster is blocking outbound connections to GitHub. And it suggested to check the egress configuration.
Yet, the issue was resolved with an ingress rule configuration:
need to define in values.yaml.
argo-cd default provide subdomain but in our case it was /argocd
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/backend-protocol: HTTP
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
nginx.ingress.kubernetes.io/rewrite-target: /
path: /argocd
hosts:
- www.example.com
and this I have defined under templates >> argocd-server-deployment.yaml
containers:
- name: argocd-server
image: {{ .Values.server.image.repository }}:{{ .Values.server.image.tag }}
imagePullPolicy: {{ .Values.server.image.pullPolicy }}
command:
- argocd-server
- --staticassets - /shared/app - --repo-server - argocd-repo-server:8081 - --insecure - --basehref - /argocd
The same case includes an instance very similar to yours:
In any case, do check your git configuration (git config -l) as seen in the ArgoCD cluster, to look for any insteadOf which would change automatically github.com into a local URL (as seen here)
I'm getting no healthy upstream error. when accessing ambassador. Pods/Services and Loadbalancer seems to be all fine and healthy. Ambassador is on top of aks.
At the moment I have got multiple services running in the Kubernetes cluster and each service has it's on Mapping with its own prefix. Is it possible to point out multiple k8s services to the same mapping so that I don't have too many prefixes? And all my k8s services will be under the same ambassador prefix?
By Default ambassador is taking me through https which is creating certificate issues, although I will be bringing https in near future for now I'm just looking to prove the concept so how can I disable HTTPS and do HTTP only ambassador?
No healthy upstream typically means that, for whatever reason, Ambassador cannot find the service listed in the mapping. The first thing I usually do when I see this is to run kubectl exec -it -n ambassador {my_ambassador_pod_name} -- sh and try to curl -v my-service where "my-service" is the Kube DNS name of the service you are trying to hit. Depending on the response, it can give you some hints on why Ambassador is failing to see the service.
Mappings work on a 1-1 basis with services. If your goal, however, is to avoid prefix usage, there are other ways Ambassador can match to create routes. One common way I've seen is to use host-based routing (https://www.getambassador.io/docs/latest/topics/using/headers/host/) and create subdomains for either individual or logical sets of services.
AES defaults to redirecting to HTTPS, but this behavior can be overwritten by applying a host with insecure routing behavior. A very simple one that I commonly use is this:
---
apiVersion: getambassador.io/v2
kind: Host
metadata:
name: wildcard
namespace: ambassador
spec:
hostname: "*"
acmeProvider:
authority: none
requestPolicy:
insecure:
action: Route
selector:
matchLabels:
hostname: wildcard
TL;DR: How can we configure istio sidecar injection/istio-proxy/envoy-proxy/istio egressgateway to allow long living (>3 hours), possibly idle, TCP connections?
Some details:
We're trying to perform a database migration to PostgreSQL which is being triggered by one application which has Spring Boot + Flyway configured, this migration is expected to last ~3 hours.
Our application is deployed inside our kubernetes cluster, which has configured istio sidecar injection. After exactly one hour of running the migration, the connection is always getting closed.
We're sure it's istio-proxy closing the connection as we attempted the migration from a pod without istio sidecar injection and it was running for longer than one hour, however this is not an option going forward as this may imply some downtime in production which we can't consider.
We suspect this should be configurable in istio proxy setting the parameter idle_timeout - which was implemented here. However this isn't working, or we are not configuring it properly, we're trying to configure this during istio installation by adding --set gateways.istio-ingressgateway.env.ISTIO_META_IDLE_TIMEOUT=5s to our helm template.
If you use istio version higher than 1.7 you might try use envoy filter to make it work. There is answer and example on github provided by #ryant1986.
We ran into the same problem on 1.7, but we noticed that the ISTIO_META_IDLE_TIMEOUT setting was only getting picked up on the OUTBOUND side of things, not the INBOUND. By adding an additional filter that applied to the INBOUND side of the request, we were able to successfully increase the timeout (we used 24 hours)
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: listener-timeout-tcp
namespace: istio-system
spec:
configPatches:
- applyTo: NETWORK_FILTER
match:
context: SIDECAR_INBOUND
listener:
filterChain:
filter:
name: envoy.filters.network.tcp_proxy
patch:
operation: MERGE
value:
name: envoy.filters.network.tcp_proxy
typed_config:
'#type': type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy
idle_timeout: 24h
We also created a similar filter to apply to the passthrough cluster (so that timeouts still apply to external traffic that we don't have service entries for), since the config wasn't being picked up there either.
for ingress gateway, we use env.ISTIO_META_IDLE_TIMEOUT to set the idle-timeout for TCP or HTTP protocol.
for sidecar, you can use the similar envoyfilter (listener-timeout-tcp) to configure INBOUND direction or OUTBOUND direction.
I am aware that client affinity is possible for a LoadBalancer type service in Kubernetes. The thing is that this affinity doesn't forbid that two different clientes access the same pod.
Is it possible to associate a pod exclusively always to the same client?
Thanks in advance and have a really nice day!
To only allow a specific external client/s to access a specific Pod/Deployment you can use whitelisting/source ranges. Restrictions can be applied to LoadBalancers as loadBalancerSourceRanges. You add a section to the Service like:
loadBalancerSourceRanges:
- 130.211.204.1/32
- 130.211.204.2/32
But not all cloud providers currently support it.
Alternatively you could expose the Pod with an Ingress and apply whitelisting on the Ingress. For whitelisting with an nginx Ingress you can add annotation to the Ingress such as nginx.ingress.kubernetes.io/whitelist-source-range: 49.36.X.X/32
No, this would imply that you’re running one copy of the service for every client which is a very non standard way to do things so you’ll have to build it yourself.
Not exactly to a POD.
You can use session affinity based on Client IP, that is of course only if the Client IP is static and only one client per IP.
apiVersion: v1
kind: Service
metadata:
name: wlp-service
labels:
app: wlp-service
spec:
type: LoadBalancer
sessionAffinity: ClientIP
ports:
- port: 443
targetPort: 7443
name: https
- port: 80
targetPort: 7080
name: http
selector:
app: POD_NAME
Second option is session affinity based on Cookies. This will work if there are several clients from the same IP, as cookies are stored locally on Client computer.
You will need to use an Ingress object and generate cookies. Your Ingress deployment will need to have:
Annotations:
affinity: cookie
session-cookie-hash: sha1/md5/index #choose one
session-cookie-name: INGRESSCOOKIE #name used in cookie value
You can read more about those two way on Redirect your users to the same pod by using session affinity on Kubernetes by medium.com
If I'm not mistaken Session Affinity will work only if IPVS kernel modules are installed on the node before running kube-proxy.
Run kube-proxy in IPVS Mode
Currently, local-up scripts, GCE scripts, and kubeadm support switching IPVS proxy mode via exporting environment variables (KUBE_PROXY_MODE=ipvs) or specifying flag (--proxy-mode=ipvs). Before running IPVS proxier, please ensure IPVS required kernel modules are already installed.
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack_ipv4
Finally, for Kubernetes v1.10, feature gate SupportIPVSProxyMode is set to true by default. For Kubernetes v1.11, the feature gate is entirely removed. However, you need to enable --feature-gates=SupportIPVSProxyMode=true explicitly for Kubernetes before v1.10.
Please check this StackOverflow question Is it possible to route traffic to a specific Pod?, also you can read more about IPVS on IPVS-Based In-Cluster Load Balancing Deep Dive
I am a newbie to Kubernetes and trying to learn calico networking.
I am following this documentation (https://docs.aws.amazon.com/eks/latest/userguide/calico.html)
and I tried to create a networkpolicy for the traffic to flow between backend to client :
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
namespace: stars
name: backend-client
spec:
podSelector:
matchLabels:
role : client
ingress:
- from:
- namespaceSelector:
matchLabels:
role: backend
ports:
- protocol: TCP
port: 9000
I finished all the 10 steps in the documentation, and i tried to test by creating a policy that would send traffic from the backend to the client with the above policy.
When i applied the policy there was no error , but i don't see the traffic/connection between the two.
Please let me know what is wrong.
Creating NetworkPolicy alone will not help in ensuring that the NetworkPolicy is enforced. We should configure the network plugin like Calico which is integrated with Kubernetes and executes the necessary operations to achieve the intent of the given Network Policy
https://kubernetes.io/docs/concepts/services-networking/network-policies/ says
"Network policies are implemented by the network plugin, so you must be using a networking solution which supports NetworkPolicy - simply creating the resource without a controller to implement it will have no effect."
I believe you need to put your policy in the client namespace instead of the stars namespace. I don't believe there are any pods with role: client in the stars namespace. A pod selector like you've specified only applies to pods in the namespace the policy is in.
While I don't think it is as direct as it could be the Kubernetes Network Policy docs do mention that a NetworkPolicy applies in the given namespace. I suggest you check them out if you haven't already.
I hope that helps.