I'd like to launch a clustered Socket IO application in Kubernetes. When I create a service (whether NodePort or LoadBalancer) the client application keeps getting disconnected and it reconnects again with the following logs:
undefined
oah4g28zZCw36g1MAAAm
undefined
undefined
oac4g28zZCw36g1MFAAAx
undefined
and this happens rapidly.
However, when I connect to a single Pod directly, the problem goes away and the connection becomes stable.
How I am creating the service is by the following command:
kubectl expose deployment xxx --type=LoadBalancer --port=80 --target-port=3000
I know that something such as a KeepAlive or Timeout configuration is missing in the service, but how can I add those or better said properly configure the service for Socket IO?
You can use the sessionAffinity: ClientIP, which will manage the session from K8s service.
kind: Service
apiVersion: v1
metadata:
name: example
spec:
selector:
app: example
ports:
- name: http
protocol: TCP
port: 80
targetPort: 80
sessionAffinity: ClientIP
sessionAffinityConfig:
clientIP:
timeoutSeconds: 1800
just for ref : Does the ws websocket server library requires sticky session when it is used behind a load balancer?
Related
I have a container with IBM MQ (Docker image ibmcom/mq/9.2.2.0-r1) exposing two ports (9443 - admin, 1414 - application).
All required setup in OpenShift is done (Pod, Service, Routes).
There are two routes, one for each port.
https://route-admin.my.domain
https://route-app.my.domain
pointing to the ports accordingly (external ports are default http=80, https=443).
Admin console is accessible through the first route, hence, MQ is up and running.
I tried to connect as a client (JMS 2.0, com.ibm.mq.allclient:9.2.2.0) using standard approach:
var fctFactory = JmsFactoryFactory.getInstance(WMQConstants.WMQ_PROVIDER);
var conFactory = fctFactory.createConnectionFactory();
// ... other props
conFactory.setObjectProperty(WMQConstants.WMQ_HOST_NAME, "route-app.my.domain");
conFactory.setObjectProperty(WMQConstants.WMQ_PORT, 443);
and failed to connect. Also tried to redefine route as HTTP and use port 80, and again without success.
If it helps let's assume we use the latest version of MQ Explorer as a client.
Each time the same connection error appears:
...
Caused by: com.ibm.mq.MQException: JMSCMQ0001:
IBM MQ call failed with compcode '2' ('MQCC_FAILED') reason '2009' ('MQRC_CONNECTION_BROKEN').
...
Caused by: com.ibm.mq.jmqi.JmqiException:
CC=2;RC=2009;AMQ9204: Connection to host 'route-app.my.domain(443)' rejected.
[1=com.ibm.mq.jmqi.JmqiException[CC=2;RC=2009;AMQ9208:
Error on receive from host 'route-app.my.domain/10.227.248.2:443 (route-app.my.domain)'.
[1=-1,2=ffffffff,3=route-app.my.domain/10.227.248.2:443 (route-app.my.domain),4=TCP]],
3=route-app.my.domain(443),5=RemoteConnection.receiveTSH]
...
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2009;AMQ9208:
Error on receive from host 'route-app.my.domain/10.227.248.2:443
Maybe, this article could give some hints about error code 2009, but still not sure what exactly affects connection errors from the OpenShift side.
Previously, I always connected to IBM MQ specifying a port value explicitly, but here is a bit different situation.
How to connect to IBM MQ in OpenShift cluster through TCP?
Configurations in OpenShift are as follows:
kind: Pod
apiVersion: v1
metadata:
name: ibm-mq
labels:
app: ibm-mq
spec:
containers:
- resources:
limits:
cpu: '1'
memory: 600Mi
requests:
cpu: '1'
memory: 600Mi
name: ibm-mq
ports:
- containerPort: 1414
protocol: TCP
- containerPort: 9443
protocol: TCP
containerStatuses:
image: 'nexus-ci/docker-lib/ibm_mq:latest'
---
kind: Service
apiVersion: v1
metadata:
name: ibm-mq
spec:
ports:
- name: admin
protocol: TCP
port: 9443
targetPort: 9443
- name: application
protocol: TCP
port: 1414
targetPort: 1414
selector:
app: ibm-mq
---
kind: Route
apiVersion: route.openshift.io/v1
metadata:
name: ibm-mq-admin
spec:
host: ibm-mq-admin.my-domain.com
to:
kind: Service
name: ibm-mq
weight: 100
port:
targetPort: admin
tls:
termination: passthrough
insecureEdgeTerminationPolicy: None
wildcardPolicy: None
---
kind: Route
apiVersion: route.openshift.io/v1
metadata:
name: ibm-mq-app
spec:
host: ibm-mq-app.my-domain.com
to:
kind: Service
name: ibm-mq
weight: 100
port:
targetPort: application
tls:
termination: passthrough
insecureEdgeTerminationPolicy: None
wildcardPolicy: None
---
UPDATE: Ended up with creating and deploying to OpenShift a small web-application receiving HTTP requests and interacting with MQ via JMS (put/get text messages), like:
POST /queue/{queueName}/send + <body>;
GET /queue/{queueName}/receive.
It interacts with MQ inside the OpenShift cluster using TCP, and accepts external HTTP connections as a regular web application.
Other solutions seem to take too much efforts, but I accepted one of them as it is theoretically correct and straightforward.
I'm not sure to fully understand your setup, but"Routes"only route HTTP traffic (On ports 80 or 443 onyl), not TCP traffic.
If you want to access your MQ server from outside the cluster, there are a few solutions, one is to create a service of type: "NodePort"
Doc: https://docs.openshift.com/container-platform/4.7/networking/configuring_ingress_cluster_traffic/configuring-ingress-cluster-traffic-nodeport.html
Your Service is not a NodePort Service. In your case, it should be something like
kind: Service
apiVersion: v1
metadata:
name: ibm-mq
spec:
type: NodePort
ports:
- port: 1414
targetPort: 1414
nodePort: 30001
selector:
app: ibm-mq
Then access from outside with anyname.<cluster domaine>:30001
And delete the useless corresponding route. As said before, I assumed you read in the doc I pointed to you that says that route only route HTTP traffic on port 80 or 443.
Doc: https://kubernetes.io/docs/concepts/services-networking/service/#nodeport
The following Java system property will be read by IBM MQ classes for JMS at 9.2.1 and higher to tell it to set the SNI header to the hostname of the remote system when initiating a TLS connection:
com.ibm.mq.cfg.SSL.OutboundSNI=HOSTNAME
To set this programmatically just use the System.setProperty method for example:
System.setProperty("com.ibm.mq.cfg.SSL.OutboundSNI","HOSTNAME");
NOTE: the string HOSTNAME is literal and not meant to be replaced by a actual hostname.
If you can not move to a com.ibm.mq.allclient.jar from 9.2.1 or later, then in 9.2.0.0 and later you could instead use com.ibm.mq.cfg.SSL.AllowOutboundSNI=NO, but this is deprecated in 9.2.1 and later.
I am using HAProxy as the ingress-controller in my GKE clusters. And exposing HAProxy service as LoadBalancer service(Internal).
Recently, I experienced an issue, where the HA-Proxy service changed its EXTERNAL-IP, and traffic stopped routing to HAProxy. This issue occurred multiple times on different days(now it has stopped). I had to manually add that new External-IP to the frontend of that Loadbalancer to allow traffic to HAProxy.
There were two pods running for HAProxy, and both had been running for days, and there was nothing in their logs. I assume it was something related to Service or GCP LB and not HAProxy itself.
I am afraid that I don't have any logs related to that.
I still don't know, what caused the service IP to change. As there were no recent changes, and the cluster and all services were running for many days properly, and suddenly this occurred.
Has anyone faced a similar issue earlier? Or what can I do to avoid such issue in future?
What could have caused the IP to change?
This is how my service is configured:
---
apiVersion: v1
kind: Service
metadata:
labels:
run: haproxy-ingress
name: haproxy-ingress
namespace: haproxy-controller
annotations:
cloud.google.com/load-balancer-type: "Internal"
networking.gke.io/internal-load-balancer-allow-global-access: "true"
cloud.google.com/network-tier: "Premium"
spec:
selector:
run: haproxy-ingress
type: LoadBalancer
ports:
- name: http
port: 80
protocol: TCP
targetPort: 80
- name: https
port: 443
protocol: TCP
targetPort: 443
- name: stat
port: 1024
protocol: TCP
targetPort: 1024
Found some logs:
Warning SyncLoadBalancerFailed 30m (x3570 over 13d) service-controller Error syncing load balancer: failed to ensure load balancer: googleapi: Error 409: IP_IN_USE_BY_ANOTHER_RESOURCE - IP '10.17.129.17' is already being used by another resource.
Normal EnsuringLoadBalancer 3m33s (x3576 over 13d) service-controller Ensuring load balancer
The Short answer is: External IP for the service are ephemeral.
Because HA-Proxy controller pods are recreated the HA-Proxy service is created with an ephemeral IP.
To avoid this issue, I would recommend using a static IP that you can reference in the loadBalancerIP field.
This can be done by following steps:
Reserve a static IP. (link)
Use this IP, to create a service (link)
Example YAML:
apiVersion: v1
kind: Service
metadata:
name: helloweb
labels:
app: hello
spec:
selector:
app: hello
tier: web
ports:
- port: 80
targetPort: 8080
type: LoadBalancer
loadBalancerIP: "YOUR.IP.ADDRESS.HERE"
Unfortunately without logs it's hard to say anything for sure. You should check the audit logs that GKE ships to Cloud Logging as that might give you some idea of what happened. One option is the GCP "oops"'d the GLB and GKE recreated it, thus giving it a new IP. I've never heard of that happening with LBs though (it happens pretty often with nodes, but not LBs). A more common case would be you ran some kubectl command that inadvertently removed the Service object and then it was recreated by some management layer you have set up (Argo, Flux, Helm Operator, whatever) but delete+recreate again means it's a new LB with a new IP. The latter case should be visible in the audit logs so check those out for sure.
I am trying to setup K8S to work with two Windows Nodes (2019). Everything seems to be working well and the containers are working and accessible using k8s service. But, once I introduce configuration for readiness (or liveness) probes - all fails. The exact error is:
Readiness probe failed: Get http://10.244.1.28:80/test.txt: dial tcp 10.244.1.28:80: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
When I try the url from k8s master, it works well and I get 200. However I read that the kubelet is the one executing the probe and indeed when trying from the Windows Node - it cannot be reached (which seems weird because the container is running on that same node). Therefore I assume that the problem is related to some network configuration.
I have a HyperV with External network Virtual Switch configured. K8S is configured to use flannel overlay (vxlan) as instructed here: https://learn.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/network-topologies.
Any idea how to troubleshoot and fix this?
UPDATE: providing the yaml:
apiVersion: v1
kind: Service
metadata:
name: dummywebapplication
labels:
app: dummywebapplication
spec:
ports:
# the port that this service should serve on
- port: 80
targetPort: 80
selector:
app: dummywebapplication
type: NodePort
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
app: dummywebapplication
name: dummywebapplication
spec:
replicas: 2
template:
metadata:
labels:
app: dummywebapplication
name: dummywebapplication
spec:
containers:
- name: dummywebapplication
image: <my image>
readinessProbe:
httpGet:
path: /test.txt
port: 80
initialDelaySeconds: 15
periodSeconds: 30
timeoutSeconds: 60
nodeSelector:
beta.kubernetes.io/os: windows
And one more update. In this doc (https://kubernetes.io/docs/setup/windows/intro-windows-in-kubernetes/) it is written:
My Windows node cannot access NodePort service
Local NodePort access from the node itself fails. This is a known
limitation. NodePort access works from other nodes or external
clients.
I don't know if this is related or not as I could not connect to the container from a different node as stated above. I also tried a service of LoadBalancer type but it didn't provide a different result.
The network configuration assumption was correct. It seems that for 'overlay', by default, the kubelet on the node cannot reach the IP of the container. So it keeps returning timeouts and connection refused messages.
Possible workarounds:
Insert an 'exception' into the ExceptionList 'OutBoundNAT' of C:\k\cni\config on the nodes. This is somewhat tricky if you start the node with start.ps1 because it overwrites this file everytime. I had to tweak 'Update-CNIConfig' function in c:\k\helper.psm1 to re-insert the exception similar to the 'l2bridge' in that file.
Use 'l2bridge' configuration. Seems like 'overlay' is running in a more secured isolation, but l2bridge is not.
I have a setup Metallb as LB with Nginx Ingress installed on K8S cluster.
I have read about session affinity and its significance but so far I do not have a clear picture.
How can I create a single service exposing multiple pods of the same application?
After creating the single service entry point, how to map the specific client IP to Pod abstracted by the service?
Is there any blog explaining this concept in terms of how the mapping between Client IP and POD is done in kubernetes?
But I do not see Client's IP in the YAML. Then, How is this service going to map the traffic to respective clients to its endpoints? this is the question I have.
kind: Service
apiVersion: v1
metadata:
name: my-service
spec:
selector:
app: my-app
ports:
- name: http
protocol: TCP
port: 80
targetPort: 80
sessionAffinity: ClientIP
sessionAffinityConfig:
clientIP:
timeoutSeconds: 10000
Main concept of Session Affinity is to redirect traffic from one client always to specific node. Please keep in mind that session affinity is a best-effort method and there are scenarios where it will fail due to pod restarts or network errors.
There are two main types of Session Affinity:
1) Based on Client IP
This option works well for scenario where there is only one client per IP. In this method you don't need Ingress/Proxy between K8s services and client.
Client IP should be static, because each time when client will change IP he will be redirected to another pod.
To enable the session affinity in kubernetes, we can add the following to the service definition.
service.spec.sessionAffinity: ClientIP
Because community provided proper manifest to use this method I will not duplicate.
2) Based on Cookies
It works when there are multiple clients from the same IP, because it´s stored at web browser level. This method require Ingress object. Steps to apply this method with more detailed information can be found here under Session affinity based on Cookie section.
Create NGINX controller deployment
Create NGINX service
Create Ingress
Redirect your public DNS name to the NGINX service public/external IP.
About mapping ClientIP and POD, according to Documentation
kube-proxy is responsible for SessionAffinity. One of Kube-Proxy job
is writing to IPtables, more details here so thats how it is
mapped.
Articles which might help with understanding Session Affinity:
https://sookocheff.com/post/kubernetes/building-stateful-services/
https://medium.com/#diegomrtnzg/redirect-your-users-to-the-same-pod-by-using-session-affinity-on-kubernetes-baebf6a1733b
follow the service reference for session affinity
kind: Service
apiVersion: v1
metadata:
name: my-service
spec:
selector:
app: my-app
ports:
- name: http
protocol: TCP
port: 80
targetPort: 80
sessionAffinity: ClientIP
sessionAffinityConfig:
clientIP:
timeoutSeconds: 10000
I am using stateful in kubernetes.
I write an application which will have leader and follower (using Go)
Leader is for writing and reading.
Follower is just for reading.
In the application code, I used "http.Redirect(w, r, url, 307)" function to redirect the writing from follower to leader.
if I use a jump pod to test the application (try to access the app to read and write), my application can work well, the follower can redirect to the leader
kubectl run -i -t --rm jumpod --restart=Never --image=quay.io/mhausenblas/jump:0.2 -- sh
But when I deploy a service (to access from outside).
apiVersion: v1
kind: Service
metadata:
name: service-name
spec:
selector:
app: app-name
ports:
- port: 80
targetPort: 9876
And access to application by this link:
curl -L -XPUT -T /tmp/put-name localhost:8001/api/v1/namespaces/default/services/service-name/proxy/
Because service will randomly select a pod each time we access to the application. When it accesses to leader, it work well (no problem happen), But if it accesses to the follower, the follower will need to redirect to leader, I met this error:
curl: (6) Could not resolve host: pod-name.svc-name.default.svc.cluster.local
What I tested:
I can use this link to access when I used jump pod
I accessed to per pod, look up the DNS. I can find the DNS name by "nslookup" command
I tried to fix the IP of leader in my code. In my code, the follower will redirect to a IP (not a domain like above). But It still met this error:
curl: (7) Failed to connect to 10.244.1.71 port 9876: No route to host
Anybody know this problem. Thank you!
In order for pod DNS to work you must create headless service:
apiVersion: v1
kind: Service
metadata:
name: svc-name-headless
spec:
clusterIP: None
selector:
app: app-name
ports:
- port: 80
targetPort: 9876
And then in StatefulSet spec you must refer to this service:
spec:
serviceName: svc-name-headless
Read more here: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#stable-network-id
Alternatively you may specify what particular pod will be selected by Service like that:
apiVersion: v1
kind: Service
metadata:
name: svc-name
spec:
selector:
statefulset.kubernetes.io/pod-name: pod-name-0
ports:
- port: 80
targetPort: 9876
When you are accessing your cluster services from outside, a DNS names reserved normally for inner cluster use (e.g. pod-name.svc-name.default.svc.cluster.local) are not recognized by clients (e.g. Web browser).
Context:
You are trying to expose access to PODs, controlled by StatefullSet with service of ClusterIP type
Solution:
Change ClusterIP (assumed by default when not specified) to NodePort or LoadBalancer