Our Kubernetes cluster includes an nginx load balancer that forwards the requests to other pods.
However, the nginx sees local source IPs and therefore cannot set the correct X-Real-IP header. I tried setting the externalTrafficPolicy value of nginx to "Local" but the IP does not change.
Section of the nginx service config:
"selector": {
"app": "nginx-ingress",
"component": "controller",
"release": "loping-lambkin"
},
"clusterIP": "10.106.1.182",
"type": "LoadBalancer",
"sessionAffinity": "None",
"externalTrafficPolicy": "Local",
"healthCheckNodePort": 32718
Result:
GET / HTTP/1.1
Host: example.com:444
X-Request-ID: dd3310a96bf154d2ac38c8877dec312c
X-Real-IP: 10.39.0.0
X-Forwarded-For: 10.39.0.0
We use a bare metal cluster with metallb.
I found out that weave needs to be configured using NO_MASQ_LOCAL=1 to respect the externalTrafficPolicy property
This appears to be a bug in the IPVS implementation for services of type LoadBalancer : https://github.com/google/metallb/issues/290
Related
I have 2 worker nodes in a Kubernetes cluster. The worker nodes are on the same L2 domain.
$]cat ipvlanconf1.yaml
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: ipvlanconf1
namespace: cncf
spec:
config: '{
"cniVersion": "0.3.0",
"type": "ipvlan",
"master": "enp1s0.10",
"mode": "l3",
"vlan": 10,
"ipam": {
"type": "whereabouts",
"range": "10.1.1.1/24",
"gateway": "10.1.1.254"
}
}'
Pod00 on Worker-node0 is using IPVLAN. So, net1 gets 10.1.1.1
Pod01 on Worker-node1 is using IPVLAN. So, net1 gets 10.1.1.2
I want to able to ping 10.1.1.1 <---> 10.1.1.2 and it should carry the VLAN header. I don't see any in the tcpdump.
Questions:
I assumed that the VLAN header is inserted by the Pod itself. However, in the IPVLAN CNI I don't see any code where VLAN information is taken via config. Is my understanding correct?
Should interfaces in pod be explicitly configured as vlan-subinterfaces (net1.10) or should I do it on the worker node (enp1s0.10)?
What should I use as 'master' interface? enp1s0 or enp1s0.10?
Thanks
I have my hyperledger fabric blockchain deployed on k8s in the namespace: hlf-blockchain and my client app is deployed is in another namespace: hlf-app
The cpp-profile template is below. url-> grpcs://<service-name>.<namespace>:<port> which enables cross namespace communication.
{
"name": "test-network",
"version": "1.0.0",
"client": {
"organization": "MyOrg",
"connection": {
"timeout": {
"peer": {
"endorser": "10000"
}
}
}
},
"organizations": {
"TboxOrg": {
"mspid": "MyOrg",
"peers": [
"peer0",
"peer1",
"peer2"
],
"certificateAuthorities": [
"ca-myorg"
]
}
},
"peers": {
"peer0": {
"url": "grpcs://peer0.hlf-blockchain:${P0PORT}",
"tlsCACerts": {
"pem": "${PEERPEM}"
},
"grpcOptions": {
"ssl-target-name-override": "peer0",
"hostnameOverride": "peer0",
"request-timeout": 10000,
"grpc.keepalive_time_ms": 60000
}
},
"peer1": {
"url": "grpcs://peer1.hlf-blockchain:${P1PORT}",
"tlsCACerts": {
"pem": "${PEERPEM}"
},
"grpcOptions": {
"ssl-target-name-override": "peer1",
"hostnameOverride": "peer1",
"request-timeout": 10000,
"grpc.keepalive_time_ms": 60000
}
},
"peer2-tbox": {
"url": "grpcs://peer2.hlf-blockchain:${P2PORT}",
"tlsCACerts": {
"pem": "${PEERPEM}"
},
"grpcOptions": {
"ssl-target-name-override": "peer2",
"hostnameOverride": "peer2",
"request-timeout": 10000,
"grpc.keepalive_time_ms": 60000
}
}
},
"certificateAuthorities": {
"ca-tboxorg": {
"url": "https://ca-myorg.hlf-blockchain:${CAPORT}",
"caName": "ca-myorg",
"tlsCACerts": {
"pem": ["${CAPEM}"]
},
"httpOptions": {
"verify": false
}
}
}
}
From my client-app using fabrid-sdk-go I am able to connect to the network using the gateway. While invoking the chaincode I am getting the following error:
Endorser Client Status Code: (2) CONNECTION_FAILED. Description: dialing connection on target [peer0:7051]: connection is in TRANSIENT_FAILURE\nTransaction processing for endorser
I am able to invoke the transactions using cli command from the same namespace: hfl-blockchain
My peer service configuration:
kind: Service
apiVersion: v1
metadata:
name: peer0
labels:
app: peer0
spec:
selector:
name: peer0
type: ClusterIP
ports:
- name: grpc
port: 7051
protocol: TCP
- name: event
port: 7061
protocol: TCP
- name: couchdb
port: 5984
protocol: TCP
I believe this error is due to communication error between different namespace, which the client apps gets from the cpp-profile.
What's the correct way to configure the peer service or the cpp connection profile?
You are correct, the discovery service is returning network URLs that are unreachable from the hlf-blockchain namespace.
It is possible to run a Gateway client in a different namespace from the Fabric network. If you are using Kube DNS, each
of the fabric nodes can be referenced with a fully qualified host name <service-name>.<namespace>.svc.cluster.local.
In order to connect a gateway client across namespaces, you will need to introduce the .svc.cluster.local
Fully Qualified Domain Name to the fabric URLs returned by discovery:
In your TLS CA enrollments, make sure that the certificate signing requests include a valid Subject Alternate Name
with the FQDN. For example, if your peer0 TLS certificate is only valid for the host peer0, then the grpcs://
connection will be rejected in the TLS handshake when connecting to grpcs://peer0.hlf-blockchain.svc.cluster.local.
In the Gateway Client Connection Profile, use the FQDN when connecting to the discovery peers. In addition
to the peer url attribute, make sure to address host names in the grpcOptions stanzas.
Discovery will return the peer host names as specified in the core.yaml peer.gossip.externalendpoint
(CORE_PEER_GOSSIP_EXTERNALENDPOINT env) parameter. Make sure that this specifies the FQDN for all peers
visible to discovery.
Discovery will return the orderer host names as specified in the configtx.yaml organization OrdererEndpoints stanza.
Make sure that these URLs specify the FQDN for all orderers.
Regarding the general networking, make sure to double-check that the gateway client application has visibility and a
network route to the pods running fabric services in a different namespace. Depending on your Calico configuration
and Kube permissions, it's possible that traffic is getting blocked before it ever reaches the Fabric services.
Please, feel free to correct me on this.
So the problem lies with the discovery service. When the client tries to establish the connection to peer through gateway (which is embedded in the peer) in order to invoke transaction, the client receives the network topology- through the Discovery service which describes the no. of endorser peers, their url and other metadata.
This metadata is built from the configtx/core.yaml where we specify the peers and orderer (including their host and port).
Since my client app is in different namespace than the hyperledger fabric blockchain, the client needs the url in grpcs://.: but my configuration in the configtx/core.yaml were : which works if the client app is also in the same namespace as the blockchain nodes.
Also, one might think to name the service as peer0-myorg.<namespace> but k8s does not allows . in the name.
To fix my issue, I just moved the client app in the same namespace as that of blockchain network. Depending on your use-case if you're using DNS name for the services then you might not face this.
I'm working on helm-charts and it should do the following:
Allow external requests over HTTPS only (to a specific service).
Allow internal requests over HTTP (to the same specific service).
I tried creating two ingress in which:
The first ingress is external and accepts only HTTPS requests (which is working as expected). I used the following anntations:
kubernetes.io/ingress.class: nginx
ingress.citrix.com/ssl-passthrough: "true"
nginx.ingress.kubernetes.io/secure-backends: "true"
nginx.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
The second ingress is internal (with annotation kubernetes.io/ingress.class: "gce-internal"). This one is aimed to accept requests only from inside Minikube cluster. But it's is not working.
Running nslookup <host-name> from outside Minikube cluster returns ** server can't find <host-name>: NXDOMAIN.
Running nslookup <host-name> from inside MInikube cluster (minikube ssh) returns ;; connection timed out; no servers could be reached.
Running curl -kv "<host-name>" returns curl: (6) Could not resolve host: <host-name>.
Any ideas on how to fix this?
For internal communication ideally, you should be using the service name as the hostname.
However you can crate the internal ingress it is possible, but it will give you one single IP address. Using that IP address you can call the service.
Considering you are already running the ingress controller of Nginx and service.
You can edit the ingress service and inject or update this annotation to the service.
annotations:
cloud.google.com/load-balancer-type: "Internal"
this above line will create the Load Balancer service with the internal IP for ingress. You can create the ingress and resolve it using the curl.
https://stackoverflow.com/a/62559152/5525824
still with GCP if you are not getting the static IP address you can use this snippet
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-ingress
annotations:
kubernetes.io/ingress.regional-static-ip-name: STATIC_IP_NAME
kubernetes.io/ingress.class: "gce-internal"
read more at : https://cloud.google.com/kubernetes-engine/docs/how-to/internal-load-balance-ingress#static_ip_addressing
I deployed a grpc service on eks and expose the service using ingress. I deployed an demo https app, it worked. However, I have a problem with the grpc app. The service is running but when I log the service I get an error. The grpc request does not even go to the server. The log is as following
level=info msg="grpc: Server.Serve failed to create ServerTransport:
connection error: desc = \"transport: http2Server.HandleStreams
received bogus greeting from client: \\"GET / HTTP/1.1\\r\\nHost:
19\\"\"" system=system
It seems it should receive http2 but it just has HTTP/1.1??
For ingress I tried
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:xxxx
alb.ingress.kubernetes.io/load-balancer-attributes: 'routing.http2.enabled=true'
for service.yml
annotations:
service.alpha.kubernetes.io/app-protocols: '{"grpc":"HTTP2", "http": "HTTP2"}'
For the service deployed it seems fine. Once I have ingress deployed it keeps have the error above.
Not sure if Ingress supports HTTP2/gRPC. If you are using GKE, you could try ESP
I use Istio service mesh to solve this problem. Istio virtual service can route HTTP1.1 HTTP2 GRPC traffic
By setting service port name to grpc or prefixing it with grpc-. Istio will configure the service with HTTP2 protocol
I cannot reach the following Kubernetes service when externalTrafficPolicy: Local is set. I access it directly through the NodePort but always get a timeout.
{
"kind": "Service",
"apiVersion": "v1",
"metadata": {
"name": "echo",
"namespace": "default",
"selfLink": "/api/v1/namespaces/default/services/echo",
"uid": "c1b66aca-cc53-11e8-9062-d43d7ee2fdff",
"resourceVersion": "5190074",
"creationTimestamp": "2018-10-10T06:14:33Z",
"labels": {
"k8s-app": "echo"
}
},
"spec": {
"ports": [
{
"name": "tcp-8080-8080-74xhz",
"protocol": "TCP",
"port": 8080,
"targetPort": 3333,
"nodePort": 30275
}
],
"selector": {
"k8s-app": "echo"
},
"clusterIP": "10.101.223.0",
"type": "NodePort",
"sessionAffinity": "None",
"externalTrafficPolicy": "Local"
},
"status": {
"loadBalancer": {}
}
}
I know that for this pods of the service need to be available on a node because traffic is not routed to other nodes. I checked this.
Not sure where you are connecting from and what command you are typing to test connectivity or what's your environment like. But this is most likely due to this known issue where the node ports are not reachable with externalTrafficPolicy set to Local if the kube-proxy cannot find the IP address for the node where it's running on.
This link sheds more light into the problem. Apparently --hostname-override on the kube-proxy is not working as of K8s 1.10. You have to specify the HostnameOverride option in the kube-proxy ConfigMap. There's also a fix described here that will make it upstream at some point in the future from this writing.
As Jagrut said, the link shared in Rico's answer does not contain the desired section with the patch anymore, so I'll share a different thread where stacksonstacks' answer worked for me: here . This solution consists in editing kube-proxy.yaml to include the HOST_IP argument.
In my case, request to node cluster/public ip which own deployment/pod.
spec:
clusterIP: 10.9x.x.x <-- request this ip
clusterIPs:
- 10.9x.x.x
externalTrafficPolicy: Local
or
NAME STATUS ... EXTERNAL-IP
master-node Ready 3.2x.x.x
worker-node Ready 13.1x.x.x <-- request this ip
additionally, alway to request same node's ip, use nodeSelector in Deployment.