Summary
I have a simple Istio enabled k8s cluster consists of only:
A Java web server.
A Redis master instance.
Normally, the web server can read and write from Redis. However, Kiali shows a disconnected graph similar to (https://kiali.io/documentation/latest/faq/#disconnected-tcp). As a result, I tried to explicitly turn on mTLS by using STRICT mode. However, Kiali seems to continue to show disconnected graph
Set up:
Kubernetes version 1.18.0
Minikube version 1.18.0
Istio version 1.9
I followed Istio's Getting Started page to install Istio.
$ istioctl install --set profile=demo -y
$ kubectl apply -f samples/addons
Java Server code snippet (redis.clients.jedis.Jedis)
Jedis redis = new Jedis("redis-master");
redis.set(key, value);
mTLS
apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
name: "default"
spec:
mtls:
mode: STRICT
Questions
My understanding is that by default, mTLS should have been turned on by default. Is this not the case for non-HTTP TCP traffic?
Is there anything special I need to do to enable mTLS for non-HTTP TCP traffic? (e.g. change the port on the Service to 443 from 6379? Set up a VirtualService?).
According to istio documentation you have to configure redis to make it work with istio.
Similar to other services deployed in an Istio service mesh, Redis instances need to listen on 0.0.0.0. However, each Redis slave instance should announce an address that can be used by master to reach it, which cannot also be 0.0.0.0.
Use the Redis configuration parameter replica-announce-ip to announce the correct address. For example, set replica-announce-ip to the IP address of each Redis slave instance using these steps:
Pass the pod IP address through an environment variable in the env subsection of the slave StatefulSet definition:
- name: "POD_IP"
valueFrom:
fieldRef:
fieldPath: status.podIP
Also, add the following under the command subsection:
echo "" >> /opt/bitnami/redis/etc/replica.conf
echo "replica-announce-ip $POD_IP" >> /opt/bitnami/redis/etc/replica.conf
Related
I am running Kafka connect worker from Openshift cluster and able to access the connector APIs from the PODs terminal like below (listener in connect-distributed.properties as https://localhost:20000):
sh-4.2$ curl -k -X GET https://localhost:20000
{"version":"5.5.0-ce","commit":"dad78e2df6b714e3","kafka_cluster_id":"XojxTYmbTXSwHguxJ_flWg"}
I have created OpenShift routes with below config:
- kind: Route
apiVersion: v1
metadata:
name: '${APP_SHORT_NAME}-route'
labels:
app: '${APP_SHORT_NAME}'
annotations:
description: Route for application's http service.
spec:
host: '${APP_SHORT_NAME}.${ROUTE_SUFFIX}'
port:
targetPort: 20000-tcp
tls:
termination: reencrypt
destinationCACertificate: '${DESTINATION_CA_CERTIFICATE}'
to:
kind: Service
name: '${APP_NAME}-service'
Port 20000 is exposed from Dockerfile but the route URL is throwing below error instead:
Possible reasons you are seeing this page:
The host doesn't exist. Make sure the hostname was typed correctly and that a route matching this hostname exists.
The host exists, but doesn't have a matching path. Check if the URL path was typed correctly and that the route was created using the desired path.
Route and path matches, but all pods are down. Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running
Same OpenShift route URL works fine with a normal Spring boot service but Kafka connect worker URL is not getting bind to the Route URL create as above. (Note the Kafka connect worker is running fine along with the logs in the OpenShift pods)
Set the listeners back to the default of bind-address 0.0.0.0 so that external connections can be accepted.
If you need a different port, you can use port forwarding in the k8s service, rather than do that at the application config.
If you're not using Strimzi (which doesn't look like you are, based on 5.5.0-ce), you'll also want to add this env-var so a cluster can be formed.
- name: CONNECT_REST_ADVERTISED_HOST_NAME
valueFrom:
fieldRef:
fieldPath: status.podIP
I have 2 AKS clusters, Cluster 1 and Cluster 2 both running Istio 1.14 minimal out-of-the-box (default configs).
Everything on Cluster 1 works as expected (after deploying Istio).
On Cluster 2, all HTTPS outbound connections initiated from my services (injected with istio-proxy) fail.
curl http://www.google.com #works
curl https://www.google.com #fails
If I create a service entry for google, then the https curl works:
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: google
spec:
hosts:
- www.google.com
ports:
- number: 443
name: https
protocol: HTTPS
resolution: DNS
location: MESH_EXTERNAL
Both Istio installations are out-of-the-box so meshConfig.outboundTrafficPolicy.mode is set to ALLOW_ANY (double-checked).
I read online that there were some Istio bugs that would cause this behavior, but I don't think it's the case here. I also compared the Istio configs between the 2 clusters and they really seem to be the same.
I'm starting to think the problem may lie in some cluster configs because I know there are some differences between the 2 clusters here.
How would you go about troubleshooting this? Do you think the issue is related to Istio or Cluster services/configs? What should I look into first?
You are correct. By default ALLOW_ANY is value set for meshConfig.outboundTrafficPolicy.mode. This can be verified in the cluster by running below command.
kubectl get configmap istio -n istio-system -o yaml | grep -o "mode: ALLOW_ANY"
Please also refer the istio documentation for the options available in accessing external services
I am exploring the istio service mesh on my k8s cluster hosted on EKS(Amazon).
I tried deploying istio-1.2.2 on a new k8s cluster with the demo.yml file used for bookapp demonstration and most of the use cases I understand properly.
Then, I deployed istio using helm default profile(recommended for production) on my existing dev cluster with 100s of microservices running and what I noticed is my services can can call http endpoints but not able to call external secure endpoints(https://www.google.com, etc.)
I am getting :
curl: (35) error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong
version number
Though I am able to call external https endpoints from my testing cluster.
To verify, I check the egress policy and it is mode: ALLOW_ANY in both the clusters.
Now, I removed the the istio completely from my dev cluster and install the demo.yml to test but now this is also not working.
I try to relate my issue with this but didn't get any success.
https://discuss.istio.io/t/serviceentry-for-https-on-httpbin-org-resulting-in-connect-cr-srvr-hello-using-curl/2044
I don't understand what I am missing or what I am doing wrong.
Note: I am referring to this setup: https://istio.io/docs/setup/kubernetes/install/helm/
This is most likely a bug in Istio (see for example istio/istio#14520): if you have any Kubernetes Service object, anywhere in your cluster, that listens on port 443 but whose name starts with http (not https), it will break all outbound HTTPS connections.
The instance of this I've hit involves configuring an AWS load balancer to do TLS termination. The Kubernetes Service needs to expose port 443 to configure the load balancer, but it receives plain unencrypted HTTP.
apiVersion: v1
kind: Service
metadata:
name: breaks-istio
annotations:
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:...
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
spec:
selector: ...
ports:
- name: http-ssl # <<<< THIS NAME MATTERS
port: 443
targetPort: http
When I've experimented with this, changing that name: to either https or tcp-https seems to work. Those name prefixes are significant to Istio, but I haven't immediately found any functional difference between telling Istio the port is HTTPS (even though it doesn't actually serve TLS) vs. plain uninterpreted TCP.
You do need to search your cluster and find every Service that listens to port 443, and make sure the port name doesn't start with http-....
I've created a private GKE cluster with Istio through the Cloud Console UI. The cluster is set up with VPC Peering to be able to reach another private GKE cluster in another Google Cloud Project.
I've created a Deployment (called website) with a Service in Kubernetes in the staging namespace. My goal is to expose this service to the outside world with Istio, using the Envoy proxy. I've created the necessary VirtualService and Gateway to do so, following this guide.
When running "kubectl exec ..." to access a pod in the private cluster, I can successfully connect to the internal IP address of the website service, and see the output of that service with "curl".
I have set up a NAT Gateway so pods in the private cluster can connect to the Internet. I confirmed this by curl-ing various non-Google web pages from within the website pod.
However, I can't connect to the website service from the outside, using the External IP of the istio-ingressgateway service, as the guide above mentions. Instead, curl-ing that External IP leads to a timeout.
I've put the full YAML config for all related resources in a private Gist, here: https://gist.github.com/marceldegraaf/0f36ca817a8dba45ac97bf6b310ca282
I'm wondering if I'm missing something in my config here, or if my use case is actually impossible?
Looking at your Gist I suspect the problem lies in the joining up of the Gateway to the istio-ingressgateway.
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: website-gateway
namespace: staging
labels:
version: v1
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
In particular I'm not convinced the selector part is correct.
You should be able to do something like
kubectl describe po -n istio-system istio-ingressgateway-rrrrrr-pppp
to find out what the selector is trying to match in the Istio Ingress Gateway pod.
I had the same problem. On my case, the istio virtual service dont find my service.
Try this on your VirtualService:
route:
- destination:
host: website
port:
number: 80
From verifying all options, the only way to have the private GKE cluster with Istio to be exposed to externally is to use Cloud NAT.
Since the Master node within GKE is a managed service, there are current limits when using Istio with a private cluster. The only workaround that would accomplish your use case is to use Cloud NAT. I have also attached an article on how to get started using Cloud NAT here.
I am using the concourse helm build provided at https://github.com/kubernetes/charts/tree/master/stable/concourse to setup concourse inside of our kubernetes cluster. I have been able to get the setup working and I am able to access it within the cluster but I am having trouble accessing it outside the cluster. The notes from the build show that I can just use kubectl port-forward to get to the webpage but I don't want to have all of the developers have to forward the port just to get to the web ui. I have tried creating a service that has a node port like this:
apiVersion: v1
kind: Service
metadata:
name: concourse
namespace: concourse-ci
spec:
ports:
- port: 8080
name: atc
nodePort: 31080
- port: 2222
name: tsa
nodePort: 31222
selector:
app: concourse-web
type: NodePort
This allows me to get to the webpage and interact with it in most ways but then when I try to look at build status it never loads the events that happened. Instead a network request for /api/v1/builds/1/events is stuck in pending and the steps of the build never load. Any ideas what I can do to be able to completely access concourse external to the cluster?
EDIT: It seems like the events network request normally responds with a text/event-stream data type and maybe the Kubernetes service isn't handling an event stream correctly. Or there is something about concourse that handles event-streams different than the norm.
After plenty of investigation I have found that the the nodePort service is actually working and it is just my antivirus (Sophos) that is silently blocking the response from the events request.
Also, you can expose your port through loadbalancer in kubernetes.
kubectl get deployments
kubectl expose deployment <web pod name> --port=80 --target-port=8080 --name=expoport --type=LoadBalancer
It will create a public IP for you, and you will be able to access concourse on port 80.
not sure since I'm also a newbie but... you can configure your chart by providing your own version of https://github.com/kubernetes/charts/blob/master/stable/concourse/values.yaml
helm install stable/concourse -f custom_values.yaml
there is a 'externalURL' param, maybe worth trying to set it to your URL
## URL used to reach any ATC from the outside world.
##
# externalURL:
In addition, ... if you are on GKE, .... you can use an internal loadbalancer, ... set it up in your values.yaml file
service:
## For minikube, set this to ClusterIP, elsewhere use LoadBalancer or NodePort
## ref: https://kubernetes.io/docs/user-guide/services/#publishing-services---service-types
##
#type: ClusterIP
type: LoadBalancer
## When using web.service.type: LoadBalancer, sets the user-specified load balancer IP
# loadBalancerIP: 172.217.1.174
## Annotations to be added to the web service.
##
annotations:
# May be used in example for internal load balancing in GCP:
cloud.google.com/load-balancer-type: Internal