Istio mTLS issue with bookinfo example application - kubernetes

Question

I've installed Istio 1.1 RC on a fresh GKE cluster, using Helm, and enabled mTLS (some options omitted like Grafana and Kiali):
helm template istio/install/kubernetes/helm/istio \
--set global.mtls.enabled=true \
--set global.controlPlaneSecurityEnabled=true \
--set istio_cni.enabled=true \
--set istio-cni.excludeNamespaces={"istio-system"} \
--name istio \
--namespace istio-system >> istio.yaml
kubectl apply -f istio.yaml
Next, I installed the Bookinfo example app like this:
kubectl label namespace default istio-injection=enabled
kubectl apply -f istio/samples/bookinfo/platform/kube/bookinfo.yaml
kubectl apply -f istio/samples/bookinfo/networking/bookinfo-gateway.yaml
kubectl apply -f istio/samples/bookinfo/networking/destination-rule-all-mtls.yaml
Then I've gone about testing by following the examples at: https://istio.io/docs/tasks/security/mutual-tls/
My results show the config is incorrect, but the verification guide above doesn't provide any hints about how to fix or diagnose issues. Here's what I see:
istio/bin/istioctl authn tls-check productpage.default.svc.cluster.local
Stderr when execute [/usr/local/bin/pilot-discovery request GET /debug/authenticationz ]: gc 1 #0.015s 6%: 0.016+1.4+1.0 ms clock, 0.064+0.31/0.45/1.6+4.0 ms cpu, 4->4->1 MB, 5 MB goal, 4 P
gc 2 #0.024s 9%: 0.007+1.4+1.0 ms clock, 0.029+0.15/1.1/1.1+4.3 ms cpu, 4->4->2 MB, 5 MB goal, 4 P
HOST:PORT STATUS SERVER CLIENT AUTHN POLICY DESTINATION RULE
productpage.default.svc.cluster.local:9080 OK mTLS mTLS default/ default/istio-system
This appears to show that mTLS is OK. And the previous checks all pass, like checking the cachain is present, etc. The above check passes for all the bookinfo components.
However, the following checks show an issue:
1: Confirm that plain-text requests fail as TLS is required to talk to httpbin with the following command:
kubectl exec $(kubectl get pod -l app=productpage -o jsonpath={.items..metadata.name}) -c istio-proxy -- curl http://productpage:9080/productpage -o /dev/null -s -w '%{http_code}\n'
200 <== Error. Should fail.
2: Confirm TLS requests without client certificate also fail:
kubectl exec $(kubectl get pod -l app=productpage -o jsonpath={.items..metadata.name}) -c istio-proxy -- curl https://productpage:9080/productpage -o /dev/null -s -w '%{http_code}\n' -k
000 <=== Correct behaviour
command terminated with exit code 35
3: Confirm TLS request with client certificate succeed:
kubectl exec $(kubectl get pod -l app=productpage -o jsonpath={.items..metadata.name}) -c istio-proxy -- curl https://productpage:9080/productpage -o /dev/null -s -w '%{http_code}\n' --key /etc/certs/key.pem --cert /etc/certs/cert-chain.pem --cacert /etc/certs/root-cert.pem -k
000 <=== Incorrect. Should succeed.
command terminated with exit code 35
What else can I do to debug my installation? I've followed the installation process quite carefully. Here's my cluster info:
Kubernetes master is running at https://<omitted>
calico-typha is running at https://<omitted>/api/v1/namespaces/kube-system/services/calico-typha:calico-typha/proxy
GLBCDefaultBackend is running at https://<omitted>/api/v1/namespaces/kube-system/services/default-http-backend:http/proxy
Heapster is running at https://<omitted>/api/v1/namespaces/kube-system/services/heapster/proxy
KubeDNS is running at https://<omitted>/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://<omitted>/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy
Kubernetes version: 1.11.7-gke.4
I guess I'm after either a more comprehensive guide or some specific things I can check.
Edit: Additional info:
Pod status:
default ns:
NAME READY STATUS RESTARTS AGE
details-v1-68868454f5-l7srt 2/2 Running 0 3h
productpage-v1-5cb458d74f-lmf7x 2/2 Running 0 2h
ratings-v1-76f4c9765f-ttstt 2/2 Running 0 2h
reviews-v1-56f6855586-qszpm 2/2 Running 0 2h
reviews-v2-65c9df47f8-ztrss 2/2 Running 0 3h
reviews-v3-6cf47594fd-hq6pc 2/2 Running 0 2h
istio-system ns:
NAME READY STATUS RESTARTS AGE
grafana-7b46bf6b7c-2qzcv 1/1 Running 0 3h
istio-citadel-5bf5488468-wkmvf 1/1 Running 0 3h
istio-cleanup-secrets-release-1.1-latest-daily-zmw7s 0/1 Completed 0 3h
istio-egressgateway-cf8d6dc69-fdmw2 1/1 Running 0 3h
istio-galley-5bcd455cbb-7wjkl 1/1 Running 0 3h
istio-grafana-post-install-release-1.1-latest-daily-vc2ff 0/1 Completed 0 3h
istio-ingressgateway-68b6767bcb-65h2d 1/1 Running 0 3h
istio-pilot-856849455f-29nvq 2/2 Running 0 2h
istio-policy-5568587488-7lkdr 2/2 Running 2 3h
istio-sidecar-injector-744f68bf5f-h22sp 1/1 Running 0 3h
istio-telemetry-7ffd6f6d4-tsmxv 2/2 Running 2 3h
istio-tracing-759fbf95b7-lc7fd 1/1 Running 0 3h
kiali-5d68f4c676-qrxfd 1/1 Running 0 3h
prometheus-c4b6997b-6d5k9 1/1 Running 0 3h
Example destinationrule:
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
creationTimestamp: "2019-02-21T15:15:09Z"
generation: 1
name: productpage
namespace: default
spec:
host: productpage
subsets:
- labels:
version: v1
name: v1
trafficPolicy:
tls:
mode: ISTIO_MUTUAL

Answer 1

If you are using Istio 1.1 RC, you should be looking at the docs at https://preliminary.istio.io/ instead of https://istio.io/. The preliminary.istio.io site is always the working copy of the docs, corresponding to the next to be Istio release (1.1 currently).
That said, those docs are currently changing a lot day-to-day as they are being cleaned up and corrected during final testing before 1.1 is released, probably in the next couple of weeks.
A possible explanation for the plain text http request returning 200 in you test is that you may be running with permissive mode.