Istio on GKE - Admission webhook not working to inject sidecar-porxy - kubernetes

Question

I have been trying to set up Istio on my existing GKE cluster.
I hev followed the steps mentioned on the Istio website for the installation - prerequisite.
https://istio.io/latest/docs/setup/platform-setup/gke/
I have a private cluster so I added the firewall rule mention in the prerequisite.
gke-aiq-kubernetes-0a227ee8-all default INGRESS 1000 tcp,udp,icmp,esp,ah,sctp False
gke-aiq-kubernetes-0a227ee8-master default INGRESS 1000 tcp:10250,tcp:443,tcp:15017 False
gke-aiq-kubernetes-0a227ee8-vms default INGRESS 1000 tcp:1-65535,udp:1-65535,icmp False
and then install the istio with the demo profile.
istioctl install --set profile=demo
and then verify the intallation
istioctl verify-install
Which says everything is succeeded.
I labeled by namecpace "instio-inject=enabled" so that I will automatically get the sidecar porxy injected.
But when I trying to deploy something in the namespace I am getting following error:
Error from server (InternalError): error when creating "pod-pending.yaml": Internal error occurred: failed calling webhook "sidecar-injector.istio.io": Post https://istiod.istio-system.svc:443/inject?timeout=30s: context deadline exceeded
What I understand from this is that there is some connectivity issue, but I am not sure how to debug this.
I tried the debugging page from istio:
https://github.com/istio/istio/wiki/Troubleshooting-Istio#diagnostics
and after running the command:
kubectl get --raw /api/v1/namespaces/istio-system/services/https:istiod:https-webhook/proxy/inject -v4
I am conformend that this is connectivity issue:
t -v5
I1113 23:20:11.241079 40356 helpers.go:199] server response object: [{
"metadata": {},
"status": "Failure",
"message": "the server is currently unable to handle the request",
"reason": "ServiceUnavailable",
"details": {
"causes": [
{
"reason": "UnexpectedServerResponse",
"message": "Error trying to reach service: 'dial tcp 10.48.3.25:15017: i/o timeout'"
}
]
},
"code": 503
}]
F1113 23:20:11.241367 40356 helpers.go:114] Error from server (ServiceUnavailable): the server is currently unable to handle the request
Need help, I am new to GKE.

Answer 1

I figured out the issue.
I have 7 different GCP projects configured with my gcloud profile and I was in a different project while running the gcloud command. (all gke clusters have the same name)
Steps from the command line:
I logged in to the GCP UI from the browser search the firewall under VPC and opened the ports manually there and it worked.
gcloud compute firewall-rules list --filter="name~gke-<clustername>-[0-9a-z]*-master"
Then get the firewall rule name.
gcloud compute firewall-rules update <firewall rule name> --allow tcp:10250,tcp:443,tcp:15017
The idea here is to add the tcp:15017 which is required by the admission hook.