I am totally a newbie in key-cloak and keycloak-gatekeeper
I am running keycloak on my machine with address localhost:8080 where as my gatekeeper is running on minikube environment
Below are the my configuration and deployment file
apiVersion: apps/v1
kind: Deployment
metadata:
name: gatekeeperdeployment
labels:
app: gatekeeperapp
spec:
replicas: 1
selector:
matchLabels:
app: gatekeeperapp
template:
metadata:
labels:
app: gatekeeperapp
spec:
containers:
- image: keycloak/keycloak-gatekeeper:7.0.0
imagePullPolicy: IfNotPresent
name: gatekeeperalone
ports:
- containerPort: 3000
args:
- --config=/etc/secrets/config.yml
- --redirection-url=http://127.0.0.1:3000
- --upstream-url=http://192.168.99.101:30732
- --resources=uri=/prods
- --enable-logging=true
- --enable-json-logging=true
- --verbose=true
volumeMounts:
- name: secret
mountPath: /etc/secrets
volumes:
- name: secret
secret:
secretName: secret-gatekeeper
and my config file is as:
client-id: ems
client-secret: 17a9da01-880a-432c-ba5a-8cbf131d5634
discovery-url: http://127.0.0.1:8180/auth/realms/ems_realm
redirection-url: http://127.0.0.1:3000
upstream-url: http://192.168.99.101:30732
skip-upstream-tls-verify: true
skip-openid-provider-tls-verify: true
encryption-key: 1234567898765431
listen: 3000
secure-cookie: false
enable-logging: true
enable-json-logging: true
enable-default-deny: true
enable-refresh-tokens: true
enable-session-cookies: true
debug: true
ingress.enabled: true
resources:
- uri: /prods
i have set the localhost path in discovery-url parameter.
So, with above configuration i am not able to communicate with the keycloak. Please provide some guidance over this behaviour. Any kind of help is much appreciated.
Are you really sure that Keycloak is running on the 127.0.0.1:8180 (discovery-url) in the gatekeeper container? Containers are running in the own network namespaces, so localhost in the container is different as localhost in the OS. Configure that discovery-url correctly or configure used network namespace of the host (e.g. use host network if keycloak is running in the host network).
Related
I am trying to connect to Firestore from code running on GKE Container. Simple REST GET api is working fine, but when I access the Firestore from read/write, I am getting Missing or insufficient permissions.
An unhandled exception was thrown by the application.
Info
2021-06-06 21:21:20.283 EDT
Grpc.Core.RpcException: Status(StatusCode="PermissionDenied", Detail="Missing or insufficient permissions.", DebugException="Grpc.Core.Internal.CoreErrorDetailException: {"created":"#1623028880.278990566","description":"Error received from peer ipv4:172.217.193.95:443","file":"/var/local/git/grpc/src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Missing or insufficient permissions.","grpc_status":7}")
at Google.Api.Gax.Grpc.ApiCallRetryExtensions.<>c__DisplayClass0_0`2.<<WithRetry>b__0>d.MoveNext()
Update I am trying to provide secret to pod with service account credentails.
Here is the k8 file which deploys a pod to cluster with no issues when no secrets are provided and I can do Get Operations which don't hit Firestore, and they work fine.
kind: Deployment
apiVersion: apps/v1
metadata:
name: foo-worldmanagement-production
spec:
replicas: 1
selector:
matchLabels:
app: foo
role: worldmanagement
env: production
template:
metadata:
name: worldmanagement
labels:
app: foo
role: worldmanagement
env: production
spec:
containers:
- name: worldmanagement
image: gcr.io/foodev/foo/master/worldmanagement.21
resources:
limits:
memory: "500Mi"
cpu: "300m"
imagePullworld: Always
readinessProbe:
httpGet:
path: /api/worldManagement/policies
port: 80
ports:
- name: worldmgmt
containerPort: 80
Now, if I try to mount secret, the pod never gets created fully, and it eventually fails
kind: Deployment
apiVersion: apps/v1
metadata:
name: foo-worldmanagement-production
spec:
replicas: 1
selector:
matchLabels:
app: foo
role: worldmanagement
env: production
template:
metadata:
name: worldmanagement
labels:
app: foo
role: worldmanagement
env: production
spec:
volumes:
- name: google-cloud-key
secret:
secretName: firestore-key
containers:
- name: worldmanagement
image: gcr.io/foodev/foo/master/worldmanagement.21
volumeMounts:
- name: google-cloud-key
mountPath: /var/
env:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /var/key.json
resources:
limits:
memory: "500Mi"
cpu: "300m"
imagePullworld: Always
readinessProbe:
httpGet:
path: /api/worldManagement/earth
port: 80
ports:
- name: worldmgmt
containerPort: 80
I tried to deploy the sample application and it works fine.
If I keep only the following the yaml file, the container gets deployed properly
- name: google-cloud-key
secret:
secretName: firestore-key
But once I add the following to yaml, it fails
volumeMounts:
- name: google-cloud-key
mountPath: /var/
env:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /var/key.json
And I can see in GCP events that the container is not able to find the google-cloud-key. Any idea how to troubleshoot this issue, i.e why I am not able to mount the secrets, I can bash into the pod if needed.
I am using multi stage docker file made of
From mcr.microsoft.com/dotnet/sdk:5.0 AS build
FROM mcr.microsoft.com/dotnet/aspnet:5.0 AS runtime
Thanks
Looks like they key itself might not be correctly visible to the pod. I would start by getting into the pod with kubectl exec --stdin --tty <podname> -- /bin/bash and ensuring that the /var/key.json (per your config) is accessible and has the correct credentials.
The following would be a good way to mount the secret:
volumeMounts:
- name: google-cloud-key
mountPath: /var/run/secret/cloud.google.com
env:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /var/run/secret/cloud.google.com/key.json
The above assumes your secret was created with a command like:
kubectl --namespace <namespace> create secret generic firestore-key --from-file key.json
Also it is important to check your Workload Identity setup. The Workload Identity | Kubernetes Engine Documentation has a good section on this.
Learning Kubernetes by setting up two pods, each running an elastic-search and a kibana container respectively.
My configuration file is able to setup both pods as well as create two services to access these applications on host machine's web browser.
Issue is that i don't know how to make Kibana container communicate with ES application/pod.
Earlier while learning Docker i crafted a docker-compose app configuration and now basically trying to do the same using Kubernetes ( docker-compose config pasted below ) .
Came across a blog that suggested using Deployment instead of Pod. Again not sure how would one make Kibana talk to ES
Kubernetes configuation yaml:
apiVersion: v1
kind: Pod
metadata:
name: pod-elasticsearch
labels:
app: myapp
spec:
hostname: "es01-docker-local"
containers:
- name: myelasticsearch-container
image: myelasticsearch
imagePullPolicy: Never
volumeMounts:
- name: my-volume
mountPath: /home/newuser
volumes:
- name: my-volume
emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
name: myelasticsearch-service
spec:
type: NodePort
ports:
- targetPort: 9200
port: 9200
nodePort: 30015
selector:
app: myapp
---
apiVersion: v1
kind: Pod
metadata:
name: pod-kibana
labels:
app: myapp
spec:
containers:
- name: mykibana-container
image: mykibana
imagePullPolicy: Never
volumeMounts:
- name: my-volume
mountPath: /home/newuser
volumes:
- name: my-volume
emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
name: mykibana-service
spec:
type: NodePort
ports:
- targetPort: 5601
port: 5601
nodePort: 30016
selector:
app: myapp
For reference below is the docker-compose that i am trying to replicate on Kubernetes
version: "2.2"
services:
elasticsearch:
image: myelasticsearch
container_name: myelasticsearch-container
restart: always
hostname: 'es01.docker.local'
ports:
- '9200:9200'
- '9300:9300'
volumes:
- myVolume:/home/newuser/
environment:
- discovery.type=single-node
kibana:
depends_on:
- elasticsearch
image: mykibana
container_name: mykibana-container
restart: always
ports:
- '5601:5601'
volumes:
- myVolume:/home/newuser/
environment:
ELASTICSEARCH_URL: http://es01:9200
ELASTICSEARCH_HOSTS: http://es01:9200
volumes:
myVolume:
networks:
myNetwork:
ES Pod description:
% kubectl describe pod/pod-elasticsearch
Name: pod-elasticsearch
Namespace: default
Priority: 0
Node: docker-desktop/192.168.65.3
Start Time: Sun, 10 Jan 2021 23:06:18 -0800
Labels: app=myapp
Annotations: <none>
Status: Running
IP: 10.x.0.yy
IPs:
IP: 10.x.0.yy
In kubernetes Pod/Deployment/DaemonSet... in the same cluster can communicate with each other with no problem because it has a flat network architecture .One way for these resources to call each other directly is by the name of Kubernetes service of each resource.
For example any resource in the cluster can call your kibana-app directly by service name you give it to it mykibana-service.name-of-namespace.
So for kibana pod to communicate with elasticsearch it can use http://name-of-service-of-elasticsearch.name-of-namespace:9200 namespace is be default if you dont specify where you create your service => http://name-of-service-of-elasticsearch.default:9200 or http://name-of-service-of-elasticsearch:9200
The concern you raised on what type of your resource you have to create (pod, deployment,daemonset or statefulSet) is not important for these resources to communicate with each other.
If you re having problem converting docker-compose to manifest file you can start with Kompose you can do kompose convert where is your docker-compose is located .
Here sample
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: elasticsearch
name: elasticsearch
namespace: default
spec:
selector:
matchLabels:
app: elasticsearch
template:
metadata:
labels:
app: elasticsearch
spec:
containers:
- image: myelasticsearch:yourtag #fix this
name: elasticsearch
ports:
- containerPort: 9200
- containerPort: 9300
volumeMounts:
- mountPath: /home/newuser/
name: my-volume
volumes:
- name: my-volume
emptyDir: {} # I wouldnt use emptydir
---
apiVersion: v1
kind: Service
metadata:
labels:
app: elasticsearch
name: elasticsearch
namespace: default
spec:
ports:
- port: 9200
name: "9200"
targetPort: 9200
- port: 9300
name: "9300"
targetPort: 9300
selector:
app: elasticsearch
type: ClusterIP #you dont need to make expose your service publicly
#####################################
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: kibana
name: kibana
namespace: default
spec:
selector:
matchLabels:
app: kibana
template:
metadata:
labels:
app: kibana
spec:
containers:
- env:
- name: ELASTICSEARCH_URL
value: http://elasticsearch:9200/ #elasticsearch is the same name as service resrouce name
- name: ELASTICSEARCH_HOSTS
value: http://elasticsearch:9200
image: mykibana:yourtagname #fix this
name: kibana
---
apiVersion: v1
kind: Service
metadata:
labels:
app: kibana
name: kibana
namespace: default
spec:
ports:
- port: 5601
protocol: TCP
targetPort: 5601
selector:
app: kibana
type: NodePort
You can choose whats adequate for your app , for example in elasticsearch you can use StatefulSet ,Deployment, in ElasticSearch, and you can you use Deployment for Kibana , Also you can change the type of volume .
Also the mynetwork that you created in docker-compose can be translated network policy where you can isolate your resources (for example isolated mynetwork namespace) because these resources are not isolated if they are created in the same cluster by default.
Hope I helped
If you want to deploy Elasticsearch and Kibana in Kubernetes the usual way then you have to take care of some core Elasticsearch cluster configuration like:
cluster.initial_master_nodes [7.0] Added in 7.0.
network.host
network.publish_host
Also you would have to carefully setup the network.host so that even after accidental pod restarts the network.host remains the same.
While deploying Kibana you need provide Elasticsearch service and also manually configure the SSL certificates if Elasticsearch has SSL enabled.
So to install Elastic Stack on Kubernetes then you should probably prefer
Elastic Cloud on Kubernetes (ECK). The documentation provided by Elastic is easy to understand.
Elastic Cloud on Kubernetes (ECK) uses Kubernetes Operators to make installation easier and it automatically takes care of core cluster configuration.
ECK installation will create a default user called "elastic" and you can retrieve its password from secrets. It also creates self-signed certificates which can be found in secrets.
For deploying Kibana you can just provide "elasticsearchRef" in your YAML file and it will automatically configure the Elasticsearch endpoints. You can use the default "elastic" user to login to Kibana.
I have a gRPC server that works fine on my local machine. I can send grpc requests from a python app and get back the right responses.
I put the server into a GKE cluster (with only one node). I had a normal TCP load balancer in front of the cluster. In this setup my local client was able to get the correct response from some requests, but not others. I think it was the gRPC streaming that didn't work.
I assumed that this is because the streaming requires an HTTP/2 connection which requires SSL.
The standard load balancer I got in GKE didn't seem to support SSL, so I followed the docs to set up an ingress load balancer which does. I'm using a Lets-Encrypt certificate with it.
Now all gRPC requests return
status = StatusCode.UNAVAILABLE
details = "Socket closed"
debug_error_string =
"{"created":"#1556172211.931158414","description":"Error received from
peer
ipv4:ip.of.ingress.service:443", "file":"src/core/lib/surface/call.cc", "file_line":1041,"grpc_message":"Socket closed","grpc_status":14}"
The IP address is the external IP address of my ingress service.
The ingress yaml looks like this:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: rev79-ingress
annotations:
kubernetes.io/ingress.global-static-ip-name: "rev79-ip"
ingress.gcp.kubernetes.io/pre-shared-cert: "lets-encrypt-rev79"
kubernetes.io/ingress.allow-http: "false" # disable HTTP
spec:
rules:
- host: sub-domain.domain.app
http:
paths:
- path: /*
backend:
serviceName: sandbox-nodes
servicePort: 60000
The subdomain and domain of the request from my python app match the host in the ingress rule.
It connects to a node-port that looks like this:
apiVersion: v1
kind: Service
metadata:
name: sandbox-nodes
spec:
type: NodePort
selector:
app: rev79
environment: sandbox
ports:
- protocol: TCP
port: 60000
targetPort: 9000
The node itself has two containers and looks like this:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: rev79-sandbox
labels:
app: rev79
environment: sandbox
spec:
replicas: 1
template:
metadata:
labels:
app: rev79
environment: sandbox
spec:
containers:
- name: esp
image: gcr.io/endpoints-release/endpoints-runtime:1.31
args: [
"--http2_port=9000",
"--service=rev79.endpoints.rev79-232812.cloud.goog",
"--rollout_strategy=managed",
"--backend=grpc://0.0.0.0:3011"
]
ports:
- containerPort: 9000
- name: rev79-uac-sandbox
image: gcr.io/rev79-232812/uac:latest
imagePullPolicy: Always
ports:
- containerPort: 3011
env:
- name: RAILS_MASTER_KEY
valueFrom:
secretKeyRef:
name: rev79-secrets
key: rails-master-key
The target of the node port is the ESP container which connects to the gRPC service deployed in the cloud, and the backend which is a Rails app that implements the backend of the API. This rails app isn't running the rails server, but a specialised gRPC server that comes with the grpc_for_rails gem
The grpc_server in the Rails app doesn't record any action in the logs, so I don't think the request gets that far.
kubectl get ingress reports this:
NAME HOSTS ADDRESS PORTS AGE
rev79-ingress sub-domain.domain.app my.static.ip.addr 80 7h
showing port 80, even though it's set up with SSL. That seems to be a bug. When I check with curl -kv https://sub-domain.domain.app the ingress server handles the request fine, and uses HTTP/2. It reurns an HTML formatted server error, but I'm not sure what generates that.
The API requires an API key, which the python client inserts into the metadata of each request.
When I go to the endpoints page of my GCP console I see that the API is not registering any requests since putting in the ingress loadbalancer, so it looks like the requests are not reaching the EPS container.
So why am I getting "socket closed" errors with gRPC?
I said I would come back and post an answer here once I got it working. It looks like I never did. Being a man of my word I'll post now my config files which are working for me.
in my deployment I've put a liveness and readiness probe for the ESP container. This made deployments happen smoothly without downtime:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: rev79-sandbox
labels:
app: rev79
environment: sandbox
spec:
replicas: 3
template:
metadata:
labels:
app: rev79
environment: sandbox
spec:
volumes:
- name: nginx-ssl
secret:
secretName: nginx-ssl
- name: gcs-creds
secret:
secretName: rev79-secrets
items:
- key: gcs-credentials
path: "gcs.json"
containers:
- name: esp
image: gcr.io/endpoints-release/endpoints-runtime:1.45
args: [
"--http_port", "8080",
"--ssl_port", "443",
"--service", "rev79-sandbox.endpoints.rev79-232812.cloud.goog",
"--rollout_strategy", "managed",
"--backend", "grpc://0.0.0.0:3011",
"--cors_preset", "cors_with_regex",
"--cors_allow_origin_regex", ".*",
"-z", " "
]
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 5
periodSeconds: 5
failureThreshold: 3
readinessProbe:
httpGet:
path: /healthz
port: 8080
timeoutSeconds: 5
failureThreshold: 1
volumeMounts:
- name: nginx-ssl
mountPath: /etc/nginx/ssl
readOnly: true
ports:
- containerPort: 8080
- containerPort: 443
protocol: TCP
- name: rev79-uac-sandbox
image: gcr.io/rev79-232812/uac:29eff5e
imagePullPolicy: Always
volumeMounts:
- name: gcs-creds
mountPath: "/app/creds"
ports:
- containerPort: 3011
name: end-grpc
- containerPort: 3000
env:
- name: RAILS_MASTER_KEY
valueFrom:
secretKeyRef:
name: rev79-secrets
key: rails-master-key
This is my service config that exposes the deployment to the load balancer:
apiVersion: v1
kind: Service
metadata:
name: rev79-srv-ingress-sandbox
labels:
type: rev79-srv
annotations:
service.alpha.kubernetes.io/app-protocols: '{"rev79":"HTTP2"}'
cloud.google.com/neg: '{"ingress": true}'
spec:
type: NodePort
ports:
- name: rev79
port: 443
protocol: TCP
targetPort: 443
selector:
app: rev79
environment: sandbox
And this is my ingress:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: rev79-ingress
annotations:
kubernetes.io/ingress.global-static-ip-name: "rev79-global-ip"
spec:
tls:
- secretName: sandbox-api-rev79-app-tls
rules:
- host: sandbox-api.rev79.app
http:
paths:
- backend:
serviceName: rev79-srv-ingress-sandbox
servicePort: 443
I'm using cert-manager to manage the certificates.
It was a long time agao now. I can't remember if there was anything else I did to solve the issue I was having
I have a (containerized) web service talking to an external CloudSQL service in Google Cloud. I've used the sidecar pattern in which a Google Cloud SQL Proxy container is next to the web service and authenticates+proxies to the external CloudSQL service. This works fine. Let's call this Deployment "deployment-api" with containers "api" + "pg-proxy"
The problem occurs when I want to deploy the application on my local minikube cluster which needs to have different configuration due to the service talking to a local postgres server on my computer. If I deploy "deployment-api" as is to minikube, it tries to run the "pg-proxy" container which barfs and the entire pod goes into a crash loop. Is there a way for me to selectively NOT deploy "pg-proxy" container without having two definitions for the Pod, e.g., using selectors/labels? I do not want to move pg-proxy container into its own deployment.
apiVersion: apps/v1
kind: Deployment
metadata:
name: deployment-api
namespace: ${MY_ENV}
labels:
app: api
env: ${MY_ENV}
spec:
replicas: ${REPLICAS}
selector:
matchLabels:
app: api
env: ${MY_ENV}
template:
metadata:
labels:
app: api
env: ${MY_ENV}
spec:
containers:
- name: pg-proxy
ports:
- containerPort: 5432
image: gcr.io/cloudsql-docker/gce-proxy:1.11
command: ["/cloud_sql_proxy",
"-instances=<redacted>:${MY_ENV}-app=tcp:5432",
"-credential_file=/secrets/cloudsql/${MY_ENV}-sql-credentials.json"]
securityContext:
runAsUser: 2 # non-root user
allowPrivilegeEscalation: false
volumeMounts:
- name: ${MY_ENV}-cloudsql-instance-credentials
mountPath: /secrets/cloudsql
readOnly: true
- name: api
image: ${DOCKER_IMAGE_PREFIX}api:${TAG}
imagePullPolicy: ${PULL_POLICY}
ports:
- containerPort: 50051
volumes:
- name: ${MY_ENV}-cloudsql-instance-credentials
secret:
secretName: ${MY_ENV}-cloudsql-instance-credentials
In raw Kubernetes means? No.
But I strongly encourage you to use Helm to deploy your application(s). With helm you can easily adapt manifest based on variables provided for each environment (or defaults). For example with variable postgresql.proxy.enabled: true in defaults and
{{- if .Values.postgresql.proxy.enabled }}
- name: pg-proxy
...
{{- end }}
in helm template you could disable this block completely on dev env by setting the value to false.
I would like to connect my Kubernetes cluster to Google Cloud SQL.
I have at least 10 different deployed pods which presently connect to MySQL [docker image deployed to k8s] using a JDBC url + username/password.
It it possible to use a single instance of the Google Cloud SQL Proxy and connect all the pods through this proxy to the Cloud SQL database? Ideally I would like to replace the mysql running in the container with the proxy.
I would prefer not having to run the proxy inside each deployment. The only samples I found seem to indicate the proxy needs to be declared in each deployment.
I found a solution.
Deploy the proxy with the yml below, and expose the deployment as a service. Most importantly, make the proxy listen on 0.0.0.0, instead of default 127.0.0.1. All the secrets as per the Google Cloud sql documentation
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: mysql
spec:
replicas: 1
template:
metadata:
name: mysql
labels:
name: mysql
spec:
containers:
- image: b.gcr.io/cloudsql-docker/gce-proxy:1.05
name: cloudsql-proxy
command: ["/cloud_sql_proxy", "--dir=/cloudsql",
"-instances=MYSQL:ZONE:DATABASE_INSTANCE=tcp:0.0.0.0:3306",
"-credential_file=/secrets/cloudsql/credentials.json"]
volumeMounts:
- name: cloudsql-oauth-credentials
mountPath: /secrets/cloudsql
readOnly: true
- name: ssl-certs
mountPath: /etc/ssl/certs
ports:
- containerPort: 3306
name: mysql
volumes:
- name: cloudsql-oauth-credentials
secret:
secretName: cloudsql-oauth-credentials
- name: ssl-certs
hostPath:
path: /etc/ssl/certs
The solution is slightly more expensive than having the proxy in the same deployment as the client software, since there is an extra TCP connection.
However there are many benefits:
Much simpler and doesn't require modifying existing K8S deployment files
Allows switching the implementation to a MySQL Docker container or using the Google Cloud SQL proxy without any modifications to the client configuration.
You can create a deployment and a service to expose the cloudsql proxy to other pods like so:
apiVersion: v1
kind: Service
metadata:
name: cloudsqlproxy
spec:
ports:
- port: 3306
targetPort: database-port
selector:
app: cloudsqlproxy
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: cloudsqlproxy
spec:
template:
metadata:
labels:
app: cloudsqlproxy
spec:
volumes:
- name: service-account-token
secret:
secretName: service-account-token
containers:
- name: cloudsql-proxy
image: gcr.io/cloudsql-docker/gce-proxy:1.11
imagePullPolicy: Always
command:
- /cloud_sql_proxy
- -instances=<project>:<cloudsqlinstance>=tcp:0.0.0.0:3306
- -credential_file=/secrets/cloudsql/credentials.json
ports:
- name: database-port
containerPort: 3306
volumeMounts:
- name: service-account-token
mountPath: /secrets/cloudsql
readOnly: true
So within any of your pods, the database your MYSQL_HOST:MYSQL_PORT will be cloudsqlproxy:3306
For multiple databases through the same proxy, you'd have the same deployment structure for the proxy, except that you will now expose 2 ports from the pod, like so:
apiVersion: extensions/v1beta1
...
spec:
template:
...
spec:
volumes:
...
containers:
- name: cloudsql-proxy
...
ports:
- name: database-port1
containerPort: 3306
- name: database-port2
containerPort: 3307
...
Then you'd create 2 services to for discovery on those ports like so:
apiVersion: v1
kind: Service
metadata:
name: cloudsqlproxy-db1
spec:
ports:
- port: 3306
targetPort: database-port1
selector:
app: cloudsqlproxy
---
apiVersion: v1
kind: Service
metadata:
name: cloudsqlproxy-db2
spec:
ports:
- port: 3306
targetPort: database-port2
selector:
app: cloudsqlproxy
So, with both services set to port 3306, you can connect to each database on that port:
mysql --host=cloudsqlproxy-db1 --port=3306 ...
mysql --host=cloudsqlproxy-db2 --port=3306 ...
Reference: https://github.com/GoogleCloudPlatform/cloudsql-proxy/blob/master/Kubernetes.md
With Google "Private IP" the cloud proxy is now irrelevant!