I have a custom Postgres instance running on a GCE VM. I am not using CloudSQL. I'd like to use the functionality provided by the cloudsql-proxy, but when I specify my custom instance the proxy fails.
googleapi: Error 404: The Cloud SQL instance does not exist., instanceDoesNotExist
It appears that only CloudSQL instances work and I don't understand the limitation. It seems like the proxy should work on any VM with port 5432 open.
Cloud SQL Proxy can only be used with Cloud SQL instances.
In the documentation About the Cloud SQL Proxy there is no mention about using it for custom databases inside GCE.
As it is stated in the documentation:
The Cloud SQL Proxy provides secure access to your Cloud SQL Second
Generation instances without having to whitelist IP addresses or
configure SSL.
However, I found some documentation about Access control overview where you can find an alternative to it.
Related
Does anyone know the pros and cons for installing the CloudSQL-Proxy (that allows us to connect securely to CloudSQL) on a Kubernetes cluster as a service as opposed to making it a sidecar against the application container?
I know that it is mostly used as a sidecar. I have used it as both (in non-production environments), but I never understood why sidecar is more preferable to service. Can someone enlighten me please?
The sidecar pattern is preferred because it is the easiest and more secure option. Traffic to the Cloud SQL Auth proxy is not encrypted or authenticated, and relies on the user to restrict access to the proxy (typically be running local host).
When you run the Cloud SQL proxy, you are essentially saying "I am user X and I'm authorized to connect to the database". When you run it as a service, anyone that connects to that database is connecting authorized as "user X".
You can see this warning in the Cloud SQL proxy example running as a service in k8s, or watch this video on Connecting to Cloud SQL from Kubernetes which explains the reason as well.
The Cloud SQL Auth proxy is the recommended way to connect to Cloud SQL, even when using private IP. This is because the Cloud SQL Auth proxy provides strong encryption and authentication using IAM, which can help keep your database secure.
When you connect using the Cloud SQL Auth proxy, the Cloud SQL Auth proxy is added to your pod using the sidecar container pattern. The Cloud SQL Auth proxy container is in the same pod as your application, which enables the application to connect to the Cloud SQL Auth proxy using localhost, increasing security and performance.
As sidecar is a container that runs on the same Pod as the application container, because it shares the same volume and network as the main container, it can “help” or enhance how the application operates. In Kubernetes, a pod is a group of one or more containers with shared storage and network. A sidecar is a utility container in a pod that’s loosely coupled to the main application container.
Sidecar Pros: Scales indefinitely as you increase the number of pods. Can be injected automatically. Already used by serviceMeshes.
Sidecar Cons: A bit difficult to adopt, as developers can't just deploy their app, but deploy a whole stack in a deployment. It consumes much more resources and it is harder to secure because every Pod must deploy the log aggregator to push the logs to the database or queue.
Refer to the documentation for more information.
Is it possible to establish connection from my localhost app to a replica-set postgres kubernetes? or what solution I need to do for having a mirror of my production database?
Thanks in advance
What you need is a so-called PostgreSQL Kubernetes operator that will be responsible for building Kubernetes objects based on your requests.
You can have a look at OperatorHub.io, they have some PostgreSQL operators.
Maybe an easier solution is KubeDB and the KubeDB PostgreSQL implementation.
The operator will also create a Kubernetes Service that will create a resolvable name linked to the Kubernetes Pods of your PostgreSQL cluster. KubeDB doc explains how to connect to the database in their documentation.
Now coming to your question :
Is it possible to establish connection from my localhost app [...]
You can access the Kubernetes service from outside but you will have to create a Kubernetes Load Balancer. See this blog article which explains it in details.
So, I created a postgreSQL instance in Google Cloud, and I have a Kubernetes Cluster with containers that I would like to connect to it. I know that the cloud sql proxy sidecar is one method, but the documentation says that I should be able to connect to the private IP as well.
I notice that a VPC peering connection was automatically created for me. It's set for a destination network of 10.108.224.0/24, which is where the instance is, with a "Next hop region" of us-central1, where my K8s cluster is.
And yet when I try the private IP via TCP on port 5432, I time out. I see nothing in the documentation about have to modify firewall rules to make this work, but I tried that anyway, finding the firewall interface in GCP rather clumsy and confusing compared with writing my own rules using iptables, but my attempts failed.
Beyond going to the cloud sql sidecar, does anyone have an idea why this would not work?
Thanks.
Does your GKE cluster meet the environment requirements for private IP? It needs to be a VPC enabled cluster on the same VPC and region as your Cloud SQL instance.
In the end, the simplest thing to do was to just use the google cloud sql proxy. As opposed to a sidecar, I have multiple containers needing db access so I put the proxy into my cluster as its own container with a service, and it seems to just work.
If your instance of cloud SQL or compute both in the same VPC then only you can create a VPC peering over private IP.
From cloud SQL compute VM you can choose the VPC and subnet and also setup same for the GKE and you can make the connection from pod to cloud sql.
I am following the link below to create a Cloud SQL Proxy.
https://cloud.google.com/sql/docs/mysql/connect-container-engine
When I get to the step to create the service account, I am unable to see any Cloud SQL roles, even though I have a MySQL instance associated with the project and I have enabled Cloud SQL administration API as described in the previous step.
Also, the whole process seems to be quite long-winded. Is there a way to connect directly from the container cluster to Cloud SQL without using the proxy? If yes, how do I find the IP address of the Cloud SQL instance? Also, how do I get the container cluster IP to white-list?
Many thanks
Is it possible to grant blanket access to my CloudSQL instance from ALL (current and future) GCE instances? I've tried adding the /16 internal network block address for my project's instances (copied from the "networks" tab under "Compute Engine": 10.240.0.0/16) but that won't save - it appears that I can only add single-machine (/32) IP addresses.
You need to use the external IP of your machine, although they are both (GCE and Cloud SQL) in Google's datacenters, you cannot communicate between the two using internal IPs.
I do not think there is a native way to allow access from any instance in your project. The only way would be to make your own app to run on one of your instances and use the GCE api to periodically query running instances, get their external ip's, and then use the CloudSQL API to modify the security configuration on the CloudSQL instance.
You could improve this slightly creating a pool of static IP's that you assign to your GCE machines that are going to access your CloudSQL instance, that way the IP's would not change, the side affect is that you would be charged for IP's that you have reserved but do not have allocated to instances.
Apart from that you would have to put a rule to allow any IP access (e.g. 1.0.0.0/0), which would not be a good idea.