I've built a Vapor app and am trying to deploy it to Google cloud run. At the same time I'm trying to connect my app to a Cloud SQL instance using unix sockets as it's documented here. There was also a issue opened on Vapor's postgres library here that mentions being successful using unix sockets to connect from cloud run.
Setup
My code looks like this:
let postgresConfig = PostgresConfiguration(unixDomainSocketPath: Environment.get("DB_SOCKET_PATH") ?? "/cloudsql",
username: Environment.get("DATABASE_USERNAME") ?? "vapor_username",
password: Environment.get("DATABASE_PASSWORD") ?? "vapor_password",
database: Environment.get("DATABASE_NAME") ?? "vapor_database")
app.databases.use(.postgres(configuration: postgresConfig), as: .psql)
I've also tested to see if the environment variables are there using this snippet, which didn't throw the fatalError.
if Environment.get("DB_SOCKET_PATH") == nil {
fatalError("No environment variables found...")
}
My DB_SOCKET_PATH looks like cloudsql/project-id:us-central1:sql-instance-connection-name/.s.PGSQL.5432
I've also set up the correct user and database in the Cloud SQL instance, as well as enabled public ip connections.
Whats happening?
When I deploy this image to google cloud run, it throws the error: Fatal error: Error raised at top level: connect(descriptor:addr:size:): No such file or directory (errno: 2): file Swift/ErrorType.swift, line 200
What have you done?
I tried testing unix socket connections on my local machine, and I found that when I had the incorrect unix socket, it would through this error.
I also tried commenting out all migration and connection code, which fixed the error, and narrowed it down to the PostgresConfiguration code.
What are you trying to do?
I'm trying to figure out how to connect my cloud run app to a cloud sql instance. Am I missing a configuration somewhere on my instances? Or am I making a mistake with my unix path/vapor implementation?
Related
Background
I have a NestJS project with Prisma ORM, and I am continually receiving the error:
PrismaClientInitializationError: Can't reach database server at `localhost`:`5432`
This is happening during the Cloud Build Deploy step.
Since this is a containerized application (attempting to) run in a Cloud Run instance, I'm supposed to use a socket connection. Here's the documentation from Prisma on connecting to a Postgres DB through a socket connection: https://www.prisma.io/docs/concepts/database-connectors/postgresql#connecting-via-sockets
Connecting via sockets
To connect to your PostgreSQL database via sockets, you must add a host field as a query parameter to the connection URL (instead of setting it as the host part of the URI). The value of this parameter then must point to the directory that contains the socket, e.g.: postgresql://USER:PASSWORD#localhost/database?host=/var/run/postgresql/
Note that localhost is required, the value itself is ignored and can be anything.
I've done this to the letter, as described in the Cloud SQL documentation, with the exception that I percent-encoded my path to the directory containing the socket. I've included and excluded the trailing slash.
So my host var looks like this, mapped from the percent-encoded values:
/cloudsql/<MY CLOUD SQL CONNECTION NAME>/<DB>
I've read over the Cloud Run documentation, and in my mind, I should expect a different error if the instance itself can't connect to the Cloud SQL instance. I've followed the "Make sure you have the appropriate permissions and connection" from the documentation a few times now.
Is there anything obvious that I'm missing? Am I wrong about an error related to Cloud Run instance just not connecting with Cloud SQL instance?
Things I've tried & things I know
I CAN connect directly to the Cloud SQL instance locally through psql
I CAN run a local server with the Cloud SQL instance public IP and establish a client connection & interact with the database
I CAN successfully create an image and run a container from that image locally
My big concern
It doesn't make sense to me in which order things should connect to the Cloud SQL instance. To me, the Cloud Run - Cloud SQL connection MUST be established before the application run inside the Cloud Run instance can establish its connection through the socket to the Cloud SQL instance. -- Am I thinking through that correctly?
I have a postgresql Google Cloud SQL instance and I'm trying to connect a FastAPI application running in Google Cloud Run to it, however I'm getting ConnectionRefusedError: [Errno 111] Connection refused errors.
My application uses the databases package for async database connections:
database = databases.Database(sqlalchemy_database_uri)
Which then tries to connect on app startup through:
#app.on_event("startup")
async def startup() -> None:
if not database.is_connected:
await database.connect() <--- this is where the error is raised
From reading through the documentation here it suggests forming the connection string like so:
"postgresql+psycopg2://user:pass#/dbname?unix_sock=/cloudsql/PROJECT_ID:REGION:INSTANCE_NAME/.s.PGSQL.5432"
I've tried several different variations of the url, with host instead of unix_sock as the sqlalchemy docs seem to suggest, as well as removing the .s.PGSQL.5432 at the end as I've seen some other SO posts suggest, all to no avail.
I've added the Cloud SQL connection to the instance in the Cloud Run dashboard and added a Cloud SQL Client role to the service account.
I'm able to connect to the databases locally with the Cloud SQL Auth Proxy.
I'm at a bit of a loss on how to fix this, or even how to debug it as there doesn't seem to be any easy way to ssh into container and try out some things. Any help would be greatly appreciated, thanks!
UPDATE
I'm able to connect directly with sqlalchemy with:
from sqlalchemy import create_engine
engine = create_engine(url)
engine.connect()
Where url is any of these formats:
"postgresql://user:pass#/db_name?host=/cloudsql/PROJECT_ID:REGION:INSTANCE_NAME"
"postgresql+psycopg2://user:pass#/db_name?host=/cloudsql/PROJECT_ID:REGION:INSTANCE_NAME"
"postgresql+pg8000://user:pass#/db_name?unix_sock=/cloudsql/PROJECT_ID:REGION:INSTANCE_NAME/.s.PGSQL.5432"
Is there something due to databases's async nature that's causing issues?
turns out this was a bug with the databases package. this should now be resolved with https://github.com/encode/databases/pull/423
I have a PostgreSQL database deployed in Google Cloud that I am trying to connect to from a Cloud Run instance. I have tried the following two packages, both of them eventually leading to the same exception:
https://pub.dev/packages/postgres
https://pub.dev/packages/database_adapter_postgre
The exception I am getting is:
SocketException: Failed host lookup: '/cloudsql/{INSTANCE_CONNECTION_NAME}' (OS Error: Name or service not known, errno = -2)
I get here both times when trying to establish the connection, so in the case of the first package:
connection = new PostgreSQLConnection(
'/cloudsql/{INSTANCE_CONNECTION_NAME}',
5432,
'postgres',
username: 'username',
password: 'password');
await connection.open(); // <-- exception thrown here
I have tried changing the host string to /cloudsql/INSTANCE_CONNECTION_NAME}/.s.PGSQL.5432, but that did not work. My first thought were permissions, the service account the Cloud Run instance is using (xxx-compute#developer.gserviceaccount.com) has the Cloud SQL Editor role (tried Client and Admin too).
Running the same database code locally from a dart console app, I can connect to my database via its public IP address as the host with both packages, so the database itself is up and running.
Can someone point me in the right direction with this exception/have an example code for any of the packages above to show how to connect it to a Cloud SQL instance from a Cloud Run?
Edit:
I tried setting up a proxy locally to test out if the connection is wrong like so:
.\cloud_sql_proxy.exe -instances={INSTANCE_CONNECTION_NAME}=tcp:5433 psql
Then changing the connection host value in the code to localhost, and the port to 5433.
To my surprise it works - so from locally I am seemingly able to connect to the DB using that connection string. It still doesn't work when I use it from a Cloud Run instance though. Any help is appreciated!
It seems dart doesn't support connection through unix socket, you need to configure a IP (public or private, as you need).
Alternatively you can use pg which support unix socket connection
Hope this helps.
Just for those who come across this question in the future:
as it stands right now, I had to resort to the suggestion posted by Daniele Ricci and use the public IP for the database. The one thing to point out here was that since Cloud Runs don't have a static IPv4 address to run from, the DB had to be set to allow connections from anywhere (had to add an authorized connection from 0.0.0.0/0), which is unsafe. Until the kind development team of dart figures out how to use UNIX sockets, this seems to be the only way of getting it to work.
Not having actually tested this myself, according to the source code of the postgres package, you have to specify that you want a Unix socket:
connection = PostgreSQLConnection(
...
isUnixSocket: true, // <-- here
);
The default is false.
The host you pass is must also be valid. The docs say:
[host] must be a hostname, e.g. "foobar.com" or IP address. Do not include scheme or port.
I was struggling with the same issue.
The solution is to create a connection as follows:
PostgreSQLConnection getProdConnection() {
final String connectionName = Platform.environment['CLOUD_SQL_CONNECTION_NAME']!;
final String databaseName = Platform.environment['DB_NAME']!;
final String user = Platform.environment['DB_USER']!;
final String password = Platform.environment['DB_PASS']!;
final String socketPath = '/cloudsql/$connectionName/.s.PGSQL.5432';
return PostgreSQLConnection(
socketPath,
5432,
databaseName,
username: user,
password: password,
isUnixSocket: true,
);
}
Then when you create a Cloud Run service, you need to define 'Enviroment variables' as follows.
You also need to select your sql instance in the 'connections' tab.
Then the last thing needed is to configure a Cloud Run service account.
Then the connection to instance should be successful and there should no longer be a need for a 0.0.0.0/0 connection.
However, if you try to run this connection locally on a Windows device during development the connection will not be allowed and you will be presented with this error message: 'Unix domain sockets are not available on this operating system.'
Therefore, I recommend that you open Google SQL networking to your public address and define a local environment using the 'Public IP address' of your SQL instance.
For more information on this topic, I can recommend these resources that have guided me to the right solution:
https://cloud.google.com/sql/docs/postgres/connect-instance-cloud-run#console_5
https://github.com/dart-lang/sdk/issues/47899
I'm trying to create a new asp.net core app which uses EF7 to access a local db database and I'm having some problems with the connection string. I've installed the local db service and I have already created several instances:
PS C:\Users\luisabreu> sqllocaldb info
MSSQLLocalDB
ProjectsV12
ProjectsV13
v11.0
MSSQLLocalDB is running v12:
PS C:\Users\luisabreu> sqllocaldb info mssqllocaldb
Name: MSSQLLocalDB
Version: 12.0.2000.8
Shared name:
Owner: GRM\luisabreu
Auto-create: Yes
State: Running
Last start time: 05/04/2016 08:46:17
Instance pipe name: np:\\.\pipe\LOCALDB#518FD662\tsql\query
This is the connection string I'm passing to EF:
{
"usersCnnString": "Data Source=(localdb)\\MSSQLLocalDB;Initial Catalog=UsersInfo;Integrated Security=True"
}
Btw, I've already tried this one too:
"usersCnnString": "Server=(localdb)\\mssqllocaldb;Database=UsersInfo;Trusted_Connection=True"
And here's the setup code I'm using for setting up ED dependencies:
services.AddEntityFramework()
.AddSqlServer()
.AddDbContext<UsersContext>(options => options.UseSqlServer(Configuration["usersCnnString"]));
Whenever I try to access the database,I end up with the following error:
System.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: SQL Network Interfaces, error: 50 - Local Database Runtime error occurred. Specified LocalDB instance name is invalid.)
I'm surely doing something wrong, but what?
thanks!
Luis
Well, it was something simpler...don't forget to kill iis express whenever you change the config of an ASP.NET Core app so that you'll get the correct config data when you re-run the app. The problem was that when I pasted the path:
Server=(localdb)\\mssqllocaldb...
into the json file, VS 2015 automatically escaped the string, turning it into:
Server=(localdb)\\\\mssqllocaldb...
Yes, I've noticed it, but only after trying to load the app on the browser. I corrected the path but didn't kill the IIS Express instance I was using to test my app. So, the error message was absolutely correct because it was still using the old path (which was trying to reach the mssqllocaldb instance running on a localdb server).
Again, stupid error. Hoping that my mistake will save someone else's 30 mins in the future...
if I create a project with user accounts using VS 2015 with RC1 the project template generates the connection string like this:
{
"Server=(localdb)\\mssqllocaldb;Database=thedbnamehere;Trusted_Connection=True;MultipleActiveResultSets=true"
}
I would try making it closer to that, but also wondering if for some reason UsersInfo might not be a valid name, could be some keyword clash
I noticed that vs uses the "userSecretsId" from project.json as the dbname, not sure if that is required but might be worth trying
I'm desperate since my Google Cloud SQL instance went down. I could connect to it yesterday without problem but since this morning i'm unable to connect to it in any way, it produces the following error: The database server returned this error: SQLSTATE[HY000] [2013] Lost connection to MySQL server at 'reading initial communication packet', system error: 0
This is what I did to try to fix this:
restart instance
added authorized ip-addresses in CIDR notation
reset root password
restored backup
pinged the ip-address and I get response
All these actions completed but i'm still unable to connect through:
PHP
MySQL workbench
Ubuntu MySQL command line
All without luck. What could I do to repair my Cloud SQL instance. Is anyone else having this problem?
I'm from the Cloud SQL team. We are looking into this issue, it should be resolved soon. See https://groups.google.com/forum/#!topic/google-cloud-sql-announce/SwomB2zuRDo. Updates will be posted on that thread (and if there's anything particularly important I'll edit this post).
The problem seems to only affect connections from outside Google Cloud. Clients connecting from App Engine and Compute Engine should work fine.
Our company has same problem.
We are unable to connect through both MySQL workbench and MySQL command line.
Our Google Appengine application has no problems to connect since its not using external IP.
there.I encountered the same problem.You need to find out your public ip address,for that type "my public ip" in Google.Now click on your Cloud SQL instance that you created,under that click on ACCESS CONTROL tab and then click on Authorization tab under that.Under Authorized network,give any name you want to the network and copy your public ip address in the network.Now save changes and try to run the command from console.It should work fine.