I'm using pg-promise in a typescript express app.
I can log on to psql by just typing psql because the linux username is the same as the database name and I'm set to use peer authentication.
I can't seem to get pg-promise to let me do this - it always fails and gives an error that password authentication failed.
I've been looking at the pgpromise docs but can't find anything - I've tried all sorts of connection strings.
I managed to make this work for me by using a Unix Domain Socket instead of a TCP socket and connecting using a connection string.
So my connection looks like this const db = pgp('socket:/var/run/postgresql'); but presumably the /var/run/postgresql could vary by system.
Related
I have a PostgreSQL database deployed in Google Cloud that I am trying to connect to from a Cloud Run instance. I have tried the following two packages, both of them eventually leading to the same exception:
https://pub.dev/packages/postgres
https://pub.dev/packages/database_adapter_postgre
The exception I am getting is:
SocketException: Failed host lookup: '/cloudsql/{INSTANCE_CONNECTION_NAME}' (OS Error: Name or service not known, errno = -2)
I get here both times when trying to establish the connection, so in the case of the first package:
connection = new PostgreSQLConnection(
'/cloudsql/{INSTANCE_CONNECTION_NAME}',
5432,
'postgres',
username: 'username',
password: 'password');
await connection.open(); // <-- exception thrown here
I have tried changing the host string to /cloudsql/INSTANCE_CONNECTION_NAME}/.s.PGSQL.5432, but that did not work. My first thought were permissions, the service account the Cloud Run instance is using (xxx-compute#developer.gserviceaccount.com) has the Cloud SQL Editor role (tried Client and Admin too).
Running the same database code locally from a dart console app, I can connect to my database via its public IP address as the host with both packages, so the database itself is up and running.
Can someone point me in the right direction with this exception/have an example code for any of the packages above to show how to connect it to a Cloud SQL instance from a Cloud Run?
Edit:
I tried setting up a proxy locally to test out if the connection is wrong like so:
.\cloud_sql_proxy.exe -instances={INSTANCE_CONNECTION_NAME}=tcp:5433 psql
Then changing the connection host value in the code to localhost, and the port to 5433.
To my surprise it works - so from locally I am seemingly able to connect to the DB using that connection string. It still doesn't work when I use it from a Cloud Run instance though. Any help is appreciated!
It seems dart doesn't support connection through unix socket, you need to configure a IP (public or private, as you need).
Alternatively you can use pg which support unix socket connection
Hope this helps.
Just for those who come across this question in the future:
as it stands right now, I had to resort to the suggestion posted by Daniele Ricci and use the public IP for the database. The one thing to point out here was that since Cloud Runs don't have a static IPv4 address to run from, the DB had to be set to allow connections from anywhere (had to add an authorized connection from 0.0.0.0/0), which is unsafe. Until the kind development team of dart figures out how to use UNIX sockets, this seems to be the only way of getting it to work.
Not having actually tested this myself, according to the source code of the postgres package, you have to specify that you want a Unix socket:
connection = PostgreSQLConnection(
...
isUnixSocket: true, // <-- here
);
The default is false.
The host you pass is must also be valid. The docs say:
[host] must be a hostname, e.g. "foobar.com" or IP address. Do not include scheme or port.
I was struggling with the same issue.
The solution is to create a connection as follows:
PostgreSQLConnection getProdConnection() {
final String connectionName = Platform.environment['CLOUD_SQL_CONNECTION_NAME']!;
final String databaseName = Platform.environment['DB_NAME']!;
final String user = Platform.environment['DB_USER']!;
final String password = Platform.environment['DB_PASS']!;
final String socketPath = '/cloudsql/$connectionName/.s.PGSQL.5432';
return PostgreSQLConnection(
socketPath,
5432,
databaseName,
username: user,
password: password,
isUnixSocket: true,
);
}
Then when you create a Cloud Run service, you need to define 'Enviroment variables' as follows.
You also need to select your sql instance in the 'connections' tab.
Then the last thing needed is to configure a Cloud Run service account.
Then the connection to instance should be successful and there should no longer be a need for a 0.0.0.0/0 connection.
However, if you try to run this connection locally on a Windows device during development the connection will not be allowed and you will be presented with this error message: 'Unix domain sockets are not available on this operating system.'
Therefore, I recommend that you open Google SQL networking to your public address and define a local environment using the 'Public IP address' of your SQL instance.
For more information on this topic, I can recommend these resources that have guided me to the right solution:
https://cloud.google.com/sql/docs/postgres/connect-instance-cloud-run#console_5
https://github.com/dart-lang/sdk/issues/47899
I connect to my PostgreSQL databases (which run on AWS EC2 instances) using certificate authentication (and not passwords). An example of a psql command I would use to connect to one of my databases is:
psql "host=<AWS EC2 instance> user=<db user> sslcert=<path to .crt> sslkey=<path to .key> sslrootcert=<path to .crt> sslmode=require dbname=<db name>"
I would like to be able to connect Tableau Desktop to one of my databases. The standard PostgreSQL connector does not allow me to specify which SSL certs to use (but it does have a Require SSL checkbox - which I think is for encryption not authentication).
So I have tried to use the ODBC connector. I make the following entries:
Connect Using
Driver: PostgreSQL Unicode
Connection Attributes
Server: <AWS EC2 instance>
Port: 5432
Database: <db name>
Username: <db user>
String Extras: sslcert=<path to .crt>; sslkey=<path to .key>; sslrootcert=<path to .crt>; sslmode=require
By much trial and error I think the String Extras require a semicolon as a delimiter, but I still cannot connect. I get the error message:
An error occurred while communicating with Other Databases (ODBC).
Unable to connect to the server. Check that the server is running and that you have access privileges to the requested database.
FATAL: connection requires a valid client certificate
Generic ODBC requires additional configuration. The driver and DSN (data source name) must be installed and configured to match the connection.
Unable to connect to the server "<AWS EC2 instance>" using the driver "PostgreSQL Unicode". Check that the server is running and that you have access privileges to the requested database.
I would be really grateful to hear from anyone who has connected Tableau Desktop to PostgreSQL using certificate authentication and could tell me what I’m doing wrong. Cheers!
I managed to fix this myself, and just in case there's someone out there who might be vaguely interested I'll go through the salient features:
1. Created a DSN (Data Store Name)
This made testing much easier rather than constantly retyping details into dialog boxes.
Rather than create manually I used this: http://www.odbcmanager.net/
On Mac OS I had to run it as sudo from the terminal or it wouldn't create anything.
I created a User DSN, which on my Mac have details stored in /Users/<user name>/.odbc.ini
It picked up the PostgreSQL Unicode driver that I'd previously installed. On Mac OS you might need to do brew install brew install psqlodbc and/or brew install unixodbc
Most of the configuration was specified by adding key/value pairs e.g. DBNAME <db name>
Set SSLMODE verify-ca (changed from require to make behaviour specific - psql assumes verify-ca/verify-full if SSL certs are supplied)
The cert paths are set with key Pqopt, value sslcert=<path to .crt> sslkey=<path to .key> sslrootcert=<path to .crt>. Must be lower case and only spaces between entries!
Here's what it added to my user's odbc.ini file:
[<DSN name>]
Driver = <path to driver, this was mine /usr/local/lib/psqlodbcw.so>
Description = <description>
SSLMODE = verify-ca
HOST = <host>
DBNAME = <database>
PORT = 5432
UID = <db user>
Pqopt = sslcert=<path to .crt> sslkey=<path to .key> sslrootcert=<path to .crt>
2. Tableau Desktop Connection
Select Other Database (ODBC) connector.
Select the DSN you created previously (it should be automatically picked up), it will automatically populate host, port, db and user fields, then click Sign In.
Once signed in behaviour was slightly different to the dedicated PostgreSQL connector. Tables did not appear on left-hand side initially. But they are accessible - they can all be displayed by clicking on the 'contains' radio button and searching with a blank name.
Hope this helps. Let me know if you've got any questions.
I am installing DSPACE which needs PostgresSQL.
Please see this link: DSPACE INSTALL
It says:
Then tighten up security a bit by editing pg_hba.conf and adding this line:
host dspace dspace 127.0.0.1 255.255.255.255 md5.
I have read on the internet as to how the above line works. It needs a connection type followed by database name/user name followed by IP address and the authentication scheme.
My question is: Should this not be a local (in place of host) connection since the dspace user is running locally?
Can someone show me step by step as to what happens when a request comes in?
Where are the instructions that the dspace user will submit a request using md5?
DSpace uses JDBC, so local won't work for it. Here local means a completely different kind of inter-process connection that has nothing to do with IP and can only address local processes: Unix sockets. JDBC uses only IP, so DSpace needs a host descriptor.
That period at the end of the line is not supposed to be part of it. The documentation was unclear there -- I've fixed it.
As Laurenz Albe noted, DSpace doesn't specify that MD5 password hashes be used. The PostgreSQL server controls which methods will be tried, based on what you specify in pg_hba.conf.
First, there is a . at the end of your pg_hba.conf line. That is a syntax error.
Whether to use local or host depends on
what API this DSPACE is using (JDBC, for example, allows no UNIX socket connections)
what connection string DSPACE is using
If you can specify a socket directory as host name, you can probably use UNIX sockets, which would be more efficient.
If DSPACE uses a driver that supports md5 authentication, the procedure is like this:
client sends server a connect packet with user and database
server requests md5 authentication and sends a random "salt" string
client computes an MD5 hash of the password with the salt from the server and sends the result to the server
server verifies that the hash is correct
I'm trying to connect to my Cloud SQL DB using SQLAlchemy from my cloud function but I can't seem to work out the correct connection string.
DATABASE_URL=postgres://$DB_USER:$_DB_PWD#/$DB_NAME?unix_socket=/cloudsql/$DB_INSTANCE
Which gives me the error:
pyscopg2.ProgrammingError: invalid dns: invalid connection option "unix_socket"
What is the correct way to connect to a Postgresql 9.6 DB over a unix socket using pyscopg2?
The special keyword needed here is host:
DATABASE_URL=postgres://user:password#/dbname?host=/path/to/db
Note that the path in host should be a path, not the socket file itself (psycopg2 assumes the socket has the standard naming convention .s.PGSQL.5432)
https://docs.sqlalchemy.org/en/latest/dialects/postgresql.html#unix-domain-connections
When trying to connect to my postgres server in Azure from psql client, I get the following error, even though I am using the correct username. How can I fix this?
psql: FATAL: Invalid Username specified. Please check the Username and retry connection. The Username should be in <username#hostname> format.
As noted in the error text, you are required to follow the <username#hostname> format when trying to connect to postgresql server, whether you are doing it from psql client or using pgadmin. Using <username#hostname> format instead of just <username> should get rid of the error.
Read the quick-start documents for Azure portal and CLI to understand more about how to create and configure your postgres server.
The # sign in username works fine for objects but not connect strings. According to URI RFC-3986 username allows hex encoding. So replace the # with %40. user#host:pw#fullhost becomes user%40host:pw#fullhost