In google cloud sql - Postgres .
[1] How to view/change the postgresql.conf parameter like sharedbuffer etc?
There is a limited set of parameters available via CloudSQL Flags: https://cloud.google.com/sql/docs/postgres/flags
Related
I have a RDS PSQL14 database on eu-central-1 and would like to connect this to Quicksight as a new data source.
However, I always get the following error:
sourceErrorCode: GENERIC_SQL_EXCEPTION
sourceErrorMessage: The authentication type 10 is not supported. Check that you have configured the pg_hba.conf file to include the client's IP address or subnet, and that it is using an authentication scheme supported by the driver.
I have no clue how could I edit the pg_hba.conf file. Though I know that the VPC in which the RDS is in would allow the connection.
Most likely this is something with sha256 vs md5 authentication or so I've read in a couple of posts, but I don't know for sure. Please someone educate me :)
This was a huge time waster.
tldr: psql 13 and up uses scram-sha-256 but older version only support md5.
The reason behind this is that AWS Quicksight is using PostgreSQL JDBC driver 42.2.1
If you try to connect this with any of the newer psql versions it will fail due to a change made to the password authentication method used in the more recent versions of PostgreSQL (scram-sha-256). However, the 42.2.x driver only supports connecting via md5 passwords.
How to solve?
1. Downgrade:
If you downgrade to version 12.9 or below the problem should sort itself out.
2. Change Auth to md5:
!! it will affect all users and connections !!
Create a new Parameter group.
Once a new Parameter group is created > Search for password_encryption > Edit parameters
Select md5 (change the "rds.accepted_password_auth_method" parameter to allow md5 + scram which would allow you to create a user with md5 for QuickSight to connect with).
Save changes
Then you can modify your database to use the DB parameter group created in step 1.
Create a new user to be used in the QuickSight authentication with the RDS instance with the necessary permissions.
3. New Quicksight user with session-level md5:
Verify current password_encryption value:
show password_encryption;
Set the session variable of the parameter to 'md5:
set password_encryption = 'md5';
Create a user and assign it the necessary credentials
create user (username) with password '(password)';
grant connect on database (database) to (username);
Use the user to connect from QuickSight and it should be able to connect successfully using the "md5" encryption and not "scram-sha-256".
I'm trying to audit connections to my postgres databases.
i got 32 databases in my installation and one postgresql.conf for all of them.
I've configured to
log_connections = on and now i got information in my file log about connections to 32 databases.
But that should i do to monitor only databases that i need?
For example i need to monitor connections only to 5 of them, other is not interesting for me.
Where should i configure it?
It would be really nice if i could do it in postgresql.conf
with the log_connections parameter, you cannot leverage the granularity to audit selective databases. I would suggest you use pg_audit extension
by default pgAudit will log all databases but you can change it to log per database by using
ALTER DATABASE <database name> set pgaudit.log='<value>';
If you are using it AWS RDS/AURORA refer https://aws.amazon.com/premiumsupport/knowledge-center/rds-postgresql-pgaudit/
For community Postgres you can use https://github.com/pgaudit/pgaudit/blob/master/README.md
Any of you with experience with PostgREST and Cloud SQL ?
I have my SQL instance ready with open access (0.0.0.0/0) and I can access it with local PostGREST using the Cloud proxy app.
Now I want to run Postgrest from an instance of the same project but
I can't find an URI format for Postgrest that supports Cloud SQL format, as
Google SQL Cloud uses only unix sockets like /cloudsql/INSTANCE_CONNECTION_NAME
Config 1
db-uri = "postgres://postgres:password#/unix(/cloudsql/INSTANCE_CONNECTION_NAME)/mydatabase"
db-schema = "api"
jwt-secret = "OOcJ7VoSY1mXqod4MKtb9WCCwt9erJkRQ2tzYmLb4Xe="
db-anon-role = "web_anon"
server-port=3000
Returns {"details":"could not translate host name \"unix(\" to address: Unknown host\n","code":"","message":"Database connection error"}
Config 2
db-uri = "postgres://postgres:password#/mydatabase?unix_socket=/cloudsql/INSTANCE_CONNECTION_NAME"
db-schema = "api"
jwt-secret = "OOcJ7VoSY1mXqod4MKtb9WCCwt9erJkRQ2tzYmLb4Xe="
db-anon-role = "web_anon"
server-port=3000
The parser rejects the question mark
{"details":"invalid URI query parameter: \"unix_socket\"\n","code":"","message":"Database connection error"}
Config 3
db-uri = "postgres://postgres:password#/mydatabase"
db-schema = "api"
jwt-secret = "OOcJ7VoSY1mXqod4MKtb9WCCwt9erJkRQ2tzYmLb4Xe="
db-anon-role = "web_anon"
server-port=3000
server-unix-socket= "/cloudsql/INSTANCE_CONNECTION_NAME"
server-unix-socket appears to only take socket lock file path. Feeding it /cloudsql/INSTANCE_CONNECTION_NAME tries to delete file as in `postgrest.exe: /cloudsql/INSTANCE_CONNECTION_NAME: DeleteFile "/cloudsql/INSTANCE_CONNECTION_NAME": invalid argument t (The filename, directory name, or volume label syntax is incorrect.)
Documentation
Cloud SQL Doc
https://cloud.google.com/sql/docs/mysql/connect-run
PostgREST
http://postgrest.org/en/v6.0/configuration.html
https://github.com/PostgREST/postgrest/issues/1186
https://github.com/PostgREST/postgrest/issues/169
Environment
PostgreSQL version:11
PostgREST version: 6.0.2
Operating system: Win10 and Alpine
First you have to add the Cloud SQL connection to the Cloud Run instance:
https://cloud.google.com/sql/docs/postgres/connect-run#configuring
After that, the DB connection will be available in the service on a Unix domain socket at path /cloudsql/<cloud_sql_instance_connection_name> and you can set the PGRST_DB_URI environment variable to reflect that.
Here's the correct format:
postgres://<pg_user>:<pg_pass>#/<db_name>?host=/cloudsql/<cloud_sql_instance_connection_name>
e.g.
postgres://postgres:postgres#/postgres?host=/cloudsql/project-id:zone-id-1:sql-instance
According with Connecting with CloudSQL, the example is:
# postgres+pg8000://<db_user>:<db_pass>#/<db_name>?unix_sock=/cloudsql//.s.PGSQL.5432
Then you can try with (Just as #marian.vladoi mentioned):
db-uri = "postgres://postgres:password#/mydatabase?unix_socket=/cloudsql/INSTANCE_CONNECTION_NAME/.s.PGSQL.5432"
Keep in mind that the connection name should include:
ProjectID:Region:DatabaseName
For example: myproject:myregion:myinstance
Anyway, you can find here more options to connect from external applications and from within Google Cloud.
I tried many variations but couldn't get it to work out of the box, however I'll post this workaround.
FWIW I was able to use an alternate socket location with postgrest locally, but then when trying to use the cloudsql location it doesn't seem to interpret it right - perhaps the colons in the socket path are throwing it off?
In any case as #Steve_Chávez mentions, this approach does work db-uri = postgres:///user:password#/dbname and defaults to the postgrest default socket location (/run/postgresql/.s.PGSQL.5432). So in the docker entrypoint we can symlink this location to the actual socket injected by Cloud Run.
First, add the following to the Dockerfile (above USER 1000):
RUN mkdir -p /run/postgresql/ && chown postgrest:postgrest /run/postgresql/
Then add an executable file at /etc/entrypoint.bash containing:
set -eEux pipefail
CLOUDSQL_INSTANCE_NAME=${CLOUDSQL_INSTANCE_NAME:-PROJECT_REGION_INSTANCE_NAME}
POSTGRES_SOCKET_LOCATION=/run/postgresql
ln -s /cloudsql/${CLOUDSQL_INSTANCE_NAME}/.s.PGSQL.5432 ${POSTGRES_SOCKET_LOCATION}/.s.PGSQL.5432
postgrest /etc/postgrest.conf
Change the Dockefile entrypoint to CMD /etc/entrypoint.sh. Then add CLOUDSQL_INSTANCE_NAME as an env var in cloud run. The PGRST_DB_URI env var is like so postgres://authenticator:password#/postgres
An alternative approach if you don't like this, would be to connect via serverless vpc connector.
I struggled with this too.
I end up doing a one-liner for DB-URI env variable
host=/cloudsql/project-id:zone:instance-id user=user port=5432 dbname=dbname password=password
However, I have postgrest running on cloud run that lets you specify the instance connection name via
INSTANCE_CONNECTION_NAME=/cloudsql/project-id:zone:instance-id
Maybe you can host it there and you end up doing it serverless Im not sure where are you running it currently.
https://cloud.google.com/sql/docs/mysql/connect-run
I don't have a lot of experience with sockets, especially google cloud ones. The Cloud SQL uses a format: mysql:unix_socket=/cloudsql/INSTANCE_CONNECTION_NAME;dbname=DATABASE
How does this get translated into making a real connection? To me it seems like it is missing a domain name.
https://cloud.google.com/appengine/docs/standard/php/cloud-sql/using-cloud-sql-mysql
env_variables:
# Replace USER, PASSWORD, DATABASE, and CONNECTION_NAME with the
# values obtained when configuring your Cloud SQL instance.
MYSQL_DSN: mysql:unix_socket=/cloudsql/INSTANCE_CONNECTION_NAME;dbname=DATABASE
MYSQL_USER: USER
MYSQL_PASSWORD: PASSWORD
Sockets on Linux are just folders that are CHMOD 777
In this case you need to create a directory /cloudsql and CHMOD 777.
Better documentation found for this by visiting https://cloud.google.com/appengine/docs/flexible/php/using-cloud-sql and clicking UNIX Sockets as the doc option.
Also you need to download the cloud sql proxy app, these are all just settings for it.
We have a hosted PostgreSQL, with no access to the system or *.conf files.
I do have a admin access and can connect to it using Oracle SQL developer.
Can I run any command to increase the max_connections. All other parameters seems to be ok shared mem and buffers can hold more connections so there is not problem there.
Changing max_connection parameter needs a Postgres restart
Commands
Check max_connection just to keep current value in mind
SHOW max_connections;
Change max_connection value
ALTER SYSTEM SET max_connections TO '500';
Restart PostgreSQL server
Apparently, the hosted Postgres we are using does not provide this option. (compose.io)
So the work around is to use a pgbouncer to manage you connections better.