psql can connect to a unix domain socket, but py-postgresql with the same parameters gets 'Permission denied' - postgresql

Problem description:
My system user is milosz, which is mapped to the PostgreSQL user project_great in pg_ident.conf. I am using peer authentication to connect to a PostgreSQL database over a unix domain socket. This connection method works when using psql, but fails to work when using py-postgresql using the same parameters from within a Python script.
Here I am successfully connecting to the database using psql:
$ psql -U project_great \
> -d project_great \
> -h /var/run/postgresql
psql (9.3.4)
Type "help" for help.
project_great=>
Here is database_test.py:
#!/usr/bin/env python3
import postgresql
params = {
'user': 'project_great',
'database': 'project_great',
'unix': '/var/run/postgresql',
}
connection = postgresql.open(**params)
Here I am attempting to connect to the database by running ./database_test.py:
$ ./database_test.py
Traceback (most recent call last):
File "./database_test.py", line 11, in <module>
sys.exit(main(sys.argv))
File "./database_test.py", line 13, in main
connection = postgresql.open(**params)
File "/home/milosz/devel/project_great/.virtualenv/lib/python3.3/site-packages/postgresql/__init__.py", line 94, in open
c.connect()
File "/home/milosz/devel/project_great/.virtualenv/lib/python3.3/site-packages/postgresql/driver/pq3.py", line 2422, in connect
self._establish()
File "/home/milosz/devel/project_great/.virtualenv/lib/python3.3/site-packages/postgresql/driver/pq3.py", line 2548, in _establish
self.typio.raise_client_error(could_not_connect, creator = self, cause = exc)
File "/home/milosz/devel/project_great/.virtualenv/lib/python3.3/site-packages/postgresql/driver/pq3.py", line 514, in raise_client_error
raise client_error
postgresql.exceptions.ClientCannotConnectError: could not establish connection to server
CODE: 08001
LOCATION: CLIENT
CONNECTION: [failed]
failures[0]:
socket'/var/run/postgresql'
Traceback (most recent call last):
File "/home/milosz/devel/project_great/.virtualenv/lib/python3.3/site-packages/postgresql/protocol/client3.py", line 136, in connect
self.socket = self.socket_factory(timeout = timeout)
File "/home/milosz/devel/project_great/.virtualenv/lib/python3.3/site-packages/postgresql/python/socket.py", line 64, in __call__
s.connect(self.socket_connect)
PermissionError: [Errno 13] Permission denied
The above exception was the direct cause of the following exception:
postgresql.exceptions.ConnectionRejectionError: Permission denied
CODE: 08004
LOCATION: CLIENT
CONNECTOR: [Unix] pq://project_great#[unix::var:run:postgresql]/project_great
category: None
DRIVER: postgresql.driver.pq3.Driver
Since the parameters to the two connections are ostensibly the same and my permissions on the socket and its containing directory are fairly open, I do not know what the issue is. Using TCP is not a solution; I want to use unix domain sockets. The py-postgresql documentation indicates that connecting using unix domain sockets should work.
Configuration:
pg_hba.conf:
# TYPE DATABASE USER ADDRESS METHOD OPTION
local all all peer map=default
pg_ident.conf:
# MAPNAME SYSTEM-USERNAME PG-USERNAME
default postgres postgres
default milosz project_great
postgresql.conf:
...
port = 5432
unix_socket_directories = '/var/run/postgresql'
...
Here are the permissions on my socket directory:
$ ll /var/run/postgresql/
total 8
drwxrwsr-x 2 postgres postgres 100 May 17 00:20 ./
drwxr-xr-x 31 root root 900 May 17 00:41 ../
-rw-r--r-- 1 postgres postgres 5 May 17 00:20 9.3-main.pid
srwxrwxrwx 1 postgres postgres 0 May 17 00:20 .s.PGSQL.5432=
-rw------- 1 postgres postgres 70 May 17 00:20 .s.PGSQL.5432.lock
The PostgreSQL user project_great has been granted all privileges on the database project_great and both the user and the database exist.
I do not have a ~/.pgpass.
Environment:
Ubuntu 13.10
Python 3.3
PostgreSQL 9.3
py-postgresql 1.1.0

Craig Ringer suggested to run both programs under strace to find out if there is a relevant difference in the system calls. I grepped for /var/run/postgresql and found that while psql ran connect like so:
connect(4, {sa_family=AF_LOCAL, sun_path="/var/run/postgresql/.s.PGSQL.5432"}, 110) = 0
./database_test.py ran connect like so:
connect(4, {sa_family=AF_LOCAL, sun_path="/var/run/postgresql"}, 21) = -1 EACCES (Permission denied)
wherein lay the issue.
While psql expects the path to the directory in which the unix domain socket is located, py-postgresql expects the full path to the socket. Therefore, the fix was to modify database_test.py to read like so:
#!/usr/bin/env python3
import postgresql
params = {
'user': 'project_great',
'database': 'project_great',
'unix': '/var/run/postgresql/.s.PGSQL.5432',
}
connection = postgresql.open(**params)
Honestly, I feel silly for not having tried the full path as an earlier debugging step!

Related

could not access file "pg_wait_sampling": No such file or directory postgresql

after I use the following command then I cannot relogin postgresql
alter system set shared_preload_libraries = 'pg_wait_sampling';
exit trying login again. Then error happens. The following is error code:
2022-03-15 11:35:30.726 IST [975] FATAL: could not access file "pg_wait_sampling": No such file or directory
2022-03-15 11:35:30.727 IST [975] LOG: database system is shut down
pg_ctl: could not start server
Examine the log output.
postgresql version: 14.2. system: wsl ubuntu.
in conf file /etc/postgresql/14/main/postgresql.conf":
#shared_preload_libraries='' #(change requires restart)
According to PostgreSQL document alter command store in postgresql.auto.conf file.
You have to change this file and remove pg_wait_sampling form this file

Connecting Apache Superset with PostgreSQL

Suppose I run my Apache Superset on top of the Docker and I want this to connect with my local postgreSQL server. I used the following URI but I got an error:
postgresql+psycopg2://username:password#localhost:5432/mydb
The error is:
ERROR: {"error": "Connection failed!\n\nThe error message returned was:\n(psycopg2.OperationalError) could not connect to server: Connection refused\n\tIs the server running on host \"localhost\" (127.0.0.1) and accepting\n\tTCP/IP connections on port 5432?\ncould not connect to server: Cannot assign requested address\n\tIs the server running on host \"localhost\" (::1) and accepting\n\tTCP/IP connections on port 5432?\n\n(Background on this error at: http://sqlalche.me/e/e3q8)", "stacktrace": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py\", line 2265, in _wrap_pool_connect\n return fn()\n File \"/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py\", line 303, in unique_connection\n return _ConnectionFairy._checkout(self)\n File \"/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py\", line 760, in _checkout\n fairy = _ConnectionRecord.checkout(pool)\n File \"/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py\", line 492, in checkout\n rec = pool._do_get()\n File \"/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py\", line 238, in _do_get\n return self._create_connection()\n File \"/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py\", line 308, in _create_connection\n return _ConnectionRecord(self)\n File \"/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py\", line 437, in __init__\n self.__connect(first_connect_check=True)\n File \"/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py\", line 639, in __connect\n connection = pool._invoke_creator(self)\n File \"/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py\", line 114, in ...
How can I solve it?
Instead of using localhost or 127.0.0.1, open up your pgAdmin.
The servers are on the left.
Click the dropdown.
Right click on the now opened cluster (level above "Databases") & open properties.
Navigate to the opened connection tab and the Hostname/Address is your replacement for "localhost"
Also make sure the final part of your connection string is pointed at your database which is one level below "Databases" in your pgAdmin.
I encountered the same problem with connecting superset to local database (postgresql), and after consulting many sites on the internet this trick solved it.Instead of local host, try to put this in SQLalchemy URI:
postgresql+psycopg2://user:password#host.docker.internal:5432/database
I understand that It is a bad practice action to connect the Docker with a host database so I changed my opinion and use the postgres image inside the docker and push my data to that postgres server.
It would be helpful if you notify me if I am wrong.

how to fix "OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly"

Services
My service based on flask + postgresql + gunicorn + supervisor + nginx
When deploying by docker, after running the service, then accessing the api, sometimes it told the error message, and sometimes it workes well.
And the sqlachemy connect database add the parameters 'sslmode:disable'.
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
Return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
Distilled_params,
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
e, statement, parameters, cursor, context
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1466, in _handle_dbapi_exception
Util.raise_from_cause(sqlalchemy_exception, exc_info)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
Reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
Cursor, statement, parameters, context
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
Cursor.execute(statement, parameters)
OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
Information
Docker for Mac: version: 2.0.0.3 (31259)
macOS: version 10.14.2
Python: version 2.7.15
Recurrence method
When view port information by command
lsof -i:5432
the port 5432 is postgresql database default port,if the outputconsole was
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
postgres 86469 user 4u IPv6 0xxddd 0t0 TCP *:postgresql (LISTEN)
postgres 86469 user 5u IPv4 0xxddr 0t0 TCP *:postgresql (LISTEN)
it would display the error message:
OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly
but if the outputconsolelog show this:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
com.docke 62421 user 26u IPv4 0xe93 0t0 TCP 192.168.2.7:6435->192.168.2.7:postgresql (ESTABLISHED)
postgres 86460 user 4u IPv6 0xed3 0t0 TCP *:postgresql (LISTEN)
postgres 86460 user 5u IPv4 0xe513 0t0 TCP *:postgresql (LISTEN)
postgres 86856 user 11u IPv4 0xfe93 0t0 TCP 192.168.2.7:postgresql->192.168.2.7:6435 (ESTABLISHED)
the situation, the api would work well.
Becauce of Docker for mac?
Refer link https://github.com/docker/for-mac/issues/2442 , the issue can not solve my problem.
Notice a similar problem?
Refer link Python & Sqlalchemy - Connection pattern -> Disconnected from the remote server randomly
also this issue can not solve my problem.
Solution
flask_sqlachemy need the parameter pool_pre_ping
from flask_sqlalchemy import SQLAlchemy as _BaseSQLAlchemy
class SQLAlchemy(_BaseSQLAlchemy):
def apply_pool_defaults(self, app, options):
super(SQLAlchemy, self).apply_pool_defaults(self, app, options)
options["pool_pre_ping"] = True
db = SQLAlchemy()
Same logic for sqlalchemy.orm, ( on which flask_sqlalchemy is based btw )
engine = sqlalchemy.create_engine(connection_string, pool_pre_ping=True)
More protection strategies can be setup such as it is described in the doc: https://docs.sqlalchemy.org/en/13/core/pooling.html#disconnect-handling-pessimistic
For example, here is my engine instantiation:
engine = sqlalchemy.create_engine(connection_string,
pool_size=10,
max_overflow=2,
pool_recycle=300,
pool_pre_ping=True,
pool_use_lifo=True)
sqlalchemy.orm.sessionmaker(bind=engine, query_cls=RetryingQuery)
For RetryingQuery code, cf: Retry failed sqlalchemy queries
I'm posting my own answer to this, since none of the above addressed my particular setup (Postgres 12.2, SQLAlchemy 1.3).
To stop the OperationalErrors, I had to pass in some additional connect_args to create_engine:
create_engine(
connection_string,
pool_pre_ping=True,
connect_args={
"keepalives": 1,
"keepalives_idle": 30,
"keepalives_interval": 10,
"keepalives_count": 5,
}
)
Building on the Solution in the answer and the info from #MaxBlax360's answer. I think the proper way to set these config values in Flask-SQLAlchemy is by setting app.config['SQLALCHEMY_ENGINE_OPTIONS']:
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
app = Flask(__name__)
# pool_pre_ping should help handle DB connection drops
app.config['SQLALCHEMY_ENGINE_OPTIONS'] = {"pool_pre_ping": True}
app.config['SQLALCHEMY_DATABASE_URI'] = \
f'postgresql+psycopg2://{POSTGRES_USER}:{dbpass}#{POSTGRES_HOST}:{POSTGRES_PORT}/{POSTGRES_DBNAME}'
db = SQLAlchemy(app)
See also Flask-SQLAlchemy docs on Configuration Keys
My db configuration:
app = Flask(__name__)
app.config['SQLALCHEMY_ENGINE_OPTIONS'] = {"pool_pre_ping": True}
app.config['SQLALCHEMY_DATABASE_URI'] = os.environ['DATABASE_URL']
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
# Play with following options:
app.config['SQLALCHEMY_POOL_SIZE'] = 10
app.config['SQLALCHEMY_MAX_OVERFLOW'] = 20
app.config['SQLALCHEMY_POOL_RECYCLE'] = 1800
db = SQLAlchemy(app)

FATAL: GTM error, could not obtain snapshot. Current XID = 0, Autovac = 0

After running virtually any command (e.g. createdb test) from GTM host inside pgxc_ctl tool, I am getting the following error:
FATAL: GTM error, could not obtain snapshot. Current XID = 0, Autovac = 0
PostgresXL is configured and installed on 4 nodes (/etc/hosts):
172.23.102.115 coordinator001
172.23.102.116 datanode001
172.23.103.0 datanode002
172.23.102.114 gtm001
So that each can ssh into another one and pg_hba.conf contains:
host all all 0.0.0.0/0 trust
GTM node has this configuration
Would appreciate any tips or ideas where to dig next.
[edit]
Getting this error during the direct connection:
psql -h coordinator001 -p 6668 -U postgres testdb
psql: FATAL: GTM error, could not obtain snapshot. Current XID = 0, Autovac = 0

Heroku postgres connection: "Is the server running locally and accepting connections on Unix domain socket"

I am setting up a new dev environment, followed the django setup tutorial and am having issues. Here is what I get when I try to run syncdb
Running `python doccal/manage.py syncdb` attached to terminal... up, run.1
Traceback (most recent call last):
File "doccal/manage.py", line 14, in <module>
execute_manager(settings)
File "/app/.heroku/venv/lib/python2.7/site-packages/django/core/management/__i
nit__.py", line 459, in execute_manager
utility.execute()
File "/app/.heroku/venv/lib/python2.7/site-packages/django/core/management/__i
nit__.py", line 382, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/app/.heroku/venv/lib/python2.7/site-packages/django/core/management/bas
e.py", line 196, in run_from_argv
self.execute(*args, **options.__dict__)
File "/app/.heroku/venv/lib/python2.7/site-packages/django/core/management/bas
e.py", line 232, in execute
output = self.handle(*args, **options)
File "/app/.heroku/venv/lib/python2.7/site-packages/django/core/management/bas
e.py", line 371, in handle
return self.handle_noargs(**options)
File "/app/.heroku/venv/lib/python2.7/site-packages/django/core/management/com
mands/syncdb.py", line 57, in handle_noargs
cursor = connection.cursor()
File "/app/.heroku/venv/lib/python2.7/site-packages/django/db/backends/__init_
_.py", line 306, in cursor
cursor = self.make_debug_cursor(self._cursor())
File "/app/.heroku/venv/lib/python2.7/site-packages/django/db/backends/postgre
sql_psycopg2/base.py", line 177, in _cursor
self.connection = Database.connect(**conn_params)
File "/app/.heroku/venv/lib/python2.7/site-packages/psycopg2/__init__.py", lin
e 179, in connect
connection_factory=connection_factory, async=async)
psycopg2.OperationalError: could not connect to server: No such file or director
y
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
I have setup this same project before using the same steps and have never had a problem. I did, a few weeks ago, get an email that Heroku was migrating away from shared databases and assume this is somehow involved.
Also, I did notice two NEW steps in the tutorial, namely, installing dj-database-url and adding these lines to settings.py
import dj_database_url
DATABASES = {'default': dj_database_url.config(default='postgres://localhost')}
I have tried to run this both with and without these lines and get the same issue regardless.
Another post suggested the fix was to do this
heroku addons:add shared-database
Tried, get a message that shared-database is deprecated and to use heroku-postgresql, but that had no effect.
Thanks for any help
config HOST as localhost like :
'HOST': 'localhost',
The simplest(not comprehensive but one to get you up and running as quick as possible) solution to this error is, in your settings.py file , in the part where you have set up database settings revert the settings to
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': BASE_DIR / 'db.sqlite3',
}
}
make migrations (i.e python manage.py makemigrations and then python manage.py migrate)
push your changes to GitHub, in your Heroku account, create a new app then move to the setting portion of your new app, connect to GitHub and deploy from GitHub, the database resources will then be provisioned for you (i.e a PostgreSQL database will be created with the same schema and data as your local SQLite database).