I'm having trouble connecting to a SQL database in Juptyer Notebook.
I have changed the port to match the port on my computer (because it was not the default).
dbconn = psycopg2.connect(dbname='mimic',user=[hidden], host='localhost', password=[hidden], port=5433)
I'm using the exact username and password I use to login into the database online, so I expect it to successfully set up the connection, however Jupyter Notebook keeps giving me an Operational Error:
~\AppData\Local\Continuum\anaconda3\lib\site-packages\psycopg2\__init__.py in connect(dsn, connection_factory, cursor_factory, **kwargs)
124
125 dsn = _ext.make_dsn(dsn, **kwargs)
--> 126 conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
127 if cursor_factory is not None:
128 conn.cursor_factory = cursor_factory
OperationalError: FATAL: password authentication failed for user
Related
I've set up a PostgreSQL database running on Docker and am currently trying to get psycopg2-binary to run a script to create tables in the PostgreSQL db. I keep getting this error and I don't know/understand what to do next. Please help!
My script:
import psycopg2
import os
from dotenv import load_dotenv
HOST = os.environ.get("POSTGRES_HOST")
USER = os.environ.get("POSTGRES_USER")
PASSWORD = os.environ.get("POSTGRES_PASSWORD")
DATABASE = os.environ.get("POSTGRES_DB")
PORT = os.environ.get("POSTGRES_PORT")
class Connection():
def __init__(self):
self.conn = psycopg2.connect(
database=DATABASE,
user=USER,
password=PASSWORD,
host=HOST,
port=PORT
)
self.cursor = self.conn.cursor()
Error:
File "venv/lib/python3.8/site-packages/psycopg2/__init__.py", line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: connection to server on socket "/tmp/.s.PGSQL.5432" failed: No such file or directory
Is the server running locally and accepting connections on that socket?
Suppose I run my Apache Superset on top of the Docker and I want this to connect with my local postgreSQL server. I used the following URI but I got an error:
postgresql+psycopg2://username:password#localhost:5432/mydb
The error is:
ERROR: {"error": "Connection failed!\n\nThe error message returned was:\n(psycopg2.OperationalError) could not connect to server: Connection refused\n\tIs the server running on host \"localhost\" (127.0.0.1) and accepting\n\tTCP/IP connections on port 5432?\ncould not connect to server: Cannot assign requested address\n\tIs the server running on host \"localhost\" (::1) and accepting\n\tTCP/IP connections on port 5432?\n\n(Background on this error at: http://sqlalche.me/e/e3q8)", "stacktrace": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py\", line 2265, in _wrap_pool_connect\n return fn()\n File \"/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py\", line 303, in unique_connection\n return _ConnectionFairy._checkout(self)\n File \"/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py\", line 760, in _checkout\n fairy = _ConnectionRecord.checkout(pool)\n File \"/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py\", line 492, in checkout\n rec = pool._do_get()\n File \"/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py\", line 238, in _do_get\n return self._create_connection()\n File \"/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py\", line 308, in _create_connection\n return _ConnectionRecord(self)\n File \"/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py\", line 437, in __init__\n self.__connect(first_connect_check=True)\n File \"/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py\", line 639, in __connect\n connection = pool._invoke_creator(self)\n File \"/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py\", line 114, in ...
How can I solve it?
Instead of using localhost or 127.0.0.1, open up your pgAdmin.
The servers are on the left.
Click the dropdown.
Right click on the now opened cluster (level above "Databases") & open properties.
Navigate to the opened connection tab and the Hostname/Address is your replacement for "localhost"
Also make sure the final part of your connection string is pointed at your database which is one level below "Databases" in your pgAdmin.
I encountered the same problem with connecting superset to local database (postgresql), and after consulting many sites on the internet this trick solved it.Instead of local host, try to put this in SQLalchemy URI:
postgresql+psycopg2://user:password#host.docker.internal:5432/database
I understand that It is a bad practice action to connect the Docker with a host database so I changed my opinion and use the postgres image inside the docker and push my data to that postgres server.
It would be helpful if you notify me if I am wrong.
Services
My service based on flask + postgresql + gunicorn + supervisor + nginx
When deploying by docker, after running the service, then accessing the api, sometimes it told the error message, and sometimes it workes well.
And the sqlachemy connect database add the parameters 'sslmode:disable'.
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
Return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
Distilled_params,
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
e, statement, parameters, cursor, context
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1466, in _handle_dbapi_exception
Util.raise_from_cause(sqlalchemy_exception, exc_info)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
Reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
Cursor, statement, parameters, context
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
Cursor.execute(statement, parameters)
OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
Information
Docker for Mac: version: 2.0.0.3 (31259)
macOS: version 10.14.2
Python: version 2.7.15
Recurrence method
When view port information by command
lsof -i:5432
the port 5432 is postgresql database default port，if the outputconsole was
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
postgres 86469 user 4u IPv6 0xxddd 0t0 TCP *:postgresql (LISTEN)
postgres 86469 user 5u IPv4 0xxddr 0t0 TCP *:postgresql (LISTEN)
it would display the error message:
OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly
but if the outputconsolelog show this:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
com.docke 62421 user 26u IPv4 0xe93 0t0 TCP 192.168.2.7:6435->192.168.2.7:postgresql (ESTABLISHED)
postgres 86460 user 4u IPv6 0xed3 0t0 TCP *:postgresql (LISTEN)
postgres 86460 user 5u IPv4 0xe513 0t0 TCP *:postgresql (LISTEN)
postgres 86856 user 11u IPv4 0xfe93 0t0 TCP 192.168.2.7:postgresql->192.168.2.7:6435 (ESTABLISHED)
the situation, the api would work well.
Becauce of Docker for mac?
Refer link https://github.com/docker/for-mac/issues/2442 , the issue can not solve my problem.
Notice a similar problem?
Refer link Python & Sqlalchemy - Connection pattern -> Disconnected from the remote server randomly
also this issue can not solve my problem.
Solution
flask_sqlachemy need the parameter pool_pre_ping
from flask_sqlalchemy import SQLAlchemy as _BaseSQLAlchemy
class SQLAlchemy(_BaseSQLAlchemy):
def apply_pool_defaults(self, app, options):
super(SQLAlchemy, self).apply_pool_defaults(self, app, options)
options["pool_pre_ping"] = True
db = SQLAlchemy()
Same logic for sqlalchemy.orm, ( on which flask_sqlalchemy is based btw )
engine = sqlalchemy.create_engine(connection_string, pool_pre_ping=True)
More protection strategies can be setup such as it is described in the doc: https://docs.sqlalchemy.org/en/13/core/pooling.html#disconnect-handling-pessimistic
For example, here is my engine instantiation:
engine = sqlalchemy.create_engine(connection_string,
pool_size=10,
max_overflow=2,
pool_recycle=300,
pool_pre_ping=True,
pool_use_lifo=True)
sqlalchemy.orm.sessionmaker(bind=engine, query_cls=RetryingQuery)
For RetryingQuery code, cf: Retry failed sqlalchemy queries
I'm posting my own answer to this, since none of the above addressed my particular setup (Postgres 12.2, SQLAlchemy 1.3).
To stop the OperationalErrors, I had to pass in some additional connect_args to create_engine:
create_engine(
connection_string,
pool_pre_ping=True,
connect_args={
"keepalives": 1,
"keepalives_idle": 30,
"keepalives_interval": 10,
"keepalives_count": 5,
}
)
Building on the Solution in the answer and the info from #MaxBlax360's answer. I think the proper way to set these config values in Flask-SQLAlchemy is by setting app.config['SQLALCHEMY_ENGINE_OPTIONS']:
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
app = Flask(__name__)
# pool_pre_ping should help handle DB connection drops
app.config['SQLALCHEMY_ENGINE_OPTIONS'] = {"pool_pre_ping": True}
app.config['SQLALCHEMY_DATABASE_URI'] = \
f'postgresql+psycopg2://{POSTGRES_USER}:{dbpass}#{POSTGRES_HOST}:{POSTGRES_PORT}/{POSTGRES_DBNAME}'
db = SQLAlchemy(app)
See also Flask-SQLAlchemy docs on Configuration Keys
My db configuration:
app = Flask(__name__)
app.config['SQLALCHEMY_ENGINE_OPTIONS'] = {"pool_pre_ping": True}
app.config['SQLALCHEMY_DATABASE_URI'] = os.environ['DATABASE_URL']
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
# Play with following options:
app.config['SQLALCHEMY_POOL_SIZE'] = 10
app.config['SQLALCHEMY_MAX_OVERFLOW'] = 20
app.config['SQLALCHEMY_POOL_RECYCLE'] = 1800
db = SQLAlchemy(app)
Problem description:
My system user is milosz, which is mapped to the PostgreSQL user project_great in pg_ident.conf. I am using peer authentication to connect to a PostgreSQL database over a unix domain socket. This connection method works when using psql, but fails to work when using py-postgresql using the same parameters from within a Python script.
Here I am successfully connecting to the database using psql:
$ psql -U project_great \
> -d project_great \
> -h /var/run/postgresql
psql (9.3.4)
Type "help" for help.
project_great=>
Here is database_test.py:
#!/usr/bin/env python3
import postgresql
params = {
'user': 'project_great',
'database': 'project_great',
'unix': '/var/run/postgresql',
}
connection = postgresql.open(**params)
Here I am attempting to connect to the database by running ./database_test.py:
$ ./database_test.py
Traceback (most recent call last):
File "./database_test.py", line 11, in <module>
sys.exit(main(sys.argv))
File "./database_test.py", line 13, in main
connection = postgresql.open(**params)
File "/home/milosz/devel/project_great/.virtualenv/lib/python3.3/site-packages/postgresql/__init__.py", line 94, in open
c.connect()
File "/home/milosz/devel/project_great/.virtualenv/lib/python3.3/site-packages/postgresql/driver/pq3.py", line 2422, in connect
self._establish()
File "/home/milosz/devel/project_great/.virtualenv/lib/python3.3/site-packages/postgresql/driver/pq3.py", line 2548, in _establish
self.typio.raise_client_error(could_not_connect, creator = self, cause = exc)
File "/home/milosz/devel/project_great/.virtualenv/lib/python3.3/site-packages/postgresql/driver/pq3.py", line 514, in raise_client_error
raise client_error
postgresql.exceptions.ClientCannotConnectError: could not establish connection to server
CODE: 08001
LOCATION: CLIENT
CONNECTION: [failed]
failures[0]:
socket'/var/run/postgresql'
Traceback (most recent call last):
File "/home/milosz/devel/project_great/.virtualenv/lib/python3.3/site-packages/postgresql/protocol/client3.py", line 136, in connect
self.socket = self.socket_factory(timeout = timeout)
File "/home/milosz/devel/project_great/.virtualenv/lib/python3.3/site-packages/postgresql/python/socket.py", line 64, in __call__
s.connect(self.socket_connect)
PermissionError: [Errno 13] Permission denied
The above exception was the direct cause of the following exception:
postgresql.exceptions.ConnectionRejectionError: Permission denied
CODE: 08004
LOCATION: CLIENT
CONNECTOR: [Unix] pq://project_great#[unix::var:run:postgresql]/project_great
category: None
DRIVER: postgresql.driver.pq3.Driver
Since the parameters to the two connections are ostensibly the same and my permissions on the socket and its containing directory are fairly open, I do not know what the issue is. Using TCP is not a solution; I want to use unix domain sockets. The py-postgresql documentation indicates that connecting using unix domain sockets should work.
Configuration:
pg_hba.conf:
# TYPE DATABASE USER ADDRESS METHOD OPTION
local all all peer map=default
pg_ident.conf:
# MAPNAME SYSTEM-USERNAME PG-USERNAME
default postgres postgres
default milosz project_great
postgresql.conf:
...
port = 5432
unix_socket_directories = '/var/run/postgresql'
...
Here are the permissions on my socket directory:
$ ll /var/run/postgresql/
total 8
drwxrwsr-x 2 postgres postgres 100 May 17 00:20 ./
drwxr-xr-x 31 root root 900 May 17 00:41 ../
-rw-r--r-- 1 postgres postgres 5 May 17 00:20 9.3-main.pid
srwxrwxrwx 1 postgres postgres 0 May 17 00:20 .s.PGSQL.5432=
-rw------- 1 postgres postgres 70 May 17 00:20 .s.PGSQL.5432.lock
The PostgreSQL user project_great has been granted all privileges on the database project_great and both the user and the database exist.
I do not have a ~/.pgpass.
Environment:
Ubuntu 13.10
Python 3.3
PostgreSQL 9.3
py-postgresql 1.1.0
Craig Ringer suggested to run both programs under strace to find out if there is a relevant difference in the system calls. I grepped for /var/run/postgresql and found that while psql ran connect like so:
connect(4, {sa_family=AF_LOCAL, sun_path="/var/run/postgresql/.s.PGSQL.5432"}, 110) = 0
./database_test.py ran connect like so:
connect(4, {sa_family=AF_LOCAL, sun_path="/var/run/postgresql"}, 21) = -1 EACCES (Permission denied)
wherein lay the issue.
While psql expects the path to the directory in which the unix domain socket is located, py-postgresql expects the full path to the socket. Therefore, the fix was to modify database_test.py to read like so:
#!/usr/bin/env python3
import postgresql
params = {
'user': 'project_great',
'database': 'project_great',
'unix': '/var/run/postgresql/.s.PGSQL.5432',
}
connection = postgresql.open(**params)
Honestly, I feel silly for not having tried the full path as an earlier debugging step!
I'm using Net::FTP::Foreign with password authentication. and I get error about pty and permission error. firstly, cannot connect to pty slave. secondly, Foreign seems not use password authentication mode. IO:Pty is installed. the machine is solaris.
Code:
my %args = (password => 'pwd', user => 'usr', port => 22, more => '-v');
log->info("Starting SFTP...\n");
my $sftp = Net::SFTP::Foreign->new($O->{sftp_server}, %args);
is there any wrong with my script or any configuration issue on my server.
thanks
Log:
warning: TIOCSCTTY failed, slave might not be set as controlling
terminal: Inappropriate ioctl for device at
/usr/pkgs/Export/US/SunOS5.10_i86pc/lib/perl5/CPAN/i86pc-solaris/IO/Pty.pm
line 121. Error: could not connect pty as controlling terminal! ssh:
tty: node name or service name not known Uncaught exception from user
code: Unable to establish SFTP connection: SSH slave exited
unexpectedly with error code 255 at
/usr/pkgs/Export/OPS/SunOS5.10_i86pc/cbin/mkt_daily_new_issue_download.pl
line 162 at
/usr/pkgs/Export/US/SunOS5.10_i86pc/lib/perl5/CPAN/Net/SFTP/Foreign/Common.pm
line 103
Net::SFTP::Foreign::Common::die_on_error('Net::SFTP::Foreign=HASH(0x84cfa78)',
'Unable to establish SFTP connection') called at
/usr/pkgs/Export/OPS/SunOS5.10_i86pc/cbin/mkt_daily_new_issue_download.pl
line 162