Why do psycopg2 and pyodbc connect to databases differently? And how could I translate a connection? - postgresql

So I've been trying to find a good module for querying information at my company's POSTGRES databases.
There are so many modules around, but I can only seem to get pyodbc to work.
Why is pyodbc's form of connecting different than psycopg2, which seems to be the superior module?
And how would I translate my connection query from one to the other?
I can connect to my company's server like this with **pyodbc** no problem.
pyodbc.connect("DRIVER={SQLServer};SERVER=some\servername;DATABASE=someDatabaseName;Trusted_Connection=yes")
But this doesn't work for psycopg2 with:
psycopg2.connect(database="someDatabaseName", host="some\servername")
psycopg2 returns:
OperationalError: could not translate host name "some\servername" to
address: Unknown host
Of topic:
I hope someone can help, my heroku server simply does not want to install pyodbc

pyodbc is a Python module that can be used to connect to many different types of databases using the ODBC API. Specifically, pyodbc is compliant with the DB API 2.0 spec.
Your working code sample uses an ODBC implementation of a SQL Server driver to connect to a SQL Server instance. Drivers are DBMS-specific, ODBC is not.

I had the same issue. I wanted to re-use the ODBC connection details as used by pyodbc to connect to a PostgreSQL DB using psycopg2.
The connection details can be found in the Windows registry.
Basically, the following code will extract the required details from the registry:
import os
from winreg import (
ConnectRegistry,
HKEY_LOCAL_MACHINE,
OpenKeyEx,
QueryValueEx,
)
def odbc_dsn_2_psyopg2dsn(odbc_dsn, user, pwd ):
"""
This function converts a windows ODBC DSN to a dsn
that can be used by psycopg2.
It reads connection parameters from the windows registry,
username and password are often not in there,
and can be supllied as parameters.
"""
# get hostname, port, sslmode from odbc registry
hreg = ConnectRegistry(None,HKEY_LOCAL_MACHINE)
key = f"SOFTWARE\\ODBC\\ODBC.INI\\{odbc_dsn}"
hkey = OpenKeyEx(hreg, key)
hostname = QueryValueEx(hkey, "Servername")[0]
portnumber = QueryValueEx(hkey, "Port")[0]
sslmode = QueryValueEx(hkey, "SSLmode")[0]
database = QueryValueEx(hkey, "Database")[0]
#set defaults for portnmumber and SSL mode (my pref, maybe in your case there are in the registry too)
if not portnumber:
portnumber = "5432"
if not sslmode:
sslmode = 'Prefer'
# using all of the above, create psycopg2 connection-dsn
return f"dbname={database} user={user} password={pwd} host={hostname} port={portnumber} sslmode={sslmode}"

Related

Error to connect Postgresql DB from Oracle dblink using Oracle Database Gateway

I'm trying to connect from Oracle Database (dblink) to Postgresql database use Oracle Database Gateway (dg4msql 19), on the Oracle Database Gateway for MS SQL Server all work fine. On the Oracle DB create dblink, but when try to connect get Error message:
ORA-28500 Oracle ODBC SQL Server Wire Protocol driver Socket closed
08S01 Oracle ODBC SQL Server Wire protocol driver
The server does not support SSL {HY000} !
listener work, tnsnames.ora also give correct answer. May be in initdg4msql.ora file I have to use some string to connect without SSL?
HS_FDS_CONNECT_INFO = postgre_servet.net:5432//database - I don`t known that is correct for PostgreSQL, or for PostgreSQL should use only ODBC connection?
How correctly use HS_FDS_CONNECT_INFO = postgre_servet.net:5432//database, or for PostgreSQL should I use only ODBC connection?
The PostgreSQL server is not configured to accept SSL connections.
First, verify that PostgreSQL was built with SSL support: that can be seen from the output of pg_config --configure (should contain --with-openssl).
To configure SSL support, set ssl = on in postgresql.conf and provide a server certificate and private key (parameters ssl_vert_file and ssl_key_file). Reload PostgreSQL to activate.
For details, read the documentation.

Tableau Desktop connection to PostgreSQL DB using Certificate Authentication over SSL/TLS

I connect to my PostgreSQL databases (which run on AWS EC2 instances) using certificate authentication (and not passwords). An example of a psql command I would use to connect to one of my databases is:
psql "host=<AWS EC2 instance> user=<db user> sslcert=<path to .crt> sslkey=<path to .key> sslrootcert=<path to .crt> sslmode=require dbname=<db name>"
I would like to be able to connect Tableau Desktop to one of my databases. The standard PostgreSQL connector does not allow me to specify which SSL certs to use (but it does have a Require SSL checkbox - which I think is for encryption not authentication).
So I have tried to use the ODBC connector. I make the following entries:
Connect Using
Driver: PostgreSQL Unicode
Connection Attributes
Server: <AWS EC2 instance>
Port: 5432
Database: <db name>
Username: <db user>
String Extras: sslcert=<path to .crt>; sslkey=<path to .key>; sslrootcert=<path to .crt>; sslmode=require
By much trial and error I think the String Extras require a semicolon as a delimiter, but I still cannot connect. I get the error message:
An error occurred while communicating with Other Databases (ODBC).
Unable to connect to the server. Check that the server is running and that you have access privileges to the requested database.
FATAL: connection requires a valid client certificate
Generic ODBC requires additional configuration. The driver and DSN (data source name) must be installed and configured to match the connection.
Unable to connect to the server "<AWS EC2 instance>" using the driver "PostgreSQL Unicode". Check that the server is running and that you have access privileges to the requested database.
I would be really grateful to hear from anyone who has connected Tableau Desktop to PostgreSQL using certificate authentication and could tell me what I’m doing wrong. Cheers!
I managed to fix this myself, and just in case there's someone out there who might be vaguely interested I'll go through the salient features:
1. Created a DSN (Data Store Name)
This made testing much easier rather than constantly retyping details into dialog boxes.
Rather than create manually I used this: http://www.odbcmanager.net/
On Mac OS I had to run it as sudo from the terminal or it wouldn't create anything.
I created a User DSN, which on my Mac have details stored in /Users/<user name>/.odbc.ini
It picked up the PostgreSQL Unicode driver that I'd previously installed. On Mac OS you might need to do brew install brew install psqlodbc and/or brew install unixodbc
Most of the configuration was specified by adding key/value pairs e.g. DBNAME <db name>
Set SSLMODE verify-ca (changed from require to make behaviour specific - psql assumes verify-ca/verify-full if SSL certs are supplied)
The cert paths are set with key Pqopt, value sslcert=<path to .crt> sslkey=<path to .key> sslrootcert=<path to .crt>. Must be lower case and only spaces between entries!
Here's what it added to my user's odbc.ini file:
[<DSN name>]
Driver = <path to driver, this was mine /usr/local/lib/psqlodbcw.so>
Description = <description>
SSLMODE = verify-ca
HOST = <host>
DBNAME = <database>
PORT = 5432
UID = <db user>
Pqopt = sslcert=<path to .crt> sslkey=<path to .key> sslrootcert=<path to .crt>
2. Tableau Desktop Connection
Select Other Database (ODBC) connector.
Select the DSN you created previously (it should be automatically picked up), it will automatically populate host, port, db and user fields, then click Sign In.
Once signed in behaviour was slightly different to the dedicated PostgreSQL connector. Tables did not appear on left-hand side initially. But they are accessible - they can all be displayed by clicking on the 'contains' radio button and searching with a blank name.
Hope this helps. Let me know if you've got any questions.

ODBC connection from Postgres database into Power bi error

Hi I am having issues connecting to my postgres database, I am using this connection string on my 64 bit machine. I have downloaded the updated 64 bit postgres driver.
driver used: database=Travel_Information;driver={PostgreSQL Unicode(x64)};port=5432;server=travel-information.chda0n3dhd8k.eu-west-1.rds.amazonaws.com
Issue when selecting tables to import
Any suggestions?
In the DSN select "(None)" instead of "dBASE Files" and your connection string should look like this.
database=dbname;driver={PostgreSQL Unicode(x64)};port=5432;server=server.eu-w1.compute.server.com;sslmode=require;
I had to use sslmode=require because I am connecting to a postgresql server instance on heroku server, and it requires it.
It is an SSL issue
I added sslmode=require at the end of connection string and it succeeded.
I tried before with the original Postgredb connector and various Npgsql versions but it didn't worked.
Although there's still an option to import the certificate from the server to your client To get the SSL cert for the PostgreSQL server you might need to speak to the DBA/sysadmin if you don’t have access. In the configuration folder for the PostgreSQL server there is a postgresql.conf file, search it for ssl settings, there is one with the location of the ssl cert. Note NOT the key file which contains the private key, only the cert file which contains the public key. copy it or its content to the machine running PowerBI and import using that procedure

Connect to a System DSN via UCanAccess

I'm currently upgrading a project from Java 7 to Java 8, thus I switched from using the JDBC-ODBC Bridge to using UCanAccess. The database I want to connect to is registered as a system DSN so I connected to it like this:
Database.forURL("jdbc:odbc:MyDB" + ";DB_CLOSE_DELAY=-1;charSet=Cp1250", driver = "sun.jdbc.odbc.JdbcOdbcDriver")
Now I changed that to
Database.forURL("jdbc:ucanaccess:MyDB" + ";DB_CLOSE_DELAY=-1;charSet=Cp1250", driver = "net.ucanaccess.jdbc.UcanaccessDriver")
but that gives me the error:
Driver net.ucanaccess.jdbc.UcanaccessDriver does not know how to handle URL jdbc:ucanaccess:MyDB;DB_CLOSE_DELAY=-1;charSet=Cp1250
Is there a way to access a DSN via UCanAccess?
Is there a way to access a DSN via UCanAccess?
Not directly. UCanAccess is a JDBC driver that does not use ODBC, so it has no knowledge of ODBC DSNs.
Your UCanAccess connection string needs to include the path to the Access database file, e.g.,
jdbc:ucanaccess://C:/path/to/mydata.accdb
If necessary, you could retrieve that file path from the configuration information for an ODBC DSN. On Windows, the information for a System DSN named MyDB would be in the Windows registry under
HKEY_LOCAL_MACHINE\SOFTWARE\ODBC\ODBC.INI\MyDb
or perhaps
HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\ODBC\ODBC.INI\MyDb

How to connect to postgresql, without specifying plain-text password, with libpq's pgpass?

Postgres' C library libpq documentation talks about a more secure way to connect to a DB without specifying password in source code.
I was not able to find any examples of how to do it. How to make my Postgre Server use this file? Please help.
You don't import it into your Python program. The point of .pgpass is that it is a regular file subject to the system's file permissions, and the libpq driver which libraries such as psycopg2 use to connect to Postgres will look to this file for the password instead of requiring the password to be in the source code or prompting for it.
Also, this is not a server-side file, but a client-side one. So, on a *nix box, you would have a ~/.pgpass file containing the credentials for the various connections you want to be able to make.
Edit in response to comment from OP:
Two main things need to happen in order for psycopg2 to correctly authenticate via .pgpass:
Do not specify a password in the string passed to psycopg2.connect
Make sure the correct entry is added to the .pgpass file for the user who will be connecting via psycopg2.
For example, to make this work for all databases for a particular user on localhost port 5432, you would add the following line to that user's .pgpass file:
localhost:5432:*:<username>:<password>
And then the connect call would be of this form:
conn = psycopg2.connect("host=localhost dbname=<dbname> user=<username>")
The underlying libpq driver that psycopg2 uses will then utilize the .pgpass file to get the password.
just adding to #khampson's answer, I could only get this to work (in windows) if I added the PGPASSFILE environment variable (and subsequently restart pycharm) even though my pgpass file is in the default location (%APPDATA%\postgresql\pgpass.conf).
Well. You asked for .pgpass but...
There is also "The Connection Service File"
By default, the per-user service file is named ~/.pg_service.conf.
A different file name can be specified by setting the environment variable PGSERVICEFILE
Look at
https://www.postgresql.org/docs/15/libpq-pgservice.html
Format is ".INI". Parameters are connection parameters.
Example:
# this is a comment in my service file
[a_service]
host=dbserver.loc
port=5432
dbname=thedbname
user=thebest
password=bho
application_name=ifyoulike
# Here is another service
[another_service]
host=etc
port=etc
Access via psql:
psql service=a_service
Access via psycopg2:
db = psycopg2.connect('service=a_service')
Access via sqlalchemy:
eng = sqlalchemy.create_engine('postgresql:///?service=another_service')
#!/usr/bin/python
import psycopg2
import sys
import pprint
def main():
conn_string = "host='127.0.0.1' dbname='yourdatabsename' user='yourusername' password='yourpassword'"
print "Connecting to database\n ->%s" % (conn_string)
conn = psycopg2.connect(conn_string)
cursor = conn.cursor()
cursor.execute("SELECT * FROM tablename")
records = cursor.fetchall()
pprint.pprint(records)
if __name__ == "__main__":
main()