I am trying to access a database via postgresql2 with my jupyter notebook but I receive the following error message.
OperationalError: could not create SSL context: no such file
import pandas as pd
import psycopg2
#Connect to postgres
conn_string = "host='xx' sslmode='require' \
dbname='dbname' port='xx' user='xx' \
password='xx'"
#Create rework dataset
conn = psycopg2.connect(conn_string)
postgreSQL_select_Query = u'SELECT * FROM "xx"."yy"'
conn.set_client_encoding('UNICODE')
cursor = conn.cursor()
cursor.execute(postgreSQL_select_Query)
colnames = [desc[0] for desc in cursor.description]
df_imp = cursor.fetchall()
df = pd.DataFrame(data=df_imp, columns=colnames)
Expected result is the access to the database and generation of dataframe.
Actual result is OperationalError: could not create SSL context: no such file by step conn = psycopg2.connect(conn_string)
---------------------------------------------------------------------------
OperationalError Traceback (most recent call last)
<ipython-input-2-932b2fb01c9f> in <module>
5
6 #Create rework dataset
----> 7 conn = psycopg2.connect(conn_string)
8 postgreSQL_select_Query = u'SELECT * FROM "xx"."xx"'
9 conn.set_client_encoding('UNICODE')
~\AppData\Local\Continuum\anaconda3\lib\site-packages\psycopg2\__init__.py in connect(dsn, connection_factory, cursor_factory, **kwargs)
128
129 dsn = _ext.make_dsn(dsn, **kwargs)
--> 130 conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
131 if cursor_factory is not None:
132 conn.cursor_factory = cursor_factory
OperationalError: could not create SSL context: No such process
After trying several solutions, the problem was the version of psycopg2 library.
conda update does not install the latest version of the library. However, pip does it and then my code works again!
Related
edit:
In postgresql.conf, the log_statement is set to:
#log_statement = 'none' # none, ddl, mod, all
My objective is to COPY a .cvs file containing ~300k records to Postgres.
I am running the script below and nothing happens; no error or warning but still the csv is not uploaded.
Any thoughts?
import psycopg2
# Try to connect
try:
conn = psycopg2.connect(database="<db>", user="<user>", password="<pwd>", host="<host>", port="<port>")
print("Database Connected....")
except:
print("Unable to Connect....")
cur = conn.cursor()
try:
sqlstr = "COPY \"HISTORICALS\".\"HISTORICAL_DAILY_MASTER\" FROM STDIN DELIMITER ',' CSV"
with open('/Users/kevin/Dropbox/Stonks/HISTORICALS/dump.csv') as f:
cur.copy_expert(sqlstr, f)
conn.commit()
print("COPY pass")
except:
print("Unable to COPY...")
# Close communication with the database
cur.close()
conn.close()
This is what my .csv looks like
Thanks!
Kevin
I suggest you to load in first time your df with pandas
import pandas as pd
import psycopg2
conn = psycopg2.connect(database="<db>", user="<user>", password="<pwd>", host="<host>", port="<port>")
cur = conn.cursor()
df = pd.read_csv('data.csv')
cur.copy_from(df, schema , null='', sep=',/;', columns=(df.columns))
For the part columns=(df.columns) I forgot if they want turple or list but should work with a conversion and you should read this
Pandas dataframe to PostgreSQL table using psycopg2 without SQLAlchemy? who could help you
I am referring https://github.com/zubkov-andrei/pg_profile for generating awr like report.
Steps which I have followed are as below :
1) Enabled below parameters inside postgresql.conf (located inside D:\Program Files\PostgreSQL\9.6\data)
track_activities = on
track_counts = on
track_io_timing = on
track_functions = on
shared_preload_libraries = 'pg_stat_statements'
pg_stat_statements.max = 1000
pg_stat_statements.track = 'top'
pg_stat_statements.save = off
pg_profile.topn = 20
pg_profile.retention = 7
2) Manually copied all the file beginning with pg_profile to D:\Program Files\PostgreSQL\9.6\share\extension
3) From pgAdmin4 console executed below commands successfully
CREATE EXTENSION dblink;
CREATE EXTENSION pg_stat_statements;
CREATE EXTENSION pg_profile;
4) To see which node is already present I executed SELECT * from node_show();
which resulted in
node_name as local
connstr as dbname=postgres port=5432
enabled as true
5) To create a snapshot I executed SELECT * from snapshot('local');
but getting below error
ERROR: could not establish connection
DETAIL: fe_sendauth: no password supplied
CONTEXT: SQL statement "SELECT dblink_connect('node_connection',node_connstr)"
PL/pgSQL function snapshot(integer) line 38 at PERFORM
PL/pgSQL function snapshot(name) line 9 at RETURN
SQL state: 08001
Once I am able to generate multiple snapshot then I guess I should be able to generate report.
just use SELECT * from snapshot()
look at the code of the function. It calls the other one with node as parameter.
I ran into a 500 server error when trying to connect to PostgreSQL from wsgi file using psycopg2.
import psycopg2
def application(environ, start_response):
try:
conn = psycopg2.connect("database = testdb, user = postgres, password = secret")
execept:
print "I am unable to connect to the database"
status = '200 OK'
output = 'Hello Udacity, Robert!'
response_headers = [('Content-type', 'text/plain'), ('Content-Length', str(len(output)))]
start_response(status, response_headers)
return [output]
After the creation of a database ('udacity' in my case), a table ('hello' in my case) and the population ('Hello world' in my case):
# psql environment
CREATE DATABASE udacity;
CREATE TABLE hello(
word text);
INSERT INTO hello VALUES ('Hello world');
You have to install python dependencies:
# shell environment
sudo apt install python-psycopg2 libpq-dev
sudo apt install python-pip
pip install psycopg2
Then, the next myapp.wsgi script works for me:
import psycopg2
def application(environ, start_response):
status = '200 OK'
output = 'Original message'
response_headers = [('Content-type', 'text/plain'), ('Content-Length', str(len(output)))]
start_response(status, response_headers)
# DB connection
try:
connection = psycopg2.connect("dbname='udacity' user='ubuntu' host='localhost' password='udacity'")
cursor = connection.cursor()
cursor.execute("""SELECT * from hello""")
rows = cursor.fetchall()
for row in rows:
output = row[0]
except:
output = 'E: Python script'
return [output]
I am using Mac Yosemite.
I have installed the packages postgresql, psycopg2, and simplejson using conda install "package name".
After the installation I have imported these packages. I tried to create a json file with my amazon redshift credentials
{
"user_name": "YOUR USER NAME",
"password": "YOUR PASSWORD",
"host_name": "YOUR HOST NAME",
"port_num": "5439",
"db_name": "YOUR DATABASE NAME"
}
I used with
open("Credentials.json") as fh:
creds = simplejson.loads(fh.read())
But this is throwing error. These were the instructions given on a website. I tried searching other websites but no site gives a good explanation.
Please let me know the ways I can connect the Jupyter to amazon redshift.
There's a nice guide from RJMetrics here: "Setting up Your Analytics Stack with Jupyter Notebook & AWS Redshift". It uses ipython-sql
This works great and displays results in a grid.
In [1]:
import sqlalchemy
import psycopg2
import simplejson
%load_ext sql
%config SqlMagic.displaylimit = 10
In [2]:
with open("./my_db.creds") as fh:
creds = simplejson.loads(fh.read())
connect_to_db = 'postgresql+psycopg2://' + \
creds['user_name'] + ':' + creds['password'] + '#' + \
creds['host_name'] + ':' + creds['port_num'] + '/' + creds['db_name'];
%sql $connect_to_db
In [3]:
% sql SELECT * FROM my_table LIMIT 25;
Here's how I do it:
----INSERT IN CELL 1-----
import psycopg2
redshift_endpoint = "<add your endpoint>"
redshift_user = "<add your user>"
redshift_pass = "<add your password>"
port = <your port>
dbname = "<your db name>"
----INSERT IN CELL 2-----
from sqlalchemy import create_engine
from sqlalchemy import text
engine_string = "postgresql+psycopg2://%s:%s#%s:%d/%s" \
% (redshift_user, redshift_pass, redshift_endpoint, port, dbname)
engine = create_engine(engine_string)
----INSERT IN CELL 3 - THIS EXAMPLE WILL GET ALL TABLES FROM YOUR DATABASE-----
sql = """
select schemaname, tablename from pg_tables order by schemaname, tablename;
"""
----LOAD RESULTS AS TUPLES TO A LIST-----
tables = []
output = engine.execute(sql)
for row in output:
tables.append(row)
tables
--IF YOU'RE USING PANDAS---
raw_data = pd.read_sql_query(text(sql), engine)
The easiest way is to use this extension -
https://github.com/sat28/jupyter-redshift
The sample notebook shows how it loads redshift utility as an IPython Magic.
Edit 1
Support for writing back to redshift database has also been added.
The question says it all. Using the Python API as an example:
import ibm_db
#setup stuff
conn = ibm_connect(DATABASE, user, password)
stmt = ibm_db.exec_immediate(conn, 'DELETE FROM sometable')
print(ibm_db.num_rows(stmt)) # prints -1
Why doesn't it print the actual count of rows deleted?
This is not an answer, really, just showing that the function does work and that SQLCODEs cause python exceptions. The fact that num_rows() does not work for you may indicate that the functionality may not be supported by the database you're connected to. You may want to describe your environment in detail: the DB2 server version and platform, whether it's a local or remote database, the DB2 client version (if different from the server) etc.
>>> import ibm_db
>>> conn = ibm_db.connect('TEST',user,password)
>>> stmt = ibm_db.exec_immediate(conn, 'create table t (f int)')
>>> print ibm_db.num_rows(stmt) # DDL statement - num_rows not applicable
-1
>>> stmt = ibm_db.exec_immediate(conn,'delete from t')
>>> print ibm_db.num_rows(stmt) # 0 rows deleted
0
>>> stmt = ibm_db.exec_immediate(conn,'delete from x') # nonexistent table - exception
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
Exception: [IBM][CLI Driver][DB2/LINUXX8664] SQL0204N "USER.X" is an undefined name. SQLSTATE=42704 SQLCODE=-204
>>> stmt = ibm_db.exec_immediate(conn,'insert into t(f) values (1),(2),(3)')
>>> print ibm_db.num_rows(stmt) # 3 rows inserted
3
>>> stmt = ibm_db.exec_immediate(conn,'delete from t')
>>> print ibm_db.num_rows(stmt) # 3 rows deleted
3
>>> ibm_db.close(conn)
True
>>>
A negative SQL Code in DB2 means an error:
SQL codes