I'm trying to connect to a postgres-DB, which unfortunately has a name with a whitespace in it:
%load_ext sql
from sqlalchemy import create_engine
%sql postgresql://postgres:dbpass#localhost/Test DB
(psycopg2.OperationalError) FATAL: database "Test DB" does not exist
I've tried to follow some tipps on the internet and used:
import urllib.parse
urllib.parse.quote_plus("Test DB")
which simply results in a string "Test+DB" (this does not work).
How can I adress the database, without changing its name?
Best regards!
I was able to solve it by using sqlalchemy's create_engine(), therefore being able to simply save the string (with its whitespace) as a variable (eg database_name):
import sqlalchemy as db
database_name = 'Test DB'
engine = db.create_engine('postgresql://' + 'user_name' + ':' + 'password' + '#localhost/' + database_name)
connection = engine.connect()
s = 'SELECT id FROM user'
df = pd.read_sql_query(s, engine)
df.head()
Hope it helps, best regards.
Related
I'm trying to connect to a postgres database by using psycopg2. I ask for the user and user password via input. But I think, this way doesnt let me connect to the database. Is there another way to place the values?
I try to connect like this:
connection = psycopg2.connect("host=localhost dbname=dbname user=%s password=%s", (username, password))
This is the error I get:
connection = _connect(dsn, connection_factory=connection_factory, **kwasync)
TypeError: 'tuple' object is not callable
There are a couple of ways to build the connection string from inputs, first:
# From input
user_name = 'some_name'
pwd = 'some_pwd'
connection = psycopg2.connect(host=localhost, dbname=dbname, user=user_name, password=pwd)
Second from here Make dsn:
from psycopg2.extensions import make_dsn
dsn = make_dsn('host=localhost dbname=dbname', user=user_name, password=pwd)
connection = psycopg2.connect(dsn)
UPDATE, forgot the most obvious way:
dsn = 'host=localhost dbname=dbname user=' + user_name + ' password=' + pwd
connection = psycopg2.connect(dsn)
I have a psycopg2 connection which I am using to connect to postgresql from pyspark. Here is my code -
host = 'IP Address'
port = 'Port'
user = 'postgres'
db = 'postgres'
password = 'password'
def move_records(main_table,stg_table):
try:
connection = psycopg2.connect(host=host,
database=db,
user=user,
password=password,
#driver = driver,
port = port)
cursor = connection.cursor()
move_query = "INSERT INTO " +main_table+ " select * from "+stg_table+" where country ='USA'"
cursor.execute(move_query)
connection.commit()
logger.debug("Record moved successfully")
except (Exception, psycopg2.DatabaseError) as error :
logger.error("%s Error in transction Reverting all other operations of a transaction ", error)
global flag
flag = False
connection.rollback()
finally:
if(connection):
cursor.close()
connection.close()
logger.debug("PostgreSQL connection is closed")
move_records(table_1,table_2)
But I keep getting error below error on line if(connection):
"UnboundLocalError","evalue":"local variable 'connection' referenced before assignment"
Can not figure out what is the issue. Need help.
I am no expert in Python but I have worked on similar thing, connecting from Python to Postgres in in AWS Lambda using psycopg2.
I believe the error lies somewhere in scope of variable. You need to declare all variables(host, port, user, db, password) inside function once again as global or nonlocal and then try to run function.
For your reference, check out this link:-
my table in postgresql v11 contains a json field, i would to store this data in an external file JSON using python (lib: SQLALSHEMY). i try to do it but i have a difficulty:
import psycopg2
from app import db
from models import Geotab
from sqlalchemy.dialects.postgresql import JSON
from sqlalchemy.sql import select
try:
connection = psycopg2.connect(user = "postgres",
password = "admin",
host = "127.0.0.1",
port = "5432",
database = "catalogue")
cursor = connection.cursor()
# Print PostgreSQL Connection properties
print ( connection.get_dsn_parameters(),"\n")
# Print PostgreSQL version
cursor.execute("SELECT version();")
record = cursor.fetchone()
print("You are connected to - ", record,"\n")
except (Exception, psycopg2.Error) as error :
print ("Error while connecting to PostgreSQL", error)
data=db.session.query(Geotab).\
filter(Geotab.dataJson['']) #.astext.cast(JSON))
print (data)
after executing code:
user=postgres password=admin host=127.0.0.1 port=5432 dbname=catalogue
{'user': 'postgres', 'dbname': 'catalogue', 'host': '127.0.0.1', 'port': '5432', 'tty': '', 'options': '', 'sslmode': 'prefer', 'sslcompression': '0', 'krbsrvname': 'postgres', 'target_session_attrs': 'any'}
You are connected to - ('PostgreSQL 11.4, compiled by Visual C++ build 1914,
64-bit',)
SELECT geocat.id AS geocat_id, geocat.url AS geocat_url, geocat.date AS
geocat_date, geocat."dataJson" AS "geocat_dataJson"
FROM geocat
WHERE geocat."dataJson" -> %(dataJson_1)s
geotab : class in models.py
dataJson : column type json
table name: geocat
help me please
i find the solution, thank you:
from sqlalchemy import create_engine
from app import db
from models import Geotab
from sqlalchemy import Integer
import json
from sqlalchemy.dialects.postgresql import JSON
engine = create_engine('postgresql://postgres:admin#127.0.0.1:5432/catalogue', echo =
True)
conn = engine.connect()
data=db.session.query(Geotab).\
filter(Geotab.id==1)
print (data)
result = db.session.query(Geotab).all()
metadata={}
for row in result:
print ("dataJson:",row.dataJson)
metadata=row.dataJson
with open('metadata.json', 'w') as outfile:
json.dump(metadata, outfile)
When I am trying to authenticate mongodb by using pymongo, it shows the below error,
command SON([('saslStart', 1), ('mechanism', 'SCRAM-SHA-1'),
('autoAuthorize', 1), ('payload',
Binary('n,,n=user,r=Mzk1NDU4ODUwNzU4', 0))]) on namespace
the_database.$cmd failed: Authentication failed.
My authentication code
client.fw_mongo.authenticate('user', 'password' )
fw_mongo is my mongodb name
What is wrong with this code?
Your code is correct, but your password is not the same as you configured MongoDB with.
try:
from pymongo import MongoClient
Client = MongoClient("mongodb://" + username + ":" + password + "server_details")
def collection_object():
db = Client.db_name
collection = db.collection_name
return collection
I have a small flask application which I am deploying to Heroku.
My local configuration looks like this:
from flask import Flask
from flask.ext.mongoengine import MongoEngine
app = Flask(__name__)
app.debug = True
app.config["MONGODB_SETTINGS"] = {'DB': "my_app"}
app.config["SECRET_KEY"] = "secretpassword"
db = MongoEngine(app)
So, I know that I need to configure the app to use the Mongo URI method of connection, and I have my connection info:
mongodb://<user>:<password>#alex.mongohq.com:10043/app12345678
I am just a little stuck as to the syntax for modifying my app to connect through the URI.
So I got it working (finally):
from flask import Flask
from mongoengine import connect
app = Flask(__name__)
app.config["MONGODB_DB"] = 'app12345678'
connect(
'app12345678',
username='heroku',
password='a614e68b445d0d9d1c375740781073b4',
host='mongodb://<user>:<password>#alex.mongohq.com:10043/app12345678',
port=10043
)
Though I anticipate that various other configurations will work.
When you look at the flask-mongoengine code, you can see what configuration variables are available
So this should work:
app.config["MONGODB_HOST"] = 'alex.mongohq.com/app12345678'
app.config["MONGODB_PORT"] = 10043
app.config["MONGODB_DATABASE"] = 'dbname'
app.config["MONGODB_USERNAME"] = 'user'
app.config["MONGODB_PASSWORD"] = 'password'
db = MongoEngine(app)
I'm not sure, if app123 is the app or the database name. You might have to fiddle arround a little to get the connection. I had the same problem with Mongokit + MongoLab on Heroku :)
Also you could use the URI like this.
app.config["MONGODB_SETTINGS"] = {'DB': "my_app", "host":'mongodb://<user>:<password>#alex.mongohq.com:10043/app12345678'}
I have actually no idea, at what point "MONGODB_SETTINGS" is read, but it seemed to work, when I tried it in the shell.
I figured out how to use the flask.ext.mongoengine.MongoEngine wrapper class to do this rather than mongoengine.connect():
from flask import Flask
from flask.ext.mongoengine import MongoEngine
app = Flask(__name__)
HOST = '<hostname>' # ex: 'oceanic.mongohq.com'
db_settings = {
'MONGODB_DB': '<database>',
'MONGODB_USERNAME': '<username>',
'MONGODB_PASSWORD': '<password>',
'MONGODB_PORT': <port>,
}
app.config = dict(list(app.config.items()) + list(db_settings.items()))
app.config["MONGODB_HOST"] = ('mongodb://%(MONGODB_USERNAME)s:%(MONGODB_PASSWORD)s#'+
HOST +':%(MONGODB_PORT)s/%(MONGODB_DB)s') % db_settings
db = MongoEngine(app)
if __name__ == '__main__':
app.run()
If you're using mongohq, app.config["MONGODB_HOST"] should match the Mongo URI under Databases->Admin->Overview.
You can then follow MongoDB's tumblelog tutorial using this setup to write your first app called tumblelog.
Using python's nifty object introspection (python oh how I love you so), you can see how the MongoEngine wrapper class accomplishes this:
from flask.ext.mongoengine import MongoEngine
import inspect
print(inspect.getsource(MongoEngine))
...
conn_settings = {
'db': app.config.get('MONGODB_DB', None),
'username': app.config.get('MONGODB_USERNAME', None),
'password': app.config.get('MONGODB_PASSWORD', None),
'host': app.config.get('MONGODB_HOST', None),
'port': int(app.config.get('MONGODB_PORT', 0)) or None
}
...
self.connection = mongoengine.connect(**conn_settings)
...
self.app = app