I am trying to replace a piece of sql code with a python variable that I will ask a user to generate using a raw_input.
Below is the code i'm using which works great if I set mypythonvariable manually i.e. inputting 344 into the sql code, but if I set the sql as is to mypythonvariable it doesn't work.
The whole sql query is then converted into a pandas dataframe for further messing about with.
Any help on how to do be appreciated.
UPDATE: I just added the %s code into the statement and i'm now getting the error message '': not all arguments converted during string formatting
'
conn = pg.connect(host = "localhost",
port = 1234,
dbname = "somename",
user = "user",
password = "pswd")
mypythonvariable = raw_input("What is your variable number? ")
sql = """
SELECT
somestuff
FROM
sometable
WHERE
something = %s
"""
df = pd.read_sql_query(sql, con=conn,params=mypythonvariable)
thanks to all that looked.
I found the solution.Looks like the params need to be passed as a list.
conn = pg.connect(host = "localhost",
port = 1234,
dbname = "somename",
user = "user",
password = "pswd")
mypythonvariable = raw_input("What is your variable number? ")
sql = """
SELECT
somestuff
FROM
sometable
WHERE
something = %s
"""
df = pd.read_sql_query(sql, con=conn,params=[mypythonvariable])
Related
I have been using this to connect to our organisation's cluster to query the data. is it possible to convert the cursor output directly into pyspark dataframe.
jarFile="/dataflame/nas_nfs/tmp/lib/olympus-jdbc-driver.jar"
url = "olympus:jdbcdriver"
env='uat'
print("Using environment", env.upper())
className = "net.vk.olympus.jdbc.driver.OlympusDriver"
conn = jaydebeapi.connect(className, url,{'username':userid,'password':pwd,'ENV':env,'datasource':'HIVE','EnableLog':'false'},jarFile)
cursor = conn.cursor()
query = "select * from abc.defcs123 limit 5"
cursor.execute(query)
pandas_df = as_pandas(cursor)
print(pandas_df)
I am trying to pipe a webscraped pandas dataframe into a MySql table with mysql.connector but I can't seem to link df values to the %s variable. The connection is good (I can add individual rows) but it just returns errors when I replace the value witht he %s.
cnx = mysql.connector.connect(host = 'ip', user = 'user', passwd = 'pass', database = 'db')
cursor = cnx.cursor()
insert_df = ("""INSERT INTO table"
"(page_1, date_1, record_1, task_1)"
"VALUES ('%s','%s','%s','%s')""")
cursor.executemany(insert_df, df)
cnx.commit()
cnx.close()
This returns "ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()."
If I add any additional oiperations it returns "ProgrammingError: Parameters for query must be an Iterable."
I am very new to this so any help is appreciated
Work around for me was to redo my whole process. I ran sqlalchemy, all the documentation makes this very easy. message if you want the code I used.
I'm using psycopg2 to query a database that starts with a number + ".district", so my code goes like:
number = 2345
cur = conn.cursor()
myquery = """ SELECT *
FROM {0}.districts
;""".format(number)
cur.execute("""{0};""".format(query))
data = cur.fetchall()
conn.close()
And i keep receiving the following psycopg2 error..
psycopg2.ProgrammingError: syntax error at or near "2345."
LINE 1: SELECT * FROM 2345.districts...
Thought it was the type of data the problem, maybe int(number) or str(number)..but no, same error appears.
¿ What am i doing wrong ?
The way you are trying to use to pass parameters is not supported. Please read the docs.
I am working with the script below.
If I change the script so I avoid the bytea datatype, I can easily copy data from my postgres table into a python variable.
But if the data is in a bytea postgres column, I encounter a strange object called memory which confuses me.
Here is the script which I run against anaconda python 3.5.2:
# bytea.py
import sqlalchemy
# I should create a conn
db_s = 'postgres://dan:dan#127.0.0.1/dan'
conn = sqlalchemy.create_engine(db_s).connect()
sql_s = "drop table if exists dropme"
conn.execute(sql_s)
sql_s = "create table dropme(c1 bytea)"
conn.execute(sql_s)
sql_s = "insert into dropme(c1)values( cast('hello' AS bytea) );"
conn.execute(sql_s)
sql_s = "select c1 from dropme limit 1"
result = conn.execute(sql_s)
print(result)
# <sqlalchemy.engine.result.ResultProxy object at 0x7fcbccdade80>
for row in result:
print(row['c1'])
# <memory at 0x7f4c125a6c48>
How to get the data which is inside of memory at 0x7f4c125a6c48 ?
You can cast it use python bytes()
for row in result:
print(bytes(row['c1']))
EDIT: I now have a better idea of what is going wrong. When I perform that query in plain old psql, I get the following result:
lwm#verbos
=# SELECT * FROM gerund LIMIT1;
infinitive │ gerund │ gerund_english
────────────┼─────────────┼────────────────
abandonar │ abandonando │ abandoning
So, I am getting back 3 strings? However, I say that I am getting back IO [Only String]. I am sure it is my type signature here that is messing things up ...
I am trying to make a simple query using the postgresql-simple library with Haskell. My code is pasted below along with the error I am seeing. Anyone got any ideas?
My database is called verbos and within it, I have a table called gerund. I am able to run a query_ that contains: conn "SELECT 2 + 2" and that works fine. I can also connect to my database with the default data as specified with the default information (password = 'postgres' : psql -h localhost -p 5432 -U postgres (from the docs[1])
{-# LANGUAGE OverloadedStrings #-}
module Main where
import Control.Monad
import Control.Applicative
import Database.PostgreSQL.Simple
main = do
conn <- connect defaultConnectInfo {
connectPassword = "postgres",
connectDatabase = "verbos"
}
mapM_ print =<< (query_ conn "SELECT * FROM gerund LIMIT 1" :: IO [Only String])
Gives me the following error:
ConversionFailed {errSQLType = "3 values: [(Basic {typoid = Oid 1043,
typcategory = 'S', typdelim = ',', typname = \"varchar\"},Just
\"abandonar\"),(Basic {typoid = Oid 1043, typcategory = 'S', typdelim
= ',', typname = \"varchar\"},Just \"abandonando\"),(Basic {typoid = Oid 1043, typcategory = 'S', typdelim = ',', typname =
\"varchar\"},Just \"abandoning\")]", errSQLTableOid = Nothing,
errSQLField = "", errHaskellType = "1 slots in target type",
errMessage = "mismatch between number of columns to convert and number
in target type"}
OK, Thanks to #AlpMestanogullari, #muistooshort, I got an answer here. My final code is:
{-# LANGUAGE OverloadedStrings #-}
module Main where
import Control.Applicative
import Database.PostgreSQL.Simple
import Database.PostgreSQL.Simple.FromRow
data Gerund = Gerund {
f :: String,
s :: String,
t :: String
} deriving (Show)
instance FromRow Gerund where
fromRow = Gerund <$> field <*> field <*> field
main = do
conn <- connect defaultConnectInfo {
connectPassword = "postgres",
connectDatabase = "verbos"
}
mapM_ print =<< (query_ conn q :: IO [Gerund])
where q = "SELECT * FROM gerund LIMIT 1"
Notes:
Knowing that my result contained 3 result columns, I needed to define a type that had 'space' for the results (f, s and t in the Gerund type)
I followed the docs[1] for FromRow closely to get my type and instance defined.
You need to import import Database.PostgreSQL.Simple.FromRow to access things like field.