I have a table defined as
CREATE TABLE users (id SERIAL PRIMARY KEY, val BYTEA);
Then I want to serialize my data structure with binary and store in the table, and then retrieve and deserialize back.
{-# LANGUAGE OverloadedStrings, DeriveAnyClass #-}
import Control.Monad (forM_)
import Data.Binary (encode, decode, Binary)
import Database.PostgreSQL.Simple
import GHC.Generics (Generic)
data User = { name :: Text, email :: Text } deriving (Show, Generic, Binary)
main = do
conn <- connect --...
let encoded = encode User {name = "me", email = "me#home.net" }
execute conn "INSERT INTO users(val) values(?)" $ Only encoded
rs <- query_ conn "SELECT id, val FROM users"
forM_ rs $ \(id,val) ->
putStrLn $ (show (id :: Int)) ++ ": " ++ show (decode val :: User)
But I get error Data.Binary.Get.runGet at position 0: not enough bytes.
Query
SELECT * FROM users;
gives
id | val
----+-----
1 | \x
I can't figure out how to map ByteStrings to 'BYTEA`s. According to the docs everything should be OK. What am I doing wrong?
Fixed by replacing the line
execute conn "INSERT INTO users(val) values(?)" $ Only encoded
with
execute conn "INSERT INTO users(val) values(?)" $ Only $ Binary encoded
It's because toField(ByteString) yields Escape whereas toField(Binary ByteString) yields EscapeByteA
Related
I'm studying PostgreSQL with Haskell with this lib: https://hackage.haskell.org/package/postgresql-simple-0.4.10.0/docs/Database-PostgreSQL-Simple.html#v:query
While I could select an user like this:
(query_ conn "SELECT * FROM users WHERE NAME == john" :: IO [Only Int]) >>= mapM_ print
using query_:
query_ :: FromRow r => Connection -> Query -> IO [r]
I think I should use query:
query :: (ToRow q, FromRow r) => Connection -> Query -> q -> IO [r]
to pass a list of values. However, how do I pass this list?
For example, for INSERT, I was able to do this:
(execute conn "INSERT INTO users (NAME, PASSWORD) VALUES (?,?)") (["john", "123456"]::[String]) >>= print
but what is the equivalent for SELECT?
I'm not sure I understand your question, since you ask about lists and I don't see how they enter into the picture. But the parameterized version of your select query is this:
query conn "SELECT * FROM users where NAME == ?" (Only ("john" :: String))
I have this table in postgres
CREATE TABLE target (
a json
b integer
c text []
id integer
CONSTRAINT id_fkey FOREIGN KEY (id)
REFERENCES public.other_table(id) MATCH SIMPLE
ON UPDATE NO ACTION
ON DELETE NO ACTION,
)
Which I would like to insert data to from psycopg2 using
import psycopg2
import psycopg2.extras as extras
# data is of the form dict, integer, list(string), string <- used to get fkey id
data = [[extras.Json([{'a':1,'b':2}, {'d':3,'e':2}]), 1, ['hello', 'world'], 'ident1'],
[extras.Json([{'a':4,'b':3}, {'d':1,'e':9}]), 5, ['hello2', 'world2'], 'ident2']]
# convert data to list of tuples containing objects
x = [tuple(u) for u in data]
# insert data to the database
query = ('WITH ins (a, b, c, ident) AS '
'(VALUES %s) '
'INSERT INTO target (a, b, c, id) '
'SELECT '
'ins.a '
'ins.b '
'ins.c '
'other_table.id'
'FROM '
'ins '
'LEFT JOIN other_table ON ins.ident = other_table.ident;')
cursor = conn.cursor()
extras.execute_values(cursor, query, x)
When I run this I get the error: column "a" is of type json but expression is of type text. I tried to solve this by adding a type cast in the SELECT statement but then I got the same error for c and then for b.
Originally I thought the problem lies in the WITH statement but based on the answers to my previous question this seems to not be the case Postgres `WITH ins AS ...` casting everything as text
It seems that execute_values is sending all the values as text with ' '.
Main Question: How can I get execute_values to send the values based on their python data type rather than just as text?
Sub questions:
How can I confirm that execute_values is in fact sending the values as text with quotation marks?
What is the purpose of the template argument of execute_values https://www.psycopg.org/docs/extras.html and could that be of help?
The issue, as Adrian Klaver points out in their comment, and also seen in this answer, is that the typing is lost in the CTE.
We can show this with an example in the psql shell:
CREATE TABLE test (col1 json);
WITH cte (c) AS (VALUES ('{"a": 1}'))
INSERT INTO test (col) SELECT c FROM cte;
resulting in
ERROR: column "col" is of type json but expression is of type text
whereas this version, with the type specified, succeeds:
WITH cte(c) AS (VALUES ('{"a": 1}'::json))
INSERT INTO test (col) SELECT c FROM cte;
We can mimic this in execute_valuesby providing the typing information in the template argument:
extras.execute_values(cursor, query, data, template='(%s::json, %s, %s, %s)')
I would like to insert values into postgresql 11.0 table. However, when I am trying to do that I am getting following error:
TypeError: not all arguments converted during string formatting
I am running following code:
#CREATE TABLE
try:
connect_str = "dbname='xx' user='xx' host='xx' " "password='xx' port = xx"
conn = psycopg2.connect(connect_str)
except:
print("Unable to connect to the database")
cursor = conn.cursor()
cursor.execute("""DROP TABLE IF EXISTS tbl""")
try:
cursor.execute("""
CREATE TABLE IF NOT EXISTS tbl(
entry_id CHARACTER VARYING NOT NULL,
name CHARACTER VARYING NOT NULL,
class CHARACTER VARYING NOT NULL,
ko_id CHARACTER VARYING NOT NULL,
PRIMARY KEY (entry_id))
""")
except:
print("The table cannot be created!")
conn.commit()
conn.close()
cursor.close()
#INSERT DATA INTO TABLE
try:
connect_str = "dbname='xx' user='xx' host='xx' " "password='xx' port = xx"
conn = psycopg2.connect(connect_str)
except:
print("Unable to connect to the database")
cursor = conn.cursor()
with open ('file.txt') as f:
for line in f:
if re.match('^[A-Z]+',line) and line.startswith("ENTRY") or line.startswith("NAME") or line.startswith("CLASS") or line.startswith("KO_PATHWAY"):
key, value = line.split(" ", 1)
#print (key, value)
if key == "ENTRY":
cursor.execute("INSERT INTO tbl (entry_id) VALUES (%s)",('value'))
conn.commit()
conn.close()
cursor.close()
The key-value looks like this:
ENTRY map00010 Pathway
NAME Glycolysis / Gluconeogenesis
CLASS Metabolism; Carbohydrate metabolism
KO_PATHWAY ko00010
ENTRY map00011 Pathway
NAME Glycolysis
CLASS Metabolism; Carbohydrate
KO_PATHWAY ko00011
The value map00010 Pathway and map00011 Pathway should be inserted in the table and create two rows.
Any help is highly appreciated.
My question is between the huge banners in the code block below.
Forgive the code dump, this is all pasted here for anyone wanting to replicate, and this code does work as expected, although it's a bit strange. Notice the last two lines, they print proper SQL.
Goal:
I have tables with primary keys of type Text, specifically, emails. Instead of writing a new query function for each table, I took upon the task of generalizing the function, so that I could type-safely query any table that has emails.
Problem:
In order to get this to work, I had to include:
instance Default Constant CEmail (Column PGText) where
def = undefined
Which makes me think I'm doing something wrong. Any advice for building a query that can find records from any table that has Emails?
{- stack
--resolver lts-8.2
--install-ghc
exec ghci
--package aeson
--package composite-base
--package composite-aeson
--package text
--package string-conversions
--package postgres-simple
--package vinyl
-}
{-# LANGUAGE
Arrows
, DataKinds
, OverloadedStrings
, PatternSynonyms
, TypeOperators
, TemplateHaskell
, FlexibleContexts
, RankNTypes
, ConstraintKinds
, TypeSynonymInstances
, FlexibleInstances
, MultiParamTypeClasses
#-}
import Data.Vinyl (RElem)
import Data.Functor.Identity (Identity)
import Data.Vinyl.TypeLevel (RIndex)
import Composite.Aeson (JsonFormat, defaultJsonFormatRec, recJsonFormat, toJsonWithFormat)
import Composite.Opaleye (defaultRecTable)
import Composite.Record (Record, Rec(RNil), (:->), pattern (:*:))
import Composite.TH (withOpticsAndProxies)
import Control.Arrow (returnA)
import Control.Lens (view)
import Data.Int (Int64)
import Data.Proxy (Proxy(Proxy))
import Data.Text (Text)
import Opaleye
import Opaleye.Internal.TableMaker (ColumnMaker)
import Data.String.Conversions (cs)
import qualified Data.Aeson as Aeson
import qualified Database.PostgreSQL.Simple as PGS -- used for printSql
import Data.Profunctor.Product.Default (Default(def))
--------------------------------------------------
-- | Types
-- | Newtype ClearPassword so it can't be passed around as ordinary Text
newtype ClearPassword a = ClearPassword a
withOpticsAndProxies [d|
type FEmail = "email" :-> Text
type CEmail = "email" :-> Column PGText
type FAge = "age" :-> Text
type CAge = "age" :-> Column PGText
type FClearPassword = "clearpass" :-> ClearPassword Text
type CHashPassword = "hashpass" :-> Column PGText
|]
--------------------------------------------------
-- | Db Setup
-- | Helper Fn
printSql :: Default Unpackspec a a => Query a -> IO ()
printSql = putStrLn . maybe "Empty query" id . showSqlForPostgres
-- | Db Records
type DbUser = '[CEmail, CAge]
type DbPassword = '[CEmail, CHashPassword]
--------------------------------------------------
--------------------------------------------------
--
-- LOOK HERE vvvvvvvvvvvvvvvvvvvvvvvv
--
--------------------------------------------------
--------------------------------------------------
type RecWith f rs = (Default ColumnMaker (Record rs) (Record rs),
Default Constant f (Column PGText),
RElem f rs (RIndex f rs))
-- | queryByEmail needs this, but totally works if `def` is declared
-- as `undefined` ???
instance Default Constant CEmail (Column PGText) where
def = undefined
queryByEmail :: (RecWith CEmail rs) =>
Table a (Record rs) -> Text -> QueryArr () (Record rs)
queryByEmail table email = proc () -> do
u <- queryTable table -< ()
let uEmail = view cEmail u
restrict -< uEmail .=== constant email
returnA -< u
--------------------------------------------------
--------------------------------------------------
--
-- LOOK UP ^^^^^^^^^^^^^^^^^^^^^^^^
--
--------------------------------------------------
--------------------------------------------------
userTable :: Table (Record DbUser) (Record DbUser)
userTable = Table "user" defaultRecTable
-- | Password
passwordTable :: Table (Record DbPassword) (Record DbPassword)
passwordTable = Table "password" defaultRecTable
-- SELECT ... FROM "user" ...
queryUserTest = printSql $ queryByEmail userTable "hi"
-- SELECT ... FROM "password" ...
queryPasswordTest = printSql $ queryByEmail passwordTable "hi"
Drop the extraneous Default Constant f (Column PGTest) constraint and you should be good to go:
#!/usr/bin/env stack
{- stack --resolver lts-8.11 --install-ghc exec ghci --package aeson --package composite-base --package composite-aeson --package text --package string-conversions --package vinyl --package composite-opaleye -}
{-# LANGUAGE Arrows, DataKinds, OverloadedStrings, PatternSynonyms, TypeOperators, TemplateHaskell, FlexibleContexts, RankNTypes, ConstraintKinds, TypeSynonymInstances, FlexibleInstances, MultiParamTypeClasses #-}
import Composite.Opaleye (defaultRecTable)
import Composite.Record (Record, (:->))
import Composite.TH (withOpticsAndProxies)
import Control.Arrow (returnA)
import Control.Lens (view)
import Data.Profunctor.Product.Default (Default)
import Data.Text (Text)
import Data.Vinyl (RElem)
import Data.Vinyl.TypeLevel (RIndex)
import Opaleye.Internal.TableMaker (ColumnMaker)
import Opaleye
newtype ClearPassword a = ClearPassword a
withOpticsAndProxies [d|
type FEmail = "email" :-> Text
type CEmail = "email" :-> Column PGText
type FAge = "age" :-> Text
type CAge = "age" :-> Column PGText
type FClearPassword = "clearpass" :-> ClearPassword Text
type CHashPassword = "hashpass" :-> Column PGText
|]
type DbUser = '[CEmail, CAge]
type DbPassword = '[CEmail, CHashPassword]
printSql :: Default Unpackspec a a => Query a -> IO ()
printSql = putStrLn . maybe "Empty query" id . showSqlForPostgres
queryByEmail :: (RElem CEmail rs (RIndex CEmail rs), Default ColumnMaker (Record rs) (Record rs)) => Table a (Record rs) -> Text -> QueryArr () (Record rs)
queryByEmail table email = proc () -> do
u <- queryTable table -< ()
let uEmail = view cEmail u
restrict -< uEmail .=== constant email
returnA -< u
userTable :: Table (Record DbUser) (Record DbUser)
userTable = Table "user" defaultRecTable
passwordTable :: Table (Record DbPassword) (Record DbPassword)
passwordTable = Table "password" defaultRecTable
queryUserTest = printSql $ queryByEmail userTable "hi"
queryPasswordTest = printSql $ queryByEmail passwordTable "hi"
The constant email call uses the (already extant) Default Constant Text (Column PGText) constraint; were email to have type CEmail instead you would need a non-trivial non-undefined-using instance.
EDIT: I now have a better idea of what is going wrong. When I perform that query in plain old psql, I get the following result:
lwm#verbos
=# SELECT * FROM gerund LIMIT1;
infinitive │ gerund │ gerund_english
────────────┼─────────────┼────────────────
abandonar │ abandonando │ abandoning
So, I am getting back 3 strings? However, I say that I am getting back IO [Only String]. I am sure it is my type signature here that is messing things up ...
I am trying to make a simple query using the postgresql-simple library with Haskell. My code is pasted below along with the error I am seeing. Anyone got any ideas?
My database is called verbos and within it, I have a table called gerund. I am able to run a query_ that contains: conn "SELECT 2 + 2" and that works fine. I can also connect to my database with the default data as specified with the default information (password = 'postgres' : psql -h localhost -p 5432 -U postgres (from the docs[1])
{-# LANGUAGE OverloadedStrings #-}
module Main where
import Control.Monad
import Control.Applicative
import Database.PostgreSQL.Simple
main = do
conn <- connect defaultConnectInfo {
connectPassword = "postgres",
connectDatabase = "verbos"
}
mapM_ print =<< (query_ conn "SELECT * FROM gerund LIMIT 1" :: IO [Only String])
Gives me the following error:
ConversionFailed {errSQLType = "3 values: [(Basic {typoid = Oid 1043,
typcategory = 'S', typdelim = ',', typname = \"varchar\"},Just
\"abandonar\"),(Basic {typoid = Oid 1043, typcategory = 'S', typdelim
= ',', typname = \"varchar\"},Just \"abandonando\"),(Basic {typoid = Oid 1043, typcategory = 'S', typdelim = ',', typname =
\"varchar\"},Just \"abandoning\")]", errSQLTableOid = Nothing,
errSQLField = "", errHaskellType = "1 slots in target type",
errMessage = "mismatch between number of columns to convert and number
in target type"}
OK, Thanks to #AlpMestanogullari, #muistooshort, I got an answer here. My final code is:
{-# LANGUAGE OverloadedStrings #-}
module Main where
import Control.Applicative
import Database.PostgreSQL.Simple
import Database.PostgreSQL.Simple.FromRow
data Gerund = Gerund {
f :: String,
s :: String,
t :: String
} deriving (Show)
instance FromRow Gerund where
fromRow = Gerund <$> field <*> field <*> field
main = do
conn <- connect defaultConnectInfo {
connectPassword = "postgres",
connectDatabase = "verbos"
}
mapM_ print =<< (query_ conn q :: IO [Gerund])
where q = "SELECT * FROM gerund LIMIT 1"
Notes:
Knowing that my result contained 3 result columns, I needed to define a type that had 'space' for the results (f, s and t in the Gerund type)
I followed the docs[1] for FromRow closely to get my type and instance defined.
You need to import import Database.PostgreSQL.Simple.FromRow to access things like field.