postgresql-simple query error - postgresql

EDIT: I now have a better idea of what is going wrong. When I perform that query in plain old psql, I get the following result:
lwm#verbos
=# SELECT * FROM gerund LIMIT1;
infinitive │ gerund │ gerund_english
────────────┼─────────────┼────────────────
abandonar │ abandonando │ abandoning
So, I am getting back 3 strings? However, I say that I am getting back IO [Only String]. I am sure it is my type signature here that is messing things up ...
I am trying to make a simple query using the postgresql-simple library with Haskell. My code is pasted below along with the error I am seeing. Anyone got any ideas?
My database is called verbos and within it, I have a table called gerund. I am able to run a query_ that contains: conn "SELECT 2 + 2" and that works fine. I can also connect to my database with the default data as specified with the default information (password = 'postgres' : psql -h localhost -p 5432 -U postgres (from the docs[1])
{-# LANGUAGE OverloadedStrings #-}
module Main where
import Control.Monad
import Control.Applicative
import Database.PostgreSQL.Simple
main = do
conn <- connect defaultConnectInfo {
connectPassword = "postgres",
connectDatabase = "verbos"
}
mapM_ print =<< (query_ conn "SELECT * FROM gerund LIMIT 1" :: IO [Only String])
Gives me the following error:
ConversionFailed {errSQLType = "3 values: [(Basic {typoid = Oid 1043,
typcategory = 'S', typdelim = ',', typname = \"varchar\"},Just
\"abandonar\"),(Basic {typoid = Oid 1043, typcategory = 'S', typdelim
= ',', typname = \"varchar\"},Just \"abandonando\"),(Basic {typoid = Oid 1043, typcategory = 'S', typdelim = ',', typname =
\"varchar\"},Just \"abandoning\")]", errSQLTableOid = Nothing,
errSQLField = "", errHaskellType = "1 slots in target type",
errMessage = "mismatch between number of columns to convert and number
in target type"}

OK, Thanks to #AlpMestanogullari, #muistooshort, I got an answer here. My final code is:
{-# LANGUAGE OverloadedStrings #-}
module Main where
import Control.Applicative
import Database.PostgreSQL.Simple
import Database.PostgreSQL.Simple.FromRow
data Gerund = Gerund {
f :: String,
s :: String,
t :: String
} deriving (Show)
instance FromRow Gerund where
fromRow = Gerund <$> field <*> field <*> field
main = do
conn <- connect defaultConnectInfo {
connectPassword = "postgres",
connectDatabase = "verbos"
}
mapM_ print =<< (query_ conn q :: IO [Gerund])
where q = "SELECT * FROM gerund LIMIT 1"
Notes:
Knowing that my result contained 3 result columns, I needed to define a type that had 'space' for the results (f, s and t in the Gerund type)
I followed the docs[1] for FromRow closely to get my type and instance defined.
You need to import import Database.PostgreSQL.Simple.FromRow to access things like field.

Related

Passing list of values to SELECT PostgreSQL query in Haskell

I'm studying PostgreSQL with Haskell with this lib: https://hackage.haskell.org/package/postgresql-simple-0.4.10.0/docs/Database-PostgreSQL-Simple.html#v:query
While I could select an user like this:
(query_ conn "SELECT * FROM users WHERE NAME == john" :: IO [Only Int]) >>= mapM_ print
using query_:
query_ :: FromRow r => Connection -> Query -> IO [r]
I think I should use query:
query :: (ToRow q, FromRow r) => Connection -> Query -> q -> IO [r]
to pass a list of values. However, how do I pass this list?
For example, for INSERT, I was able to do this:
(execute conn "INSERT INTO users (NAME, PASSWORD) VALUES (?,?)") (["john", "123456"]::[String]) >>= print
but what is the equivalent for SELECT?
I'm not sure I understand your question, since you ask about lists and I don't see how they enter into the picture. But the parameterized version of your select query is this:
query conn "SELECT * FROM users where NAME == ?" (Only ("john" :: String))

How to insert bytea value using postgresql-simple in Haskell

I have a table defined as
CREATE TABLE users (id SERIAL PRIMARY KEY, val BYTEA);
Then I want to serialize my data structure with binary and store in the table, and then retrieve and deserialize back.
{-# LANGUAGE OverloadedStrings, DeriveAnyClass #-}
import Control.Monad (forM_)
import Data.Binary (encode, decode, Binary)
import Database.PostgreSQL.Simple
import GHC.Generics (Generic)
data User = { name :: Text, email :: Text } deriving (Show, Generic, Binary)
main = do
conn <- connect --...
let encoded = encode User {name = "me", email = "me#home.net" }
execute conn "INSERT INTO users(val) values(?)" $ Only encoded
rs <- query_ conn "SELECT id, val FROM users"
forM_ rs $ \(id,val) ->
putStrLn $ (show (id :: Int)) ++ ": " ++ show (decode val :: User)
But I get error Data.Binary.Get.runGet at position 0: not enough bytes.
Query
SELECT * FROM users;
gives
id | val
----+-----
1 | \x
I can't figure out how to map ByteStrings to 'BYTEA`s. According to the docs everything should be OK. What am I doing wrong?
Fixed by replacing the line
execute conn "INSERT INTO users(val) values(?)" $ Only encoded
with
execute conn "INSERT INTO users(val) values(?)" $ Only $ Binary encoded
It's because toField(ByteString) yields Escape whereas toField(Binary ByteString) yields EscapeByteA

How to generalize an Opaleye Query in Haskell (Using Vinyl)?

My question is between the huge banners in the code block below.
Forgive the code dump, this is all pasted here for anyone wanting to replicate, and this code does work as expected, although it's a bit strange. Notice the last two lines, they print proper SQL.
Goal:
I have tables with primary keys of type Text, specifically, emails. Instead of writing a new query function for each table, I took upon the task of generalizing the function, so that I could type-safely query any table that has emails.
Problem:
In order to get this to work, I had to include:
instance Default Constant CEmail (Column PGText) where
def = undefined
Which makes me think I'm doing something wrong. Any advice for building a query that can find records from any table that has Emails?
{- stack
--resolver lts-8.2
--install-ghc
exec ghci
--package aeson
--package composite-base
--package composite-aeson
--package text
--package string-conversions
--package postgres-simple
--package vinyl
-}
{-# LANGUAGE
Arrows
, DataKinds
, OverloadedStrings
, PatternSynonyms
, TypeOperators
, TemplateHaskell
, FlexibleContexts
, RankNTypes
, ConstraintKinds
, TypeSynonymInstances
, FlexibleInstances
, MultiParamTypeClasses
#-}
import Data.Vinyl (RElem)
import Data.Functor.Identity (Identity)
import Data.Vinyl.TypeLevel (RIndex)
import Composite.Aeson (JsonFormat, defaultJsonFormatRec, recJsonFormat, toJsonWithFormat)
import Composite.Opaleye (defaultRecTable)
import Composite.Record (Record, Rec(RNil), (:->), pattern (:*:))
import Composite.TH (withOpticsAndProxies)
import Control.Arrow (returnA)
import Control.Lens (view)
import Data.Int (Int64)
import Data.Proxy (Proxy(Proxy))
import Data.Text (Text)
import Opaleye
import Opaleye.Internal.TableMaker (ColumnMaker)
import Data.String.Conversions (cs)
import qualified Data.Aeson as Aeson
import qualified Database.PostgreSQL.Simple as PGS -- used for printSql
import Data.Profunctor.Product.Default (Default(def))
--------------------------------------------------
-- | Types
-- | Newtype ClearPassword so it can't be passed around as ordinary Text
newtype ClearPassword a = ClearPassword a
withOpticsAndProxies [d|
type FEmail = "email" :-> Text
type CEmail = "email" :-> Column PGText
type FAge = "age" :-> Text
type CAge = "age" :-> Column PGText
type FClearPassword = "clearpass" :-> ClearPassword Text
type CHashPassword = "hashpass" :-> Column PGText
|]
--------------------------------------------------
-- | Db Setup
-- | Helper Fn
printSql :: Default Unpackspec a a => Query a -> IO ()
printSql = putStrLn . maybe "Empty query" id . showSqlForPostgres
-- | Db Records
type DbUser = '[CEmail, CAge]
type DbPassword = '[CEmail, CHashPassword]
--------------------------------------------------
--------------------------------------------------
--
-- LOOK HERE vvvvvvvvvvvvvvvvvvvvvvvv
--
--------------------------------------------------
--------------------------------------------------
type RecWith f rs = (Default ColumnMaker (Record rs) (Record rs),
Default Constant f (Column PGText),
RElem f rs (RIndex f rs))
-- | queryByEmail needs this, but totally works if `def` is declared
-- as `undefined` ???
instance Default Constant CEmail (Column PGText) where
def = undefined
queryByEmail :: (RecWith CEmail rs) =>
Table a (Record rs) -> Text -> QueryArr () (Record rs)
queryByEmail table email = proc () -> do
u <- queryTable table -< ()
let uEmail = view cEmail u
restrict -< uEmail .=== constant email
returnA -< u
--------------------------------------------------
--------------------------------------------------
--
-- LOOK UP ^^^^^^^^^^^^^^^^^^^^^^^^
--
--------------------------------------------------
--------------------------------------------------
userTable :: Table (Record DbUser) (Record DbUser)
userTable = Table "user" defaultRecTable
-- | Password
passwordTable :: Table (Record DbPassword) (Record DbPassword)
passwordTable = Table "password" defaultRecTable
-- SELECT ... FROM "user" ...
queryUserTest = printSql $ queryByEmail userTable "hi"
-- SELECT ... FROM "password" ...
queryPasswordTest = printSql $ queryByEmail passwordTable "hi"
Drop the extraneous Default Constant f (Column PGTest) constraint and you should be good to go:
#!/usr/bin/env stack
{- stack --resolver lts-8.11 --install-ghc exec ghci --package aeson --package composite-base --package composite-aeson --package text --package string-conversions --package vinyl --package composite-opaleye -}
{-# LANGUAGE Arrows, DataKinds, OverloadedStrings, PatternSynonyms, TypeOperators, TemplateHaskell, FlexibleContexts, RankNTypes, ConstraintKinds, TypeSynonymInstances, FlexibleInstances, MultiParamTypeClasses #-}
import Composite.Opaleye (defaultRecTable)
import Composite.Record (Record, (:->))
import Composite.TH (withOpticsAndProxies)
import Control.Arrow (returnA)
import Control.Lens (view)
import Data.Profunctor.Product.Default (Default)
import Data.Text (Text)
import Data.Vinyl (RElem)
import Data.Vinyl.TypeLevel (RIndex)
import Opaleye.Internal.TableMaker (ColumnMaker)
import Opaleye
newtype ClearPassword a = ClearPassword a
withOpticsAndProxies [d|
type FEmail = "email" :-> Text
type CEmail = "email" :-> Column PGText
type FAge = "age" :-> Text
type CAge = "age" :-> Column PGText
type FClearPassword = "clearpass" :-> ClearPassword Text
type CHashPassword = "hashpass" :-> Column PGText
|]
type DbUser = '[CEmail, CAge]
type DbPassword = '[CEmail, CHashPassword]
printSql :: Default Unpackspec a a => Query a -> IO ()
printSql = putStrLn . maybe "Empty query" id . showSqlForPostgres
queryByEmail :: (RElem CEmail rs (RIndex CEmail rs), Default ColumnMaker (Record rs) (Record rs)) => Table a (Record rs) -> Text -> QueryArr () (Record rs)
queryByEmail table email = proc () -> do
u <- queryTable table -< ()
let uEmail = view cEmail u
restrict -< uEmail .=== constant email
returnA -< u
userTable :: Table (Record DbUser) (Record DbUser)
userTable = Table "user" defaultRecTable
passwordTable :: Table (Record DbPassword) (Record DbPassword)
passwordTable = Table "password" defaultRecTable
queryUserTest = printSql $ queryByEmail userTable "hi"
queryPasswordTest = printSql $ queryByEmail passwordTable "hi"
The constant email call uses the (already extant) Default Constant Text (Column PGText) constraint; were email to have type CEmail instead you would need a non-trivial non-undefined-using instance.

haskell postgresql-simple incompatible type _int8 and Int64 (and Integer)

The erroneous function below is part of a program called subdivide working with Postgis geospatial intersections on the server side and processing the returned array of Int64 on the client side.
It is built and run under Stack, resolving to Nightly 2016-08-02 and explicitly specifying architecture x86_64.
I get the following runtime error executing the Postgres query defined as "intersectionsSql" (see the comment RUNTIME ERROR HERE):
"Created table: server : [Only {fromOnly = \"PostgreSQL 9.6beta2 on x86_64-pc-linux-gnu, compiled by gcc (Debian 4.9.2-10) 4.9.2, 64-bit\"}] quadrant: BOX3D(-180.0 90.0, -90.0 45.0)"
subdivide: Incompatible {errSQLType = "_int8", errSQLTableOid = Nothing, errSQLField = "object_ids", errHaskellType = "Int64", errMessage = "types incompatible"}
I have tried Integer, Int64 and Int, all with the same result, which is counter-intuitive as those Haskell types should all be compatible with _int8 according to the PostgreSQL-simple instance documentation:
https://hackage.haskell.org/package/postgresql-simple-0.5.0.0/candidate/docs/Database-PostgreSQL-Simple-FromField.html
The SQL query should return a single row of postgres bigint[], which I have confirmed via PGAdmin.
Any ideas?
Also any comments around how I have written the code - its over a decade since last I worked with GHC and times have changed.
Thanks for your consideration.
Mike Thomas
accumulateIntersections :: Identifier -> Identifier -> ConnectInfo -> ((Double,Double),(Double,Double)) -> IO ()
accumulateIntersections sourceTable accumulationTable connectionInfo q =
let
theBox = makeBox3D (fst (fst q)) (snd (fst q)) (fst (snd q)) (snd (snd q))
theValue = (Only theBox)
dropTable = [sql| DROP TABLE IF EXISTS ? CASCADE |]
createTable = [sql| CREATE TABLE ? ( quadrant_id BIGSERIAL, area_metres_squared FLOAT8, shape GEOMETRY, object_ids BIGINT[] ) |]
aggregateSql = [sql| DROP AGGREGATE IF EXISTS _array_agg (anyarray);
CREATE AGGREGATE _array_agg(anyarray) (SFUNC = array_cat, STYPE = anyarray);
|]
intersectionsSql = [sql| SELECT _array_agg (object_ids) object_ids
FROM ?
WHERE ST_Intersects(ST_SetSRID ( ?::box3d, 4326 ), shape)
|]
insertIntersections = [sql| INSERT INTO ? (shape, object_ids)
VALUES ( ST_SetSRID ( ?::box3d, 4326 )
, ? ) |]
in
do
connection <- connect connectionInfo
execute_ connection aggregateSql
postgresVersion <- (query_ connection "SELECT version()" :: IO [Only String])
i0 <- execute connection dropTable (Only accumulationTable)
i1 <- execute connection createTable (Only accumulationTable)
print ("Created table: server : " ++ (show postgresVersion) ++ " quadrant: " ++ theBox)
is :: [Only Int64] <- query connection intersectionsSql (sourceTable, theBox) -- RUNTIME ERROR HERE
print ("Intersections done.")
ids::[Int64] <- forM is (\(Only id) -> return id)
print ("Ids done.")
close connection
return ()
See the above comment relayed from LP Smith, who I contacted when no answers were forthcoming here. It resolves my issue.
The key was to recognize that _int8 represents an array of 8 byte integers, rather than thinking, as I had done, that it was an internal representation for a single 8 byte integer. Leon's suggested change was to substitute "[Only (Vector Int64)]" for "[Only Int64]" in the line marked above as the point of the runtime error.
Thank you Leon.

passing python variable to sql statement psycopg2 pandas

I am trying to replace a piece of sql code with a python variable that I will ask a user to generate using a raw_input.
Below is the code i'm using which works great if I set mypythonvariable manually i.e. inputting 344 into the sql code, but if I set the sql as is to mypythonvariable it doesn't work.
The whole sql query is then converted into a pandas dataframe for further messing about with.
Any help on how to do be appreciated.
UPDATE: I just added the %s code into the statement and i'm now getting the error message '': not all arguments converted during string formatting
'
conn = pg.connect(host = "localhost",
port = 1234,
dbname = "somename",
user = "user",
password = "pswd")
mypythonvariable = raw_input("What is your variable number? ")
sql = """
SELECT
somestuff
FROM
sometable
WHERE
something = %s
"""
df = pd.read_sql_query(sql, con=conn,params=mypythonvariable)
thanks to all that looked.
I found the solution.Looks like the params need to be passed as a list.
conn = pg.connect(host = "localhost",
port = 1234,
dbname = "somename",
user = "user",
password = "pswd")
mypythonvariable = raw_input("What is your variable number? ")
sql = """
SELECT
somestuff
FROM
sometable
WHERE
something = %s
"""
df = pd.read_sql_query(sql, con=conn,params=[mypythonvariable])