Using postgres 9.3, go 1.6
I've been trying to use transactions with the go pq library.
// Good
txn, _ := db.Begin()
txn.Query("UPDATE t_name SET a = 1")
err := txn.Commit() // err is nil
// Bad
txn, _ := db.Begin()
txn.Query("UPDATE t_name SET a = $1", 1)
err := txn.Commit() // Gives me a "unexpected command tag Q" error
// although the data is committed
For some reason, when I execute a Query with parameters, I always get an unexpected command tag Q error from the Commit(). What is this error (what is Q?) and why am I getting it?
I believe this is where the error is created.
To start of i agree whit Dmitri from the comments, in this case you should probably use Exec.
However after receiving this same issue I started digging:
Query returns 2 arguments a Rows pointer and an error. What you always have to do with a Rows object is to close it when you are don with it:
// Fixed
txn, _ := db.Begin()
rows, _ := txn.Query("UPDATE t_name SET a = $1", 1)
//Read out rows
rows.Close() //<- This will solve the error
err := txn.Commit()
I was however unable to see any difference in the traffic to the database when using rows.Close() witch indicates to me that this might be a bug in pq.
Related
I am getting this error trying to scan in postgres rows.
can't scan into dest[0]: unknown oid 1003 cannot be scanned into string
This is the code I that throws that error.
sourceKeys := make([]string, 0, 1)
targetSchema := ""
targetTable := ""
targetKeys := make([]string, 0, 1)
fks := make([]ForeignKey, 0)
for rows.Next() {
err := rows.Scan(
sourceKeys,
&targetSchema,
&targetTable,
targetKeys,
)
}
…….
}
When I look at postgres documentation (https://pkg.go.dev/github.com/lib/pq/oid) I can see that oid=1003 is for T__name types. How can I read in a name without using the pq library?
I'm running the same query against a local postgresql instance using a golang application, and using psql. The timings differ greatly and I'm wondering why. Using explain/analyze the query took 1ms, using database/sql in golang, it took 24ms. I've added my code snippets below. I realize that explain/analyze may not be equivalent to querying the database directly, and there may be some network latency involved as well, however the discrepancy is still significant. Why is there such a discrepancy?
edit: I've tried the above with a sample size of 10+ queries, and the discrepancy still holds true.
postgres=# \timing
Timing is on.
postgres=# select 1;
?column?
----------
1
(1 row)
Time: 2.456 ms
postgres=# explain analyze select 1;
QUERY PLAN
------------------------------------------------------------------------------------
Result (cost=0.00..0.01 rows=1 width=4) (actual time=0.002..0.002 rows=1 loops=1)
Planning Time: 0.017 ms
Execution Time: 0.012 ms
(3 rows)
Time: 3.748 ms
package main
import (
"database/sql"
"fmt"
_ "github.com/lib/pq"
"time"
)
func main() {
// setup database connection
db, err := sql.Open("postgres", "host='localhost' port=5432 user='postgres' password='' dbname='postgres' sslmode=disable")
if err != nil {
panic(err)
}
// query database
firstQueryStart := time.Now()
_, err = db.Query("select 1;")
firstQueryEnd := time.Now()
if err != nil {
panic(err)
}
fmt.Println(fmt.Sprintf("first query took %s", firstQueryEnd.Sub(firstQueryStart).String()))
//run the same query a second time and measure the timing
secondQueryStart := time.Now()
_, err = db.Query("select 1;")
secondQueryEnd := time.Now()
if err != nil {
panic(err)
}
fmt.Println(fmt.Sprintf("second query took %s", secondQueryEnd.Sub(secondQueryStart).String()))
}
first query took 13.981435ms
second query took 13.343845ms
Note #1: sql.DB does not represent a connection, instead it represents a pool of connections.
Note #2: sql.Open initializes the pool but it does not have to actually open a connection, it's allowed to only validate the dsn input and then the opening of the connections will be handled lazily by the pool.
The reason your 1st db.Query is slow is because you're starting off with a fresh connection pool, one that has 0 idle (but open) connections, and therefore the 1st db.Query will need to first establish a new connection to the server and only after that will it be able to execute the sql statement.
The reason your 2nd db.Query is also slow is because the connection created by the 1st db.Query has not been released back to the pool, and therefore your 2nd db.Query will also need to first establish a new connection to the server before it can execute the sql statement.
To release a connection back to the pool you need to first retain the primary return value of db.Query and then invoke the Close method on it.
To start off with a pool that has at least one available connection, call Ping right after initializing the pool.
Example:
func main() {
// setup database connection
db, err := sql.Open("postgres", "postgres:///?sslmode=disable")
if err != nil {
panic(err)
} else if err := db.Ping(); err != nil {
panic(err)
}
for i := 0; i < 5; i++ {
// query database
firstQueryStart := time.Now()
rows, err := db.Query("select 1;")
firstQueryEnd := time.Now()
if err != nil {
panic(err)
}
// put the connection back to the pool so
// that it can be reused by next iteration
rows.Close()
fmt.Println(fmt.Sprintf("query #%d took %s", i, firstQueryEnd.Sub(firstQueryStart).String()))
}
}
Times on my machine (without db.Ping only #0 is slow)
query #0 took 6.312676ms
query #1 took 102.88µs
query #2 took 66.702µs
query #3 took 64.694µs
query #4 took 208.016µs
Times on my machine (with db.Ping #0 is a lot faster than without)
query #0 took 284.846µs
query #1 took 78.349µs
query #2 took 76.518µs
query #3 took 81.733µs
query #4 took 103.862µs
A note on prepared statements:
If you're executing a simple query with no arguments e.g. db.Query("select 1 where true") then you really are executing just a simple query.
If, however, you're executing a query with arguments e.g. db.Query("select 1 where $1", true) then, in actuality, you are creating and executing a prepared statement.
See 4.2. Value Expressions, it says:
A value expression is one of the following: ...
A positional parameter reference, in the body of a function definition or prepared statement
...
Also Positional Parameters says:
A positional parameter reference is used to indicate a value that is
supplied externally to an SQL statement. Parameters are used in SQL
function definitions and in prepared queries. Some client libraries
also support specifying data values separately from the SQL command
string, in which case parameters are used to refer to the out-of-line
data values.
How the postgres' message-flow protocol specifies simple queries and extended queries
The extended query protocol breaks down the above-described simple
query protocol into multiple steps. The results of preparatory steps
can be re-used multiple times for improved efficiency. Furthermore,
additional features are available, such as the possibility of
supplying data values as separate parameters instead of having to
insert them directly into a query string.
And finally, under the covers of the lib/pq driver:
...
// Check to see if we can use the "simpleQuery" interface, which is
// *much* faster than going through prepare/exec
if len(args) == 0 {
return cn.simpleQuery(query)
}
if cn.binaryParameters {
cn.sendBinaryModeQuery(query, args)
cn.readParseResponse()
cn.readBindResponse()
rows := &rows{cn: cn}
rows.rowsHeader = cn.readPortalDescribeResponse()
cn.postExecuteWorkaround()
return rows, nil
}
st := cn.prepareTo(query, "")
st.exec(args)
return &rows{
cn: cn,
rowsHeader: st.rowsHeader,
}, nil
...
I am trying to execute the following query against the PostgreSQL database in Go using pq driver:
SELECT COUNT(id)
FROM tags
WHERE id IN (1, 2, 3)
where 1, 2, 3 is passed at a slice tags := []string{"1", "2", "3"}.
I have tried many different things like:
s := "(" + strings.Join(tags, ",") + ")"
if err := Db.QueryRow(`
SELECT COUNT(id)
FROM tags
WHERE id IN $1`, s,
).Scan(&num); err != nil {
log.Println(err)
}
which results in pq: syntax error at or near "$1". I also tried
if err := Db.QueryRow(`
SELECT COUNT(id)
FROM tags
WHERE id IN ($1)`, strings.Join(stringTagIds, ","),
).Scan(&num); err != nil {
log.Println(err)
}
which also fails with pq: invalid input syntax for integer: "1,2,3"
I also tried passing a slice of integers/strings directly and got sql: converting Exec argument #0's type: unsupported type []string, a slice.
So how can I execute this query in Go?
Pre-building the SQL query (preventing SQL injection)
If you're generating an SQL string with a param placeholder for each of the values, it's easier to just generate the final SQL right away.
Note that since values are strings, there's place for SQL injection attack, so we first test if all the string values are indeed numbers, and we only proceed if so:
tags := []string{"1", "2", "3"}
buf := bytes.NewBufferString("SELECT COUNT(id) FROM tags WHERE id IN(")
for i, v := range tags {
if i > 0 {
buf.WriteString(",")
}
if _, err := strconv.Atoi(v); err != nil {
panic("Not number!")
}
buf.WriteString(v)
}
buf.WriteString(")")
Executing it:
num := 0
if err := Db.QueryRow(buf.String()).Scan(&num); err != nil {
log.Println(err)
}
Using ANY
You can also use Postgresql's ANY, whose syntax is as follows:
expression operator ANY (array expression)
Using that, our query may look like this:
SELECT COUNT(id) FROM tags WHERE id = ANY('{1,2,3}'::int[])
In this case you can declare the text form of the array as a parameter:
SELECT COUNT(id) FROM tags WHERE id = ANY($1::int[])
Which can simply be built like this:
tags := []string{"1", "2", "3"}
param := "{" + strings.Join(tags, ",") + "}"
Note that no check is required in this case as the array expression will not allow SQL injection (but rather will result in a query execution error).
So the full code:
tags := []string{"1", "2", "3"}
q := "SELECT COUNT(id) FROM tags WHERE id = ANY($1::int[])"
param := "{" + strings.Join(tags, ",") + "}"
num := 0
if err := Db.QueryRow(q, param).Scan(&num); err != nil {
log.Println(err)
}
This is not really a Golang issue, you are using a string to compare to integer (id) in your SQL request. That means, SQL receive:
SELECT COUNT(id)
FROM tags
WHERE id IN ("1, 2, 3")
instead of what you want to give it. You just need to convert your tags into integer and passe it to the query.
EDIT:
Since you are trying to pass multiple value to the query, then you should tell it:
params := make([]string, 0, len(tags))
for i := range tags {
params = append(params, fmt.Sprintf("$%d", i+1))
}
query := fmt.Sprintf("SELECT COUNT(id) FROM tags WHERE id IN (%s)", strings.Join(params, ", "))
This will end the query with a "($1, $2, $3...", then convert your tags as int:
values := make([]int, 0, len(tags))
for _, s := range tags {
val, err := strconv.Atoi(s)
if err != nil {
// Do whatever is required with the error
fmt.Println("Err : ", err)
} else {
values = append(values, val)
}
}
And finally, you can use it in the query:
Db.QueryRow(query, values...)
This should do it.
Extending #icza solution, you can use pq.Array instead of building the params yourself.
So using his example, the code can look like this:
tags := []string{"1", "2", "3"}
q := "SELECT COUNT(id) FROM tags WHERE id = ANY($1::int[])"
num := 0
if err := Db.QueryRow(q, pq.Array(tags)).Scan(&num); err != nil {
log.Println(err)
}
I'm using https://godoc.org/github.com/go-pg/pg a bunch of other places in the code so I'm hoping I don't have to switch to another client.
I can't get the ORM to write this query (below) correctly, so I just want to pass it thru as a custom string. But I can't figure out how to get the results into my []string slice.
tokens := []string{}
qry := `SELECT p.token
FROM pntokens p
join
(VALUES ('123'), ('456'), ('789')) AS t (userid)
on p.userid = t.userid ;`
I've tried:
err := db.Model(&Pntoken{}, qry).Select(&tokens)
err := db.Query([]string{}, qry, nil).Select(&tokens)
_, err := db.Exec(qry)
res, err := db.Model((*Pntoken)(nil)).Exec(qry)
But cannot get the tool out of my way enough to just get some simple results into my slice.
All tips appreciated!
Use like this for lib https://github.com/go-pg/pg:
tokens := []string
qry := `SELECT p.token FROM pntokens p
join (VALUES ('123'), ('456'), ('789')) AS t (userid)
on p.userid = t.userid ;`
_, err := pc.DB.Query(&tokens,"qry",nil)
if err != nil {
return tokens, err
}
I am trying to execute the following query against the PostgreSQL database in Go using pq driver:
SELECT COUNT(id)
FROM tags
WHERE id IN (1, 2, 3)
where 1, 2, 3 is passed at a slice tags := []string{"1", "2", "3"}.
I have tried many different things like:
s := "(" + strings.Join(tags, ",") + ")"
if err := Db.QueryRow(`
SELECT COUNT(id)
FROM tags
WHERE id IN $1`, s,
).Scan(&num); err != nil {
log.Println(err)
}
which results in pq: syntax error at or near "$1". I also tried
if err := Db.QueryRow(`
SELECT COUNT(id)
FROM tags
WHERE id IN ($1)`, strings.Join(stringTagIds, ","),
).Scan(&num); err != nil {
log.Println(err)
}
which also fails with pq: invalid input syntax for integer: "1,2,3"
I also tried passing a slice of integers/strings directly and got sql: converting Exec argument #0's type: unsupported type []string, a slice.
So how can I execute this query in Go?
Pre-building the SQL query (preventing SQL injection)
If you're generating an SQL string with a param placeholder for each of the values, it's easier to just generate the final SQL right away.
Note that since values are strings, there's place for SQL injection attack, so we first test if all the string values are indeed numbers, and we only proceed if so:
tags := []string{"1", "2", "3"}
buf := bytes.NewBufferString("SELECT COUNT(id) FROM tags WHERE id IN(")
for i, v := range tags {
if i > 0 {
buf.WriteString(",")
}
if _, err := strconv.Atoi(v); err != nil {
panic("Not number!")
}
buf.WriteString(v)
}
buf.WriteString(")")
Executing it:
num := 0
if err := Db.QueryRow(buf.String()).Scan(&num); err != nil {
log.Println(err)
}
Using ANY
You can also use Postgresql's ANY, whose syntax is as follows:
expression operator ANY (array expression)
Using that, our query may look like this:
SELECT COUNT(id) FROM tags WHERE id = ANY('{1,2,3}'::int[])
In this case you can declare the text form of the array as a parameter:
SELECT COUNT(id) FROM tags WHERE id = ANY($1::int[])
Which can simply be built like this:
tags := []string{"1", "2", "3"}
param := "{" + strings.Join(tags, ",") + "}"
Note that no check is required in this case as the array expression will not allow SQL injection (but rather will result in a query execution error).
So the full code:
tags := []string{"1", "2", "3"}
q := "SELECT COUNT(id) FROM tags WHERE id = ANY($1::int[])"
param := "{" + strings.Join(tags, ",") + "}"
num := 0
if err := Db.QueryRow(q, param).Scan(&num); err != nil {
log.Println(err)
}
This is not really a Golang issue, you are using a string to compare to integer (id) in your SQL request. That means, SQL receive:
SELECT COUNT(id)
FROM tags
WHERE id IN ("1, 2, 3")
instead of what you want to give it. You just need to convert your tags into integer and passe it to the query.
EDIT:
Since you are trying to pass multiple value to the query, then you should tell it:
params := make([]string, 0, len(tags))
for i := range tags {
params = append(params, fmt.Sprintf("$%d", i+1))
}
query := fmt.Sprintf("SELECT COUNT(id) FROM tags WHERE id IN (%s)", strings.Join(params, ", "))
This will end the query with a "($1, $2, $3...", then convert your tags as int:
values := make([]int, 0, len(tags))
for _, s := range tags {
val, err := strconv.Atoi(s)
if err != nil {
// Do whatever is required with the error
fmt.Println("Err : ", err)
} else {
values = append(values, val)
}
}
And finally, you can use it in the query:
Db.QueryRow(query, values...)
This should do it.
Extending #icza solution, you can use pq.Array instead of building the params yourself.
So using his example, the code can look like this:
tags := []string{"1", "2", "3"}
q := "SELECT COUNT(id) FROM tags WHERE id = ANY($1::int[])"
num := 0
if err := Db.QueryRow(q, pq.Array(tags)).Scan(&num); err != nil {
log.Println(err)
}