Higher than expected latency in GO/Postgres Connection [duplicate] - postgresql

Question

I'm running the same query against a local postgresql instance using a golang application, and using psql. The timings differ greatly and I'm wondering why. Using explain/analyze the query took 1ms, using database/sql in golang, it took 24ms. I've added my code snippets below. I realize that explain/analyze may not be equivalent to querying the database directly, and there may be some network latency involved as well, however the discrepancy is still significant. Why is there such a discrepancy?
edit: I've tried the above with a sample size of 10+ queries, and the discrepancy still holds true.
postgres=# \timing
Timing is on.
postgres=# select 1;
?column?
----------
1
(1 row)
Time: 2.456 ms
postgres=# explain analyze select 1;
QUERY PLAN
------------------------------------------------------------------------------------
Result (cost=0.00..0.01 rows=1 width=4) (actual time=0.002..0.002 rows=1 loops=1)
Planning Time: 0.017 ms
Execution Time: 0.012 ms
(3 rows)
Time: 3.748 ms
package main
import (
"database/sql"
"fmt"
_ "github.com/lib/pq"
"time"
)
func main() {
// setup database connection
db, err := sql.Open("postgres", "host='localhost' port=5432 user='postgres' password='' dbname='postgres' sslmode=disable")
if err != nil {
panic(err)
}
// query database
firstQueryStart := time.Now()
_, err = db.Query("select 1;")
firstQueryEnd := time.Now()
if err != nil {
panic(err)
}
fmt.Println(fmt.Sprintf("first query took %s", firstQueryEnd.Sub(firstQueryStart).String()))
//run the same query a second time and measure the timing
secondQueryStart := time.Now()
_, err = db.Query("select 1;")
secondQueryEnd := time.Now()
if err != nil {
panic(err)
}
fmt.Println(fmt.Sprintf("second query took %s", secondQueryEnd.Sub(secondQueryStart).String()))
}
first query took 13.981435ms
second query took 13.343845ms

Answer 1

Note #1: sql.DB does not represent a connection, instead it represents a pool of connections.
Note #2: sql.Open initializes the pool but it does not have to actually open a connection, it's allowed to only validate the dsn input and then the opening of the connections will be handled lazily by the pool.
The reason your 1st db.Query is slow is because you're starting off with a fresh connection pool, one that has 0 idle (but open) connections, and therefore the 1st db.Query will need to first establish a new connection to the server and only after that will it be able to execute the sql statement.
The reason your 2nd db.Query is also slow is because the connection created by the 1st db.Query has not been released back to the pool, and therefore your 2nd db.Query will also need to first establish a new connection to the server before it can execute the sql statement.
To release a connection back to the pool you need to first retain the primary return value of db.Query and then invoke the Close method on it.
To start off with a pool that has at least one available connection, call Ping right after initializing the pool.
Example:
func main() {
// setup database connection
db, err := sql.Open("postgres", "postgres:///?sslmode=disable")
if err != nil {
panic(err)
} else if err := db.Ping(); err != nil {
panic(err)
}
for i := 0; i < 5; i++ {
// query database
firstQueryStart := time.Now()
rows, err := db.Query("select 1;")
firstQueryEnd := time.Now()
if err != nil {
panic(err)
}
// put the connection back to the pool so
// that it can be reused by next iteration
rows.Close()
fmt.Println(fmt.Sprintf("query #%d took %s", i, firstQueryEnd.Sub(firstQueryStart).String()))
}
}
Times on my machine (without db.Ping only #0 is slow)
query #0 took 6.312676ms
query #1 took 102.88µs
query #2 took 66.702µs
query #3 took 64.694µs
query #4 took 208.016µs
Times on my machine (with db.Ping #0 is a lot faster than without)
query #0 took 284.846µs
query #1 took 78.349µs
query #2 took 76.518µs
query #3 took 81.733µs
query #4 took 103.862µs
A note on prepared statements:
If you're executing a simple query with no arguments e.g. db.Query("select 1 where true") then you really are executing just a simple query.
If, however, you're executing a query with arguments e.g. db.Query("select 1 where $1", true) then, in actuality, you are creating and executing a prepared statement.
See 4.2. Value Expressions, it says:
A value expression is one of the following: ...
A positional parameter reference, in the body of a function definition or prepared statement
...
Also Positional Parameters says:
A positional parameter reference is used to indicate a value that is
supplied externally to an SQL statement. Parameters are used in SQL
function definitions and in prepared queries. Some client libraries
also support specifying data values separately from the SQL command
string, in which case parameters are used to refer to the out-of-line
data values.
How the postgres' message-flow protocol specifies simple queries and extended queries
The extended query protocol breaks down the above-described simple
query protocol into multiple steps. The results of preparatory steps
can be re-used multiple times for improved efficiency. Furthermore,
additional features are available, such as the possibility of
supplying data values as separate parameters instead of having to
insert them directly into a query string.
And finally, under the covers of the lib/pq driver:
...
// Check to see if we can use the "simpleQuery" interface, which is
// *much* faster than going through prepare/exec
if len(args) == 0 {
return cn.simpleQuery(query)
}
if cn.binaryParameters {
cn.sendBinaryModeQuery(query, args)
cn.readParseResponse()
cn.readBindResponse()
rows := &rows{cn: cn}
rows.rowsHeader = cn.readPortalDescribeResponse()
cn.postExecuteWorkaround()
return rows, nil
}
st := cn.prepareTo(query, "")
st.exec(args)
return &rows{
cn: cn,
rowsHeader: st.rowsHeader,
}, nil
...