Is it possible to insert multiple rows into Postgres database at once? Could someone please suggest if there is a way to insert a slice of slices into database. I have created a slice for each row and created a another slice(multiple rows) by appending all the row slices to it. how do I insert the slice(multiple rows) into db?
When I create a row slice, I'm using row := []interface{}{} . Because I have fields which are strings and int in each row. Looks like I
get an error when I'm inserting data and the error is unsupported type []interface {}, a slice of interface
Implementation:
rowdata := []interface{}{}
row := []interface{}{data.ScenarioUUID, data.Puid, data.Description, data.Status, data.CreatedBy, data.CreatedAt, data.UpdatedBy, data.UpdatedAt, data.ScopeStartsAt, data.ScopeEndsAt, Metric, MetricName, Channel, date, timeRangeValue}
rowdata = append(rowdata, row)
qry2 := `INSERT INTO sample (scenarioUuid,
puId,
description,
status,
createdBy,
createdAt,
updatedBy,
updatedAt,
scopeStartsAt,
scopeEndsAt,
metric,
metric_name,
channel,
time,
value) VALUES ($1, $2, $3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15)`
if _, err := db.Exec(qry2, rowdata); err != nil {
panic(err)
You could do something like this:
samples := // the slice of samples you want to insert
query := `insert into samples (<the list of columns>) values `
values := []interface{}{}
for i, s := range samples {
values = append(values, s.<field1>, s.<field2>, < ... >)
numFields := 15 // the number of fields you are inserting
n := i * numFields
query += `(`
for j := 0; j < numFields; j++ {
query += `$`+strconv.Itoa(n+j+1) + `,`
}
query = query[:len(query)-1] + `),`
}
query = query[:len(query)-1] // remove the trailing comma
db.Exec(query, values...)
https://play.golang.org/p/YqNJKybpwWB
Related
I have a simple select statement:
Select * FROM X where X.name in ("bob", "joe") and X.phone='123'
That works fine in postgres,
In my Go code I have the following code:
var phone string = "123"
var names []string = []string{"bob", "joe"}
sqlStatement := `Select * FROM X where X.name in ($1) and X.phone=$2`
rows, sqlerr := db.Query(sqlStatement, names, phone)
but for some reason I error out from that sql.
unsupported Scan, storing driver.Value type into type *string
how can i use my names array in side the sqlstatement?
note: if i do a fmt.printf and paste the sql statement into postgres, i do get data back + if i take out the $1, and manually input the strings
Copying and pasting fragments from some working Go PostgreSQL code:
import (
"database/sql"
"github.com/lib/pq"
)
query := `
. . .
WHERE code = ANY($1)
. . .
`
codes := []string{"JFK", "LGA", "EWR"}
rows, err := db.Query(query, pq.Array(codes))
I solved this using http://jmoiron.github.io/sqlx/#inQueries
var phone string = "123"
var names []string = []string{"bob", "joe"}
sqlStatement := `Select * FROM X where X.name in (?) and X.phone=?`
sqlStatement, args, err := sqlx.In(sqlStatement, names, phone)
sqlStatement = db.Rebind(sqlStatement)
rows, sqlerr := db.Queryx(sqlStatement, args...)
this now returns correctly.
another way to solve this was to use fmt.Sprintf() and converting the ? to %s
I am trying to execute the following query against the PostgreSQL database in Go using pq driver:
SELECT COUNT(id)
FROM tags
WHERE id IN (1, 2, 3)
where 1, 2, 3 is passed at a slice tags := []string{"1", "2", "3"}.
I have tried many different things like:
s := "(" + strings.Join(tags, ",") + ")"
if err := Db.QueryRow(`
SELECT COUNT(id)
FROM tags
WHERE id IN $1`, s,
).Scan(&num); err != nil {
log.Println(err)
}
which results in pq: syntax error at or near "$1". I also tried
if err := Db.QueryRow(`
SELECT COUNT(id)
FROM tags
WHERE id IN ($1)`, strings.Join(stringTagIds, ","),
).Scan(&num); err != nil {
log.Println(err)
}
which also fails with pq: invalid input syntax for integer: "1,2,3"
I also tried passing a slice of integers/strings directly and got sql: converting Exec argument #0's type: unsupported type []string, a slice.
So how can I execute this query in Go?
Pre-building the SQL query (preventing SQL injection)
If you're generating an SQL string with a param placeholder for each of the values, it's easier to just generate the final SQL right away.
Note that since values are strings, there's place for SQL injection attack, so we first test if all the string values are indeed numbers, and we only proceed if so:
tags := []string{"1", "2", "3"}
buf := bytes.NewBufferString("SELECT COUNT(id) FROM tags WHERE id IN(")
for i, v := range tags {
if i > 0 {
buf.WriteString(",")
}
if _, err := strconv.Atoi(v); err != nil {
panic("Not number!")
}
buf.WriteString(v)
}
buf.WriteString(")")
Executing it:
num := 0
if err := Db.QueryRow(buf.String()).Scan(&num); err != nil {
log.Println(err)
}
Using ANY
You can also use Postgresql's ANY, whose syntax is as follows:
expression operator ANY (array expression)
Using that, our query may look like this:
SELECT COUNT(id) FROM tags WHERE id = ANY('{1,2,3}'::int[])
In this case you can declare the text form of the array as a parameter:
SELECT COUNT(id) FROM tags WHERE id = ANY($1::int[])
Which can simply be built like this:
tags := []string{"1", "2", "3"}
param := "{" + strings.Join(tags, ",") + "}"
Note that no check is required in this case as the array expression will not allow SQL injection (but rather will result in a query execution error).
So the full code:
tags := []string{"1", "2", "3"}
q := "SELECT COUNT(id) FROM tags WHERE id = ANY($1::int[])"
param := "{" + strings.Join(tags, ",") + "}"
num := 0
if err := Db.QueryRow(q, param).Scan(&num); err != nil {
log.Println(err)
}
This is not really a Golang issue, you are using a string to compare to integer (id) in your SQL request. That means, SQL receive:
SELECT COUNT(id)
FROM tags
WHERE id IN ("1, 2, 3")
instead of what you want to give it. You just need to convert your tags into integer and passe it to the query.
EDIT:
Since you are trying to pass multiple value to the query, then you should tell it:
params := make([]string, 0, len(tags))
for i := range tags {
params = append(params, fmt.Sprintf("$%d", i+1))
}
query := fmt.Sprintf("SELECT COUNT(id) FROM tags WHERE id IN (%s)", strings.Join(params, ", "))
This will end the query with a "($1, $2, $3...", then convert your tags as int:
values := make([]int, 0, len(tags))
for _, s := range tags {
val, err := strconv.Atoi(s)
if err != nil {
// Do whatever is required with the error
fmt.Println("Err : ", err)
} else {
values = append(values, val)
}
}
And finally, you can use it in the query:
Db.QueryRow(query, values...)
This should do it.
Extending #icza solution, you can use pq.Array instead of building the params yourself.
So using his example, the code can look like this:
tags := []string{"1", "2", "3"}
q := "SELECT COUNT(id) FROM tags WHERE id = ANY($1::int[])"
num := 0
if err := Db.QueryRow(q, pq.Array(tags)).Scan(&num); err != nil {
log.Println(err)
}
I am using Golang and Postgres to filter some financial data. I have a Postgres database which has a single table containing a single Stock Market (if that's the correct term). This table has columns for id, symbol, date, open, high, low, close and volume. The total number of rows is 6,610,598 and the number of distinct stocks (symbols) is 2174.
Now what I want to do is to filter the data from that table, and save to another table. So the first one contains raw data and second one contains cleaned data.
We have three parameters, a date (EVALDATE) and 2 integers (MINCTD & MINDP). First, we have to select only those stocks that will pass our minimum calendar trading days parameter. So that will be selected by (NOTE: we use golang for this)
symbols []string got its value from ( Select distinct symbol from table_name; )
[]filteredSymbols
var symbol, date string
var open, high, low, close float64
var volume int
for _, symbol := range symbols {
var count int
query := fmt.Sprintf("Select count(*) from table_name where symbol = '%s' and date >= '%s';", symbol, EVALDATE)
row := db.QueryRow(query)
if err := row.Scan(&count); err != nil ........
if count >= MINCTD
filteredSymbols = append(filteredSymbols, symbol)
}
Basically, the operation above only asks for those symbols which has enough number of rows from the EVALDATE up to current date (latest date in data) that will satisfy MINCTD. The operation above took 30 minutes
If a symbol satisfies the first filter above, it will undergo a second filter which will test if within that period (EVALDATE to LATEST_DATE) it has enough rows that contain complete data (no OHLC without values). So the query below is used to filter the symbols which passed the filter above:
Select count(*) from table_name where symbol='symbol' and date>= 'EVALDATE' and open != 0 and high != 0 and low != 0 and close != 0;
This query took 36 minutes.
After getting the slice of symbols which passed both filter, I will then grab their data again using postgres query then begin a bulk insert to another table.
So 1 hour and 6 minutes is not very acceptable. What should I do then? Grab all data then filter using Golang in memory?
Couple of things I note from the question.
Try to avoid scanning 6 million+ rows to arrive at 2174 values (i.e. avoid Select distinct symbol from table_name;). Do you not have (or can you build) a "master table" of symbols with a primary key of the symbols?
Combine your queries to test the data such as the following:
select
count(*) c1
, count(case when open != 0 and high != 0 and low != 0 and close != 0 then 1 end) as c2
from table_name
where symbol='symbol'
and date>= 'EVALDATE'
An index on (symbol, date) would assist performance.
In Go, clean 7,914,698 rows for 3,142 symbols in 28.7 seconds, which is better than 3,960 seconds (1 hour and 6 minutes) for 6,610,598 rows for 2,174 symbols.
Output:
$ go run clean.go
clean: 7914698 rows 28.679295705s
$ psql
psql (9.6.6)
peter=# \d clean
Table "public.clean"
Column | Type | Modifiers
--------+------------------+-----------
id | integer |
symbol | text | not null
date | date | not null
close | double precision |
volume | integer |
open | double precision |
high | double precision |
low | double precision |
Indexes:
"clean_pkey" PRIMARY KEY, btree (symbol, date)
peter=# SELECT COUNT(*) FROM clean;
count
---------
7914698
peter=# SELECT COUNT(DISTINCT symbol) FROM clean;
count
-------
3142
peter=# \q
$
clean.go:
package main
import (
"database/sql"
"fmt"
"strconv"
"time"
_ "github.com/lib/pq"
)
func clean(db *sql.DB, EVALDATE time.Time, MINCTD, MINDP int) (int64, time.Duration, error) {
start := time.Now()
tx, err := db.Begin()
if err != nil {
return 0, 0, err
}
committed := false
defer func() {
if !committed {
tx.Rollback()
}
}()
{
const query = `DROP TABLE IF EXISTS clean;`
if _, err := tx.Exec(query); err != nil {
return 0, 0, err
}
}
var nRows int64
{
const query = `
CREATE TABLE clean AS
SELECT id, symbol, date, close, volume, open, high, low
FROM unclean
WHERE symbol IN (
SELECT symbol
FROM unclean
WHERE date >= $1
GROUP BY symbol
HAVING
COUNT(*) >= $2
AND
COUNT(CASE WHEN NOT (open >0 AND high >0 AND low >0 AND close >0) THEN 1 END) <= $3
)
ORDER BY symbol, date
;
`
EVALDATE := EVALDATE.Format("'2006-01-02'")
MINCTD := strconv.Itoa(MINCTD)
MINDP := strconv.Itoa(MINDP)
res, err := tx.Exec(query, EVALDATE, MINCTD, MINDP)
if err != nil {
return 0, 0, err
}
nRows, err = res.RowsAffected()
if err != nil {
return 0, 0, err
}
}
{
const query = `ALTER TABLE clean ADD PRIMARY KEY (symbol, date);`
_, err := tx.Exec(query)
if err != nil {
return 0, 0, err
}
}
if err = tx.Commit(); err != nil {
return 0, 0, err
}
committed = true
since := time.Since(start)
{
const query = `ANALYZE clean;`
if _, err := db.Exec(query); err != nil {
return nRows, since, err
}
}
return nRows, since, nil
}
func main() {
db, err := sql.Open("postgres", "user=peter password=peter dbname=peter")
if err != nil {
fmt.Println(err)
return
}
defer db.Close()
var ( // one year
EVALDATE = time.Now().AddDate(-1, 0, 0)
MINCTD = 240
MINDP = 5
)
nRows, since, err := clean(db, EVALDATE, MINCTD, MINDP)
if err != nil {
fmt.Println(err)
return
}
fmt.Println("clean:", nRows, "rows", since)
return
}
Playground: https://play.golang.org/p/qVOQQ6mcU-1
References:
Technical Analysis of the Financial Markets: A Comprehensive Guide to Trading Methods and Applications, John J. Murphy.
An Introduction to Database Systems, 8th Edition, C.J. Date.
PostgreSQL: Introduction and Concepts, Bruce Momjian.
PostgreSQL 9.6.6 Documentation, PostgreSQL.
I'm trying to pass a variable number of arguments to db.Query() in Go. I'm doing something like:
var values []interface{}
query := []string{"SELECT * FROM table"}
sep := "WHERE"
if ... {
values = append(values, something)
query = append(query, fmt.Sprintf(` %s field_a=$%d`, sep, len(values))
sep = "AND"
}
if ... {
values = append(values, something)
query = append(query, fmt.Sprintf(` %s field_b=$%d`, sep, len(values))
sep = "AND"
}
// Add an arbitrary number of conditional arguments...
rows, err := db.Query(strings.Join(query, " "), values...)
The query looks fine, and the values are all there, but I'm not getting anything back from the query. Some of the values are integers, some strings. When I manually try the exact same query (copy/paste) in psql, I substituting the actual values for $1, $2, etc, I get correct results. What am I missing?
Edit
Here's an example of what the final query (the result of strings.Join()) should look like:
SELECT
table1.field1, table1.field2, table1.field3,
table2.field4, table2.field6, table2.field6
FROM table1, table2
WHERE
table1.field2=$1 AND
table2.field3 IN ($2, $3)
When I try:
SELECT
table1.field1, table1.field2, table1.field3,
table2.field4, table2.field6, table2.field6
FROM table1, table2
WHERE
table1.field2='something' AND
table2.field3 IN (40, 50)
from psql, it works fine. When I call:
var values []interface{}
values = append(values, "something")
values = append(values, 40)
values = append(values, 50)
db.Query(`SELECT
table1.field1, table1.field2, table1.field3,
table2.field4, table2.field6, table2.field6
FROM table1, table2
WHERE
table1.field2=$1 AND
table2.field3 IN ($2, $3)`, values...)
from Go, I get a Rows object that returns false the first time rows.Next() is called, and a nil error.
I am trying to execute the following query against the PostgreSQL database in Go using pq driver:
SELECT COUNT(id)
FROM tags
WHERE id IN (1, 2, 3)
where 1, 2, 3 is passed at a slice tags := []string{"1", "2", "3"}.
I have tried many different things like:
s := "(" + strings.Join(tags, ",") + ")"
if err := Db.QueryRow(`
SELECT COUNT(id)
FROM tags
WHERE id IN $1`, s,
).Scan(&num); err != nil {
log.Println(err)
}
which results in pq: syntax error at or near "$1". I also tried
if err := Db.QueryRow(`
SELECT COUNT(id)
FROM tags
WHERE id IN ($1)`, strings.Join(stringTagIds, ","),
).Scan(&num); err != nil {
log.Println(err)
}
which also fails with pq: invalid input syntax for integer: "1,2,3"
I also tried passing a slice of integers/strings directly and got sql: converting Exec argument #0's type: unsupported type []string, a slice.
So how can I execute this query in Go?
Pre-building the SQL query (preventing SQL injection)
If you're generating an SQL string with a param placeholder for each of the values, it's easier to just generate the final SQL right away.
Note that since values are strings, there's place for SQL injection attack, so we first test if all the string values are indeed numbers, and we only proceed if so:
tags := []string{"1", "2", "3"}
buf := bytes.NewBufferString("SELECT COUNT(id) FROM tags WHERE id IN(")
for i, v := range tags {
if i > 0 {
buf.WriteString(",")
}
if _, err := strconv.Atoi(v); err != nil {
panic("Not number!")
}
buf.WriteString(v)
}
buf.WriteString(")")
Executing it:
num := 0
if err := Db.QueryRow(buf.String()).Scan(&num); err != nil {
log.Println(err)
}
Using ANY
You can also use Postgresql's ANY, whose syntax is as follows:
expression operator ANY (array expression)
Using that, our query may look like this:
SELECT COUNT(id) FROM tags WHERE id = ANY('{1,2,3}'::int[])
In this case you can declare the text form of the array as a parameter:
SELECT COUNT(id) FROM tags WHERE id = ANY($1::int[])
Which can simply be built like this:
tags := []string{"1", "2", "3"}
param := "{" + strings.Join(tags, ",") + "}"
Note that no check is required in this case as the array expression will not allow SQL injection (but rather will result in a query execution error).
So the full code:
tags := []string{"1", "2", "3"}
q := "SELECT COUNT(id) FROM tags WHERE id = ANY($1::int[])"
param := "{" + strings.Join(tags, ",") + "}"
num := 0
if err := Db.QueryRow(q, param).Scan(&num); err != nil {
log.Println(err)
}
This is not really a Golang issue, you are using a string to compare to integer (id) in your SQL request. That means, SQL receive:
SELECT COUNT(id)
FROM tags
WHERE id IN ("1, 2, 3")
instead of what you want to give it. You just need to convert your tags into integer and passe it to the query.
EDIT:
Since you are trying to pass multiple value to the query, then you should tell it:
params := make([]string, 0, len(tags))
for i := range tags {
params = append(params, fmt.Sprintf("$%d", i+1))
}
query := fmt.Sprintf("SELECT COUNT(id) FROM tags WHERE id IN (%s)", strings.Join(params, ", "))
This will end the query with a "($1, $2, $3...", then convert your tags as int:
values := make([]int, 0, len(tags))
for _, s := range tags {
val, err := strconv.Atoi(s)
if err != nil {
// Do whatever is required with the error
fmt.Println("Err : ", err)
} else {
values = append(values, val)
}
}
And finally, you can use it in the query:
Db.QueryRow(query, values...)
This should do it.
Extending #icza solution, you can use pq.Array instead of building the params yourself.
So using his example, the code can look like this:
tags := []string{"1", "2", "3"}
q := "SELECT COUNT(id) FROM tags WHERE id = ANY($1::int[])"
num := 0
if err := Db.QueryRow(q, pq.Array(tags)).Scan(&num); err != nil {
log.Println(err)
}