Within a transaction I'm inserting a row.
How can I access and return the ID of the inserted row.
As you can see in the code below(See under comment // Return last Inserted ID.) I tried to use the LastInsertedId() function, but it gives me an error back.
Btw, I'm using Postgres.
What am I missing here?
Thx!
/**
* Creates order and returns its ID.
*/
func createOrder(w http.ResponseWriter, r *http.Request) (orderID int) {
// Begin.
tx, err := db.Begin()
if err != nil {
log.Fatal(err)
}
// Db query.
sqlQuery := `INSERT INTO ORDER_CUSTOMER
(CUSTOMER_ID)
VALUES ($1)
RETURNING ID`
// Prepare.
stmt, err := tx.Prepare(sqlQuery)
if err != nil {
log.Fatal(err)
return
}
// Defer Close.
defer stmt.Close()
customerEmail := validateSession(r)
ID := getIDFromCustomer(customerEmail)
order := order{}
order.CustomerID = ID
// Exec.
ret, err := stmt.Exec(order.CustomerID)
// Rollback.
if err != nil {
tx.Rollback()
e := errors.New(err.Error())
msg.Warning = e.Error()
tpl.ExecuteTemplate(w, "menu.gohtml", msg)
return
}
// Return last Inserted ID.
lastID, err := ret.LastInsertId()
if err != nil {
orderID = 0
} else {
orderID = int(lastID)
}
// Commit.
tx.Commit()
return orderID
} // createOrder
Here is a working solution for now, further improvement is welcomed.
/**
* Creates order and returns its ID.
*/
func createOrder(w http.ResponseWriter, r *http.Request) (orderID int) {
// Begin.
tx, err := db.Begin()
if err != nil {
log.Fatal(err)
}
// Db query.
sqlQuery := `INSERT INTO ORDER_CUSTOMER
(CUSTOMER_ID)
VALUES ($1)
RETURNING ID`
// Prepare.
stmt, err := tx.Prepare(sqlQuery)
if err != nil {
log.Fatal(err)
return
}
// Defer Close.
defer stmt.Close()
customerEmail := validateSession(r)
ID := getIDFromCustomer(customerEmail)
order := order{}
order.CustomerID = ID
// Exec.
_, err = stmt.Exec(order.CustomerID)
// Rollback.
if err != nil {
tx.Rollback()
e := errors.New(err.Error())
msg.Warning = e.Error()
tpl.ExecuteTemplate(w, "menu.gohtml", msg)
return
}
// Return last Inserted ID.
//lastID, err := ret.LastInsertId()
err = stmt.QueryRow(order.CustomerID).Scan(&orderID)
if err != nil {
orderID = 0
}
// Commit.
tx.Commit()
return orderID
} // createOrder
This happens because the postgresql driver you are using for go doesn't supports the LastInsertedId() function. You didn't say which driver you are using but I have had this issue working with github.com/lib/pq.
The answer to this is to use QueryRow insted of Exec in your original example. Just make sure you are using RETURNING ID on your query and treat it as if it was a select.
Here is an example (I didn't test this and I might be missing something but you get the idea):
func createOrder(w http.ResponseWriter, r *http.Request) (orderID int) {
// Begin.
tx, err := db.Begin()
if err != nil {
log.Fatal(err)
}
// Db query.
sqlQuery := `INSERT INTO ORDER_CUSTOMER
(CUSTOMER_ID)
VALUES ($1)
RETURNING ID`
// Prepare.
stmt, err := tx.Prepare(sqlQuery)
if err != nil {
log.Fatal(err)
return
}
// Defer Close.
defer stmt.Close()
customerEmail := validateSession(r)
ID := getIDFromCustomer(customerEmail)
order := order{}
order.CustomerID = ID
// Exec.
var orderID int // Or whatever type you are using
err := stmt.QueryRow(order.CustomerID).Scan(&orderID)
// Rollback.
if err != nil {
//if something goes wrong set the orderID to 0 as in your original code
orderID = 0
tx.Rollback()
e := errors.New(err.Error())
msg.Warning = e.Error()
tpl.ExecuteTemplate(w, "menu.gohtml", msg)
return
}
// Commit.
tx.Commit()
return orderID
} // createOrder
Related
The error
(*errors.errorString) *{s: "pq: unexpected DataRow in simple query execution"}
appears after the line with the commentary. Didn't find any solution online. Since stackoverflow asks for more details, this is an update query that is supposed to update a todo and a list of subtasks in the database. The exact error is in the question topic. I post the complete code for the function that returns the error.
func (t *TodoTable) UpdateTodo(ctx context.Context, todo *Todo, t_id int) error {
tx, err := t.sqlxdb.BeginTxx(ctx, &sql.TxOptions{})
if err != nil {
return err
}
rollback_err := func(err error) error {
if err2 := tx.Rollback(); err2 != nil {
return fmt.Errorf("%v; %v", err, err2)
}
return err
}
row := tx.QueryRowxContext(ctx, "UPDATE todos SET todo_name=$1, deadline=$2, updated_at=$3 WHERE todo_id=$4 returning todo_id", todo.TodoName, todo.Deadline, todo.UpdatedAt, t_id)
if row.Err() != nil {
return rollback_err(err)
}
var subs_ids []int
// Getting subs ids from database
query := fmt.Sprintf("SELECT sub_id FROM subs WHERE todo_id=%d", t_id)
// THE ERROR COMES AFTER EXECUTING THE LINE BELOW
rows, err := tx.Query(query)
if err != nil {
rollback_err(err)
}
if rows != nil {
for rows.Next() {
var sub_id int
err = rows.Scan(&sub_id)
if err != nil {
rollback_err(err)
}
subs_ids = append(subs_ids, sub_id)
}
if err := tx.Commit(); err != nil {
return rollback_err(err)
}
}
// Updating subs
for i, sub := range todo.Subs {
_, err = tx.ExecContext(ctx, fmt.Sprintf("UPDATE subs SET sub_name='%s' WHERE sub_id=%d", sub.Sub_name, subs_ids[i]))
if err != nil {
return rollback_err(err)
}
}
return nil
}
I have a function that updates records in table and requests it using pgx Postgres driver. This function hangs on commit. Is there any ideas why does it happen? Why I can't use transactions in this case?
Of course, as the query is atomic I can remove transactions. But it's still unclear—why it happens and what to do if I need a transaction.
func (r *Repository) GetUpdatedItems(ctx context.Context, filters []string) ([]Item, error) {
conn, err := r.pool.Acquire(ctx)
// error handling
defer conn.Release()
tx, err := conn.Begin(ctx)
// error handling
defer func() {
closeCtx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()
_ = tx.Rollback(closeCtx)
}()
query := fmt.Sprintf(`UPDATE %s
SET fieldOne = $1, fieldTwo = $2
WHERE otheField = '' OR otherField IS NULL
RETURNING fieldOne, fieldTwo, otheField, someMoreField;`,
r.tableName, sqlArray(aggregatesTypes))
rows, err := conn.Query(ctx, query, filters[0], filters[1])
// error handling
defer rows.Close()
var retItems []reaper.Item
for rows.Next() {
var fieldOne string
var fieldTwo string
var otheField string
var someMoreField string
if err := rows.Scan(&id, &fieldOne, &fieldTwo, &otheField, &someMoreField); err != nil {
return nil, fmt.Errorf("failed to scan item: %w", err)
}
item := Item{
FieldOne: fieldOne,
FieldTwo: fieldTwo,
OtheField: otheField,
SomeMoreField: someMoreField
}
retItems = append(retItems, item)
}
if err := tx.Commit(ctx); err != nil {
return nil, fmt.Errorf("failed to commit transaction: %w", err)
}
return retItems, nil
}
tx.Query must be used instead of conn.Query.
Thanks for the help!
After a POST request, i was expecting to have a last inserted record marshalled into json, but instead returns an empty body. What am i not doing well?
package models
import (
"encoding/json"
"errors"
"flag"
"fmt"
"log"
"net/http"
"strconv"
"github.com/go-chi/chi"
"github.com/google/jsonapi"
"github.com/thedevsaddam/renderer"
"github.com/xo/dburl"
)
var rnd = renderer.New()
var flagVerbose = flag.Bool("v", false, "verbose")
var FlagURL = flag.String("url", "postgres://postgres:#127.0.0.1/sweb", "url")
// Page represents a row from 'public.pages'.
type Page struct {
Tag string `jsonapi:"attr,tag"` // tag
Body string `jsonapi:"attr,body"` // body
Slug string `jsonapi:"attr,slug"` // slug
Title string `jsonapi:"attr,title"` // title
ID int `jsonapi:"primary,pages"` // id
Link string `jsonapi:"attr,link"` // link
// xo fields
_exists, _deleted bool
}
func (page Page) JSONAPILinks() *jsonapi.Links {
return &jsonapi.Links{
"self": fmt.Sprintf("https://%d", page.ID),
}
}
I believe this is the culprit. After inserting a record, it should return the last inserted record as specified.
func (p *Page) PInsert(db XODB) (*Page, error) {
var err error
// if already exist, bail
if p._exists {
return p, errors.New("insert failed: already exists")
}
// sql insert query, primary key provided by sequence
const sqlstr = `INSERT INTO public.pages (` +
`tag, body, slug, title` +
`) VALUES (` +
`$1, $2, $3, $4` +
`) RETURNING id, tag, body, title`
// run query
XOLog(sqlstr, p.Tag, p.Body, p.Slug, p.Title)
err = db.QueryRow(sqlstr, p.Tag, p.Body, p.Slug, p.Title).Scan(&p.ID, &p.Tag, &p.Body, &p.Title)
if err != nil {
return p, err
}
// set existence
p._exists = true
return p, nil
}
Update updates the Page in the database and return last inserted records.
The same should apply for the Update function
func (p *Page) Update(db XODB) (*Page, error) {
var err error
// if doesn't exist, bail
if !p._exists {
return p, errors.New("update failed: does not exist")
}
// if deleted, bail
if p._deleted {
return p, errors.New("update failed: marked for deletion")
}
// sql query
const sqlstr = `UPDATE public.pages SET (` +
`tag, body, slug, title` +
`) = ( ` +
`$1, $2, $3, $4` +
`) WHERE id = $5`
// run query
XOLog(sqlstr, p.Tag, p.Body, p.Slug, p.Title, p.ID)
_, err = db.Exec(sqlstr, p.Tag, p.Body, p.Slug, p.Title, p.ID)
return p, err
}
func (p *Page) PSave(db XODB) (*Page, error) {
if p.Exists() {
return p.Update(db)
}
return p.PInsert(db)
}
func NewPage(w http.ResponseWriter, r *http.Request) {
db, err := dburl.Open(*FlagURL)
defer db.Close()
if err != nil {
log.Fatal(err)
}
var page Page
//page := new(Page)
if err := jsonapi.UnmarshalPayload(r.Body, &page); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
p, err := page.PSave(db)
if err != nil {
fmt.Println(err)
if err := jsonapi.MarshalPayload(w, p); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
fmt.Println(err)
}
}
w.Header().Set("Content-Type", jsonapi.MediaType)
w.WriteHeader(http.StatusCreated)
}
This is the last function i believe the issue is happening from. the last inserted record supposed to be marshalled into json.
Your last section of code contains a number of mistakes. The relevant section (without the useless and obfuscating Printlns) is:
p, err := page.PSave(db)
if err != nil {
if err := jsonapi.MarshalPayload(w, p); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
}
w.Header().Set("Content-Type", jsonapi.MediaType)
w.WriteHeader(http.StatusCreated)
And the primary mistake is that json.MarshalPayload is only called when err != nil. In other words, you only serialize the page if you failed to save it.
The secondary mistake is that jsonapi.MarshalPayload will call Write on the http.ResponseWriter. This turns all subsequent calls to Header().Set and WriteHeader into no-ops.
More correct code would look like this.
// 1. Save the page in the database, bail on error
p, err := page.PSave(db)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
// 2. Marshal the page into an intermediate buffer, bail on error
var buf bytes.Buffer
if err := jsonapi.MarshalPayload(&buf, p); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
// 3. Write the entire response; failures to write the intermediate buffer
// cannot be communicated over HTTP
w.Header().Set("Content-Type", jsonapi.MediaType)
w.WriteHeader(http.StatusCreated)
if _, err := buf.WriteTo(w); err != nil {
log.Printf("failed to write response: %v", err)
return
}
I am processing a bunch of files and then dumping the results in PostgreSQL. I would like to process many workers at the same time but keep getting the error "pq: sorry, too many clients already". This seems to happen whenever workers is > 100 or so. (For simplicity, the code below demonstrates the process but instead of processing a file I am simply inserting 1M rows in each table).
Since I am reusing the same *db why am I getting this error? Does each transaction count as a client or am I doing something wrong?
package main
import (
"database/sql"
"flag"
"fmt"
"log"
"sync"
"github.com/lib/pq"
)
func process(db *sql.DB, table string) error {
if _, err := db.Exec(fmt.Sprintf(`DROP TABLE IF EXISTS %v;`, table)); err != nil {
return err
}
col := "age"
s := fmt.Sprintf(`
CREATE TABLE %v (
pk serial PRIMARY KEY,
%v int NOT NULL
)`, table, col)
_, err := db.Exec(s)
if err != nil {
return err
}
tx, err := db.Begin()
if err != nil {
return err
}
defer func() {
if err != nil {
tx.Rollback()
return
}
err = tx.Commit()
}()
stmt, err := tx.Prepare(pq.CopyIn(table, col))
if err != nil {
return err
}
defer func() {
err = stmt.Close()
}()
for i := 0; i < 1e6; i++ {
if _, err = stmt.Exec(i); err != nil {
return err
}
}
return err
}
func main() {
var u string
flag.StringVar(&u, "user", "", "user")
var pass string
flag.StringVar(&pass, "pass", "", "pass")
var host string
flag.StringVar(&host, "host", "", "host")
var database string
flag.StringVar(&database, "database", "", "database")
var workers int
flag.IntVar(&workers, "workers", 10, "workers")
flag.Parse()
db, err := sql.Open("postgres",
fmt.Sprintf(
"user=%s password=%s host=%s database=%s sslmode=require",
u, pass, host, database,
),
)
if err != nil {
log.Fatalln(err)
}
defer db.Close()
db.SetMaxIdleConns(0)
var wg sync.WaitGroup
ch := make(chan int)
for i := 0; i < workers; i++ {
wg.Add(1)
go func() {
defer wg.Done()
for i := range ch {
table := fmt.Sprintf("_table%d", i)
log.Println(table)
if err := process(db, table); err != nil {
log.Fatalln(err)
}
}
}()
}
for i := 0; i < 300; i++ {
ch <- i
}
close(ch)
wg.Wait()
}
I realize I can simply increase the posgresql settings but would like to understand the question: How to increase the max connections in postgres?
Since I am reusing the same *db why am I getting this error?
I suspect the Postgress driver is using a separate connections for each of your workers which is a smart decision for most cases.
Does each transaction count as a client or am I doing something wrong?
In your case yes each transaction count as a client, because you are calling process() as a goroutine. You are creating as many concurrent transactions as workers. Since each of your transactions is long all of them are probably using an individual connection to the database at the same time and hence you hit a limit.
go func() {
defer wg.Done()
for i := range ch {
table := fmt.Sprintf("_table%d", i)
log.Println(table)
if err := process(db, table); err != nil {
log.Fatalln(err)
}
}
}()
I've written a toy app to experiment with using Postgresql through sqlx. I got a mass insert working using
pq.CopyIn
as content of a prepared statement
stmt, _ := tx.Preparex(pq.CopyIn(tablename, column, column, ...)
I would then proceed to add rows to the mass insert I'm creating.
tx.Exec(..., ..., ...)
then finally execute the prepared statement
stmt.Exec()
This worked perfectly before, but now I've come back to it and try and execute this code, it hangs on the
stmt.Exec
Am I missing something in my code or is this all to do with the Database Engine, being unresponsive.
Here's my full code for this.
package main
import (
_ "database/sql"
"fmt"
"log"
"encoding/json"
"io/ioutil"
"os"
"github.com/jmoiron/sqlx"
"github.com/lib/pq"
)
var schema = `
CREATE TABLE IF NOT EXISTS contact (
id Serial,
first_name text,
last_name text,
email text
);`
type Contact struct {
Id int `json:"-"`
First_name string `json:"first_name"`
Last_name string `json:"last_name"`
Email string `json:"email"`
}
type Contacts struct {
Contacts []Contact `json:"contacts"`
}
func (c *Contacts) createFromJSON(json_str []byte) error {
b := []byte(json_str)
err := json.Unmarshal(b, &c)
if err != nil {
log.Fatal(err)
}
return err
}
func (c *Contacts) save(db *sqlx.DB) error {
tx := db.MustBegin()
stmt, _ := tx.Preparex(pq.CopyIn("contact", "first_name", "last_name", "email"))
for _, contact := range c.Contacts {
tx.Exec(contact.First_name, contact.Last_name, contact.Email)
}
_, err := stmt.Exec()
if err != nil {
log.Fatal(err)
return err
}
err = stmt.Close()
if err != nil {
log.Fatal(err)
return err
}
tx.Commit()
return nil
}
func connect() (*sqlx.DB, error) {
db, err := sqlx.Connect("postgres", "user=pqgotest dbname=pqgotest sslmode=disable")
if err != nil {
log.Fatal(err)
}
return db, err
}
func createTables(db *sqlx.DB) {
db.MustExec(schema)
}
func main() {
db, err := connect()
if err != nil {
os.Exit(1)
}
createTables(db)
contactsJson, e := ioutil.ReadFile("./contacts.json")
if e != nil {
fmt.Printf("File error: %v\n", e)
os.Exit(1)
}
tx := db.MustBegin()
tx.MustExec("DELETE FROM contact")
tx.Commit()
contacts := new(Contacts)
contacts.createFromJSON(contactsJson)
contacts.save(db)
people := new(Contacts)
db.Select(people.Contacts, "SELECT * FROM contact ORDER BY email,id ASC")
for _, contact := range people.Contacts {
contact_json, err := json.Marshal(contact)
if err != nil {
log.Fatal(err)
os.Exit(1)
}
fmt.Printf("%s\n", contact_json)
}
}
I could include the contents of the contacts.json file as well, if that will help.
UPDATE
Yes it was obvious in the end. I was creating a statement from tx,
stmt, _ := tx.Preparex(pq.CopyIn(tablename, column, column, ...)
and further additions to this should be to stmt
stmt.Exec(..., ..., ...)
Also another error not directly related to the question is where I insert an array of contacts into the Contacts field of the struct Contacts
people := new(Contacts)
db.Select(people.Contacts, "SELECT * FROM contact ORDER BY email,id ASC")
should be passing a pointer to the Select method of db of the Contacts array field of Contacts, like so
db.Select(&people.Contacts, "SELECT * FROM contact ORDER BY email,id ASC")
In case people try and run this code later and wonder why it's not printing the results to the console.
From Bulk imports part in https://godoc.org/github.com/lib/pq, it should be
stmt.Exec(contact.First_name, contact.Last_name, contact.Email)