Consider the following code
func main() {
ctx := context.Background()
tx, err := db.BeginTx(ctx, nil)
row := tx.QueryRow("SELECT timestamp, my_id, my_value FROM my_table ORDER BY timestamp ASC limit 1 ")
var my_id int
var my_time time.Time
var my_value string
err = row.Scan(&my_time, &my_id, &my_value)
if err != nil {
tx.Rollback()
return
}
_, err = tx.ExecContext(ctx, "UPDATE my_table SET status = 'start' WHERE my_id = $1", my_id)
if err != nil {
tx.Rollback()
return
}
err = tx.Commit()
if err != nil {
log.Fatal(err)
}
// do something with my_value that takes a long time
}
Which select the row with the oldest timestamp and set the status to start and do something with the return value, I run this transaction in a multi threaded/server environment, how do I make sure that each thread is getting a unique my_id and that no two thread is processing the same my_id?
I don't think the select statement locks the row that got returned during the first select, so multiple thread can attempt to update the same row.
I could modify the update statement to be
UPDATE my_table SET status = 'start' WHERE my_id = $1 AND status <> `start`
but then I have to reselect another id, is there a way to avoid it?
Use pessimistic locking to put an UPDATE lock on the row when reading:
SELECT ... FOR UPDATE
That will prevent concurrent data modifications.
Related
I'am trying to create registration in my Telegram Bot with Golang and Postgres. When user writes "register", bot has to check if user's uuid already exists in DB, and if not to create row with his uuid.
Here is my function to check if uuid already exists in DB:
func IsUserInDB(uuid int64) (bool, error) {
var exists bool
query := fmt.Sprintf("SELECT EXISTS(SELECT 1 FROM users WHERE uuid = %d);", uuid)
err := Db.QueryRow(query).Scan(&exists)
return exists, err
}
Here is my function for adding user's uuid to DB:
func AddUserToDB(column string, row interface{}) error {
query := fmt.Sprintf("INSERT INTO users (%s) VALUES (%v);", column, row)
_, err := Db.Exec(query)
return err
}
And the logic for bot:
func (b *Bot) handleMessages(message *tgbotapi.Message) error {
switch message.Text {
case "register":
exists, err := data.IsUserInDB(message.From.ID)
if err != nil {
return err
}
if !exists {
err := data.AddUserToDB("uuid", message.From.ID)
return err
}
return nil
default:
msg := tgbotapi.NewMessage(message.Chat.ID, "unknown message...")
_, err := b.bot.Send(msg)
return err
}
}
First time, when I send "register", bot successfully adds user's id to db, but the problem happens if I try to send "register" 1 more time. IsUserInDB() returns me false and bot adds 1 more row with the same uuid. So, I think problem is with my IsUserInDb() function
Why not just a unique index on your users table?
CREATE UNIQUE INDEX unq_uuid ON users (uuid);
Then you don't have to check, you just try to insert and it will return an error if it already exists.
I am inserting some data in table using SQLx like this
func (*RideRepositoryImpl) insert(entity interface{}, tx persistence.Transaction) (sql.Result, error) {
ride := entity.(*model.Ride)
placeHolders := repository.InsertPlaceholders(len(rideColumns))
sql := fmt.Sprintf("INSERT INTO %s(%s) VALUES(%s)", TableName, strings.Join(Columns, ","), placeHolders)
return tx.Exec(sql, ride.ID.String(), ride.DeviceIotID, ride.VehicleID.String(), ride.UserID.String(),ride.AdditionComments)
}
and calling this function like this
func (p *RideRepositoryImpl) Save(ride *model.Ride, tx persistence.Transaction) error {
return repository.Save(ride, p.insert, tx)
Now I want to get UUID of saved record instantly after saving this record . Is there any clean way to do this instantly ?
PostgreSQL has the RETURNING clause for this.
Sometimes it is useful to obtain data from modified rows while they
are being manipulated. The INSERT, UPDATE, and DELETE commands
all have an optional RETURNING clause that supports this. Use of
RETURNING avoids performing an extra database query to collect the
data, and is especially valuable when it would otherwise be difficult
to identify the modified rows reliably.
// add the RETURNING clause to your INSERT query
sql := fmt.Sprintf("INSERT INTO %s(%s) VALUES(%s) RETURNING <name_of_uuid_column>", TableName, strings.Join(Columns, ","), placeHolders)
// use QueryRow instead of Exec
row := tx.QueryRow(sql, ride.ID.String(), ride.DeviceIotID, ride.VehicleID.String(), ride.UserID.String(),ride.AdditionComments)
// scan the result of the query
var uuid string
if err := row.Scan(&uuid); err != nil {
panic(err)
}
// ...
For additional INSERT-specific info related to RETURNING you can go to the INSERT documentation and search the page for "returning" with CTRL/CMD+F.
If, in addition, you need your function to still return an sql.Result value to satisfy some requirement, then you can return your own implementation.
var _ sql.Result = sqlresult{} // compiler check
type sqlresult struct { lastid, nrows int64 }
func (r sqlresult) LastInsertId() (int64, error) { return r.lastid, nil }
func (r sqlresult) RowsAffected() (int64, error) { return r.nrows, nil }
func (*RideRepositoryImpl) insert(entity interface{}, tx persistence.Transaction) (sql.Result, error) {
ride := entity.(*model.Ride)
placeHolders := repository.InsertPlaceholders(len(rideColumns))
sql := fmt.Sprintf("INSERT INTO %s(%s) VALUES(%s) RETURNING <name_of_uuid_column>", TableName, strings.Join(Columns, ","), placeHolders)
row := tx.QueryRow(sql, ride.ID.String(), ride.DeviceIotID, ride.VehicleID.String(), ride.UserID.String(),ride.AdditionComments)
if err := row.Scan(&ride.<NameOfUUIDField>); err != nil {
return nil, err
}
return sqlresult{0, 1}, nil
}
I have a Golang function to fetch all the records from Postgres database, this function is simply using :
SELECT * from stock_transactions
I want to apply filter to this function to fetch records with some conditions, in-short I want to use :
SELECT * from stock_transactions WHERE symbol = $symb
The problem is to handle the case where if $symb = null the query should act as SELECT * from stock_transactions. I can write an if-else clause for the same but if the number of parameters are more than 2 it could be messy. Is there a better way to handle this?
My function:
func showstocks (w http.ResponseWriter, r *http.Request){
var err error
if r.Method != "GET" {
http.Error(w, http.StatusText(405), http.StatusMethodNotAllowed)
return
}
rows, err := db.Query("SELECT * FROM stock_transaction ORDER BY id DESC")
if err != nil {
http.Error(w, http.StatusText(500), 500)
return
}
defer rows.Close()
sks := make([]stockdata, 0)
for rows.Next() {
sk := stockdata{}
err := rows.Scan(&sk.Sname, &sk.Ttype, &sk.Uprice, &sk.Qty, &sk.Bfee, &sk.Ddate)
if err != nil {
http.Error(w, http.StatusText(500), 500)
return
}
sks = append(sks, sk)
}
if err = rows.Err(); err != nil {
http.Error(w, http.StatusText(500), 500)
return
}
tpl.ExecuteTemplate(w, "dashboard.gohtml", sks)
}
//Suggested by #mkopriva. Tried and tested.
package main
import (
"fmt"
)
func main() {
// ...
// use interface{} because string types cannot be nil
var stock_symbol interface{} // the "zero-value of interface types is nil, so stock_symbol here is nil
// if empty then stock_symbol will be left as nil
if val := r.FormValue("stock_symbol"); len(val) > 0 {
stock_symbol = val // set to the provided value
}
// ...
// now stock_symbol is either the provided string value or nil
db.Query("SELECT ... WHERE (stock_symbol = $1 OR $1 IS NULL)", stock_symbol)
}
You can use the (stock_symbol = $1 OR $1 IS NULL) trick as you have already seen, which is programmatically convenient. However this can often lead to inefficient queries, as the query planner may not be smart enough to optimize them correctly. It might be better to write some code which removes the tautological phrases and also removes the corresponding bind variables, rather than passing each of them to the database. It is slightly messy, but it should be encapsulated into a function, not rewritten each time.
I use a temporary table to hold a range of ID's so I can use them in several other queries without adding a long list of ID's to every query.
I'm building this in GO and this is new for me. Creating the temporary table works, fetching the ID's succeed and also adding those IDs to the temporary table is successful. But when I use the temporary table I get this error:
pq: relation "temp_id_table" does not exist
This is my code (EDITED: added transaction):
//create context
ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
defer cancel()
// create database connection
psqlInfo := fmt.Sprintf("host=%s port=%s user=%s "+
"password=%s dbname=%s sslmode=disable",
c.Database.Host, c.Database.Port, c.Database.User, c.Database.Password, c.Database.DbName)
db, err := sql.Open("postgres", psqlInfo)
err = db.PingContext(ctx)
tx, err := db.BeginTx(ctx, &sql.TxOptions{Isolation: sql.LevelSerializable})
// create temporary table to store ids
_, err = tx.ExecContext(ctx, "CREATE TEMPORARY TABLE temp_id_table (id int)")
// fetch all articles of set
newrows, err := db.QueryContext(ctx, "SELECT id FROM article WHERE setid = $1", SetId)
var tempid int
var ids []interface{}
for newrows.Next() {
err := newrows.Scan(&tempid)
ids = append(ids, tempid)
}
// adding found ids to temporary table so we can use it in other queries
var buffer bytes.Buffer
buffer.WriteString("INSERT INTO temp_id_table (id) VALUES ")
for i := 0; i < len(ids); i++ {
if i>0 {
buffer.WriteString(",")
}
buffer.WriteString("($")
buffer.WriteString(strconv.Itoa(i+1))
buffer.WriteString(")")
}
_, err = db.QueryContext(ctx, buffer.String(), ids...)
// fething article codes
currrows, err := db.QueryContext(ctx, "SELECT code FROM article_code WHERE id IN (SELECT id FROM temp_id_table)")
(I simplified the code and removed all error handling to make the code more readable)
When I change it to a normal table everything works fine. What do I do wrong?
EDIT 05-06-2019:
I created a simple test program to test new input from the comments below:
func main() {
var codes []interface{}
codes = append(codes, 111)
codes = append(codes, 222)
codes = append(codes, 333)
config := config.GetConfig();
// initialising variables
ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
defer cancel()
// create database connection
log.Printf("create database connection")
db, err := connection.Create(config, ctx)
defer db.Close()
if err != nil {
log.Fatal(err)
}
// create transaction
log.Printf("create transaction")
tx, err := db.BeginTx(ctx, &sql.TxOptions{Isolation: sql.LevelReadUncommitted})
if err != nil {
log.Fatal(err)
}
// create temporary table to store IB codes
log.Printf("create temporary table to store codes")
_, err = tx.ExecContext(ctx, "CREATE TEMPORARY TABLE tmp_codes (code int)")
if err != nil {
log.Fatal(err)
}
// adding found IB codes to temporary table so we can fetch the current articles
log.Printf("adding codes to temporary table so we can fetch the current articles")
_, err = tx.QueryContext(ctx, "INSERT INTO tmp_codes (code) VALUES ($1),($2),($3)", codes...)
if err != nil {
log.Fatal(err)
}
testcodes, err := tx.QueryContext(ctx, "SELECT * FROM tmp_codes")
if err != nil {
log.Fatal(err)
}
defer testcodes.Close()
var testcount int
for testcodes.Next() {
testcount++
}
log.Printf(fmt.Sprintf("%d items in temporary table before commit, %d ibcodes added", testcount, len(codes)))
// close transaction
log.Printf("commit transaction")
tx.Commit()
}
The problem is the connection pool. You're not guaranteed to use the same server connection for each query. To guarantee this, you can start a transaction with Begin or BeginTx.
The returned sql.Tx object is guaranteed to use the same connection for its lifetime.
Related:
SQL Server Temp Tables and Connection Pooling
I have the following sql statement:
SELECT pk, up FROM mytable WHERE 2 > 1 LIMIT 10
This is just for simplicity, obviously. I am able to parameterize any of the integers:
SELECT pk, up FROM mytable WHERE 2 > $1 LIMIT 10
BUT, when I try to parameterize the operator, eg:
SELECT pk, up FROM mytable WHERE 2 $1 1 LIMIT 10
I get:
pq: syntax error at or near "$1"
Full Code:
package main
import (
"database/sql"
_ "github.com/lib/pq"
"log"
)
func main() {
log.SetFlags(log.Lshortfile)
Db, err := sql.Open("postgres", "user=yoitsmeletmein password=supersecretyo host=what.a.host dbname=mydb sslmode=require")
if err != nil {
log.Fatal("Cannot connect to db: ", err)
}
q := `SELECT pk FROM mytable WHERE 2 $1 1 LIMIT 10`
params := []interface{}{">"}
rows, err := Db.Query(q, params...)
if err != nil {
log.Println(err)
} else {
defer rows.Close()
for rows.Next() {
var pk int64
if err := rows.Scan(&pk); err != nil {
log.Fatal(err)
}
log.Println(pk)
}
}
}
Prepared statements allow to parametrize values, nothing else. It wouldn't make sense to parametrize operators to begin with, a statement cannot be prepared without knowing involved operators. And it would be potentially dangerous, opening vectors for SQL injection.
To switch operators, you'll have to concatenate a new query string in your client or use dynamic SQL with a server-side procedural language, the default being plpgsql.