I use postgres COPY in my Go backend. Copy is the only operation inside transaction. Should I roll it back if it failed?
func (pc *Postgres) Copy(records [][]interface{}) error {
tx, err := pc.db.Begin()
if err != nil {
return errors.Wrap(err, "can't open transaction")
}
stmt, err := tx.Prepare(pq.CopyIn(pc.table, pc.columns...))
if err != nil {
return errors.Wrap(err, "can't prepare stmt")
}
for _, record := range records {
if _, err := stmt.Exec(record...); err != nil {
return errors.Wrap(err, "error exec record")
}
}
if _, err = stmt.Exec(); err != nil {
return errors.Wrap(err, "error exec stmt")
}
if err = stmt.Close(); err != nil {
return errors.Wrap(err, "error close stmt")
}
if err = tx.Commit(); err != nil {
return errors.Wrap(err, "error commit transaction")
}
return nil
}
As far as I understand if \copy fails transaction will be aborted(link) and rolled back.
However in officials lib/pq examples I see they always use rollback(but they have more than one operation).
Could somebody please guide me through these nuances?
It looks like part of the confusion is because \COPY is not the same thing as COPY.
\COPY (as referenced in this question) is a command in psql that "Performs a frontend (client) copy". In simple terms you run psql (a terminal-based front-end to PostgreSQL) on a computer and can \COPY data between the database and a file stored locally (or accessible from your machine). This command is part of psql and not something you can use via lib/pq.
COPY is a Postgres SQL command that runs on the server; the file you are copying "must be accessible by PostgreSQL user (the user ID the server runs as) and the name must be specified from the viewpoint of the server". This is what you are calling in your application (from the pq docs "CopyIn uses COPY FROM internally" implementation here and here)
So, as the answer referenced by #mh-cbon states:
the deferred rollback is there to ensure that the transaction is rolled back in case of an early return.
Consider:
tx, err := pc.db.Begin()
if err != nil {
return errors.Wrap(err, "can't open transaction")
}
stmt, err := tx.Prepare(pq.CopyIn(pc.table, pc.columns...))
if err != nil {
return errors.Wrap(err, "can't prepare stmt")
}
If the Prepare fails you have created a transaction and then return without closing it; this leaves the transaction open which is not a good thing. Adding a defer tx.Rollback() ensures that does not happen.
Related
I have the two tables in postgres database, which is inside docker - data_table where is stored usual types and related data_files_table where is stored file data in bytea type. When I'm inserting in data_table and data_files_table in one transaction from golang service using sqlx, sometimes I get an error:
unexpected EOF
on query of insert to data_files_table. Often, it occurs on data bigger than 7 MB.
In postgres logs I found row, that might be relates:
could not receive data from client: Connection reset by peer
Why this occurs? Is it relate to TOAST, tcp connection killing, or something else?
Here is my code:
func (r *repo) CreateGrantWithDocuments(
ctx context.Context,
model models.Grant,
documents []models.GrantDocument,
) (*pgtype.UUID, error) {
tx, err := r.db.Beginx() // sqlx with pgx driver
if err != nil {
return nil, fmt.Errorf("db.Beginx: %w", err)
}
defer helpdb.Rollback(ctx, r.log, tx)
var grantID pgtype.UUID
err = tx.GetContext(
ctx, &grantID, createGrantSQL, args...) // args doesn't matter, it's ok; query returns id
if err != nil {
return nil, fmt.Errorf("tx.GetContext: %w", err)
}
for _, document := range documents {
data, err := io.ReadAll(document.File)
if err != nil {
return nil, fmt.Errorf("io.ReadAll: %w", err)
}
if err = document.File.Close(); err != nil { // file is an io.ReadCloser from multipart/form-data request
return nil, fmt.Errorf("document.File.Close: %w", err)
}
_, err = tx.NamedExec(createGrantDocumentSQL, document) // ERROR: unexpected EOF
if err != nil {
return nil, fmt.Errorf("tx.NamedExec: %w", err)
}
}
if err = tx.Commit(); err != nil {
return nil, fmt.Errorf("tx.Commit: %w", err)
}
return &grantID, nil
}
I'm using dependencies:
go 1.19
github.com/jmoiron/sqlx v1.3.4 // db connects
github.com/jackc/pgx/v4 v4.14.0 // db driver
I already changed the config on app-side: set idle and open connections count and lifetime to different values, disable idle, increase and decrease lifetime, but nothing helps.
Error doesn't occur locally, only on dev-server.
This problem relates limits of docker container by cgroups. docker stats shows that memory limit is not exceeded, but in fact it is exceeded, because this command doesn't see some types of memory that postgres uses, but cgroups see.
Also, this problem may relates to network problems, but not in my case.
Use rss+cache memory types in monitoring for prevent this problem.
In my golang project that use gorm as ORM and posgress as database, in some sitution when I begin transaction to
change three tables and commiting, just one of tables changes. two other tables data does not change.
any idea how it might happen?
you can see example below
o := *gorm.DB
tx := o.Begin()
invoice.Number = 1
err := tx.Save(&invoice)
if err != nil {
err2 := tx.RollBack().Error()
return err
}
receipt.Ref = "1331"
err = tx.Save(&receipt)
if err != nil {
err2 := tx.RollBack().Error()
return err
}
payment.status = "succeed"
err = tx.Save(&payment)
if err != nil {
err2 := tx.RollBack().Error()
return err
}
err = tx.Commit()
if err != nil {
err2 := tx.Rollback()
return err
}
Just payment data changed and I'm not getting any error.
Apparently you are mistakenly using save points! In PostgreSQL, we can have nested transactions, that is, defining save points make the transaction split into parts. I am not a Golang programmer and my primary language is not Go, but as I guess the problem is "tx.save" which makes a SavePoint, and does not save the data into database. SavePoints makes a new transaction save point, and thus, the last table commits.
If you are familiar with the Node.js, then any async function callback returns an error as the first argument. In Go, we follow the same norm.
https://medium.com/rungo/error-handling-in-go-f0125de052f0
I am trying to create a function that InsertOne data by wrap transaction, and I already to replica set in mongodb also, I tried in local and and MongoDB atlas the error were same,
here is the code:
const MONGODB_URI = "mongodb://localhost:27017,localhost:27018,localhost:27019/?replicaSet=rs"
ctx := context.Background()
client, err := mongo.Connect(ctx, options.Client().ApplyURI(MONGODB_URI))
if err != nil {
panic(err)
}
db := client.Database("production")
defer db.Client().Disconnect(ctx)
col := db.Collection("people")
// transaction
err = db.Client().UseSession(ctx, func(sessionContext mongo.SessionContext) error {
err := sessionContext.StartTransaction()
if err != nil {
fmt.Println(err)
return err
}
_, err = col.InsertOne(sessionContext, req)
if err != nil {
sessionContext.AbortTransaction(sessionContext)
fmt.Println(err)
return err
} else {
sessionContext.CommitTransaction(sessionContext)
}
return nil
})
if err != nil {
return nil, err
}
I have follow the instructions of this question in Stackoverflow and I have tried also ways from this article mongodb developer
what I got is this error:
(NamespaceNotFound) Cannot create namespace
production.people in multi-document transaction.
and
multiple write errors: [{write errors:
[{Cannot create namespace spura.people in multi-document transaction.}]},
{<nil>}]"
it got error when I inserted data , is that something wrong in my code? I have tried look carefully and try the instruction of document or articles and always got that error :(
MongoDB 4.2 and lower does not permit creating collections in transactions. This restriction is lifted in 4.4.
For 4.2 and lower, create the collection ahead of time.
I am working on a go project where I need to serve files stored in mongodb. The files are stored in a GridFs. I use gopkg.in/mgo.v2 as package to connect and query the db.
I can retrieve the file from the db, that is not hard.
f, err := s.files.OpenId(id)
But how can I serve that file with http?
I work with the JulienSchmidt router to handle all the other restfull requests.
The solutions I find always use static files, not files from a db.
Thanks in advance
Tip: Recommended to use github.com/globalsign/mgo instead of gopkg.in/mgo.v2 (the latter is not maintained anymore).
The mgo.GridFile type implements io.Reader, so you could use io.Copy() to copy its content into the http.ResponseWriter.
But since mgo.GridFile also implements io.Seeker, you may take advantage of http.ServeContent(). Quoting its doc:
The main benefit of ServeContent over io.Copy is that it handles Range requests properly, sets the MIME type, and handles If-Match, If-Unmodified-Since, If-None-Match, If-Modified-Since, and If-Range requests.
Example handler serving a file:
func serveFromDB(w http.ResponseWriter, r *http.Request) {
var gridfs *mgo.GridFS // Obtain GridFS via Database.GridFS(prefix)
name := "somefile.pdf"
f, err := gridfs.Open(name)
if err != nil {
log.Printf("Failed to open %s: %v", name, err)
http.Error(w, "something went wrong", http.StatusInternalServerError)
return
}
defer f.Close()
http.ServeContent(w, r, name, time.Now(), f) // Use proper last mod time
}
its old but i got another solution with goMongo driver by importing
"go.mongodb.org/mongo-driver/mongo/gridfs"
var bucket *gridfs.Bucket //creates a bucket
dbConnection, err := db.GetDBCollection() //connect db with your your
if err != nil {
log.Fatal(err)
}
bucket, err = gridfs.NewBucket(dbConnection)
if err != nil {
log.Fatal(err)
}
name := "br100_update.txt"
downloadStream, err := bucket.OpenDownloadStreamByName(name)
if err != nil {
log.Printf("Failed to open %s: %v", name, err)
http.Error(w, "something went wrong", http.StatusInternalServerError)
return
}
defer func() {
if err := downloadStream.Close(); err != nil {
log.Fatal(err)
}
}()
// Use SetReadDeadline to force a timeout if the download does not succeed in
// 2 seconds.
if err = downloadStream.SetReadDeadline(time.Now().Add(2 * time.Second)); err
!= nil {
log.Fatal(err)
}
// this code below use to read the file
fileBuffer := bytes.NewBuffer(nil)
if _, err := io.Copy(fileBuffer, downloadStream); err != nil {
log.Fatal(err)
}
I'm quite new to both PostgreSQL and golang. Mainly, I am trying to understand the following:
Why did I need the Commit statement to close the connection and the other two Close calls didn't do the trick?
Would also appreciate pointers regarding the right/wrong way in which I'm going about working with cursors.
In the following function, I'm using gorp to make a CURSOR, query my Postgres DB row by row and write each row to a writer function:
func(txn *gorp.Transaction,
q string,
params []interface{},
myWriter func([]byte, error)) {
cursor := "DECLARE GRABDATA NO SCROLL CURSOR FOR " + q
_, err := txn.Exec(cursor, params...)
if err != nil {
myWriter(nil, err)
return
}
rows, err := txn.Query("FETCH ALL in GRABDATA")
if err != nil {
myWriter(nil, err)
return
}
defer func() {
if _, err := txn.Exec("CLOSE GRABDATA"); err != nil {
fmt.Println("Error while closing cursor:", err)
}
if err = rows.Close(); err != nil {
fmt.Println("Error while closing rows:", err)
} else {
fmt.Println("\n\n\n Closed rows without error", "\n\n\n")
}
if err = txn.Commit(); err != nil {
fmt.Println("Error on commit:", err)
}
}()
pointers := make([]interface{}, len(cols))
container := make([]sql.NullString, len(cols))
values := make([]string, len(cols))
for i := range pointers {
pointers[i] = &container[i]
}
for rows.Next() {
if err = rows.Scan(pointers...); err != nil {
myWriter(nil, err)
return
}
stringLine := strings.Join(values, ",") + "\n"
myWriter([]byte(stringLine), nil)
}
}
In the defer section, I would initially, only Close the rows, but then I saw that pg_stat_activity stay open in idle in transaction state, with the FETCH ALL in GRABDATA query.
Calling txn.Exec("CLOSE <cursor_name>") didn't help. After that, I had a CLOSE GRABDATA query in idle in transaction state...
Only when I started calling Commit() did the connection actually close. I thought that maybe I need to call Commit to execute anything on the transation, but if that's the case - how come I got the result of my queries without calling it?
you want to end transaction, not close a declared cursor. commit does it.
you can run multiple queries in one transaction - this is why you see the result without committing.
the pg_stat_activity.state values are: active when you run the statement (eg, begin transaction; or fetch cursos), idle in transaction when you don't currently run statements, but the transaction remains begun and lastly idle, after you run end or commit, so the transaction is over. After you disconnect the session ends and there's no row in pg_stat_activity at all...