So... this was apperantly an issue a few years ago, atleast there are several bugs with what I think is the same issue. In a method I have 2 transactions, one after the other (so they are not concurrent, they are sequencial) and the second transaction always fails.
This is roughly the code:
import (
"database/sql"
_ "github.com/lib/pq"
)
var db *sql.DB
func InitStorage() {
var err error
db, err = sql.Open(os.Getenv("DB_CONNECTION_DRIVER"), os.Getenv("DB_CONNECTION_STRING"))
if err != nil {
glog.Error(err)
}
if db == nil {
glog.Fatal(db)
}
tx, err := db.Begin()
if err != nil {
glog.Error(err)
} else {
glog.Info("ERROR ON BEGIN IS NiLL!!!")
}
_, err = tx.Query(`INSERT INTO urls(url_hash, url) VALUES($1, $2)`, `asdadsaaa`, `1313`)
if err != nil {
glog.Error(err)
tx.Rollback()
}
tx.Commit()
glog.Info("SECOND TRANSACTION!!!")
tx1, err := db.Begin()
if err != nil {
glog.Error("SECOND TRANSACTION HAS ERR: ", err)
} else {
glog.Info("ERROR ON BEGIN IS NiLL!!!")
}
_, err = tx1.Query(`INSERT INTO urls(url_hash, url) VALUES($1, $2)`, `asdadsaaa`, `1313`)
if err != nil {
glog.Error(err)
tx1.Rollback()
}
tx1.Commit()
}
And this is always the output (I have cut the end, since it doesn't bring useful information):
I0324 00:24:26.192256 11580 storage_rdb.go:33] ERROR ON BEGIN IS NiLL!!!
I0324 00:24:26.195134 11580 storage_rdb.go:42] SECOND TRANSACTION!!!
E0324 00:24:26.195197 11580 storage_rdb.go:45] SECOND TRANSACTION HAS ERRpq: unexpected transaction status idle in transaction
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x20 pc=0x5460cb]
goroutine 1 [running]:
panic(0x7c41c0, 0xc820010150)
/usr/local/go/src/runtime/panic.go:464 +0x3e6
database/sql.(*Tx).Query(0x0, 0x8b2e00, 0x2e, 0xc82011fd50, 0x2, 0x2, 0xc82001a540, 0x0, 0x0)
/usr/local/go/src/database/sql/sql.go:1404 +0x3b
main.InitStorage()
I assume I am doing something wrong(since the bug was found in 2013), but I can't figure out what it is. Please help.
Related
My application is using pgx to running database queries in a goroutines. However, I am getting connection busy errors. Is there a way to have a goroutine
func writeDb(dbconn *pgx.Conn) {
sqlWritePost := `QUERY_HERE`
_, err := dbconn.Exec(context.Background(), sqlWritePost, v.Url, v.Content, v.StrippedContent, v.Posthash)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
}
func main() {
var dbconn *pgx.Conn
dbconn, err := pgx.Connect(context.Background(), os.Getenv("database_string"))
if err != nil {
fmt.Println(err)
os.Exit(1)
}
...
go writeDb(dbconn)
...
}
I am receiving errors conn busy. Is there a way to structure my code to avoid this issue?
Thanks!
I have created a mongodb replica set. I am able to run transactions from the mongo shell. But when I try to do it using mongo-go-driver I always get this error (IllegalOperation) Transaction numbers are only allowed on a replica set member or mongos. I am not sure where I am going wrong. I am using this as a reference https://github.com/simagix/mongo-go-examples/blob/master/examples/transaction_test.go.
I create the client like this
client, err := mongo.NewClient(options.Client().ApplyURI("mongodb://localhost:27017,localhost:27018,localhost:27019?replicaSet=rs"))
I can connect to the individual mongodb instances, just not the replica set.
This is the transaction that I am trying to run
var session mongo.Session
coll := db.Collection("collectionname")
if session, err = client.StartSession(); err != nil {
return "", fmt.Errorf("Could not start session: %q", err)
}
if err = session.StartTransaction(); err != nil {
return "", fmt.Errorf("Could not start Transaction: %q", err)
}
if err = mongo.WithSession(ctx, session, func(sc md.SessionContext) error {
newVal, err = coll.InsertOne(sc, val) // some val that I have
if err != nil {
sc.AbortTransaction(sc)
return fmt.Errorf("Error during New address creation, aborting: %q", err)
}
if err = sc.CommitTransaction(sc); err != nil {
return fmt.Errorf("Error While commiting New address Transaction: %q", err)
}
return nil
}); err != nil {
return "", err
}
session.EndSession(ctx)
Is there something I am missing. Is there some other example maybe that I can reference. Thanks for the help/suggestions.
This is an issue with your connection code - not your transaction implementation most likely. Try using the more modern connection string for connecting to a replica set:
connectionString := "mongodb+srv://USERNAME:PASSWORD#mongoatlas.1mxpg.mongodb.net/?retryWrites=true&w=majority"
var err error
Client, err = mongo.NewClient(options.Client().ApplyURI(ConnectionString))
if err != nil {
log.Fatal(err)
}
I'm having trouble finding some examples that do three of the following things:
1) Allow raw sql transactions in golang.
2) Use prepared statements.
3) Rollback on query failures.
I would like to do something like this, but with prepared statements.
stmt, stmt_err := db.Prepare(`
BEGIN TRANSACTION;
-- Insert record into first table.
INSERT INTO table_1 (
thing_1,
whatever)
VALUES($1,$2);
-- Inert record into second table.
INSERT INTO table_2 (
thing_2,
whatever)
VALUES($3,$4);
END TRANSACTION;
`)
if stmt_err != nil {
return stmt_err
}
res, res_err := stmt.Exec(
thing_1,
whatever,
thing_2,
whatever)
When I run this, I get this error:
pq: cannot insert multiple commands into a prepared statement
What gives? Are ACID compliant transactions even possible in golang? I cannot find an example.
EDIT
no examples here.
Yes Go has a great implementation of sql transactions. We start the transaction with db.Begin and we can end it with tx.Commit if everything goes good or with tx.Rollback in case of error.
type Tx struct { }
Tx is an in-progress database transaction.
A transaction must end with a call to Commit or Rollback.
After a call to Commit or Rollback, all operations on the transaction fail with ErrTxDone.
The statements prepared for a transaction by calling the transaction's Prepare or Stmt methods are closed by the call to Commit or Rollback.
Also note that we prepare queries with the transaction variable tx.Prepare(...)
Your function may looks like this:
func doubleInsert(db *sql.DB) error {
tx, err := db.Begin()
if err != nil {
return err
}
{
stmt, err := tx.Prepare(`INSERT INTO table_1 (thing_1, whatever)
VALUES($1,$2);`)
if err != nil {
tx.Rollback()
return err
}
defer stmt.Close()
if _, err := stmt.Exec(thing_1, whatever); err != nil {
tx.Rollback() // return an error too, we may want to wrap them
return err
}
}
{
stmt, err := tx.Prepare(`INSERT INTO table_2 (thing_2, whatever)
VALUES($1, $2);`)
if err != nil {
tx.Rollback()
return err
}
defer stmt.Close()
if _, err := stmt.Exec(thing_2, whatever); err != nil {
tx.Rollback() // return an error too, we may want to wrap them
return err
}
}
return tx.Commit()
}
I have a full example here
I came up with a possible solution to rollback on any failure without any significant drawbacks. I am pretty new to Golang though, I could be wrong.
func CloseTransaction(tx *sql.Tx, commit *bool) {
if *commit {
log.Println("Commit sql transaction")
if err := tx.Commit(); err != nil {
log.Panic(err)
}
} else {
log.Println("Rollback sql transcation")
if err := tx.Rollback(); err != nil {
log.Panic(err)
}
}
}
func MultipleSqlQuriesWithTx(db *sql.DB, .. /* some parameter(s) */) (.. .. /* some named return parameter(s) */, err error) {
tx, err := db.Begin()
if err != nil {
return
}
commitTx := false
defer CloseTransaction(tx, &commitTx)
// First sql query
stmt, err := tx.Prepare(..) // some raw sql
if err != nil {
return
}
defer stmt.Close()
res, err := stmt.Exec(..) // some var args
if err != nil {
return
}
// Second sql query
stmt, err := tx.Prepare(..) // some raw sql
if err != nil {
return
}
defer stmt.Close()
res, err := stmt.Exec(..) // some var args
if err != nil {
return
}
/*
more tx sql statements and queries here
*/
// success, commit and return result
commitTx = true
return
}
This question already has answers here:
How to use global var across files in a package?
(3 answers)
Closed 3 years ago.
I keep getting this error when I run my Go code which makes queries to my local postgres database.
Error:
panic serving [::1]:56708: runtime error: invalid memory address or nil pointer dereference
goroutine 23 [running]:
net/http.func·011()
/usr/local/go/src/pkg/net/http/server.go:1100 +0xb7
runtime.panic(0x2ef0a0, 0x4d8ee4)
/usr/local/go/src/pkg/runtime/panic.c:248 +0x18d
database/sql.(*DB).conn(0x0, 0x277a1, 0x0, 0x0)
/usr/local/go/src/pkg/database/sql/sql.go:625 +0x751
database/sql.(*DB).Ping(0x0, 0x0, 0x0)
/usr/local/go/src/pkg/database/sql/sql.go:452 +0x39
main.firstHandler(0x58e9a8, 0xc208052320, 0xc2080284e0)
/Users/Tommy/Documents/gocode/server/server.go:122 +0x35
net/http.HandlerFunc.ServeHTTP(0x3c6be8, 0x58e9a8, 0xc208052320, 0xc2080284e0)
/usr/local/go/src/pkg/net/http/server.go:1235 +0x40
github.com/gorilla/mux.(*Router).ServeHTTP(0xc2080186e0, 0x58e9a8, 0xc208052320, 0xc2080284e0)
/Users/Audrey/gocode/src/github.com/gorilla/mux/mux.go:98 +0x292
net/http.(*ServeMux).ServeHTTP(0xc208022660, 0x58e9a8, 0xc208052320, 0xc2080284e0)
/usr/local/go/src/pkg/net/http/server.go:1511 +0x1a3
net/http.serverHandler.ServeHTTP(0xc208004660, 0x58e9a8, 0xc208052320, 0xc2080284e0)
/usr/local/go/src/pkg/net/http/server.go:1673 +0x19f
net/http.(*conn).serve(0xc208050500)
/usr/local/go/src/pkg/net/http/server.go:1174 +0xa7e
created by net/http.(*Server).Serve
/usr/local/go/src/pkg/net/http/server.go:1721 +0x313
Go:
func firstHandler(w http.ResponseWriter, r *http.Request) {
err := db.Ping()
if err != nil {
log.Fatal(err)
}
rows, err := db.Query("SELECT id, created_at, updated_at FROM script WHERE updated_at = $1", 3)
if err != nil {
log.Fatal(err)
}
defer rows.Close()
var created_at, updated_at, id int
for rows.Next() {
err := rows.Scan(&id, &created_at, &updated_at)
if err != nil {
log.Fatal(err)
}
fmt.Fprintf("%s %s %s", id, created_at, updated_at)
}
}
var r = mux.NewRouter()
var db *sql.DB
func main() {
db, err := sql.Open("postgres", "user=Tommy host=localhost dbname=dbgo sslmode=verify-full")
if err != nil {
log.Fatal(err)
}
defer db.Close()
r.HandleFunc("/ping", firstHandler)
http.Handle("/", r)
http.ListenAndServe(":8080", nil)
}
Help. What am I doing wrong? I referred to this also: https://gophercasts.io/lessons/4-postgres-basics.
Actually, you declare the connection with:
var db *sql.DB
but you open the connection with:
db, err := sql.Open("postgres", "user=Tommy host=localhost dbname=dbgo sslmode=verify-full")
Note the := (it combines a variable declaration with an assignment). This will actually shadow the global db variable by a local one. The connection is opened but assigned to the local variable. So the value of the global db variable is nil.
When the firstHandler function is called, its value is still nil, which triggers the panic.
Replace the := by a = (and declare the err object before).
My GO version is 1.1.1
the sever recieved messages after connection close, but NoDelay was setted.
Is there something wrong
addr, _ := net.ResolveTCPAddr("tcp", "localhost:5432")
conn, err := net.DialTCP("tcp", nil, addr)
defer conn.Close()
if err != nil {
fmt.Println("connect fail")
return
}
err = conn.SetNoDelay(true)
if err != nil {
fmt.Println(err.Error())
}
for {
var message string
_, err := fmt.Scanln(&message)
if err != nil && err.Error() != "unexpected newline" {
fmt.Println("input finished", err)
break
}
if message == "" {
fmt.Println("no input, end")
break
}
// message = fmt.Sprintf("%s\n",message)
//fmt.Fprintf(conn, message) // send immediately but following message won't send any more
conn.Write([]byte(message)) // won't send until connection close
}
There doesn't seem to be anything vitally wrong with your code so I'm guessing the error is on the server end.
If you create a local TCP server on port 5432 you can test this.
Try running the below server code and then test your client code against it. It just echos all received data to stdout.
package main
import (
"io"
"log"
"net"
"os"
)
func main() {
l, err := net.Listen("tcp", "localhost:5432")
if err != nil {
log.Fatal(err)
}
defer l.Close()
for {
conn, err := l.Accept()
if err != nil {
log.Fatal(err)
}
go func(c net.Conn) {
defer c.Close()
io.Copy(os.Stdout, c)
}(conn)
}
}
You should see each line sent to the client printed (without the newline) as soon as you hit enter.
the problem is on the server end.
func handleConnection(conn net.Conn) {
// I didn't put it in for loop
message, err := bufio.NewReader(conn).ReadString('\n')
}