I have created a mongodb replica set. I am able to run transactions from the mongo shell. But when I try to do it using mongo-go-driver I always get this error (IllegalOperation) Transaction numbers are only allowed on a replica set member or mongos. I am not sure where I am going wrong. I am using this as a reference https://github.com/simagix/mongo-go-examples/blob/master/examples/transaction_test.go.
I create the client like this
client, err := mongo.NewClient(options.Client().ApplyURI("mongodb://localhost:27017,localhost:27018,localhost:27019?replicaSet=rs"))
I can connect to the individual mongodb instances, just not the replica set.
This is the transaction that I am trying to run
var session mongo.Session
coll := db.Collection("collectionname")
if session, err = client.StartSession(); err != nil {
return "", fmt.Errorf("Could not start session: %q", err)
}
if err = session.StartTransaction(); err != nil {
return "", fmt.Errorf("Could not start Transaction: %q", err)
}
if err = mongo.WithSession(ctx, session, func(sc md.SessionContext) error {
newVal, err = coll.InsertOne(sc, val) // some val that I have
if err != nil {
sc.AbortTransaction(sc)
return fmt.Errorf("Error during New address creation, aborting: %q", err)
}
if err = sc.CommitTransaction(sc); err != nil {
return fmt.Errorf("Error While commiting New address Transaction: %q", err)
}
return nil
}); err != nil {
return "", err
}
session.EndSession(ctx)
Is there something I am missing. Is there some other example maybe that I can reference. Thanks for the help/suggestions.
This is an issue with your connection code - not your transaction implementation most likely. Try using the more modern connection string for connecting to a replica set:
connectionString := "mongodb+srv://USERNAME:PASSWORD#mongoatlas.1mxpg.mongodb.net/?retryWrites=true&w=majority"
var err error
Client, err = mongo.NewClient(options.Client().ApplyURI(ConnectionString))
if err != nil {
log.Fatal(err)
}
Related
I am expected to have 80% test coverage even for pushing the basic project structure. I am a bit confused how do I write unit tests for the following code to Connect to postgres db and ping postgres for health check. Can someone help me please.
var postgres *sql.DB
// ConnectToPostgres func to connect to postgres
func ConnectToPostgres(connStr string) (*sql.DB, error) {
db, err := sql.Open("postgres", connStr)
if err != nil {
log.Println("postgres-client ", err)
return nil, err
}
postgres = db
return db, nil
}
// PostgresHealthCheck to ping database and check for errors
func PostgresHealthCheck() error {
if err := postgres.Ping(); err != nil {
return err
}
return nil
}
type PostgresRepo struct {
db *sql.DB
}
// NewPostgresRepo constructor
func NewPostgresRepo(database *sql.DB) *PostgresRepo {
return &PostgresRepo{
db: database,
}
}
You need to use this : https://github.com/DATA-DOG/go-sqlmock
Its very easy to use. Here is an example where a controller is getting tested using a mocked SQL :
Implementation
func (up UserProvider) GetUsers() ([]models.User, error) {
var users = make([]models.User, 0, 10)
rows, err := up.DatabaseProvider.Query("SELECT firstname, lastname, email, age FROM Users;")
if err != nil {
return nil, err
}
for rows.Next() {
var u models.User = models.User{}
err := rows.Scan(&u.Name, &u.Lastname, &u.Email, &u.Age)
if err != nil {
return nil, err
}
users = append(users, u)
}
if err := rows.Err(); err != nil {
return nil, err
}
return users, nil
}
Test
func TestGetUsersOk(t *testing.T) {
db, mock := NewMock()
mock.ExpectQuery("SELECT firstname, lastname, email, age FROM Users;").
WillReturnRows(sqlmock.NewRows([]string{"firstname", "lastname", "email", "age"}).
AddRow("pepe", "guerra", "pepe#gmail.com", 34))
subject := UserProvider{
DatabaseProvider: repositories.NewMockDBProvider(db, nil),
}
resp, err := subject.GetUsers()
assert.Nil(t, err)
assert.NotNil(t, resp)
assert.Equal(t, 1, len(resp))
}
func NewMock() (*sql.DB, sqlmock.Sqlmock) {
db, mock, err := sqlmock.New()
if err != nil {
log.Fatalf("an error '%s' was not expected when opening a stub database connection", err)
}
return db, mock
}
I find that writing tests against a live database makes for more high quality tests. The challenge with Postgres is that there's no good in-memory fake that you can substitute in.
What I came up with is standing up the postgres Docker container and creating temporary databases in there. The PostgresContainer type in the github.com/bitcomplete/sqltestutil package does exactly this:
# Postgres version is "12"
pg, _ := sqltestutil.StartPostgresContainer(context.Background(), "12")
defer pg.Shutdown(ctx)
db, err := sql.Open("postgres", pg.ConnectionString())
// ... execute SQL
Per the docs, it's a good idea to set up your tests so that the container is only started once, as it can take a few seconds to start up (more if the image needs to be downloaded). It suggests some approaches for mitigating that problem.
My application is using pgx to running database queries in a goroutines. However, I am getting connection busy errors. Is there a way to have a goroutine
func writeDb(dbconn *pgx.Conn) {
sqlWritePost := `QUERY_HERE`
_, err := dbconn.Exec(context.Background(), sqlWritePost, v.Url, v.Content, v.StrippedContent, v.Posthash)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
}
func main() {
var dbconn *pgx.Conn
dbconn, err := pgx.Connect(context.Background(), os.Getenv("database_string"))
if err != nil {
fmt.Println(err)
os.Exit(1)
}
...
go writeDb(dbconn)
...
}
I am receiving errors conn busy. Is there a way to structure my code to avoid this issue?
Thanks!
I am trying to create a function that InsertOne data by wrap transaction, and I already to replica set in mongodb also, I tried in local and and MongoDB atlas the error were same,
here is the code:
const MONGODB_URI = "mongodb://localhost:27017,localhost:27018,localhost:27019/?replicaSet=rs"
ctx := context.Background()
client, err := mongo.Connect(ctx, options.Client().ApplyURI(MONGODB_URI))
if err != nil {
panic(err)
}
db := client.Database("production")
defer db.Client().Disconnect(ctx)
col := db.Collection("people")
// transaction
err = db.Client().UseSession(ctx, func(sessionContext mongo.SessionContext) error {
err := sessionContext.StartTransaction()
if err != nil {
fmt.Println(err)
return err
}
_, err = col.InsertOne(sessionContext, req)
if err != nil {
sessionContext.AbortTransaction(sessionContext)
fmt.Println(err)
return err
} else {
sessionContext.CommitTransaction(sessionContext)
}
return nil
})
if err != nil {
return nil, err
}
I have follow the instructions of this question in Stackoverflow and I have tried also ways from this article mongodb developer
what I got is this error:
(NamespaceNotFound) Cannot create namespace
production.people in multi-document transaction.
and
multiple write errors: [{write errors:
[{Cannot create namespace spura.people in multi-document transaction.}]},
{<nil>}]"
it got error when I inserted data , is that something wrong in my code? I have tried look carefully and try the instruction of document or articles and always got that error :(
MongoDB 4.2 and lower does not permit creating collections in transactions. This restriction is lifted in 4.4.
For 4.2 and lower, create the collection ahead of time.
I have mongo capped collection and a simple API, written on Go. I built and run it. When I try to sent Get request or simply go localhost:8000/logger in browser - my process closes. Debug shows this happens, while executing "find" in collection. It produces error "client is disconnected". Collection has 1 document, and debug shows it is connected with my helper.
Go version 1.13
My code:
func main() {
r := mux.NewRouter()
r.HandleFunc("/logger", getDocs).Methods("GET")
r.HandleFunc("/logger", createDoc).Methods("POST")
log.Fatal(http.ListenAndServe("localhost:8000", r))
}
func getDocs(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
var docs []models.Logger
//Connection mongoDB with helper class
collection := helper.ConnectDB()
cur, err := collection.Find(context.TODO(), bson.M{})
if err != nil {
helper.GetError(err, w)
return
}
defer cur.Close(context.TODO())
for cur.Next(context.TODO()) {
var doc models.Logger
err := cur.Decode(&doc)
if err != nil {
log.Fatal(err)
}
docs = append(docs, doc)
}
if err := cur.Err(); err != nil {
log.Fatal(err)
}
json.NewEncoder(w).Encode(docs)
}
func ConnectDB() *mongo.Collection {
client, err := mongo.NewClient(options.Client().ApplyURI("mongodb://127.0.0.1:27017"))
if err != nil {
log.Fatal(err)
}
fmt.Println("Connected to MongoDB!")
logCollection := client.Database("local").Collection("loggerCollection")
return logCollection
}
According to the documentation, the call to mongo.NewClient doesn't ensure that you can connect the Mongo server. You should first call mongo.Client.Ping() to verify if you can connect to the database or not.
client, err := mongo.NewClient(options.Client().ApplyURI("mongodb://127.0.0.1:27017"))
if err != nil {
log.Fatal(err)
}
if err := client.Ping(context.TODO(), readpref.Primary()); err != nil {
// Can't connect to Mongo server
log.Fatal(err)
}
There could be several reasons behind failing to connect, the most obvious one is incorrect setup of ports. Is your mongodb server up and listening on port 27017? Is there any change you're running mongodb with Docker and it's not forwarding to the correct port?
I faced similar issue , read #Jay answer it definitely helped , as I checked my MongoDB was running using "MongoDB Compass" , then I changed the location of my insert statement , previously I was calling before the call of "context.WithTimeout". Below is working code.
package main
import (
"context"
"log"
"time"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)
type Book struct {
Name string `json:"name,omitempty"`
PublisherID string `json:"publisherid,omitempty"`
Cost string `json:"cost,omitempty"`
StartTime string `json:"starttime,omitempty"`
EndTime string `json:"endtime,omitempty"`
}
func main() {
client, err := mongo.NewClient(options.Client().ApplyURI("mongodb://localhost:27017"))
if err != nil {
log.Fatal(err)
}
ctx, _ := context.WithTimeout(context.Background(), 10*time.Second)
err = client.Connect(ctx)
if err != nil {
log.Fatal(err)
}
defer client.Disconnect(ctx)
testCollection := client.Database("BooksCollection").Collection("BooksRead")
inserRes, err := testCollection.InsertOne(context.TODO(), Book{Name: "Harry Potter", PublisherID: "IBN123", Cost: "1232", StartTime: "2013-10-01T01:11:18.965Z", EndTime: "2013-10-01T01:11:18.965Z"})
log.Println("InsertResponse : ", inserRes)
log.Println("Error : ", err)
}
I can see document inserted in console as well as in "MongoDB Comapass."
In heiper function "ConnectDB" after "NewClient" I must use "client.Connect(context.TODO())"
before any other use of client
Context
Currently I have a REST API that manages customer's data in a db. I'm using the following stack:
Go 1.13
github.com/jinzhu/gorm v1.9.1
Postgres 11
I have the following connection settings.
// NewConnection ...
func NewConnection() (*gorm.DB, error) {
config := getConfig()
connStr := "host=xx.xx.xx port=5432 user=chavista-hatter dbname=my-db password=abc sslmode=verify-ca sslrootcert=/path/to/rcert sslcert=/path/to/cert sslkey=/path/to/key connect_timeout=0"
db, err := gorm.Open("postgres", conn)
if err != nil {
return nil, err
}
db.DB().SetMaxOpenConns(25)
db.DB().SetMaxIdleConns(25)
db.DB().SetConnMaxLifetime(5 * time.Minute)
db.SingularTable(true)
if config.LogQueries {
db = db.Debug()
}
return db, nil
}
I get a connection in the main class and inject that connection into a repository class that executes the queries through Gorm (ORM)
Main class
db, err := database.NewConnection()
if err != nil {
panic(fmt.Sprintf("failed to connect database --> %v", err))
}
fmt.Println("database connection established successfully")
defer db.Close()
customerRepo := customer.NewRepository(db)
Repository class
type repository struct {
db *gorm.DB
}
//NewRepository
func NewRepository(db *gorm.DB) Repository {
return &repository{
db: db,
}
}
func (r *repository) Register(customer *models.Customer) (string, error) {
err := r.db.Create(&customer).Error
if err != nil {
return "", err
}
return customer.key, nil
}
Problem
I sending over 500k request (INSERTS) to my db which have 512 connections available and after a few minutes the following error starts to come up repeatedly in postgres log:
unexpected EOF on client connection with an open transaction
could not receive data from client: Connection reset by peer
Any help?
How are you using GetConnection in your code? It's creating a new connection pool every time it's called - ideally you'd only want to call it once, and pass that single connection around wherever it's used.
I would try changing it to this:
var db *gorm.DB
func NewConnection() (*gorm.DB, error) {
if db != nil {
return db, nil
}
config := getConfig()
connStr := "host=xx.xx.xx port=5432 user=chavista-hatter dbname=my-db password=abc sslmode=verify-ca sslrootcert=/path/to/rcert sslcert=/path/to/cert sslkey=/path/to/key connect_timeout=0"
var err error
db, err = gorm.Open("postgres", conn)
if err != nil {
return nil, err
}
db.DB().SetMaxOpenConns(25)
db.DB().SetMaxIdleConns(25)
db.DB().SetConnMaxLifetime(5 * time.Minute)
db.SingularTable(true)
if config.LogQueries {
db = db.Debug()
}
return db, nil
}
and see if it solves the issue.