How to handle duplicate unique index error? - mongodb

I'm using MongoDB. Code to add data to the collection:
type User struct {
Firstname string `json:"firstname" bson:"firstname"`
Lastname *string `json:"lastname,omitempty" bson:"lastname"`
Username string `json:"username" bson:"username"`
RegistrationDate primitive.DateTime `json:"registrationDate" bson:"registrationData"`
LastLogin primitive.DateTime `json:"lastLogin" bson:"lastLogin"`
}
var client *mongo.Client
func AddUser(response http.ResponseWriter, request *http.Request) {
collection := client.Database("hattip").Collection("user")
var user User
_ = json.NewDecoder(request.Body).Decode(&user)
insertResult, err := collection.InsertOne(context.TODO(), user)
if err != nil {
// here i need to get the kind of error.
fmt.Println("Error on inserting new user", err)
response.WriteHeader(http.StatusPreconditionFailed)
} else {
fmt.Println(insertResult.InsertedID)
response.WriteHeader(http.StatusCreated)
}
}
func main() {
client = GetClient()
err := client.Ping(context.Background(), readpref.Primary())
if err != nil {
log.Fatal("Couldn't connect to the database", err)
} else {
log.Println("Connected!")
}
router := mux.NewRouter()
router.HandleFunc("/person", AddUser).Methods("POST")
err = http.ListenAndServe("127.0.0.1:8080", router)
if err == nil {
fmt.Println("Server is listening...")
} else {
fmt.Println(err.Error())
}
}
func GetClient() *mongo.Client {
clientOptions := options.Client().ApplyURI("mongodb://127.0.0.1:27017")
client, err := mongo.NewClient(clientOptions)
if err != nil {
log.Fatal(err)
}
err = client.Connect(context.Background())
if err != nil {
log.Fatal(err)
}
return client
}
If I add a record with a username that already exists in the database, I get -
Error on inserting new user multiple write errors: [{write errors:
[{E11000 duplicate key error collection: hattip.user index:
username_unique dup key: { username: "dd" }}]}, {}]
in the line fmt.Println("Error on inserting new user", err) The record with the string dd in the username field is already there, and the username field is a unique index.
I want to be sure that the error is exact E11000 error (a repeating collection of key errors).
So far i compare err to whole error string that appears on duplication of a unique field, but it's completely wrong. If there is a way to get error code from err object, or there are other ways to solve this problem?
Also, i found mgo package, but to use it properly i have to learn it, rewrite current code and so on, but honestly, it looks good:
if mgo.IsDup(err) {
err = errors.New("Duplicate name exists")
}

According to the driver docs, InsertOne may return a WriteException, so you can check if the error is a WriteException, and if it is so, then check the WriteErrors in it. Each WriteError contains an error code.
if we, ok:=err.(WriteException); ok {
for _,e:=range we.WriteErrors {
// check e.Code
}
}
You can write an IsDup based on this.

Related

Mongo-Driver Should I use InsertMany or InsertOne

I am creating a Rest CRUD HTTP server written in Go. My error is I am getting this message from the database connection: "context deadline exceeded".
I have a CreateUsers() function to insert multiple users into the database. Currently, I am inserting one user at time:
func CreateUsers(users []*models.User) ([]primitive.ObjectID, error) {
client, ctx, cancel := database.GetConnection()
defer cancel()
defer client.Disconnect(ctx)
var userIds []primitive.ObjectID
if len(users) == 0 {
log.Printf("No users to create")
return userIds, errors.New("no users to create")
}
for _, user := range users {
user.ID = primitive.NewObjectID()
hashedPassword, err := utils.HashPassword(user.Password)
if err != nil {
log.Printf("Error while hashing password: %v", err)
return userIds, err
}
user.Password = hashedPassword
result, err := client.Database("users").Collection("users").InsertOne(ctx, user)
if err != nil {
log.Printf("Error while creating user: %v", err)
return userIds, err
}
oid := result.InsertedID.(primitive.ObjectID)
userIds = append(userIds, oid)
}
return userIds, nil
}
My database connection (database.GetConnection) is something like:
func GetConnection() (*mongo.Client, context.Context, context.CancelFunc) {
client, err := mongo.NewClient(options.Client().ApplyURI(connectionURI))
if err != nil {
log.Fatalf("error while creating client: %v", err)
}
ctx, cancel := context.WithTimeout(context.Background(), connectionTimeout*time.Second)
err = client.Connect(ctx)
if err != nil {
log.Fatalf("cluster connection error: %v", err)
}
err = client.Ping(ctx, nil)
if err != nil {
log.Fatalf("cluster ping error")
}
log.Println("connected to mongodb")
return client, ctx, cancel
}
Note: connectionTimeout is equal to 5.
I am not sure but the time exceeded error may be related to the InsertOne() approach. So, instead of focusing in solving that error I should be asking how to parse the []models.User into bson.D{} documents to pass them as parameters to InsertMany().
I think another advantage of using InsertMany approach is the query time will be way below. Any suggestions?

MutableSideEffect() panics when setting second value

We're in the process of writing a .NET Cadence client and are a bit confused with how MutableSideEffect() is supposed to work. We've been thinking of the ID being passed as essentially a variable name and that developers should be able to update mutable values in a workflow. When we try this though, the second MutableSideEffect() call fails with this panic:
panic: adding duplicate decision DecisionType: Marker, ID: MutableSideEffect_value-1, state=Created, isDone()=false, history=[Created]
We munged the greetings workflow sample to make these calls:
package main
import (
"fmt"
"math/rand"
"time"
"go.uber.org/cadence/activity"
"go.uber.org/cadence/workflow"
"go.uber.org/zap"
)
/**
* This greetings sample workflow executes 3 activities in sequential. It gets greeting and name from 2 different activities,
* and then pass greeting and name as input to a 3rd activity to generate final greetings.
*/
// ApplicationName is the task list for this sample
const ApplicationName = "greetingsGroup"
// This is registration process where you register all your workflows
// and activity function handlers.
func init() {
workflow.Register(SampleGreetingsWorkflow)
activity.Register(getGreetingActivity)
activity.Register(getNameActivity)
activity.Register(sayGreetingActivity)
}
// SampleGreetingsWorkflow Workflow Decider.
func SampleGreetingsWorkflow(ctx workflow.Context) error {
// Get Greeting.
ao := workflow.ActivityOptions{
ScheduleToStartTimeout: time.Minute,
StartToCloseTimeout: time.Minute,
HeartbeatTimeout: time.Second * 20,
}
ctx = workflow.WithActivityOptions(ctx, ao)
logger := workflow.GetLogger(ctx)
var greetResult string
err := workflow.ExecuteActivity(ctx, getGreetingActivity).Get(ctx, &greetResult)
if err != nil {
logger.Error("Get greeting failed.", zap.Error(err))
return err
}
f := func(ctx workflow.Context) interface{} {
return rand.Intn(100)
}
e := func(a, b interface{}) bool {
if a == b {
return true
}
return false
}
var result int
sideEffectValue := workflow.MutableSideEffect(ctx, "value-1", f, e)
err = sideEffectValue.Get(&result)
if err != nil {
panic(err)
}
logger.Debug("MutableSideEffect-1", zap.Int("Value", result))
//************** THIS CALL FAILS **************
sideEffectValue = workflow.MutableSideEffect(ctx, "value-1", f, e)
err = sideEffectValue.Get(&result)
if err != nil {
panic(err)
}
logger.Debug("MutableSideEffect-2", zap.Int("Value", result))
// Get Name.
var nameResult string
err = workflow.ExecuteActivity(ctx, getNameActivity).Get(ctx, &nameResult)
if err != nil {
logger.Error("Get name failed.", zap.Error(err))
return err
}
// Say Greeting.
var sayResult string
err = workflow.ExecuteActivity(ctx, sayGreetingActivity, greetResult, nameResult).Get(ctx, &sayResult)
if err != nil {
logger.Error("Marshalling failed with error.", zap.Error(err))
return err
}
logger.Info("Workflow completed.", zap.String("Result", sayResult))
return nil
}
// Get Name Activity.
func getNameActivity() (string, error) {
return "Cadence", nil
}
// Get Greeting Activity.
func getGreetingActivity() (string, error) {
return "Hello", nil
}
// Say Greeting Activity.
func sayGreetingActivity(greeting string, name string) (string, error) {
result := fmt.Sprintf("Greeting: %s %s!\n", greeting, name)
return result, nil
}
Are we thinking about this correctly?
This is a bug in the Go client library. It happens when a MutableSideEffect with the same id is used multiple times during a single decision.
If you force a separate decision by putting workflow.Sleep(ctx, time.Second) just before the second MutableSideEffect call the problem disappears.
I filed an issue to get this fixed.
Thanks a lot for reporting!

pq: sorry, too many clients already

I am getting pq: sorry, too many clients already error when I am calling the GetMessages() multiple times.
Please find the updated code:
main() code
func main() {
dbConn, err := InitDB()
if err != nil {
Log.Error("Connection Error: ", err.Error())
return
}
defer dbConn.Close()
go run()
var input string
fmt.Scanln(&input)
}
Database connection code is:
func InitDB()(*sql.DB, error) {
connectionString := fmt.Sprintf("user=%v password='%v' dbname=%v sslmode=disable", USER, PASSWORD, DATABASE)
db, err = sql.Open(DRIVER, connectionString)
return db, err
}
run goroutine:
func run() {
for {
messages, err := GetMessages()
if err != nil {
Log.Error("Connection Error: ", err.Error())
return
}
log.Info(messages)
}
}
GetMessages() function code:
func GetMessages() (messages []string, err error) {
rows, err := db.Query(`SELECT message1, message2, message3, message4, message5,
message6, message7, message8, message9, message10, message11, message12, message13,
message14, message15, message16, message17, message18, message19, message20, message21,
message22, message23, message24, message25, message26, message27, message28, message29,
message30, message31, message32, message33, message34, message35, message36, message37,
message38, message39, message40, message41, message42, message43, message44, message45,
message46, message47, message48 FROM table1 WHERE id=1`)
if err != nil {
Log.Error("Query error", err)
return messages, err
}
var pointers []interface{}
defer rows.Close()
for rows.Next() {
pointers = make([]interface{}, 48)
messages = make([]string, 48)
for i, _ := range pointers {
pointers[i] = &messages[i]
}
err = rows.Scan(pointers...)
if err != nil {
Log.Error("Failed to scan row", err)
return messages, err
}
}
return messages, nil
}
I checked this answer and I have used scan but still it isn't working
UPDATE
Issue was in another function. I was using db.Query without closing the returned rows object and was repeatedly calling that function. I've updated my code; used db.Exec instead of db.Query and it's working now. Thank you so much #mkopriva for this answer. :)
Try setting SetMaxOpenConns. The default is 0 (unlimited). This may be causing the issue. It would help if you also had SetConnMaxLifetime; otherwise, Postgres will start holding connections longer, and you will notice an increase in memory usage.
I've had the same problem with my postgres / golang project.
Eventually, this example worked flawlessly, without "eating" any DB connections:
// example params
firstName := "Jeremy"
lastName := "Baker"
// setup statement
stmt, err := db.Prepare(
`INSERT INTO user (
firstname,
lastname) VALUES($1, $2)
RETURNING id`) // id is the primary key of table: user
if err != nil {
return err
}
defer stmt.Close()
// execute statement
var userID string
err = stmt.QueryRow(
firstName,
lastName).Scan(&userID)
if err != nil {
return err
}

Duplicate Id in mongodb with go

I'm trying to code a simple web app in Go using Mongodb.
I've created a minimalistic simple Model / Controller setup.
I can create new user using POST with url "/user" with data such as '{"pseudo": "bobby1"}'. The user is created. However, when looking inside Mongodb shell, I get:
{ "_id" : ObjectId("5616d1ea56ca4dbc03bb83bc"), "id" : ObjectId("5616d1ea5213c64824000001"), "pseudo" : "bobby2"}
The "id" field is the one coming from my struct and the "_id" field is the one from Mongodb. From looking at different sample code, it looked like I could use myself the one from Mongodb but I can't find how to do it.. ><
Since the "id" is only used by me, I can't find user by their ID since I do not have that one...
More over, when I do a GET /user, it returns the full list of user, and I only get:
{"id":"5616d1ea5213c64824000001","pseudo":"bobby2"
Not the "real" Mongodb Id...
Thanks,
Here's the code:
model/user.go
const userCollection = "user"
// Get our collection
var C *mgo.Collection = database.GetCollection(userCollection)
// User represents the fields of a user in db
type User struct {
Id bson.ObjectId `json:"id"i bson:"_id,omitempty"`
Pseudo string `json:"pseudo" bson:"pseudo"`
}
// it will return every users in the db
func UserFindAll() []User {
var users []User
err := C.Find(bson.M{}).All(&users)
if err != nil {
panic(err)
}
return users
}
// UserFIndId return the user in the db with
// corresponding ID
func UserFindId(id string) User {
if !bson.IsObjectIdHex(id) {
s := fmt.Sprintf("Id given %s is not valid.", id)
panic(errors.New(s))
}
oid := bson.ObjectIdHex(id)
u := User{}
if err := C.FindId(oid).One(&u); err != nil {
panic(err)
}
return u
}
// UserCreate create a new user on the db
func UserCreate(u *User) {
u.Id = bson.NewObjectId()
err := C.Insert(u)
if err != nil {
panic(err)
}
}
controller/user.go
/ GetAll returns all users
func (us *UserController) GetAll(w http.ResponseWriter, request *http.Request) {
// u := model.User{
// ID: "987654321",
// Pseudo: "nikko",
// }
users := model.UserFindAll()
bu, err := json.Marshal(users)
if err != nil {
fmt.Printf("[-] Error while marshaling user struct : %v\n", err)
w.Write([]byte("Error Marshaling"))
w.WriteHeader(404)
return
}
w.Header().Set("Content-type", "application/json")
w.WriteHeader(200)
fmt.Fprintf(w, "%s\n", bu)
}
// Get return a specific user according to Id
func (us *UserController) Get(w http.ResponseWriter, request *http.Request) {
vars := mux.Vars(request)
ps := vars["id"]
u := model.UserFindId(ps)
bu, err := json.Marshal(u)
if err != nil {
fmt.Printf("[-] Error while marshaling user struct : %v\n", err)
w.Write([]byte("Error Marshaling"))
w.WriteHeader(404)
return
}
w.Header().Set("Content-type", "application/json")
w.WriteHeader(200)
fmt.Fprintf(w, "%s\n", bu)
}
func (us *UserController) Post(w http.ResponseWriter, r *http.Request) {
u := model.User{}
json.NewDecoder(r.Body).Decode(&u)
model.UserCreate(&u)
bu, err := json.Marshal(u)
if err != nil {
fmt.Printf("[-] Error PUT user struct : %v\n", err)
w.WriteHeader(404)
return
}
w.Header().Set("Content-type", "application/json")
w.WriteHeader(201)
fmt.Fprintf(w, "%s\n", bu)
}
Looks like you have a stray character in your struct tags:
type User struct {
Id bson.ObjectId `json:"id"i bson:"_id,omitempty"`
Pseudo string `json:"pseudo" bson:"pseudo"`
}
That i should not exist after json:"id". It should be:
type User struct {
Id bson.ObjectId `json:"id" bson:"_id,omitempty"`
Pseudo string `json:"pseudo" bson:"pseudo"`
}

how to fetch the unknown mongo doc via mgo

Here is the piece of code that is trying to fetch all the docs from the mongodb.
func fetchAll(db *mgo.Database) map[string]interface {
var msg map[string]interface{}
err := db.C("msg").Find(nil).All(&msg)
if err != nil {
panic(err)
}
return msg
}
I got the error: syntax error: unexpected var
What is wrong here? And is there a better way to fetch the arbitrary mongo docs via mgo?
thanks
First, fix the syntax error:
func fetchAll(db *mgo.Database) map[string]interface{} {
var msg map[string]interface{}
err := db.C("msg").Find(nil).All(&msg)
if err != nil {
panic(err)
}
return msg
}
Note the {} in the function return type declaration.
But there's more. All() retrieves all documents from the result set to a slice. Change the return type to a slice of maps:
func fetchAll(db *mgo.Database) []map[string]interface{} {
var msgs []map[string]interface{}
err := db.C("msg").Find(nil).All(&msgs)
if err != nil {
panic(err)
}
return msgs
}
While we are at it, let's return the error instead of panicking.
func fetchAll(db *mgo.Database) ([]map[string]interface{}, error) {
var msgs []map[string]interface{}
err := db.C("msg").Find(nil).All(&msgs)
return msgs, err
}