I am trying to use the official mongodb driver in golang and am seeing something unexpected.
If I have a struct like
type User struct {
ID primitive.ObjectID `json:"id" bson:"_id"`
Name string `json:"name" bson:"name"`
Email string `json:"email" bson:"email"`
}
I create a new instance of this with Name and Email but omit ID expecting that the DB will fill this with its value. Instead it uses all zeroes and so the second and so on inserts fail with
multiple write errors: [{write errors: [{E11000 duplicate key error collection: collection.name index: _id_ dup key: { : ObjectId('000000000000000000000000') }}]}, {<nil>}]
If I use a *primitive.ObjectID I get the same class of error only on null instead of zeroes
multiple write errors: [{write errors: [{E11000 duplicate key error collection: collection.name index: _id_ dup key: { : null }}]}, {<nil>}]
It doesn't matter if I use the omitempty directive or not, same result.
If I omit the ID field entirely, it works, but then my struct doesn't have that data on it.
Is there a way to have the DB handle the ID? Or MUST I explicitly call the NewObjectID() function on the struct?
It doesn't matter if I use the omitempty directive or not, same result.
omitempty tag on ID should work. For example:
type User struct {
ID primitive.ObjectID `json:"id" bson:"_id,omitempty"`
Name string `json:"name" bson:"name"`
Email string `json:"email" bson:"email"`
}
collection.InsertOne(context.Background(), User{Name:"Foo", Email:"Baz"})
If you don't specify the omitepmty tag, then the behaviour that you observed is just Go structs behaviour; whereby if any of struct fields are omitted it will be zero-valued. In this case because you have specified the field type to be primitive.ObjectID, ObjectId('000000000000000000000000') is the zero value.
This is the reason why you need to generate a value first before inserting, i.e.:
collection.InsertOne(context.Background(),
User{ ID: primitive.NewObjectID(),
Name: "Foo",
Email: "Bar"})
Is there a way to have the DB handle the ID?
Technically, it's the MongoDB driver that automatically generates the ObjectId if it's not supplied before sending to server.
You can try to use bson.M instead of a struct when inserting to leave out the _id field, i.e.
collection.InsertOne(context.Background(),
bson.M{"name":"Foo", "email":"Bar"})
Code snippet above is written using mongo-go-driver v1.3.x
The omitempty struct tag should work:
type User struct {
ID primitive.ObjectID `json:"id" bson:"_id,omitempty"`
Name string `json:"name" bson:"name"`
Email string `json:"email" bson:"email"`
}
The primitive.ObjectID type implements the bsoncodec.Zeroer interface, so it should be omitted from the document if it's the empty object ID (all 0's) and the driver will generate a new one for you. Can you try this and post the output?
Related
Let's assume that I have two models,
type Customer struct {
Id int `json:"id" db:"id"`
Name string `json:"name" db:"name"`
Address Address `json:"adress"`
}
type Address struct {
Street string `json:"street" db:"street"`
City string `json:"city" db:"city"`
}
// ...
customer := models.Customer{}
err := db.Get(&customer , `select * from users where id=$1 and name=$2`, id, name)
But this scan throws an error as: missing destination name street in *models.Customer
Am I doing something wrong? As you can see I already updated the db corresponding of the value. I doubled check so case sensitivity shouldn't be a problem.
Or is it not possible using https://github.com/jmoiron/sqlx?
I can see it in the documentation but still couldn't figure out how to solve it.
http://jmoiron.github.io/sqlx/#advancedScanning
The users table is declared as:
CREATE TABLE `users` (
`id` varchar(256) NOT NULL,
`name` varchar(150) NOT NULL,
`street` varchar(150) NOT NULL,
`city` varchar(150) NOT NULL,
)
The very link you posted gives you an hint about how to do this:
StructScan is deceptively sophisticated. It supports embedded structs, and assigns to fields using the same precedence rules that Go uses for embedded attribute and method access
So given your DB schema, you can simply embed Address into Customer:
type Customer struct {
Id int `json:"id" db:"id"`
Name string `json:"name" db:"name"`
Address
}
In your original code, Address was a field with its own db tag. This is not correct, and by the way your schema has no address column at all. (it appears you edited it out of your code snippet)
By embedding the struct into Customer instead, Address fields including tags are promoted into Customer and sqlx will be able to populate them from your query result.
Warning: embedding the field will also flatten the output of any JSON marshalling. It will become:
{
"id": 1,
"name": "foo",
"street": "bar",
"city": "baz"
}
If you want to place street and city into a JSON address object as based on your original struct tags, the easiest way is probably to remap the DB struct to your original type.
You could also scan the query result into a map[string]interface{} but then you have to be careful about how Postgres data types are represented in Go.
I had the same problem and came up with a slightly more elegant solution than #blackgreen's.
He's right, the easiest way is to embed the objects, but I do it in a temporary object instead of making the original messier.
You then add a function to convert your temp (flat) object into your real (nested) one.
type Customer struct {
Id int `json:"id" db:"id"`
Name string `json:"name" db:"name"`
Address Address `json:"adress"`
}
type Address struct {
Street string `json:"street" db:"street"`
City string `json:"city" db:"city"`
}
type tempCustomer struct {
Customer
Address
}
func (c *tempCustomer) ToCustomer() Customer {
customer := c.Customer
customer.Address = c.Address
return customer
}
Now you can scan into tempCustomer and simply call tempCustomer.ToCustomer before you return. This keeps your JSON clean and doesn't require a custom scan function.
I have a Task type that has a list of Runner type objects in it. I am trying to map it to database using golang gorm but it doesn't have foreign key and i am getting invalid association during migration
My Task struct:
type Task struct {
gorm.Model
Name string `gorm:"not null;unique_index"`
Description string
Runners []Runner
}
My Runner struct:
type Runner struct {
gorm.Model
Name string `gorm:"not null;unique"`
Description string
}
My migration code:
func migrateSchema () (err error) {
db, err := context.DBProvider()
if err != nil {
return
}
db.Model(&Task{}).Related(&Runner{})
db.AutoMigrate(&Task{})
db.AutoMigrate(&Runner{})
return
}
On db.AutoMigrate(&Task{}) I get invalid association message in console and when I check the database there is no foreign key created or no reference field created on runners table
What am I doing wrong?
I had a similar issue, and it took me forever to figure it out. I believe the GORM documentation could definitely be better. Here's the relevant code snippet from the GORM site:
//User has many emails, UserID is the foreign key
type User struct {
gorm.Model
Emails []Email
}
type Email struct {
gorm.Model
Email string
UserID uint
}
db.Model(&user).Related(&emails)
//// SELECT * FROM emails WHERE user_id = 111; // 111 is user's primary key
Why your code isn't working:
First you need to add a TaskID field to your Runner struct.
db.Model(&Task{}).Related(&Runner{}) doesn't do what you think it does. If you look at the code snippet from GORM, the SELECT comment kind of explains it (not very well though). The example is assuming that the &user is already populated and has an ID of 111, then it fetches the emails storing them in &emails that match the UserID of &user.
I'm very new to golang and try to write a simple event-sourcing user-management webapi using mongodb as backing database. Now i have User, which looks something like this:
type User struct {
Id bson.ObjectId `json:"id" bson:"_id"`
UserName string `json:"username" bson:"username"`
Email string `json:"email" bson:"email"`
PwdHash string `json:"pwd_hash" bson:"pwd_hash"`
FullName string `json:"fullname" bson:"fullname"`
}
and three events, happening to user, when somebody uses api:
type UserCreatedEvent struct {
UserId bson.ObjectId `json:"id" bson:"_id"`
//time when event was issued
CreatedEpoch time.Time `json:"created_epoch" bson:"created_epoch"`
//id of user that made a change
IssuedByUserId bson.ObjectId `json:"issuedby_userid" bson:"issuedby_userid"`
}
type UserDeletedEvent struct {
UserId bson.ObjectId `json:"id" bson:"_id"`
CreatedEpoch time.Time `json:"created_epoch" bson:"created_epoch"`
//id of user that made a change
IssuedByUserId bson.ObjectId `json:"issuedby_userid" bson:"issuedby_userid"`
}
type UserUpdatedEvent struct {
UserId bson.ObjectId `json:"id" bson:"_id"`
CreatedEpoch time.Time `json:"created_epoch" bson:"created_epoch"`
//id of user that made a change
IssuedByUserId bson.ObjectId `json:"issuedby_userid" bson:"issuedby_userid"`
ChangedFieldName string `json:"changed_field_name" bson:"changed_field_name"`
NewChangedFieldValue string `json:"new_changed_field_value" bson:"new_changed_field_value"`
}
Now i'm stuck on saving and retrieving events from db. The problem is i want to store them in a single collection, so that i have a full plain history of user modifications. But i can't find how to correctly store event type name as a mongo document field and then use it in searches. What is the idiomatic go-way to do this?
I'll be gratefull for any help.
What's nice about a non-relational database is that a little redundancy is OK and is faster for retrieval since you're not joining things together. You could just add your bottom three objects as properties on your User in whatever fashion makes sense for you and your data. Then you wouldn't need to store the UserId on those objects either.
If you need to search over the Events quickly, you could create another collection to hold them. You'd be inserting to two collections, but retrieval times/logic should be pretty good/easy to write.
Your new Event would be something like:
type UserEventType int
const (
Created UserEventType = iota
Deleted
Updated
)
type UserEvent struct {
UserId bson.ObjectId `json:"id" bson:"_id"`
CreatedEpoch time.Time `json:"created_epoch" bson:"created_epoch"`
//id of user that made a change
IssuedByUserId bson.ObjectId `json:"issuedby_userid" bson:"issuedby_userid"`
ChangedFieldName string `json:"changed_field_name,omitempty" bson:"changed_field_name,omitempty"`
NewChangedFieldValue string `json:"new_changed_field_value,omitempty" bson:"new_changed_field_value,omitempty"`
EventType UserEventType `json:"user_event_type" bson:"user_event_type"`
}
Notice the omitempty on the fields that are optional depending on the type of event.
You really aren't supposed to store different objects in the same collection. Here's a line from the Mongo documentation:
MongoDB stores documents in collections. Collections are analogous to tables in relational databases.
In case you aren't familiar with relational databases, a table would essentially represent one type of object.
I have the following document in my local mongodb:
_id 25dd9d29-efd5-4b4e-8af0-360c49fdba31
name Reykjavik
initialDiseaseColouring blue
In my code I set up a city structure as the following:
type City struct {
ID bson.ObjectId `bson:"_id,omitempty"`
Name string
InitialDiseaseColouring string
}
I'm querying it using
result := City{}
collection.Find(bson.M{"name":"Reykjavik"}).One(&result)
When I try to access the initialDiseaseColouring attribute it's not there
This is result when I print it:
{ObjectIdHex("32356464396432392d656664352d346234652d386166302d333630633439666462613331") Reykjavik }
Does anyone know why?
I was following the example on https://gist.github.com/border/3489566
By default, the bson codec uses the lowercased field name as the key. Use the field tag to specify a different key:
type City struct {
ID bson.ObjectId `bson:"_id,omitempty"`
Name string
InitialDiseaseColouring string `bson:"initialDiseaseColouring"`
}
The addition of the field tag changes the key from "initialdiseasecolouring" to "initialDiseaseColouring".
I've written some simple program with golang and mgo. My question is how to properly to relationships in mgo.
1st Approach:
type User struct {
Id bson.ObjectId `json:"_id,omitempty" bson:"_id,omitempty"`
Username string `json:"username" bson:"username"`
Email string `json:"email" bson:"email"`
Password string `json:"password" bson:"password"`
Friends []User `json:"friends" bson:"friends"`
}
"Friends" is a slice of Users. I can $push a pointer to a User and it just works fine. The thing is that I only want to store a reference to the user and not nesting it:
2nd Approach:
type User struct {
Id bson.ObjectId `json:"_id,omitempty" bson:"_id,omitempty"`
Username string `json:"username" bson:"username"`
Email string `json:"email" bson:"email"`
Password string `json:"password" bson:"password"`
Friends []bson.ObjectId `json:"friends" bson:"friends"`
}
This gives me the output I want - but now it's not visible from the struct which nested structs are referenced. Does mgo provide some mechanism to deal with this?
mgo is a db driver library and not an ORM..
What I'd do is have the ids array as in the 2nd example (unexported, with lowercase) and have a Friends() method which queries the db by those ids and return a []User