Struggling with unmarshalling data in Golang from mongo, may be cause I am new to this. Just started learning golang with MongoDB
Tried with map[string]interface{} to avoid any struct related errors
var data map[string]interface{}
filter := bson.M{"profile.username": username}
singleResult := u.getCollection(client).FindOne(u.ctx, filter)
err := singleResult.Decode(data)
This fails to unmarshall with error cannot Decode to nil value
Tried with exact struct structure too.
var result *models.UserData
filter := bson.M{"profile.username": username}
singleResult := u.getCollection(client).FindOne(u.ctx, filter)
err := singleResult.Decode(result)
Fails with same error cannot Decode to nil value
Tried to find all with map[string]interface{}
var result []models.UserData
cursor, _ := u.getCollection(client).Find(u.ctx, bson.M{})
err := cursor.All(u.ctx, &result)
Works perfectly as expected
Tried to find all with exact struct structure
var data []map[string]interface{}
cursor, _ := u.getCollection(client).Find(u.ctx, bson.M{})
err := cursor.All(u.ctx, &result)
Works perfectly as expected
Now I thought may be I am not finding the data in mongo but then
filter := bson.M{"profile.username": username}
singleResult := u.getCollection(client).FindOne(u.ctx, filter)
raw, _ := singleResult.DecodeBytes()
log.Print("\n\n" + raw.String()+"\n\n")
This prints the data as expected. Although one thing I noticed all non-string values are formatted as {"$numberLong":"1"}. Still don't know if it is correct or cause of the issue.
In your first 2 examples that fail, the data passed to Decode() are both nil:
// data == nil
var data map[string]interface{}
// ...
// result == nil
var result *models.UserData
Try like
var result = &models.UserData{} // init the pointer with a block of valid allocated memory
// ...
err := singleResult.Decode(result)
In order for Decode() to write the document(s) into the passed value, it must be a (non-nil) pointer. Passing any value creates a copy, and if you pass a non-pointer, only the copy could be modified. If you pass a pointer, a copy is still made, but Decode() will modify the pointed value, not the pointer.
In your first 2 examples that fail, you pass a non-pointer (or a nil pointer):
err := singleResult.Decode(result)
Modify it to pass a (non-nil) pointer:
err := singleResult.Decode(&result)
Your last 2 examples work because you're already passing (non-nil) pointers:
err := cursor.All(u.ctx, &result)
Related
I'm trying to update a document in MongoDB with mongodb/mongo-go-driver. From its doc, one document can be replaced with:
var coll *mongo.Collection
var id primitive.ObjectID
// find the document for which the _id field matches id and add a field called "location"
// specify the Upsert option to insert a new document if a document matching the filter isn't found
opts := options.Replace().SetUpsert(true)
filter := bson.D{{"_id", id}}
replacement := bson.D{{"location", "NYC"}}
result, err := coll.ReplaceOne(context.TODO(), filter, replacement, opts)
if err != nil {
log.Fatal(err)
}
if result.MatchedCount != 0 {
fmt.Println("matched and replaced an existing document")
return
}
if result.UpsertedCount != 0 {
fmt.Printf("inserted a new document with ID %v\n", result.UpsertedID)
}
But what if I have fields more than like 20. I am wondering if I can update without re-writing all fields again, I tried something like this:
// Replacement struct
type Input struct {
Name *string `json:"name,omitempty" bson:"name,omitempty" validate:"required"`
Description *string `json:"description,omitempty" bson:"description,omitempty"`
Location *string `json:"location,omitempty" bson:"location,omitempty"`
}
...
oid, _ := primitive.ObjectIDFromHex(id) // because 'id' from request is string
filter := bson.M{"_id", oid} // throws `[compiler] [E] missing key in map literal`
// ^^^
replacement := bson.D{{"$set", input}} // throws `composite literal uses unkeyed fields`
// ^^^^^
result, err := coll.ReplaceOne(context.TODO(), filter, replacement, opts)
...
But it throws errors at filter and replacement. How can I replace a whole document properly?
bson.M is a map, so if you use it, you have to write: bson.M{"_id": oid}. See missing type in composite literal go AND missing key in map literal go.
And bson.D is a slice of structs, so if you use it, you should write bson.D{{Key:"$set", Value: input}}. Omitting the field names is not a compiler error, it's just a go vet warning.
Now on to replacing. The replacement must be a document itself, without using $set (this is not updating but replacing). For reference see MongoDB's collection.replaceOne() and the driver's doc: Collection.ReplaceOne():
The replacement parameter must be a document that will be used to replace the selected document. It cannot be nil and cannot contain any update operators (https://docs.mongodb.com/manual/reference/operator/update/).
So do it like this:
filter := bson.M{"_id": oid}
result, err := coll.ReplaceOne(context.TODO(), filter, input, opts)
Here is the complete code. This will help readers to understand the code flow.
func UpdateFunction(echoCtx echo.Context) (error) {
//Web Section
documentID := echoCtx.Param("id") //Provided in URL
var newObject myStruct
err := echoCtx.Bind(&newObject) // Json with updated values sent by web client mapped to struct
ctx := echoCtx.Request().Context()
//Database Section
database := db.Conn.Database("myDatabase")
collection := database.Collection("myCollection")
existingHexID, err := primitive.ObjectIDFromHex(documentID)
if err != nil {
fmt.Println("ObjectIDFromHex ERROR", err)
} else {
fmt.Println("ObjectIDFromHex:", existingHexID)
}
// Replacing OLD document with new using _id
filter := bson.M{"_id": newDocumentID}
result, err := collection.ReplaceOne(ctx, filter, newObject)
if err != nil {
log.Fatal(err)
}
fmt.Printf(
"insert: %d, updated: %d, deleted: %d /n",
result.MatchedCount,
result.ModifiedCount,
result.UpsertedCount,
)
return echoCtx.JSON(http.StatusOK, result.ModifiedCount)
}
I tried to follow documentation here and here but had no luck.
I want to get a singleResult from FindOne on the Collection named moviesCollection and then use Decode or Unmarshal to put those values into a struct. The values in the struct JSONData are exactly the same as in each Document
I am using the official mongodb driver github.com/mongodb/mongo-go-driver
Here is an example of what I have tried:
mongoContext, cancelContext := context.WithTimeout(context.Background(), 10*time.Second)
defer cancelContext()
mongoClient, _ := mongo.Connect(mongoContext, options.Client().ApplyURI("mongodb://localhost:27017"))
moviesCollection := mongoClient.Database("Entertainment").Collection("Movies")
moviesCollection.InsertOne(mongoContext, bson.M{"_id": "Deadpool", "Path": "path/to/file"})
singleResult := moviesCollection.FindOne(mongoContext, bson.M{"_id": "Deadpool"})
if singleResult.Err() != nil {
log.Println("Find error: ", singleResult.Err())
}
JSONData := struct {
Path string `json:"Path"`
}{}
decodeError := singleResult.Decode(&JSONData)
if decodeError != nil {
log.Println("Decode error: ", decodeError)
}
fmt.Println("Path: ", JSONData.Path)
However no errors are produced and JSON.Path produces and empty string.
I have also tried using bson.D{{"_id", "Deadpool"}} instead of bson.M{"_id": "Deadpool"}
I can confirm that JSON.Path is not empty string as I have checked the database natively using MongoDB Compass. The entry contains the following:
{"_id":"Deadpool","Path":"path/to/file"}
Internally, MongoDB uses bson. Change your struct as below should work.
From
JSONData := struct {
Path string `json:"Path"`
}{}
to
JSONData := struct {
Path string `bson:"Path"`
}{}
Hey so as simagix mentioned you should be able to change your tag from JSON to bson:
`bson:"Path"`
Another option, incase you need to obtain a more generic result is to pass it a D object like so:
JSONData := &bson.D{}
decodeError := singleResult.Decode(JSONData)
You can then obtain all the information through a map using the JSON.Data.Map function.
If you are using mongo-go-driver >= v.0.1.0 then, taking a look to the go-doc it looks pretty straightforward:
filter := bson.D{{"hello", "world"}}
err := collection.FindOne(context.Background(), filter).Decode(&result)
if err != nil { return err }
// do something with result...
So, what you need is:
package main
import (
"context"
"github.com/mongodb/mongo-go-driver/bson"
"github.com/mongodb/mongo-go-driver/mongo
)
func main() {
ctx := context.Background()
client, err := mongo.NewClient("mongodb://localhost:27017")
if err != nil {
...
}
if err := client.Connect(ctx); err != nil {
...
}
defer client.Disconnect(ctx)
collection := client.Database("myDb").Collection("movies")
filter := bson.D{{"_id", "sometitle"}}
var result JSONData
err := collection.FindOne(ctx, filter).Decode(&result)
if err != nil {
...
}
// Do Something cool here
}
I see you are using Title to filter the doc. Pay attention.
I have mongoDB in a Docker container, I can connect to and update the DB just fine, I can see the results in Compass. However when it comes to grabbing a collection and printing the results they don't print as I expect them too.
This is a snippet of my code:
db := client.Database("maccaption")
collection := client.Database("maccaption").Collection("JobBacklog")
res, err := collection.InsertOne(context.Background(), bson.M{"hello": "world"})
if err != nil {
log.Fatal(err)
}
result := struct {
Foo string
Bar string
}{}
filter := bson.D{{"hello", "world"}}
err = collection.FindOne(context.Background(), filter).Decode(&result)
if err != nil {
log.Fatal(err)
}
fmt.Println("Results", result)
I'm using the official mongo-go-driver. and following the examples here https://godoc.org/github.com/mongodb/mongo-go-driver/mongo
I know the DB is connected, I can see the update when I add to the DB and then it shows up in Compass when I run the code, but the collection.FindOne returns Results {0} when I expect it to return hello: world.
Can anyone help me with this?
Thanks!
You've inserted a document with a field hello with value "world". You're then trying to unpack that document into a struct with fields Foo and Bar. Neither of those are named Hello and neither has a bson tag, so there is nowhere it should unmarshal your hello field to. If you define instead:
result := struct{
Hello string
}
It should unmarshal as desired.
Mgo and golang question.
I run into problem again. I try to update record in the database, but running simple command visitors.UpdateId(v.Id, bson.M{"$set": zscore}); where zscore is a variable of type Zscore, does not work. However if I manually convert zscore to bson.M structure, everything works fine.
Does anybody know how to update the record in mongodb using mgo, without manually dumping structure values into bson.M?
Example:
type Zscore struct {
a float64 `bson:"a,omitempty" json:"a"`
b float64 `bson:"b,omitempty" json:"b"`
c float64 `bson:"c,omitempty" json:"c"`
}
v := Visitor{}
zscore := Zscore{}
visitors := updater.C("visitors")
for result.Next(&v) {
zscore.a = 1
zscore.b = 2
zscore.c = 0
//does not work
if err := visitors.UpdateId(v.Id, bson.M{"$set": zscore}); err != nil {
log.Printf("Got error while updating visitor: %v\n", err)
}
//works
set := bson.M{
"zscore.a": zscore.a,
"zscore.b": zscore.b,
"zscore.c": zscore.c,
}
if err := visitors.UpdateId(v.Id, bson.M{"$set": set}); err != nil {
log.Printf("Got error while updating visitor: %v\n", err)
}
}
All Go marshaling packages I'm aware of, including the bson package, will not marshal fields that are private (start with a lowercase letter). To fix the issue, just export the relevant fields by uppercasing the first letter of their name.
Also note that, besides the issue mentioned above, the first part of your example will not marshal in an equivalent way to the second part. bson.M{"$set": zscore} is equivalent to bson.M{"$set": bson.M{"a": ... etc ...}}.
I am using the GOB encoding for my project and i figured out (after a long fight) that empty strings are not encoded/decoded correctly. In my code i use a errormessage (string) to report any problems, this errormessage is most of the time empty. If i encode a empty string, it become nothing, and this gives me a problem with decoding. I don't want to alter the encoding/decoding because these parts are used the most.
How can i tell Go how to encode/decode empty strings?
Example:
Playground working code.
Playground not working code.
The problem isn't the encoding/gob module, but instead the custom MarshalBinary/UnmarshalBinary methods you've declared for Msg, which can't correctly round trip an empty string. There are two ways you could go here:
Get rid of the MarshalBinary/UnmarshalBinary methods and rely on GOB's default encoding for structures. This change alone wont' be enough because the fields of the structure aren't exported. If you're happy to export the fields then this is the simplest option: https://play.golang.org/p/rwzxTtaIh2
Use an encoding that can correctly round trip empty strings. One simple option would be to use GOB itself to encode the struct fields:
func (m Msg) MarshalBinary() ([]byte, error) {
var b bytes.Buffer
enc := gob.NewEncoder(&b)
if err := enc.Encode(m.x); err != nil {
return nil, err
}
if err := enc.Encode(m.y); err != nil {
return nil, err
}
if err := enc.Encode(m.z); err != nil {
return nil, err
}
return b.Bytes(), nil
}
// UnmarshalBinary modifies the receiver so it must take a pointer receiver.
func (m *Msg) UnmarshalBinary(data []byte) error {
dec := gob.NewDecoder(bytes.NewBuffer(data))
if err := dec.Decode(&m.x); err != nil {
return err
}
if err := dec.Decode(&m.y); err != nil {
return err
}
return dec.Decode(&m.z)
}
You can experiment with this example here: https://play.golang.org/p/oNXgt88FtK
The first option is obviously easier, but the second might be useful if your real example is a little more complex. Be careful with custom encoders though: GOB includes a few features that are intended to detect incompatibilities (e.g. if you add a field to a struct and try to decode old data), which are missing from this custom encoding.