I have unknown json structure data stored in mongodb. They have the fields to present the unix time like this:
"date": 1424803567,
I am using mgo to load them to the bson.M.
var result bson.M
iter := c.Find(q).Iter()
for iter.Next(&result) {
Those unix time fields have turned to the fload64 instead of the int.
"date": 1.424728798e+09,
So, how to prevent that happens in the case above? thanks!
Mgo does not unmarshal an integer to a float unless destination value is explicitly typed as a float by the application. Mgo is returning a float value here because the value stored in the database is a float.
You can unmarshal the float value to an integer by specifying the type using a struct:
var result struct {
Date int64 `bson:"date"`
}
for iter.Next(&result) {
...
}
Related
I am trying to insert/update data in PostgreSQL using jackc/pgx into a table that has column of custom type. This is the table type written as a golan struct:
// Added this struct as a Types in PSQL
type DayPriceModel struct {
Date time.Time `json:"date"`
High float32 `json:"high"`
Low float32 `json:"low"`
Open float32 `json:"open"`
Close float32 `json:"close"`
}
// The 2 columns in my table
type SecuritiesPriceHistoryModel struct {
Symbol string `json:"symbol"`
History []DayPriceModel `json:"history"`
}
I have written this code for inserting data:
func insertToDB(data SecuritiesPriceHistoryModel) {
DBConnection := config.DBConnection
_, err := DBConnection.Exec(context.Background(), "INSERT INTO equity.securities_price_history (symbol) VALUES ($1)", data.Symbol, data.History)
}
But I am unable to insert the custom data type (DayPriceModel).
I am getting an error
Failed to encode args[1]: unable to encode
The error is very long and mostly shows my data so I have picked out the main part.
How do I INSERT data into PSQL with such custom data types?
PS: An implementation using jackc/pgx is preferred but database/SQL would just do fine
I'm not familiar enough with pgx to know how to setup support for arrays of composite types. But, as already mentioned in the comment, you can implement the driver.Valuer interface and have that implementation produce a valid literal, this also applies if you are storing slices of structs, you just need to declare a named slice and have that implement the valuer, and then use it instead of the unnamed slice.
// named slice type
type DayPriceModelList []DayPriceModel
// the syntax for array of composites literal looks like
// this: '{"(foo,123)", "(bar,987)"}'. So the implementation
// below must return the slice contents in that format.
func (l DayPriceModelList) Value() (driver.Value, error) {
// nil slice? produce NULL
if l == nil {
return nil, nil
}
// empty slice? produce empty array
if len(l) == 0 {
return []byte{'{', '}'}, nil
}
out := []byte{'{'}
for _, v := range l {
// This assumes that the date field in the pg composite
// type accepts the default time.Time format. If that is
// not the case then you simply provide v.Date in such a
// format which the composite's field understand, e.g.,
// v.Date.Format("<layout that the pg composite understands>")
x := fmt.Sprintf(`"(%s,%f,%f,%f,%f)",`,
v.Date,
v.High,
v.Low,
v.Open,
v.Close)
out = append(out, x...)
}
out[len(out)-1] = '}' // replace last "," with "}"
return out, nil
}
And when you are writing the insert query, make sure to add an explicit cast right after the placeholder, e.g.
type SecuritiesPriceHistoryModel struct {
Symbol string `json:"symbol"`
History DayPriceModelList `json:"history"` // use the named slice type
}
// ...
_, err := db.Exec(ctx, `INSERT INTO equity.securities_price_history (
symbol
, history
) VALUES (
$1
, $2::my_composite_type[])
`, data.Symbol, data.History)
// replace my_composite_type with the name of the composite type in the database
NOTE#1: Depending on the exact definition of your composite type in postgres the above example may or may not work, if it doesn't, simply adjust the code to make it work.
NOTE#2: The general approach in the example above is valid, however it is likely not very efficient. If you need the code to be performant do not use the example verbatim.
My data structure is
{
_id: ObjectID,
...
fields: [
{ name: "Aryan" },
{ books : [ 1,2,3 ] },
]
}
In our application a user can define his own fields data structure but with a key value structure. So, we had no way of knowing the structure of the data.
So in our document struct we had
type Document struct {
Fields map[string]interface{}
}
as the second parameter returned by mongo was primitive.A ( []interface{} under the hood ), the individual item could have been an array, map, anything.
But we couldn't range over that for it being.
How can I get the individual values like the book ids [1,2,3] or the name value "Aryan" ?
After a couple of attempts to solving this, I was at a state where my current element it self is an interface{} ( [1,2,3] in this case ) and I couldn't get the individual 1,2,3s.
But finally managed to solve it
for k, val := range doc.Fields {
v := reflect.ValueOf(val)
switch reflect.TypeOf(t).Kind() {
case reflect.Slice:
// getting the individual ids
for i := 0; i < v.Len(); i++ {
fmt.Println(v.Index(i))
}
case reflect.String:
default:
// handle
}
}
Note: v.Index(i) returns a data type of Value .
to change the data type
v.Index(i).Interface().(string) // string
v.Index(i).Interface().(float64) // float
Ideally you should avoid working with interface{} type since it's very error prone and the compiler can't help you. The idiomatic way is to define a struct for your model with BSON tags like in this example`
type MyType struct {
ID primitive.ObjectID `bson:"_id,omitempty"`
Fields []Field `bson:"fields,omitempty"`
}
type Field struct {
Name string `bson:"name,omitempty"`
Books []int `bson:"books,omitempty"`
}
Field here is defined as the reunion of all possible fields which again is not ideal but at least the compiler can help you and developers know what to expect from the database document.
TL;DR: Does the MongoDB driver provide a function to marshal and unmarshal a single field of a document?
This is a pretty straightforward question, but here's some context:
I have a worker responsible for synchronizing data between 2 separated databases. When it receives an event message, signalizing some document must sync, it selects the document in the primary database, and replicates it in another (it's a whole different database, not a replica set).
The thing is: I don't know the full structure of that document, so to preserve the data, I must unmarshal this document in a map map[string]interface{}, or a bson.M that works in the same fashion. But this seems like a lot of overhead, to unmarshal all this data I'm not even using, only to marshal it back to the other database.
So I thought about creating a structure that would just store the binary value of that document, without performing any marshal or unmarshal in order to reduce the overhead, like this:
type Document = map[string]Field
type Field struct {
Type bsontype.Type
Value []byte
}
func (f Field) MarshalBSONValue() (bsontype.Type, []byte, error) {
return f.Type, f.Value, nil
}
func (f *Field) UnmarshalBSONValue(btype bsontype.Type, value []byte) error {
f.Type = btype
f.Value = value
return nil
}
With this structure I can indeed reduce how much of the data will be parsed, but now, I need to manually unmarshal the one value in this document I'll need to use.
So I'm wondering if the MongoDB driver would have some function such as:
// Hypothetical function to get the value of a BSON
var status string
if err := decodeBSON(doc['status'].Type, doc['status'].Value, &status); err != nil {
return err
}
And
// Hypothetical function to set the value of a BSON
createdAt, err := encodeBSON(bsontype.Date, time.Now())
if err != nil {
return err
}
doc["createdAt"] = Field{Type: bsontype.Date, Value: createdAt}
How can I achieve this?
The Field type in your code is equivalent to the driver's bson.RawValue type. By switching to RawValue, you can decode individual fields using the RawValue.Unmarshal method and encode fields using bson.MarshalValue, which returns the two components (type and data) that you need to construct a new RawValue.
An example of how you can use these methods to change a field based on its original value: The Field type in your code is equivalent to the driver's bson.RawValue type. By switching to RawValue, you can decode individual fields using the RawValue.Unmarshal method and encode fields using bson.MarshalValue, which returns the two components (type and data) that you need to construct a new RawValue.
An example of how you can change a field depending on its original value without unmarshalling all of the original document's fields: https://gist.github.com/divjotarora/06c5188138456070cee26024f223b3ee
I have something like...
type Item struct {
Name string `json:"name" bson:"name"`
Config map[string]interface{} `json:"config" bson:"config,inline"`
}
func (repository *ItemRepository) UpdateConfig(id string, config map[string]interface{}) {
change := mgo.Change{
Update: bson.M{"$set": bson.M{
"config": config,
}},
},
}
But I get an error Raw Unmarshal can't deal with struct values. Use a pointer.. I'm not entirely sure what this means, or what's expected. Is it possible to just store 'loose' data as a sub-document?
I fixed this by using Update instead of FindId(id).Apply(change, nil).
I tried to retrieve a document from my collection with unique id.
I have a collection with fields: name, age, city, and rank. I want to get 'city' results from mongodb using golang.
My struct code
type exp struct {
name string `bson:"name"`
age int `bson:"age"`
city string `bson:"city"`
rank int `bson:"rank"`
}
With the following code to retrieve results from mongodb:
var result []exp //my struct type
err = coll.Find(bson.M{"City":bson.M{}}).Distinct("City",&result)
fmt.Println(result)
With this code I get an empty array as the result. How would I get all the cities?
Try this code
var result []string
err = c.Find(nil).Distinct("city", &result)
if err != nil {
log.Fatal(err)
}
fmt.Println(result)
Due to restrictions in reflection, mgo (as well as encoding/json and other similar packages) is unable to use unexported fields to marshal or unmarshal data. What you need to do is to export your fields by capitalize the first letter:
type exp struct {
Name string `bson:"name"`
Age int `bson:"age"`
City string `bson:"city"`
Rank int `bson:"rank"`
}
A side note: you do not need to specify the bson tags if the desired name is the same as the lowercase field name. The documentation for bson states:
The lowercased field name is used as the key for each exported field,
but this behavior may be changed using the respective field tag.
Edit:
I just realized you did get an empty slice and not a slice with empty struct fields. My answer is then not an actual answer to the question, but it is still an issue that you need to consider.