Unmarshal Json Array of object obtained from postgresql - postgresql

I have a table in Postgres that is a Jsonb
Create Table Business(
id serial not null primary key,
id_category integer not null,
name varchar(50) not null,
owner varchar(200) not null,
coordinates jsonb not null,
reason varchar(300) not null,
foreign key(id_category) references Category(id)
);
as you can see i store the coordinates as a jsonb
ex:
Insert into Business(id_category, name, owner, coordinates, reason)
values
(1,'MyName','Owner', [{"latitude": 12.1268142, "longitude": -86.2754}]','Description')
the way that I extract the data and assign it is like this.
type Business struct {
ID int `json:"id,omitempty"`
Name string `json:"name,omitempty"`
Owner string `json:"owner,omitempty"`
Category string `json:"category,omitempty"`
Departments []string `json:"departments,omitempty"`
Location []Coordinates `json:"location,omitempty"`
Reason string `json:"reason,omitempty"`
}
type Coordinates struct {
Latitude float64 `json:"latitude,omitempty"`
Longitude float64 `json:"longitude,omitempty"`
}
func (a Coordinates) Value() (driver.Value, error) {
return json.Marshal(a)
}
func (a *Coordinates) Scan(value []interface{}) error {
b, ok := value.([]byte)
if !ok {
return errors.New("type assertion to []byte failed")
}
return json.Unmarshal(b, &a)
}
However, I keep receiving this message.
sql: Scan error on column index 3, name "coordinates": unsupported
Scan, storing driver.Value type []uint8 into type *models.Coordinates
And the controller that I use to extract the information is this.
func (b *BusinessRepoImpl) Select() ([]models.Business, error) {
business_list := make([]models.Business, 0)
rows, err := b.Db.Query("SELECT business.id, business.name, business.owner, business.coordinates, business.reason_froggy, category.category FROM business INNER JOIN category on category.id = business.id_category group by business.id, business.name, business.owner, business.coordinates, business.reason_froggy, category.category")
if err != nil {
return business_list, err
}
for rows.Next() {
business := models.Business{}
err := rows.Scan(&business.ID, &business.Name, &business.Owner, &business.Location, &business.Reason, &business.Category)
if err != nil {
break
}
business_list = append(business_list, business)
}
err = rows.Err()
if err != nil {
return business_list, err
}
return business_list, nil
}
Can anyone please tell me how to solve this issue? Retrieve the json array of objects and assign it to the coordinates field inside Business.

1.
As you can see from the documentation the Scanner interface, to be satisfied, requires the method
Scan(src interface{}) error
But your *Coordinates type implements a different method
Scan(value []interface{}) error
The types interface{} and []interface{} are two very different things.
2.
The Scanner interface must be implemented on the type of the field which you want to pass as an argument to rows.Scan. That is, you've implemented your Scan method on *Coordinates but the type of the Location field is []Coordinates.
Again, same thing, the types *Coordinates and []Coordinates are two very different things.
So the solution is to implement the interface properly and on the proper type.
Note that since Go doesn't allow adding methods to unnamed types, and []Coordinates is an unnamed type, you need to declare a new type that you'll then use in place of []Coordinates.
type CoordinatesSlice []Coordinates
func (s *CoordinatesSlice) Scan(src interface{}) error {
switch v := src.(type) {
case []byte:
return json.Unmarshal(v, s)
case string:
return json.Unmarshal([]byte(v), s)
}
return errors.New("type assertion failed")
}
// ...
type Business struct {
// ...
Location CoordinatesSlice `json:"location,omitempty"`
// ...
}
NOTE
If the business location will always have only one pair of coordinates store into the db as a jsonb object and change the Location type from CoordinatesSlice to Coordinates and accordingly move the Scanner implementation from *CoordinatesSlice to *Coordinates.

I know that this solution is really unoptimized, but it was the only way that it works.
basically i have to obtain the json and then do an unmarshal into the Location attribute.
var location string = ""
if err := json.Unmarshal([]byte(location), &business.Location); err != nil { panic(err) }

Related

Age() function not working with time.Time

Trying to use age() function from postgresql with DateOfBirth
type Person struct {
ID int
DOB time.Time
}
query := `SELECT id, AGE(dateofbirth) from person WHERE id = $1`
But got an error
Unsupported Scan, storing driver.Value type []uint8 into type *time.Time
The age() function work if I just pass the DOB as string
type Person struct {
ID int
DOB string
}
But I want to check the validity of DOB so I use time.Parse("2006-01-02", DOB)
func createPerson(w http.ResponseWriter, r *http.Request) {
err := r.ParseForm()
if err != nil {
log.Fatal(err)
}
dob := r.PostForm.Get("dob")
birthday, _ := time.Parse("2006-01-02", dob)
err = models.person.Insert(birthday)
if err != nil {
log.Fatal("Server Error: ", err)
return
}
http.Redirect(w, r, "/", http.StatusSeeOther)
}
func getPerson(w http.ResponseWriter, r *http.Request) {
id, err := strconv.Atoi(chi.URLParam(r, "id"))
if err != nil {
log.Fatal("Not found", err)
return
}
person, err = models.person.Get(id)
if err != nil {
log.Fatal("Server Error: ", err)
return
}
render.HTML(w, http.StatusOK, "person.html", person)
}
Your database driver is responsible for transforming the incoming data from the database into Go data types. I'll assume you're using pq, which has the following rules about data types:
This package returns the following types for values from the PostgreSQL backend:
integer types smallint, integer, and bigint are returned as int64
floating-point types real and double precision are returned as float64
character types char, varchar, and text are returned as string
temporal types date, time, timetz, timestamp, and timestamptz are returned as time.Time
the boolean type is returned as bool
the bytea type is returned as []byte
All other types are returned directly from the backend as []byte values in text format.
In PostgreSQL, the AGE function returns data type interval. This is not one of the directly supported data types, so pq processes it as []byte. That is why it works when you make DOB a string. Conversion between []byte and string in Go is trivial, so the driver is able to fill in the value.
The driver fails to fill the []byte into a time.Time value because it doesn't know what conversion routine to use. Actually, there is no direct conversion routine because interval represents a duration of time, while time.Time represents an instance of time. interval is more analogous to a time.Duration value. Still, the driver doesn't support any automatic conversion from interval to any type, and I'm not aware of a way to add a new conversion. You would have to implement the conversion yourself after fetching the data from the database.
You could also switch to the pgx driver, which supports an Interval type.

[]string to jsonb with Gorm and postgres

I have a Go struct which contains a slice of strings which I'd like to save as a jsonB object in Postgres with GORM.
I've come accross a solution which requires to use the GORM specific type (postgres.Jsonb) which I'd like to avoid.
When I try to run the AutoMigrate with a slice in my model, it panics and won't start although when I wrap this slice in a struct (which I'm okay doing) it will run without error but won't create the column in postgres.
type User struct {
gorm.Model
Data []string `sql:"type:"jsonb"; json:"data"`
} //Panics
type User struct {
gorm.Model
Data struct {
NestedData []string
} `sql:"type:"jsonb"; json:"data"`
} //Doesn't crash but doesn't create my column
Has anyone been able to manipulate jsonb with GORM without using the postgres.Jsonb type in models ?
The simplest way to use JSONB in Gorm is to use pgtype.JSONB.
Gorm uses pgx as it driver, and pgx has package called pgtype, which has type named pgtype.JSONB.
If you have already install pgx as Gorm instructed, you don't need install any other package.
This method should be the best practice since it using underlying driver and no custom code is needed. It can also be used for any JSONB type beyond []string.
type User struct {
gorm.Model
Data pgtype.JSONB `gorm:"type:jsonb;default:'[]';not null"`
}
Get value from DB
u := User{}
db.find(&u)
var data []string
err := u.Data.AssignTo(&data)
if err != nil {
t.Fatal(err)
}
Set value to DB
u := User{}
err := u.Data.Set([]string{"abc","def"})
if err != nil {
return
}
db.Updates(&u)
Maybe:
type DataJSONB []string
func (dj DataJSONB) Value() (driver.Value, error) {
return json.Marshal(dj)
}
func (dj *DataJSONB) Scan(value interface{}) error {
b, ok := value.([]byte)
if !ok {
return fmt.Errorf("[]byte assertion failed")
}
return json.Unmarshal(b, dj)
}
// Your bit
type User struct {
gorm.Model
Data DataJSONB `sql:"type:"jsonb"; json:"data"`
}
Define a new type:
type Data map[string]interface{}
And implement the Valuer and Scanner interfaces onto them, which allows the field to be converted to a value for the database, and scanned back into the field, respectively:
// Value converts into []byte
func (d Data) Value() (driver.Value, error) {
j, err := json.Marshal(d)
return j, err
}
// Scan puts the []byte back into Data
func (d *Data) Scan(src interface{}) error {
source, ok := src.([]byte)
if !ok {
return errors.New("Type assertion .([]byte) failed.")
}
var i interface{}
if err := json.Unmarshal(source, &i); err != nil {
return err
}
*d, ok = i.(map[string]interface{})
if !ok {
return errors.New("Type assertion .(map[string]interface{}) failed.")
}
return nil
}
Then you can define your field in your model like this:
type User struct {
gorm.Model
Data Data `type: jsonb not null default '{}'::jsonb`
}
Using the underlying map[string]interface{} type is nice too, as you can Unmarshal/Marshal any JSON to/from it.

How to insert slice

I have a postgres db that I would like to generate tables for and write to using Gorp, however I get an error message when I try to insert due to the slices contained within my structs "sql: converting argument $4 type: unsupported type []core.EmbeddedStruct, a slice of struct.
My structs look as follows:
type Struct1 struct {
ID string
Name string
Location string
EmbeddedStruct []EmbeddedStruct
}
type EmbeddedStruct struct {
ID string
Name string
struct1Id string
EmbeddedStruct2 []EmbeddedStruct2
}
type EmbeddedStruct2 struct {
ID string
Name string
embeddedStructId string
}
func (repo *PgStruct1Repo) Write(t *core.Struct1) error {
trans, err := createTransaction(repo.dbMap)
defer closeTransaction(trans)
if err != nil {
return err
}
// Check to see if struct1 item already exists
exists, err := repo.exists(t.ID, trans)
if err != nil {
return err
}
if !exists {
log.Debugf("saving new struct1 with ID %s", t.ID)
err = trans.Insert(t)
if err != nil {
return err
}
return nil
}
return nil
}
Does anyone have any experience with/or know if Gorp supports inserting slices? From what I've read it seems to only support slices for SELECT statements
Gorp supports inserting a variadic number of slices, so if you have a slice records, you can do:
err = db.Insert(records...)
However, from your question it seems you want to save a single record that has a slice struct field.
https://github.com/go-gorp/gorp
gorp doesn't know anything about the relationships between your structs (at least not yet).
So, you have to handle the relationship yourself. The way I personally would solve this issue is to have Gorp ignore the slice on the parent:
type Struct1 struct {
ID string
Name string
Location string
EmbeddedStruct []EmbeddedStruct `db:"-"`
}
And then use the PostInsert hook to save the EmbeddedStruct (side note, this is a poor name as it is not actually an embedded struct)
func (s *Struct1) PostInsert(sql gorp.SqlExecutor) error {
for i := range s.EmbeddedStruct {
s.EmbeddedStruct[i].struct1Id = s.ID
}
return sql.Insert(s.EmbeddedStruct...)
}
And then repeat the process on EmbeddedStruct2.
Take care to setup the relationships properly on the DB side to ensure referential integrity (e.g. ON DELETE CASCADE / RESTRICT), and it would probably be a good idea to wrap the whole thing in a transaction.

GoLang Parse Point DataType

In my DB i have a table that require a coordinate value.
such value is stored as a
Point DataType
however when i try to obtain such value, GO responds with an error:
unsupported Scan, storing driver.Value type []uint8 into type *models.Point
this is my Model
type Business struct {
ID int `json:"id,omitempty"`
Name string `json:"name,omitempty"`
Owner string `json:"owner,omitempty"`
Coordinates Point `json:"coordinates,omitempty"`
Reason string `json:"reason,omitempty"`
BusinessBranches []BusinessBranches `json:"reason,omitempty"`
}
and this is the Point type model
type Point struct {
X float64 `json:"lat"`
Y float64 `json:"lon"`
}
in my controller i call the selection of Business.
func (b *BusinessRepoImpl) Select() ([]models.Business, error) {
business_list := make([]models.Business, 0)
rows, err := b.Db.Query("SELECT id, name, owner, coordinates, reason FROM business")
if err != nil {
return business_list, err
}
for rows.Next() {
business := models.Business{}
err := rows.Scan(&business.ID, &business.Name, &business.Owner, &business.Coordinates, &business.Reason)
if err != nil {
fmt.Println(err)
break
}
business_list = append(business_list, business)
}
err = rows.Err()
if err != nil {
return business_list, err
}
return business_list, nil
}
anyone know how to parse such DataType, so i can read it and store it?
Thanks. :)
As i can see you need to set to set a structure with the current lat, long and then unmarshal from the database.

Golang and MongoDB: DeleteMany with filter

I try to read and write and delete data from a Go application with the official mongodb driver for go (go.mongodb.org/mongo-driver).
Here is my struct I want to use:
Contact struct {
ID xid.ID `json:"contact_id" bson:"contact_id"`
SurName string `json:"surname" bson:"surname"`
PreName string `json:"prename" bson:"prename"`
}
// xid is https://github.com/rs/xid
I omit code to add to the collection as this is working find.
I can get a list of contacts with a specific contact_id using the following code (abbreviated):
filter := bson.D{}
cursor, err := contactCollection.Find(nil, filter)
for cur.Next(context.TODO()) {
...
}
This works and returns the documents. I thought about doing the same for delete or a matched get:
// delete - abbreviated
filter := bson.M{"contact_id": id}
result, _ := contactCollection.DeleteMany(nil, filter)
// result.DeletedCount is always 0, err is nil
if err != nil {
sendError(c, err) // helper function
return
}
c.JSON(200, gin.H{
"ok": true,
"message": fmt.Sprintf("deleted %d patients", result.DeletedCount),
}) // will be called, it is part of a webservice done with gin
// get complete
func Get(c *gin.Context) {
defer c.Done()
id := c.Param("id")
filter := bson.M{"contact_id": id}
cur, err := contactCollection.Find(nil, filter)
if err != nil {
sendError(c, err) // helper function
return
} // no error
contacts := make([]types.Contact, 0)
for cur.Next(context.TODO()) { // nothing returned
// create a value into which the single document can be decoded
var elem types.Contact
err := cur.Decode(&elem)
if err != nil {
sendError(c, err) // helper function
return
}
contacts = append(contacts, elem)
}
c.JSON(200, contacts)
}
Why does the same filter does not work on delete?
Edit: Insert code looks like this:
_, _ = contactCollection.InsertOne(context.TODO(), Contact{
ID: "abcdefg",
SurName: "Demo",
PreName: "on stackoverflow",
})
Contact.ID is of type xid.ID, which is a byte array:
type ID [rawLen]byte
So the insert code you provided where you use a string literal to specify the value for the ID field would be a compile-time error:
_, _ = contactCollection.InsertOne(context.TODO(), Contact{
ID: "abcdefg",
SurName: "Demo",
PreName: "on stackoverflow",
})
Later in your comments you clarified that the above insert code was just an example, and not how you actually do it. In your real code you unmarshal the contact (or its ID field) from a request.
xid.ID has its own unmarshaling logic, which might interpret the input data differently, and might result in an ID representing a different string value than your input. ID.UnmarshalJSON() defines how the string ID will be converted to xid.ID:
func (id *ID) UnmarshalJSON(b []byte) error {
s := string(b)
if s == "null" {
*id = nilID
return nil
}
return id.UnmarshalText(b[1 : len(b)-1])
}
As you can see, the first byte is cut off, and ID.UnmarshalText() does even more "magic" on it (check the source if you're interested).
All-in-all, to avoid such "transformations" happen in the background without your knowledge, use a simple string type for your ID, and do necessary conversions yourself wherever you need to store / transmit your ID.
For the ID Field, you should use the primitive.ObjectID provided by the bson package.
"go.mongodb.org/mongo-driver/bson/primitive"
ID primitive.ObjectID `json:"_id" bson:"_id"`