I'm trying to append data to an array that belongs to a json field in postgres. While using pgAdmin I know the following query works. ~
UPDATE lesson SET data =
jsonb_set (data, '{pages, 999999}', '{"pageNum": 2, "pageType": "voc"}', True)
WHERE id = 2;
I am simply trying to get the above query to work via my rest api written in go. I am getting an error that reads "pq: invalid input syntax for type json".
my code is as follows~
_, err := db.Exec(`
UPDATE lessons SET data =
jsonb_set (data, '{pages, 999999}','{"pageNum": $1, "pageType": $2}', True)
WHERE id = $3`,
pageNum, pageType, id) // variable types are int string int
I suspect that the postgres driver isn't interpolating the the $ parameters. It will work if I use fmt.Sprinf() for the whole query but I am trying to avoid SQL injection attacks, and would like to take advantage of the built in security measures of the go sql library.
For reference my data is structured as follows~
Lessons Table
Lessons
id int
data jsonb
Go structs:
type Lesson struct {
ID int `json:"id"`
Name string `json:"name"`
Pages []Page `json:"pages"`
}
type Page struct {
PageNum int `json:"pageNum"`
PageType string `json:"pageType"`
You cannot use query parameters within a string in Postgres. Either pass the entire string to Postgres as single parameter:
str := fmt.Sprintf('{"pageNum": %d, "pageType": %q}', pageNum, pageType)
_, err := db.Exec(`
UPDATE lessons SET data =
jsonb_set (data, '{pages, 999999}', $1, True)
WHERE id = $2`,
str, id) // variable types are int string int
or use string concatenation to do it on the server side:
_, err := db.Exec(`
UPDATE lessons SET data =
jsonb_set (data, '{pages, 999999}','{"pageNum": ' || $1 || ', "pageType": ' || $2 || '}', True)
WHERE id = $3`,
pageNum, pageType, id) // variable types are int string int
The best/safest, is probably the first approach, with full JSON marshaling in your client, rather than a simple fmt.Sprintf. I leave that as an exercise for the reader.
Related
I have jsonb value stored in postgres columns which I need to convert to object to send the response
I am unable to convert JSON -> struct using json.Unmarshal method
type Result struct {
Id int
Expertise string // because json string
Languages string // because json string
}
// saving dd query data to result struct
e, _ := json.Marshal(data.Expertise)
l, _ := json.Marshal(data.Languages)
row := r.db.QueryRow(`
INSERT INTO filters (user_id, expertise, languages)
VALUES ($1, $2, $3)
ON CONFLICT (user_id) DO
UPDATE SET expertise=$2, languages=$3
RETURNING id, expertise, languages;
`, userid, e, l)
var res Result
err := row.Scan(&res.Id, &res.Expertise, &res.Languages)
Then I take this the Expertise and Languages field and UNMARSHAL THEM
// helper method
func Unmarshal(value string) interface{} {
var obj interface{}
err := json.Unmarshal([]byte(value), &obj)
return obj
}
type CreateFilterResponse struct {
Id int `json:"id"`
Expertise interface{} `json:"expertise"`
Languages interface{} `json:"languages"`
}
response := dto.CreateFilterResponse{
Id: res.Id,
Expertise: Unmarshal(res.Expertise), // 👈 not decoding, still json
Languages: Unmarshal(res.Languages), // 👈 not decoding, still json
}
I would really appreciate the help, I need the Expertise and Languages to be {} and [] respectively
This is what I am getting in response:
{"id":5,"expertise":"[{\"role\":1,\"experience\":5},
{\"role\":3,\"experience\":4}]","languages":"[1,2,3]"}
Note:
Expertise is jsonb column : []{role: int, experience: int}
Languages is jsonb column : []int
Thanks to #Blackgreen comment, the suggestion worked.
As he suggested "I believe the issue is that you are scanning the jsonb bytes into string, which causes it to be interpreted literally. Then when you pass a string into json.Unmarshal it stays a string. Change the type of Expertise and Language to []byte (or json.RawMessage)"
I did the same and it works, here is the code:
// for parsing jsonb coming from db query
type Result struct {
Id int
Expertise json.RawMessage // change
Languages json.RawMessage // change
}
// the database query itself
e, _ := json.Marshal(filters.Expertise)
l, _ := json.Marshal(filters.Languages)
row := r.db.QueryRow(`
INSERT INTO filters (user_id, expertise, languages)
VALUES ($1, $2, $3)
ON CONFLICT (user_id) DO
UPDATE SET expertise=$2, languages=$3
RETURNING id, expertise, languages;
`, userid, e, l)
var res Result
err := row.Scan(&res.Id, &res.Expertise, &res.Languages) // scanning to result
if err != nil {
fmt.Println("Unable to create a new filter ==>", err)
return nil, err
}
Then I Unmarshalled the expertise and languages jsonb values using a helper method and created a Response struct for the client:
// response struct type
type CreateFilterResponse struct {
Id int `json:"id"`
Expertise interface{} `json:"expertise"`
Languages interface{} `json:"languages"`
}
// final response struct
response := dto.CreateFilterResponse{
Id: res.Id,
Expertise: Unmarshal(res.Expertise),
Languages: Unmarshal(res.Languages),
}
// helper method
func Unmarshal(value []byte) interface{} {
var obj interface{}
json.Unmarshal(value, &obj)
return obj
}
This works for now. I still need to find an easy way to do this as this is too much boilerplate code to perform a simple task. like in JS/TS it can be done using single line of code JSON.parse(value)
The problem is in the json output. It is encoding your JSON twice, making it difficult to read and deserialize the json in your structure.
You can start by organizing your structures as follows:
type Expertise struct {
Role int `json:"role"`
Experience int `json:"Experience"`
}
type CreateFilterResponse struct {
Id int `json:"id"`
Expert []Expertise `json:"expertise"`
Languages []int `json:"languages"`
}
I made a cleaner method just to exemplify and facilitate understanding. It will remove the unnecessary quotes and escapes so you can convert your string to []bytes and unmarshal your structure.
func jsonCleaner(quoted string) []byte{
quoted = strings.ReplaceAll(quoted, "\n", "")
quoted = strings.ReplaceAll(quoted, "\\", "")
quoted = strings.ReplaceAll(quoted, "]\"", "]")
quoted = strings.ReplaceAll(quoted, "\"[", "[")
dataBytes := []byte(quoted)
fmt.Println(dataBytes)
return dataBytes
}
Your json message would go from:
{"id":5,"expertise":"[{\"role\":1,\"experience\":5},{\"role\":3,\"experience\":4}]","languages":"[1,2,3]"}
To:
{"id":5,"expertise":[{"role":1,"experience":5},"role":3,"experience":4}],"languages":[1,2,3]}
I am trying to insert a struct in mongo database.
type SecretsStruct struct {
UserID string `bson:"userid" json:"userid"`
secretOne string `bson:"secret_one" json:secret_one`
secretTwo string `bson:"secret_two" json:secret_two`
secretThree string `bson:"secret_three" json:secret_three`
}
func (c *SecretsStruct) SetSecrets(userId string, encryptedKeys
[][]byte){
c.UserID = userId
c.secretOne = hex.EncodeToString(encryptedKeys[0])
c.secretTwo = hex.EncodeToString(encryptedKeys[1])
c.secretThree = hex.EncodeToString(encryptedKeys[2])
log.Printf("This is the c %s", c)
}
g := SecretsStruct{}
g.SetSecrets(userStruct.UserID, encryptedKeys)
err = secretCollection.Insert(g)
if err != nil {
panic(err)
}
I have tried inserting the byte arrays corresponding to the secrets but of no help. The result which gets populated to the corresponding insertion operation is :
{'_id': ObjectId('5b80117c118c660aaa0c87c2'),
'userid': 'eb19d220-ef13-43aa-8a7f-f78637718000'}
On the other hand, if I try to insert same data with a map but without struct.
secretCollection.Insert(bson.M{"userid": userStruct.UserID,
"secret_one": encryptedKeys[0],
"secret_two": encryptedKeys[1],
"secret_three": encryptedKeys[2]})
The insertion operation executes successfully.
You have to export your struct fields, so that another package (in this case mgo) can access them:
type SecretsStruct struct {
UserID string `bson:"userid" json:"userid"`
SecretOne string `bson:"secret_one" json:secret_one`
SecretTwo string `bson:"secret_two" json:secret_two`
SecretThree string `bson:"secret_three" json:secret_three`
}
If I have a table that returns something like:
id: 1
names: {Jim, Bob, Sam}
names is a varchar array.
How do I scan that back into a []string in Go?
I'm using lib/pg
Right now I have something like
rows, err := models.Db.Query("SELECT pKey, names FROM foo")
for rows.Next() {
var pKey int
var names []string
err = rows.Scan(&pKey, &names)
}
I keep getting:
panic: sql: Scan error on column index 1: unsupported Scan, storing driver.Value type []uint8 into type *[]string
It looks like I need to use StringArray
https://godoc.org/github.com/lib/pq#StringArray
But, I think I'm too new to Go to understand exactly how to use:
func (a *StringArray) Scan(src interface{})
You are right, you can use StringArray but you don't need to call the
func (a *StringArray) Scan(src interface{})
method yourself, this will be called automatically by rows.Scan when you pass it anything that implements the Scanner interface.
So what you need to do is to convert your []string to *StringArray and pass that to rows.Scan, like so:
rows, err := models.Db.Query("SELECT pKey, names FROM foo")
for rows.Next() {
var pKey int
var names []string
err = rows.Scan(&pKey, (*pq.StringArray)(&names))
}
Long Story Short, use like this to convert pgSQL array to GO array, here 5th column is coming as a array :
var _temp3 []string
for rows.Next() {
// ScanRows scan a row into temp_tbl
err := rows.Scan(&_temp, &_temp0, &_temp1, &_temp2, pq.Array(&_temp3))
In detail :
To insert a row that contains an array value, use the pq.Array function like this:
// "ins" is the SQL insert statement
ins := "INSERT INTO posts (title, tags) VALUES ($1, $2)"
// "tags" is the list of tags, as a string slice
tags := []string{"go", "goroutines", "queues"}
// the pq.Array function is the secret sauce
_, err = db.Exec(ins, "Job Queues in Go", pq.Array(tags))
To read a Postgres array value into a Go slice, use:
func getTags(db *sql.DB, title string) (tags []string) {
// the select query, returning 1 column of array type
sel := "SELECT tags FROM posts WHERE title=$1"
// wrap the output parameter in pq.Array for receiving into it
if err := db.QueryRow(sel, title).Scan(pq.Array(&tags)); err != nil {
log.Fatal(err)
}
return
}
Note: that in lib/pq, only slices of certain Go types may be passed to pq.Array().
Another example in which varchar array in pgSQL in generated at runtime in 5th column, like :
--> predefined_allow false admin iam.create {secrets,configMap}
I converted this as,
Q := "SELECT ar.policy_name, ar.allow, ar.role_name, pro.operation_name, ARRAY_AGG(pro.resource_id) as resources FROM iam.authorization_rules ar LEFT JOIN iam.policy_rules_by_operation pro ON pro.id = ar.operation_id GROUP BY ar.policy_name, ar.allow, ar.role_name, pro.operation_name;"
tx := g.db.Raw(Q)
rows, _ := tx.Rows()
defer rows.Close()
var _temp string
var _temp0 bool
var _temp1 string
var _temp2 string
var _temp3 []string
for rows.Next() {
// ScanRows scan a row into temp_tbl
err := rows.Scan(&_temp, &_temp0, &_temp1, &_temp2, pq.Array(&_temp3))
if err != nil {
return nil, err
}
fmt.Println("Query Executed...........\n", _temp, _temp0, _temp1, _temp2, _temp3)
}
Output :
Query Executed...........
predefined_allow false admin iam.create [secrets configMap]
I have been trying to use the postgres IN clause in golang, but keep getting errors. This is the query I want to execute.
SELECT id1 FROM my_table WHERE type = (an int) AND id2 = (an int) AND id1 IN (list of UUIDs)
I used this code to construct this query but got the following error.
var params []interface{}
inCondition := ""
params = append(params, type)
params = append(params, id2)
for _, id := range id1 {
params = append(params, id)
if inCondition != "" {
inCondition += ", "
}
inCondition += "?"
}
query := fmt.Sprintf(`SELECT id1 FROM my_table WHERE type = ? AND id2 = ? AND id1 IN (%s)`, inCondition)
rows, err := db.Query(query, params...)
Query I got:
SELECT id1 FROM my_table WHERE type = ? AND id2 = ? AND id1 IN (?, ?, ?)
Params output:
[]interface {}=[0 7545449 d323f8d5-ab97-46a3-a34e-95ceac2f3a6a d323f8d5-ab97-46a3-a34e-95ceac2f3a6b d323f8d5-ab97-46a3-a34e-95ceac2f3a6d]
Error:
pq: syntax error at or near \"AND\""
What am I missing? or, how will I get this to work? id1 is a slice of UUIDs whose length is variable.
Ran into a similar issue. Can't remember where exactly I picked this up but I remember still running into issues when dealing with arrays of type integer vs string. What I had to do was to have a local custom type and return a driver compatible value for it. See sample below.
// Int64Array is a type implementing the sql/driver/value interface
// This is due to the native driver not supporting arrays...
type Int64Array []int64
// Value returns the driver compatible value
func (a Int64Array) Value() (driver.Value, error) {
var strs []string
for _, i := range a {
strs = append(strs, strconv.FormatInt(i, 10))
}
return "{" + strings.Join(strs, ",") + "}", nil
}
Highly recommend checking out sqlx. Wrote a simple orm wrapper called papergres to make my go + postgres life easier :) Give it a try.
Instead of ?, using $1,$2 etc as placeholder worked.
want to save the data of the following format
{"_ibj_id":"1","url_id":'1',"url":{"0":"http://0.com","1":"http:://1.com"}}
Look at my code,
type db_list struct {
Url_id int
Url map[int]string
}
func list(table *mgo.Collection) {
var doc *goquery.Document
var e error
for i := 1628644; i > 1628643; i-- {
if doc, e = goquery.NewDocument("http://www.120ask.com/list/all/" + strconv.Itoa(i)); e != nil {
panic(e.Error())
}
var save_list db_list
save_list.Url_id = i
save_list.Url = make(map[int]string)
//fmt.Println("%s", doc.Text())
doc.Find(".q-quename").Each(func(n int, s *goquery.Selection) {
href, isTrue := s.Attr("href")
if isTrue {
save_list.Url[n] = href
fmt.Println("%D : %s", n, save_list.Url[n])
}
})
fmt.Println("%D", len(save_list.Url))
//save database
table.Insert(save_list)
}
}
The database will eventually save
Please view the picture in the annex, is to save the format of the data, save the URLvalue of the property 1
You're probably after the JSON Unmarshal function in encoding/json
{"_ibj_id":"1","url_id":'1',"url":{"0":"http://0.com","1":"http:://1.com"}} is technically invalid JSON due to the single-quotes around the url_id value( '1' should be "1") but other than that, it should map nicely to the following struct:
{
id string
url_id string
urls []string
}
But you may need to experiment with the types. According to the docs for the Unmarshal function, it will use the following Go types for each JSON type:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
I'd highly recommend reading Andrew Gerrands Blog Post "JSON and Go".
It's unclear to me exactly what you are trying to do. One thing I notice though, is that in your desired format the keys for Url are string values where as in Go they are integers. You could try changing the type of Url to map[string]string and see if that solves your problem.