Could you help me please with my problem. I'm trying to use column encryption for postgres and there is one small question: how I can achieve transformation value of column (ex: "test_value") to PGP_SYM_ENCRYPT('test_value', 'KEY') in "insert" sql query?
As I understood, the custom types can be the solution for me, but some things isn't clear... Maybe anyone has an example for my case?
(I see this aws docs about pgcrypto using: https://docs.aws.amazon.com/dms/latest/sql-server-to-aurora-postgresql-migration-playbook/chap-sql-server-aurora-pg.security.columnencryption.html)
What I did:
type sstring struct {
string
}
var _ types.ValueAppender = (*sstring)(nil)
func (tm sstring) AppendValue(b []byte, flags int) ([]byte, error) {
if flags == 1 {
b = append(b, '\'')
}
b = []byte("PGP_SYM_ENCRYPT('123456', 'AES_KEY')")
if flags == 1 {
b = append(b, '\'')
}
return b, nil
}
var _ types.ValueScanner = (*sstring)(nil)
func (tm *sstring) ScanValue(rd types.Reader, n int) error {
if n <= 0 {
tm.string = ""
return nil
}
tmp, err := rd.ReadFullTemp()
if err != nil {
return err
}
tm.string = string(tmp)
return nil
}
type model struct {
ID uint `pg:"id"`
Name string `pg:"name"`
Crypto sstring `pg:"crypto,type:sstring"`
tableName struct{} `pg:"models"`
}
----------
_, err := r.h.ModelContext(ctx, model).Insert()
And... the process just do nothing. Do not respond, do not fall, do not create row in sql table.. Nothing.
Anyway. My question is how to implement wrap some column by sql function using pg-go orm.
I tried to use https://github.com/go-pg/pg/blob/v10/example_custom_test.go#L13-L49 as custom type handler example.. But smth went wrong. =(
Related
I have a function for comparing two structs and making a bson document as input to mongodb updateOne()
Example struct format
type event struct {
...
Name string
StartTime int32
...
}
Diff function, please ignore that I have not checked for no difference yet.
func diffEvent(e event, u event) (bson.M, error) {
newValues := bson.M{}
if e.Name != u.Name {
newValues["name"] = u.Name
}
if e.StartTime != u.StartTime {
newValues["starttime"] = u.StartTime
}
...
return bson.M{"$set": newValues}, nil
}
Then I generated a test function like so:
func Test_diffEvent(t *testing.T) {
type args struct {
e event
u event
}
tests := []struct {
name string
args args
want bson.M
wantErr bool
}{
{
name: "update startime",
args: args{
e: event{StartTime: 1},
u: event{StartTime: 2},
},
want: bson.M{"$set": bson.M{"starttime": 2}},
wantErr: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := diffEvent(tt.args.e, tt.args.u)
if (err != nil) != tt.wantErr {
t.Errorf("diffEvent() error = %v, wantErr %v", err, tt.wantErr)
return
}
if !reflect.DeepEqual(got, tt.want) {
t.Errorf("diffEvent() = %v, want %v", got, tt.want)
}
})
}
}
This fails with a
--- FAIL: Test_diffEvent/update_startime (0.00s)
models_test.go:582: diffEvent() = map[$set:map[starttime:2]], want map[$set:map[starttime:2]]
For me this seem to be the same. I have played around with this and bool fields, string fields, enum fields, and fields as struct or fields as arrays of structs seems to work fine with deepequal, but it gives an error for int32 fields.
As a go beginner; what am I missing here? I would assume that if bool/string works then int32 would too.
This:
bson.M{"starttime": 2}
Sets the "starttime" key to the value of the literal 2. 2 is an untyped integer constant, and since no type is provided, its default type will be used which is int.
And 2 values stored in interface values are only equal if the dynamic value stored in them have identical type and value. So a value 2 with int type cannot be equal to a value 2 of type int32.
Use explicit type to tell you want to specify a value of int32 type:
bson.M{"starttime": int32(2)}
I am using PostgreSQL as a database for my service, with a go backend.
After querying the database, I can access the rows with rows.Next() and rows.Scan(XX) :
func All(receptacle ...interface{}) (error) {
[...]
for rows.Next() {
err = rows.Scan(receptacle...)
if err != nil {
tx.Rollback()
return nil, err
}
}
rows.Close()
[...]
}
func main() {
var myInt1 int
var myInt2 int
var myString string
All(&myInt1, &myInt2, &myString)
//all 3 variables are correctly set
}
Here, receptacle is variadic.
If my query returns two int and one string, instead of doing rows.Scan(&myInt1, &myInt2, &myString), I am using the function's arguments rows.Scan(args...).
This is perfectly working when one, and only one, a row is requested from the BD.
What I would like to do is to use the same process to store multiple rows :
func main() {
var myInts1 []int
var myInts2 []int
var myStrings []string
All(&myInts1, &myInts2, &myStrings)
//myInts1 == []int{firstRowInt1, secondRowInt1, ...}
//myInts2 == []int{firstRowInt2, secondRowInt2, ...}
//myStrings == []string{firstRowString, secondRowString, ...}
}
Is there a way to tell a function to use the index X of a variadic argument, something like this: rows.Scan(receptacle[0]...) ?
My guess (and tries) is no, but wanted to confirm this hypothesis
sql
CREATE TABLE public.tiantang_page (
href varchar NOT NULL,
status int4 NOT NULL,
description varchar NOT NULL,
urls url[] NULL
);
CREATE TYPE url AS (
url varchar,
status int4);
insert composite type array
type url struct {
url string
status int
}
var urls [1]url
urls[0] = url{
url: "",
status: 0,
}
update := "UPDATE \"public\".\"tiantang_page\" SET \"urls\"=$1 where \"href\"=$2;"
r, err := db.Exec(update, pq.Array(urls),href)
if err != nil {
log.Fatal(err)
}
error
sql: converting argument $1 type: unsupported type parsetest.url, a struct
library
https://godoc.org/github.com/lib/pq
Note that custom composite types are not fully supported by lib/pq.
If all you want is to be able to store the urls then the simplest approach would be to implement the driver.Valuer interface on the url type and then use it as you do with pq.Array:
func (u url) Value() (driver.Value, error) {
return fmt.Sprintf("(%s,%d)", u.url, u.status), nil
}
// ...
r, err := db.Exec(update, pq.Array(urls), href)
more info on that can be found here: https://github.com/lib/pq/issues/544
Note that I haven't tried this with arrays, only with slices, so you may have to switch from using an array to using a slice, i.e. instead of var urls [1]url you would use var urls = make([]url, 1).
If you also want to be able to retrieve the array of urls back from the db, then you'll have to implement the sql.Scanner interface, however here the pq.Array is not very reliable and you'll have to implement the scanner on the slice type and do all the parsing yourself.
The general format of composite types is (val1, val2, ...) note that you have to put double quotes around values that contain commas or parentheses. For example to construct a value of the url type you would use the literal expression: (http://example.com,4). More info in the docs.
The format for an array of composite types is {"(val1, val2, ...)" [, ...]}, note that in this case if you need to put double quotes around the values you need to escape them. For example {"(http://example.com,4)","(\"http://example.com/?list=foo,bar,baz\",3)"}
So as you can see the more complex the data in the composite type the more complex will be the parsing as well.
Here's a crude example (does not handle quoted values):
type urlslice []url
func (s *urlslice) Scan(src interface{}) error {
var a []byte // the pq array as bytes
switch v := src.(type) {
case []byte:
a = v
case string:
a = []byte(v)
case nil:
*s = nil
return nil
default:
return fmt.Errorf("urlslice.Scan unexpected src type %T", src)
}
a = a[1 : len(a)-1] // drop curly braces
for i := 0; i < len(a); i++ {
if a[i] == '"' && (len(a) > (i+1) && a[i+1] == '(') { // element start?
i += 2 // move past `"(`
j := i // start of url.url
u := url{}
for ; i < len(a) && a[i] != ','; i++ {
}
u.url = string(a[j:i])
i += 1 // move past `,`
j = i // start of url.status
for ; i < len(a) && a[i] != ')'; i++ {
}
i64, err := strconv.ParseInt(string(a[j:i]), 10, 64)
if err != nil {
return err
}
u.status = int(i64)
*s = append(*s, u)
i += 2 // move past `)",`
}
}
return nil
}
for completeness, here's the Valuer interface implemented by the slice type, again not handling proper quoting of values that may require it:
func (s urlslice) Value() (driver.Value, error) {
data := []byte{'{'}
for _, url := range s {
data = append(data, '"', '(')
data = append(data, []byte(url.url)...)
data = append(data, ',')
data = strconv.AppendInt(data, int64(url.status), 10)
data = append(data, ')', '"', ',')
}
data[len(data)-1] = '}' // replace last ',' with '}' to close the array
return data, nil
}
With the urlslice implementing the two interfaces directly you can stop using pq.Array.
var urls = urlslice{{
url: "http://example.com",
status: 4,
}}
update := `UPDATE "public"."tiantang_page" SET "urls"=$1 where "href"=$2`
r, err := db.Exec(update, urls, href)
if err != nil {
log.Fatal(err)
}
var urls2 urlslice
selurls := `SELECT "urls" FROM "public"."tiantang_page" where "href" = $1`
if err := db.QueryRow(selurls, href).Scan(&urls2); err != nil {
log.Fatal(err)
}
Please keep in mind that both of the above examples should be considered only as hints of the direction to take in solving this problem. Not only are the two examples incomplete in that they don't handle quoted values, but they are also not very elegant implementations.
Reasonably complete composite literal parser:
type parseState int
const (
state_initial parseState = iota // start
state_value_start // no bytes read from value yet
state_value // unquoted value
state_quoted // inside quote
state_value_end // after a close quote
state_end // after close paren
)
func parseComposite(in []byte) ([]string, error) {
state := state_initial
ret := []string{}
val := []byte{}
for _, b := range in {
switch state {
case state_initial:
if b != '(' {
return nil, fmt.Errorf("initial character not ')': %v", in)
} else {
state = state_value_start
}
case state_value_start:
if b == '"' {
state = state_quoted
continue
}
fallthrough
case state_value:
if b == ',' {
ret = append(ret, string(val))
val = nil
state = state_value_start
} else if b == ')' {
ret = append(ret, string(val))
val = nil
state = state_end
} else {
val = append(val, b)
}
case state_quoted:
if b == '"' {
ret = append(ret, string(val))
val = nil
state = state_value_end
} else {
val = append(val, b)
}
case state_value_end:
if b == ',' {
state = state_value_start
} else if b == ')' {
state = state_end
} else {
return nil, fmt.Errorf("invalid delimiter after closing quote: %v", in)
}
case state_end:
return nil, fmt.Errorf("trailing bytes: %v", in)
}
}
if state != state_end {
return nil, fmt.Errorf("unterminated value: %v", in)
}
return ret, nil
}
I'm trying to store a Point variable in an SQL database using GORM on a go server, I tried looking everywhere but I haven't seen an answer that satisfies me yet
For any type that's not supported by the standard library's database/sql package, nor by a 3rd party package that depends on database/sql, like gorm does, you can implement the Valuer and Scanner interfaces to add custom support for the type.
To be able to correctly implement those two interfaces you need to first find out what the target database expects as input and what it returns as output for that type. In the case of PostgreSQL and the Point type the syntax is (x,y).
So what you can do is the following:
Declare the type.
type Point struct {
X, Y float64
}
Implement the Valuer interface.
func (p Point) Value() (driver.Value, error) {
out := []byte{'('}
out = strconv.AppendFloat(out, p.X, 'f', -1, 64)
out = append(out, ',')
out = strconv.AppendFloat(out, p.Y, 'f', -1, 64)
out = append(out, ')')
return out, nil
}
Implement the Scanner interface.
func (p *Point) Scan(src interface{}) (err error) {
var data []byte
switch src := src.(type) {
case []byte:
data = src
case string:
data = []byte(src)
case nil:
return nil
default:
return errors.New("(*Point).Scan: unsupported data type")
}
if len(data) == 0 {
return nil
}
data = data[1 : len(data)-1] // drop the surrounding parentheses
for i := 0; i < len(data); i++ {
if data[i] == ',' {
if p.X, err = strconv.ParseFloat(string(data[:i]), 64); err != nil {
return err
}
if p.Y, err := strconv.ParseFloat(string(data[i+1:]), 64); err != nil {
return err
}
break
}
}
return nil
}
I wrote a function that would return a sorted slice of strings from a map[string]Foo. I'm curious what is the best way to create a generic routine that can return a sorted slice of strings from any type that is a map with strings as keys.
Is there a way to do it using an interface specification? For example, is there any way to do something like:
type MapWithStringKey interface {
<some code here>
}
To implement the interface above, a type would need strings as keys. I could then write a generic function that returns a sorted list of keys for fulfilling types.
This is my current best solution using the reflect module:
func SortedKeys(mapWithStringKey interface{}) []string {
keys := []string{}
typ := reflect.TypeOf(mapWithStringKey)
if typ.Kind() == reflect.Map && typ.Key().Kind() == reflect.String {
switch typ.Elem().Kind() {
case reflect.Int:
for key, _ := range mapWithStringKey.(map[string]int) {
keys = append(keys, key)
}
case reflect.String:
for key, _ := range mapWithStringKey.(map[string]string) {
keys = append(keys, key)
}
// ... add more cases as needed
default:
log.Fatalf("Error: SortedKeys() does not handle %s\n", typ)
}
sort.Strings(keys)
} else {
log.Fatalln("Error: parameter to SortedKeys() not map[string]...")
}
return keys
}
Click for Go Playground version
I'm forced to code type assertions for each supported type even though at compile time, we should know the exact type of the mapWithStringKey parameter.
You cannot make partial types. But you can define an interface which serves your purpose:
type SortableKeysValue interface {
// a function that returns the strings to be sorted
Keys() []string
}
func SortedKeys(s SortableKeysValue) []string {
keys := s.Keys()
sort.Strings(keys)
return keys
}
type MyMap map[string]string
func (s MyMap) Keys() []string {
keys := make([]string, 0, len(s))
for k, _ := range s {
keys = append(keys, k)
}
return keys
}
Try it here: http://play.golang.org/p/vKfri-h4Cp
Hope that helps (go-1.1):
package main
import (
"fmt"
"reflect"
)
var m = map[string]int{"a": 3, "b": 4}
func MapKeys(m interface{}) (keys []string) {
v := reflect.ValueOf(m)
for _, k := range v.MapKeys() {
keys = append(keys, k.Interface().(string))
}
return
}
func main() {
fmt.Printf("%#v\n", MapKeys(m))
}