I'm trying to store encoded data using encoding/gob from Golang into Postgres. I'm using Gorm as well.
First, sending form data using
if err := json.NewDecoder(r.Body).Decode(model); err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
Currently client_encoding is set to UTF8 in the postgres database. Here's what I'm using to encode:
type payload struct {
Key []byte
Nonce *[nonceLength]byte
Message []byte
}
// Encrypt message
p.Message = secretbox.Seal(p.Message, plaintext, p.Nonce, key) // key set prior to this
buf := &bytes.Buffer{}
if err := gob.NewEncoder(buf).Encode(p); err != nil {
return nil, err
}
return buf.Bytes(), nil
Then I store string(buf.Bytes()) which is stored in the database column which is currently a string type. Now I'm a novice with encoding, and I think gob just has a different encoding for my database. I'm receiving this error in console:
(pq: invalid byte sequence for encoding "UTF8": 0xff)
I've been following this gist for encryption/decryption:
https://gist.github.com/fuzzyami/f3a7231037166117a6fef9607960aee7
From what I've read, I shouldn't be encoding structs into the db, in this case p, unless using gob. Correct me if I'm wrong with that (can't find the resource at the moment where I found this).
Is anyone able to point me in the right direction for storing this data in Postgres which is decrypted later? I'm clearly not understanding the encoding process, and not entirely sure where to start out here with reading resources, so any help is appreciated!
Use a bytea column in postgresql to store a []byte, and skip the conversion to string.
Took a look at https://golang.org/src/encoding/base64/example_test.go
Was able to use
return base64.StdEncoding.EncodeToString(buf.Bytes()), nil
Which successfully stored in the database.
Related
I have a custom type for the purposes of formatting a time object
type MyTime time.Time
To use this with JSON I have implemented:
func (t *MyTime) UnmarshalJSON(b []byte) error
func (t MyTime) MarshalJSON() ([]byte, error)
These functions marshal the type to/from a formatted string and work fine.
I now need to read/write this field to/from mongoDb. I have implemented
func (t *MyTime) UnmarshalBSON(b []byte) error
Which lets me read my type from mongo with no trouble (mongo doc has same format as above). this implementation simply strips the size and trailing zero bytes from the BSON string and parses as for JSON.
So far so good
My issue is how I write this value in the same format. I started by implementing:
func (t MyTime) MarshalBSON() ([]byte, error)
with a reversal of the unmarshal above (inserting size and trailing zero) and verified that the byte slice was identical to that I read earlier but this generated an error.
Unlike the JSON version, the MarshalBSON implementation will not allow me to write a string. Debugging into the mgo library shows that it is expecting me to marshal a structure, not a string field.
After googling the problem, I tried implementing
func (t *MyTime) GetBSON() (output interface{}, err error)
But this is not called by the InsertIntoMongo code.
There are multiple ways of marshalling the object but this format is defined externally and I cannot change it.
How can I achieve this last piece of the puzzle?
Namely, make BSON marshal my custom type as a string value (the reverse of the unmarshal already implemented)
edit:
My implementation of MarshalBSON mirrored the JSON as a reverse of Unmarshal.
In JSON unmarshal is passed a string (as []byte), Marshal returns the same string.
In BSON, unmarshal is passed size (4 bytes = len(string) + 1) + string + \x00. Marshal returns the same stream (verified as identical)
the call
mongoClient.Database.Collection.InsertOne(ctx, data, nil)
Where data is a structure containing a number of instances of MyTime
error = go.mongodb.org/mongo-driver/mongo.CommandError
error.Code = 22
error.Name = InvalidBSON
error.Message = invalid bson type in element with field name '021-07-16T09:12:31.793395300' in object with _id: "..."
(The field name is the string value I tried to write with the first character (byte) missing)
It should be a string, not a field
With the Marshal Function missing, the parent element (MyTime) was written to mongo as an (empty) object where it should be a string.
Slighty old question, but in case its of help...
MarshalBSON() and UnmarshalBSON() are really for things that marshal/unmarshal to BSON document. Whereas, your type MyTime time.Time is probably better marshalled/unmarshalled as a BSON value - for which there are specific methods that need to be implemented, e.g.
func (mt *MyTime) UnmarshalBSONValue(btype bsontype.Type, data []byte) error {
if btype != bsontype.String {
return errors.New("cannot unmarshal non-string bson value to MyTime")
}
vr := bsonrw.NewBSONValueReader(btype, data)
dec, err := bson.NewDecoder(vr)
if err != nil {
return err
}
var str string
err = dec.Decode(&str)
if err != nil {
return err
}
dt, err := time.Parse(time.RFC3339, str)
if err != nil {
return err
}
*mt = MyTime(dt)
return nil
}
func (mt MyTime) MarshalBSONValue() (bsontype.Type, []byte, error) {
str := time.Time(mt).Format(time.RFC3339)
return bson.MarshalValue(str)
}
I am using crypto in go to take a password and use a passphrase to encrypt the password and then store it as a string in a postgres sql database. The encryption works fine but when I try to add it to my database I get an error that seems to indicate that going from a []byte to a string type messes up the encrypted password.
func Encrypt(password string, passphrase string) string {
data := []byte(password)
block, _ := aes.NewCipher([]byte(createHash(passphrase)))
gcm, err := cipher.NewGCM(block)
if err != nil {
panic(err.Error())
}
nonce := make([]byte, gcm.NonceSize())
if _, err = io.ReadFull(rand.Reader, nonce); err != nil {
panic(err.Error())
}
ciphertext := gcm.Seal(nonce, nonce, data, nil)
return string(ciphertext)
}
func createHash(key string) string {
hasher := md5.New()
hasher.Write([]byte(key))
return hex.EncodeToString(hasher.Sum(nil))
}
When I run this code and try to add the row in the database I get the error
ERROR #22021 invalid byte sequence for encoding "UTF8"
I am just looking for an easy way in go to encrypt a string password and save the encrypted string into a table in postgres. The column in the table is of type VARCHAR but I am willing to change that as well if that is for some reason the issue. Thanks for your time!
Base64 encode it:
return base64.StdEncoding.EncodeToString(ciphertext)
Even though a string is a byte array, not all sequences of bytes are a valid UTF-8 string. Base64 encoding is commonly used to store binary data as text.
I'm trying to persist some files in MongoDB. Not interested in GridFS as files are always less than 256KB. It seems difficult to find any good example on how this typically is done.
What I do is open temporary file for reading to get an io.Reader and use bson.NewFromIOReader(io.Reader) to read bytes into bson.Raw type. However I always get an exception unexpected EOF. After some investigation it seems that this function uses io.ReadFull which may throw that exception. So I'm wondering what I do wrong. I can perform io.Copy operation on the same reader without such exception.
Any ideas?
If your files are only a few KBs, simplest, cleanest (and maybe even fastest) is to use ioutil.ReadFile() to read the complete file into a byte slice. Save the document where one of the property's value is this byte slice. No magic.
For example:
data, err := ioutil.ReadFile("filename.txt")
if err != nil {
// Handle error
}
doc := bson.M{
"data": data,
"created": time.Now(),
}
coll := ... // Obtain collection
if _, err := coll.InsertOne(context.Background(), doc); err != nil {
// handle error
}
In your actual example you may want to use a struct to model your document, e.g.:
type Doc struct {
ID primitive.ObjectID `bson:"_id"`
Data []byte `bson:"data"`
Created time.Time `bson:"created"`
}
And then use:
doc := &Doc{
Data: data,
Created: time.Now(),
}
I am using the GOB encoding for my project and i figured out (after a long fight) that empty strings are not encoded/decoded correctly. In my code i use a errormessage (string) to report any problems, this errormessage is most of the time empty. If i encode a empty string, it become nothing, and this gives me a problem with decoding. I don't want to alter the encoding/decoding because these parts are used the most.
How can i tell Go how to encode/decode empty strings?
Example:
Playground working code.
Playground not working code.
The problem isn't the encoding/gob module, but instead the custom MarshalBinary/UnmarshalBinary methods you've declared for Msg, which can't correctly round trip an empty string. There are two ways you could go here:
Get rid of the MarshalBinary/UnmarshalBinary methods and rely on GOB's default encoding for structures. This change alone wont' be enough because the fields of the structure aren't exported. If you're happy to export the fields then this is the simplest option: https://play.golang.org/p/rwzxTtaIh2
Use an encoding that can correctly round trip empty strings. One simple option would be to use GOB itself to encode the struct fields:
func (m Msg) MarshalBinary() ([]byte, error) {
var b bytes.Buffer
enc := gob.NewEncoder(&b)
if err := enc.Encode(m.x); err != nil {
return nil, err
}
if err := enc.Encode(m.y); err != nil {
return nil, err
}
if err := enc.Encode(m.z); err != nil {
return nil, err
}
return b.Bytes(), nil
}
// UnmarshalBinary modifies the receiver so it must take a pointer receiver.
func (m *Msg) UnmarshalBinary(data []byte) error {
dec := gob.NewDecoder(bytes.NewBuffer(data))
if err := dec.Decode(&m.x); err != nil {
return err
}
if err := dec.Decode(&m.y); err != nil {
return err
}
return dec.Decode(&m.z)
}
You can experiment with this example here: https://play.golang.org/p/oNXgt88FtK
The first option is obviously easier, but the second might be useful if your real example is a little more complex. Be careful with custom encoders though: GOB includes a few features that are intended to detect incompatibilities (e.g. if you add a field to a struct and try to decode old data), which are missing from this custom encoding.
I am using postgresql as my backend database.
Tried to scan a field languagespoken which is an array of text
var user userprofile
row := core.db.QueryRow(
"SELECT languagespoken FROM \"user\" WHERE id = $1",
userId,
)
err := row.Scan(&user.Languages)
if err != nil {
return user, err
}
My structure looks like this
type userprofile struct {
Languages []string `json:languages`
}
But getting the error
2014/06/30 15:27:17 PANIC: reflect.Set: **value of type []uint8 is not assignable to type []string**
/usr/lib/go/src/pkg/reflect/value.go:2198 (0x56c152)
Value.assignTo: panic(context + ": value of type " + v.typ.String() + " is not assignable to type " + dst.String())
/usr/lib/go/src/pkg/reflect/value.go:1385 (0x56966b)
Value.Set: x = x.assignTo("reflect.Set", v.typ, target)
/usr/lib/go/src/pkg/database/sql/convert.go:215 (0x492d70)
convertAssign: dv.Set(sv)
/usr/lib/go/src/pkg/database/sql/sql.go:1562 (0x49c0e5)
(*Rows).Scan: err := convertAssign(dest[i], sv)
/usr/lib/go/src/pkg/database/sql/sql.go:1630 (0x49c560)
(*Row).Scan: err := r.rows.Scan(dest...)
/home/ceresti/source/gocode/src/ceresti.kilnhg.com/ceresti/server/app/databaseapi.go:144 (0x402478)
(*coreStruct).GetUserProfile: err := row.Scan(&user.Languages)
/home/ceresti/source/gocode/src/ceresti.kilnhg.com/ceresti/server/app/restfulapi.go:327 (0x40a63c)
Getuserprofile: userprofileStruct, err := core.GetUserProfile(userId)
/usr/lib/go/src/pkg/runtime/asm_amd64.s:340 (0x4309c2)
Please let me know how to resolve this issue.
Not all sql databases specify array types (e.g., sqlite3). Go doesn't support any sql implementations directly, but it supplies an implementation-agnostic interface (in the generic sense of the word) for which third-party drivers may be written. Go doesn't impose any limitations on which types its drivers may support, so if a type doesn't cooperate, it's probably the fault of the driver.
TL;DR: Try getting it as a string
// if `string` doesn't work, try `[]uint8` or `[]byte` and then convert
// that output to a string if necessary
var languages string
if err := row.Scan(&languages); err != nil {
// handle error
}
// depending on how the string is encoded, this may or may not work
// Update: since you say your list is encoded in the form:
// `{elem1, elem2, elem3}`, we'll simply ignore the first and last
// characters and split on ", "
user.Languages = strings.Split(languages[1:len(languages)-1], ", ")
It appears you're trying to scan the entire result set of that database query in one shot. You can't do that; you need to read each row, one at a time, into a byte slice, then convert the byte slice into a string.
Since you're serializing into a []string, saving the byte slice each time is not a priority. In this case, you can use sql.RawBytes instead of []byte, which will reuse the same memory.
// error checking elided
var row sql.RawBytes
myRow.Scan(&row) // note the pointer!
str := string(row)