I am using crypto in go to take a password and use a passphrase to encrypt the password and then store it as a string in a postgres sql database. The encryption works fine but when I try to add it to my database I get an error that seems to indicate that going from a []byte to a string type messes up the encrypted password.
func Encrypt(password string, passphrase string) string {
data := []byte(password)
block, _ := aes.NewCipher([]byte(createHash(passphrase)))
gcm, err := cipher.NewGCM(block)
if err != nil {
panic(err.Error())
}
nonce := make([]byte, gcm.NonceSize())
if _, err = io.ReadFull(rand.Reader, nonce); err != nil {
panic(err.Error())
}
ciphertext := gcm.Seal(nonce, nonce, data, nil)
return string(ciphertext)
}
func createHash(key string) string {
hasher := md5.New()
hasher.Write([]byte(key))
return hex.EncodeToString(hasher.Sum(nil))
}
When I run this code and try to add the row in the database I get the error
ERROR #22021 invalid byte sequence for encoding "UTF8"
I am just looking for an easy way in go to encrypt a string password and save the encrypted string into a table in postgres. The column in the table is of type VARCHAR but I am willing to change that as well if that is for some reason the issue. Thanks for your time!
Base64 encode it:
return base64.StdEncoding.EncodeToString(ciphertext)
Even though a string is a byte array, not all sequences of bytes are a valid UTF-8 string. Base64 encoding is commonly used to store binary data as text.
Related
Our app uses the library, SecureHash object, in order to create one-way password:
https://github.com/nremond/pbkdf2-scala/blob/master/src/main/scala/io/github/nremond/SecureHash.scala
Now my problem is that my code in Go returns -1 for password check.
package main
import (
"bytes"
"crypto/rand"
"crypto/sha512"
"fmt"
"golang.org/x/crypto/pbkdf2"
"math/big"
"strings"
)
func main() {
iteration := 25000
// Hash User input
password := []byte("123")
salt := "yCyQMMMBt1TuPa1F9FeKfT0yrNIF8tLB"
key := pbkdf2.Key(password, []byte(salt), iteration, sha512.Size, sha512.New)
// COMPARE PASSWORD fetched from DB
// 123 hash in scala
tokenS := strings.Split("$pbkdf2-sha512$25000$yCyQMMMBt1TuPa1F9FeKfT0yrNIF8tLB$TtQt5BZLs4qlA0YAkcGukZwu7pkxOLcxwuoQB3qNtxM", "$")
passwordHashInDatabase := tokenS[4]
out := bytes.Compare(key, []byte(passwordHashInDatabase))
fmt.Println("out: ", out)
}
You have a couple problems:
You're not base64 decoding the salt before passing it to pbkdf2.Key() and you're not base64 decoding the key fetched from the database before comparing it to the result from pbkdf2.Key(). Also note that the Scala implementation does some character replacement before/after base64 decoding/encoding. This too needs to be replicated in the Go implementation.
The createHash() method in the Scala implementation has a dkLength parameter which defaults to 32. In the Go implementation, you're instead providing the result of sha512.Size, which is 64 and does not match. I know that the default value was used because the value from the database is 32 bytes long.
Here's a hastily fixed implemnentation:
package main
import (
"bytes"
"crypto/sha512"
"encoding/base64"
"fmt"
"log"
"strings"
"golang.org/x/crypto/pbkdf2"
)
func b64Decode(s string) ([]byte, error) {
s = strings.ReplaceAll(s, ".", "+")
return base64.RawStdEncoding.DecodeString(s)
}
func main() {
iteration := 25000
// Hash User input
password := []byte("123")
salt, err := b64Decode("yCyQMMMBt1TuPa1F9FeKfT0yrNIF8tLB")
if err != nil {
log.Fatal("Failed to base64 decode the salt: %s", err)
}
key := pbkdf2.Key(password, salt, iteration, 32, sha512.New)
// COMPARE PASSWORD fetched from DB
// 123 hash in scala
tokens := strings.Split("$pbkdf2-sha512$25000$yCyQMMMBt1TuPa1F9FeKfT0yrNIF8tLB$TtQt5BZLs4qlA0YAkcGukZwu7pkxOLcxwuoQB3qNtxM", "$")
passwordHashInDatabase, err := b64Decode(tokens[4])
if err != nil {
log.Fatal("Failed to base64 decode password hash from the database: %s", err)
}
fmt.Printf("%x\n%x\n", key, passwordHashInDatabase)
fmt.Printf("%d\n%d\n", len(key), len(passwordHashInDatabase))
out := bytes.Compare(key, passwordHashInDatabase)
fmt.Println("out: ", out)
}
Output:
4ed42de4164bb38aa503460091c1ae919c2eee993138b731c2ea10077a8db713
4ed42de4164bb38aa503460091c1ae919c2eee993138b731c2ea10077a8db713
32
32
out: 0
Go Playground
Verification fails because:
the hash to be verified, tokenS[4], has a length of 32 bytes, while the calculated hash, key, has a length of 64 bytes,
the salt is not Base64 decoded before the hash is computed,
when comparing, the hash to be verified is Base64 encoded while the calculated hash is raw.
A possible fix is:
iteration := 25000
// Hash User input
password := []byte("123")
salt, _ := base64.RawStdEncoding.DecodeString("yCyQMMMBt1TuPa1F9FeKfT0yrNIF8tLB") // Fix 1: Base64 decode salt (Base64: without padding and with . instead of +)
key := pbkdf2.Key(password, []byte(salt), iteration, sha256.Size, sha512.New) // Fix 2: Apply an output size of 32 bytes
// COMPARE PASSWORD fetched from DB
// 123 hash in scala
tokenS := strings.Split("$pbkdf2-sha512$25000$yCyQMMMBt1TuPa1F9FeKfT0yrNIF8tLB$TtQt5BZLs4qlA0YAkcGukZwu7pkxOLcxwuoQB3qNtxM", "$")
passwordHashInDatabase, _ := base64.RawStdEncoding.DecodeString(tokenS[4]) // Fix 3: Base64 decode the hash to be verified (Base64: without padding and with . instead of +)
out := bytes.Compare(key, passwordHashInDatabase) // better apply an constant-time comparison
fmt.Println("out: ", out) // 0
Although this code works for this particular token, in general a modified Base64 is applied with . instead of + (thus, if . are present, they must be replaced by + before Base64 decoding) and without padding (the latter is the reason for using base64.RawStdEncoding in above code snippet).
Note that there is a passlib implementation for Go (passlib package), but it seems to use essentially default values (e.g. an output size of 64 bytes for pbkdf2-sha512) and so cannot be applied directly here.
Nevertheless, this implementation can be used as a blueprint for your own implementation (e.g. regarding Base64 encoding, s. Base64Decode(), constant-time comparison, s. SecureCompare(), preventive against side channel attacks etc.).
I have generated my KMS key in Golang using this code
// Create the key
result, err := svc.CreateKey(&kms.CreateKeyInput{
Tags: []*kms.Tag{
{
TagKey: aws.String("CreatedBy"),
TagValue: aws.String("User"),
},
},
})
And this is the line causing the error during Decrypt
//Get body bytes
body, _ := ioutil.ReadAll(resp.Body)
//base64Text := make([]byte, base64.StdEncoding.EncodedLen(len(body)))
//base64.StdEncoding.Encode(base64Text, []byte(body))
//Needs decrypting?
var newfile *bytes.Reader
if decrypt == true {
log.Println("Now decrypting data")
log.Println("resp.SSECustomerAlgorithm : ", resp.SSECustomerAlgorithm)
log.Println("resp.SSEKMSKeyId : ", aws.StringValue(resp.SSEKMSKeyId)) //right key found.
newkms := inputfs.NewKms()
//n := map[string]*string{"CreatedBy": aws.String("User") }
params := &kms.DecryptInput{
CiphertextBlob: body,
//EncryptionContext:n,
}
output, err := newkms.Decrypt(params)
if err != nil {
log.Println("Decrypt error :", err)
}
newbody := output.Plaintext
newfile = bytes.NewReader(newbody)
}
Error at Decrypt error
Decrypt error : InvalidCiphertextException:
status code: 400, request id: b69a8634-1784-4c57-8d3d-2439041249fe
What could possibly cause this? Nothing I do seems to work. Here is a related question: Decrypting using AWS go sdk but i don't see the answer.
It turns out my objects were already decrypted.
My objects were originally KMS encrypted using S3 PUTobject operation.
AWS S3 automatically decrypts such objects on S3 GETobject operation.
I can actually get the particular object in plaintext, so this supports the above statement.
See Request Headers and the Amazon S3 API
I'm trying to store encoded data using encoding/gob from Golang into Postgres. I'm using Gorm as well.
First, sending form data using
if err := json.NewDecoder(r.Body).Decode(model); err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
Currently client_encoding is set to UTF8 in the postgres database. Here's what I'm using to encode:
type payload struct {
Key []byte
Nonce *[nonceLength]byte
Message []byte
}
// Encrypt message
p.Message = secretbox.Seal(p.Message, plaintext, p.Nonce, key) // key set prior to this
buf := &bytes.Buffer{}
if err := gob.NewEncoder(buf).Encode(p); err != nil {
return nil, err
}
return buf.Bytes(), nil
Then I store string(buf.Bytes()) which is stored in the database column which is currently a string type. Now I'm a novice with encoding, and I think gob just has a different encoding for my database. I'm receiving this error in console:
(pq: invalid byte sequence for encoding "UTF8": 0xff)
I've been following this gist for encryption/decryption:
https://gist.github.com/fuzzyami/f3a7231037166117a6fef9607960aee7
From what I've read, I shouldn't be encoding structs into the db, in this case p, unless using gob. Correct me if I'm wrong with that (can't find the resource at the moment where I found this).
Is anyone able to point me in the right direction for storing this data in Postgres which is decrypted later? I'm clearly not understanding the encoding process, and not entirely sure where to start out here with reading resources, so any help is appreciated!
Use a bytea column in postgresql to store a []byte, and skip the conversion to string.
Took a look at https://golang.org/src/encoding/base64/example_test.go
Was able to use
return base64.StdEncoding.EncodeToString(buf.Bytes()), nil
Which successfully stored in the database.
I have this code:
u := models.Users{}
u = u.FindByEmail(login.Email)
password := []byte(login.Password)
hashedPassword, err := bcrypt.GenerateFromPassword(password, bcrypt.DefaultCost)
if err != nil {
panic(err)
}
err = bcrypt.CompareHashAndPassword(hashedPassword, []byte(u.Password))
fmt.Println(err)
I end up getting this error: crypto/bcrypt: hashedPassword is not the hash of the given password
However I previously saved my model to have the same hash as "admin", but when I run my application, it tells me it is not equal.
Re-read the docs carefully.
CompareHashAndPassword compares a bcrypt hashed password with its possible plaintext equivalent. Returns nil on success, or an error on failure.
Basically, it is saying that you should compare the hash you have stored against the plain text password.
you probably want:
u := models.Users{}
u = u.FindByEmail(login.Email)
plainPassword := []byte(login.Password)
// Assumes that u.Password is the actual hash and that you didn't store plain text password.
err = bcrypt.CompareHashAndPassword([]byte(u.Password), plainPassword)
fmt.Println(err)
I am using the GOB encoding for my project and i figured out (after a long fight) that empty strings are not encoded/decoded correctly. In my code i use a errormessage (string) to report any problems, this errormessage is most of the time empty. If i encode a empty string, it become nothing, and this gives me a problem with decoding. I don't want to alter the encoding/decoding because these parts are used the most.
How can i tell Go how to encode/decode empty strings?
Example:
Playground working code.
Playground not working code.
The problem isn't the encoding/gob module, but instead the custom MarshalBinary/UnmarshalBinary methods you've declared for Msg, which can't correctly round trip an empty string. There are two ways you could go here:
Get rid of the MarshalBinary/UnmarshalBinary methods and rely on GOB's default encoding for structures. This change alone wont' be enough because the fields of the structure aren't exported. If you're happy to export the fields then this is the simplest option: https://play.golang.org/p/rwzxTtaIh2
Use an encoding that can correctly round trip empty strings. One simple option would be to use GOB itself to encode the struct fields:
func (m Msg) MarshalBinary() ([]byte, error) {
var b bytes.Buffer
enc := gob.NewEncoder(&b)
if err := enc.Encode(m.x); err != nil {
return nil, err
}
if err := enc.Encode(m.y); err != nil {
return nil, err
}
if err := enc.Encode(m.z); err != nil {
return nil, err
}
return b.Bytes(), nil
}
// UnmarshalBinary modifies the receiver so it must take a pointer receiver.
func (m *Msg) UnmarshalBinary(data []byte) error {
dec := gob.NewDecoder(bytes.NewBuffer(data))
if err := dec.Decode(&m.x); err != nil {
return err
}
if err := dec.Decode(&m.y); err != nil {
return err
}
return dec.Decode(&m.z)
}
You can experiment with this example here: https://play.golang.org/p/oNXgt88FtK
The first option is obviously easier, but the second might be useful if your real example is a little more complex. Be careful with custom encoders though: GOB includes a few features that are intended to detect incompatibilities (e.g. if you add a field to a struct and try to decode old data), which are missing from this custom encoding.