A peculiar error occurred today when was manipulating data in Postgres using database/sql and driver github.com/lib/pq. I have the following SQL schema created in Postgres:
CREATE TABLE IF NOT EXISTS bench_bytea (
id INT PRIMARY KEY,
name VARCHAR,
data BYTEA
);
A very basic table containing a data blob of type BYTEA.
I then tried to execute simple INSERT statement using the Exec() function provided by database/sql. Here it is:
psqlInfo := fmt.Sprintf("host=%s port=%d user=%s "+
"password=%s dbname=%s sslmode=disable",
host, port, user, password, dbname)
db, err := sql.Open("postgres", psqlInfo)
if err != nil {
panic(err)
}
defer db.Close()
stmt := `INSERT INTO bench_bytea (id, name, data) VALUES ($1, $2, $3) ON CONFLICT (id) DO NOTHING`
data := `{"title": "Sleeping Beauties", "genres": ["Fiction", "Thriller", "Horror"], "published": false}`
i := 0
_, err = db.Exec(stmt, i, "testing "+string(i), []byte(data))
if err != nil {
panic(err)
}
The key highlight happens on the db.Exec() line where I execute a SQL INSERT statement (in practicality i is an index of an array where I stored different testing data. I didn't want to include the other pieces of data here since it's really long and irrelevant). The error I received is:
pq: invalid byte sequence for encoding "UTF8": 0x00
Now if I change "testing "+string(i) into "testing", the error is gone. That is, if I didn't insert a concatenating strings into the name column, there's no error. What is going on here?
You can't convert an integer to a string like that. The result of string(0) is "\x00", aka a null byte, instead of "0" which is probably you want. You should use strconv.Itoa for the conversion instead.
Related
I am pretty new to Go and I am trying to find the best way to set up my db communication. Essentially I remember from my previous workplaces that in PHP you can create a class that represents a SQL table and when you need to insert data into your db you would create an object of that class with all the necessary data, call insert(), pass your object and it would insert that data into a corresponding table without you writing any SQL code, update() works in a very similar way except it would update instead of inserting. Unfortunately, I don't remember the name of that PHP framework but maybe someone knows a way to achieve something like that in Go or is it not a thing?
Lets say I have a struct:
type Patients struct {
ID int
Name string
Image string
}
Now I want to have a function that takes Patients objet as a parameter and inserts it into a patients postgres table automatically converting patient into what postgres expects:
func (patients *Patients) insert(patient Patients) {
}
And then update() would take a Patients object and basically perform this chunk of code without me writing it:
stmt := `update patients set
name = $1,
image = $2,
where id = $3
`
_, err := db.ExecContext(ctx, stmt,
patient.Name,
patient.Image,
patient.ID
)
You are looking for something called an ORM (Object Relational Mapper). There are a few in Go, but the most popular is GORM. It's a bit of a controversial topic, but I think it's a good idea to use an ORM if you're new to Go and/or databases. It will save you a lot of time and effort.
The alternative is to use the database/sql package and write your own SQL queries. This is a good idea if you're an experienced Go developer and/or database administrator. It will give you more control over your queries and will be more efficient. Recommended reading: https://www.alexedwards.net/blog/organising-database-access. Recommended libraries for this approach include sqlx and pgx.
Here is what your struct would look like as a GORM model:
type Patient struct {
ID int `gorm:"primaryKey"`
Name string
Image string
}
And here is an example program for how to insert a patient into the database:
package main
import (
"fmt"
"gorm.io/driver/postgres"
"gorm.io/gorm"
)
type Patient struct {
ID int `gorm:"primaryKey"`
Name string
Image string
}
func main() {
dsn := "host=localhost user=postgres password=postgres dbname=postgres port=5432 sslmode=disable TimeZone=UTC"
db, err := gorm.Open(postgres.Open(dsn), &gorm.Config{})
if err != nil {
panic("failed to connect database")
}
db.AutoMigrate(&Patient{})
patient := Patient{
Name: "John Smith",
Image: "https://example.com/image.png",
}
result := db.Create(&patient)
if result.Error != nil {
panic(result.Error)
}
fmt.Println(patient)
}
If instead you wanted to use sqlx, you would write something like this:
package main
import (
"database/sql"
"fmt"
"log"
_ "github.com/lib/pq"
)
type Patient struct {
ID int
Name string
Image string
}
func main() {
dsn := "host=localhost user=postgres password=postgres dbname=postgres port=5432 sslmode=disable TimeZone=UTC"
db, err := sql.Open("postgres", dsn)
if err != nil {
log.Fatal(err)
}
defer db.Close()
_, err = db.Exec(`
CREATE TABLE IF NOT EXISTS patients (
id SERIAL PRIMARY KEY,
name TEXT,
image TEXT
)
`)
if err != nil {
log.Fatal(err)
}
patient := Patient{
Name: "John Smith",
Image: "https://example.com/image.png",
}
_, err = db.Exec(`
INSERT INTO patients (name, image) VALUES ($1, $2)
`, patient.Name, patient.Image)
if err != nil {
log.Fatal(err)
}
fmt.Println(patient)
}
Of course, managing your database schema is a bit more complicated with an ORM. You can use migrations, but I prefer to use a tool called goose. It's a bit of a pain to set up, but it's very powerful and flexible. Here is an example of how to use it:
package main
import (
"fmt"
"log"
"github.com/pressly/goose"
"gorm.io/driver/postgres"
"gorm.io/gorm"
)
type Patient struct {
ID int `gorm:"primaryKey"`
Name string
Image string
}
func main() {
dsn := "host=localhost user=postgres password=postgres dbname=postgres port=5432 sslmode=disable TimeZone=UTC"
db, err := gorm.Open(postgres.Open(dsn), &gorm.Config{})
if err != nil {
panic("failed to connect database")
}
goose.SetDialect("postgres")
goose.SetTableName("schema_migrations")
err = goose.Run("up", db.DB(), "migrations")
if err != nil {
log.Fatal(err)
}
patient := Patient{
Name: "John Smith",
Image: "https://example.com/image.png",
}
result := db.Create(&patient)
if result.Error != nil {
panic(result.Error)
}
fmt.Println(patient)
}
where your migrations directory looks like this:
migrations/
00001_create_patients.up.sql
00001_create_patients.down.sql
and your migrations look like this:
-- 00001_create_patients.up.sql
CREATE TABLE patients (
id SERIAL PRIMARY KEY,
name TEXT,
image TEXT
);
-- 00001_create_patients.down.sql
DROP TABLE patients;
I hope this helps! Let me know if you have any questions.
I think what you're looking for is an ORM. An ORM is a library that essentially does this, taking language structures and automatically handling the SQL logic for you.
The most popular library for this in Go is GORM. Here's a link to their home page: https://gorm.io/. I've used it heavily in production and it's been a good experience!
The docs have a good example of what it'll look like.
Hope this helps.
I have a Postgres database, normally when you execute commands from psql you get some sort of an output like in the example bellow
test=> CREATE SCHEMA IF NOT EXISTS algo;
NOTICE: schema "algo" already exists, skipping
CREATE SCHEMA
test=> CREATE SCHEMA IF NOT EXISTS algo2;
CREATE SCHEMA
I didn't find a way to get these output messages when using an sql client in go. The db.Exec function is returning sql.Result which has LastInsertId and RowsAffected but no output message. Is it possible to do it with sql client in go?
here is my Go code.
package main
import (
"database/sql"
"fmt"
)
func main() {
psqlInfo := fmt.Sprintf("host=%s port=%d user=%s "+" dbname=%s password=%s sslmode=disable", host, port, user, dbname, password)
db, err := sql.Open("postgres", psqlInfo)
if err != nil {
panic(err)
}
res, err := db.Exec("CREATE SCHEMA IF NOT EXISTS algo;")
if err != nil {
panic(err)
}
lastInsert, _ := res.LastInsertId()
RoswAffected, _ := res.RowsAffected()
fmt.Println(lastInsert, RoswAffected)
// output: 0 0
}
I am trying to update a record in a postgres table with an array (slice) of values. The table has the following DDL:
CREATE TABLE slm_files (
id uuid DEFAULT gen_random_uuid() PRIMARY KEY,
filename character varying NOT NULL,
status character varying NOT NULL,
original_headers text[]
);
and the Go code I have is as follows:
package main
import (
"context"
"database/sql"
"encoding/json"
"fmt"
"os"
"strings"
"time"
"github.com/aws/aws-lambda-go/events"
"github.com/aws/aws-lambda-go/lambda"
"github.com/lib/pq"
)
type message struct {
ID string `json:"id"`
Filename string `json:"filename"`
Status string `json:"status"`
OriginalHeaders []string `json:"OriginalHeaders"`
}
func main() {
host := os.Getenv("PGhost")
port := 5432
user := os.Getenv("PGuser")
password := os.Getenv("PGpassword")
dbname := os.Getenv("PGdbname")
pgConString := fmt.Sprintf("port=%d host=%s user=%s "+
"password=%s dbname=%s sslmode=disable",
port, host, user, password, dbname)
msgBody := `update_headers___
{
"id": "76b67119-d8c1-4a20-b53e-49e4972e2f19",
"filename": "SLM1171_inputData_preNCOA-5babc88b-1d14-468d-bf6e-c3b36ce90d95.csv",
"status": "Submitted",
"OriginalHeaders": [
"city",
"state",
"zipcode",
"full_name",
"individual_id"
]
}`
fmt.Println("Processing file", msgBody)
queryMethod := strings.Split(msgBody, "___")[0]
fieldDict := strings.Split(msgBody, "___")[1]
db, err := sql.Open("postgres", pgConString)
if err != nil {
panic(err)
}
fmt.Println("Connected Successfully")
defer db.Close()
body := message{}
json.Unmarshal([]byte(fieldDict), &body)
fmt.Println(queryMethod)
fmt.Println(body)
var sqlStatement string
switch queryMethod {
case "update_ncoa":
sqlStatement = fmt.Sprintf(`UPDATE slm_files SET status = '%s', updated_at = '%s' where id = '%s';`,
body.Status,
body.UpdatedAt,
body.ID,
)
case "update_headers":
sqlStatement = fmt.Sprintf(`UPDATE slm_files SET original_headers = '%s', updated_at = '%s' where id = '%s';`,
pq.Array(body.OriginalHeaders),
body.UpdatedAt,
body.ID,
)
}
fmt.Println(sqlStatement)
_, err = db.Query(sqlStatement)
if err != nil {
fmt.Println("Failed to run query", err)
return
}
}
fmt.Println("Query executed!")
return
}
but I keep getting the error
pq: malformed array literal: "&[first_name last_name city state zipcode full_name individual_id]": Error
null
I have read a few things on the internet that lead me to using pq.Array() but that doesnt seem to work.
I have read about the difference in format between Go arrays and Postgres arrays, so I had hoped that letting the pq.Array function would sort it out but apparently not.
As Peter advised, there's a lot to fix up with that database handling. And it's definitely worth redoing those SQL statements to not use Sprintf to make the query.
But in terms of just getting something working with postgres arrays and the pq library, you need to use the Value() method of pq.Array to get the postgres format. Change your update statement for the headers to something like this:
arrayVal, _ := pq.Array(body.OriginalHeaders).Value()
sqlStatement = fmt.Sprintf(`UPDATE slm_files SET original_headers = '%s', updated_at = '%s' where id = '%s';`,
arrayVal,
body.UpdatedAt,
body.ID,
)
And it's worth checking the return from the Value() method to make sure there are no errors, I just ignored it for the sake of a simple example.
type User struct {
Email string `json:"email"`
Password string `json:"password"`
}
db, err := sql.Open("postgres", os.Getenv("DATABASE_URL"))
if err != nil {
log.Fatal(err)
}
fmt.Println("email: ", reflect.TypeOf(usr.Email)) //string
fmt.Println("salt: ", reflect.TypeOf(salt)) //[]uint8
fmt.Println("hash: ", reflect.TypeOf(hash)) //string
sql := `INSERT INTO public."Users" (email, password, salt) VALUES ($1, $2, $3)`
_, err = db.Exec(sql, usr.Email, hash, salt)
throws error: "pq: invalid byte sequence for encoding "UTF8": 0x97"
my table: "email" type: TEXT, "password" type: TEXT, "salt" type: smallint[] (thinking this might be the cause of the error but I am not sure what to use instead)
PostgreSQL bytea = []unit8 Golang
changed type and issue was resolved!
I created a simple sql database with a BYTEA field,
create table testhex (testhex bytea);
insert into testhex (testhex) values ('\x123456');
and then I tried to query it from Go.
package main
import (
"database/sql"
_ "github.com/lib/pq"
)
func main(){
var err error
db, err := sql.Open("postgres", "dbname=testhex sslmode=disable")
if err != nil {
panic(err)
}
var result string
err = db.QueryRow("select testhex from testhex where testhex = $1", `\x123456`).Scan(&result)
if err != nil {
panic(err)
}
}
It doesn't find the row. What am I doing wrong?
When you ran the following query:
insert into testhex (testhex) values ('\x123456');
You inserted the 3 byte sequence [0x12 0x34 0x56] into the table. For the database query you're executing with QueryRow though, you're searching for the 8 character literal string \x123456 so you get no matches in the result.
When you use positional arguments with QueryRow, it is the database adapter's job to convert them to a form the database understands (either by sending them to the database as bound parameters, or by substituting them into the query with appropriate escaping). So by passing an already escaped value you will run into this sort of problem.
Instead, try passing []byte{0x12, 0x34, 0x56} as the positional argument, which should match what is in the database.