I am pretty new to Go and I am trying to find the best way to set up my db communication. Essentially I remember from my previous workplaces that in PHP you can create a class that represents a SQL table and when you need to insert data into your db you would create an object of that class with all the necessary data, call insert(), pass your object and it would insert that data into a corresponding table without you writing any SQL code, update() works in a very similar way except it would update instead of inserting. Unfortunately, I don't remember the name of that PHP framework but maybe someone knows a way to achieve something like that in Go or is it not a thing?
Lets say I have a struct:
type Patients struct {
ID int
Name string
Image string
}
Now I want to have a function that takes Patients objet as a parameter and inserts it into a patients postgres table automatically converting patient into what postgres expects:
func (patients *Patients) insert(patient Patients) {
}
And then update() would take a Patients object and basically perform this chunk of code without me writing it:
stmt := `update patients set
name = $1,
image = $2,
where id = $3
`
_, err := db.ExecContext(ctx, stmt,
patient.Name,
patient.Image,
patient.ID
)
You are looking for something called an ORM (Object Relational Mapper). There are a few in Go, but the most popular is GORM. It's a bit of a controversial topic, but I think it's a good idea to use an ORM if you're new to Go and/or databases. It will save you a lot of time and effort.
The alternative is to use the database/sql package and write your own SQL queries. This is a good idea if you're an experienced Go developer and/or database administrator. It will give you more control over your queries and will be more efficient. Recommended reading: https://www.alexedwards.net/blog/organising-database-access. Recommended libraries for this approach include sqlx and pgx.
Here is what your struct would look like as a GORM model:
type Patient struct {
ID int `gorm:"primaryKey"`
Name string
Image string
}
And here is an example program for how to insert a patient into the database:
package main
import (
"fmt"
"gorm.io/driver/postgres"
"gorm.io/gorm"
)
type Patient struct {
ID int `gorm:"primaryKey"`
Name string
Image string
}
func main() {
dsn := "host=localhost user=postgres password=postgres dbname=postgres port=5432 sslmode=disable TimeZone=UTC"
db, err := gorm.Open(postgres.Open(dsn), &gorm.Config{})
if err != nil {
panic("failed to connect database")
}
db.AutoMigrate(&Patient{})
patient := Patient{
Name: "John Smith",
Image: "https://example.com/image.png",
}
result := db.Create(&patient)
if result.Error != nil {
panic(result.Error)
}
fmt.Println(patient)
}
If instead you wanted to use sqlx, you would write something like this:
package main
import (
"database/sql"
"fmt"
"log"
_ "github.com/lib/pq"
)
type Patient struct {
ID int
Name string
Image string
}
func main() {
dsn := "host=localhost user=postgres password=postgres dbname=postgres port=5432 sslmode=disable TimeZone=UTC"
db, err := sql.Open("postgres", dsn)
if err != nil {
log.Fatal(err)
}
defer db.Close()
_, err = db.Exec(`
CREATE TABLE IF NOT EXISTS patients (
id SERIAL PRIMARY KEY,
name TEXT,
image TEXT
)
`)
if err != nil {
log.Fatal(err)
}
patient := Patient{
Name: "John Smith",
Image: "https://example.com/image.png",
}
_, err = db.Exec(`
INSERT INTO patients (name, image) VALUES ($1, $2)
`, patient.Name, patient.Image)
if err != nil {
log.Fatal(err)
}
fmt.Println(patient)
}
Of course, managing your database schema is a bit more complicated with an ORM. You can use migrations, but I prefer to use a tool called goose. It's a bit of a pain to set up, but it's very powerful and flexible. Here is an example of how to use it:
package main
import (
"fmt"
"log"
"github.com/pressly/goose"
"gorm.io/driver/postgres"
"gorm.io/gorm"
)
type Patient struct {
ID int `gorm:"primaryKey"`
Name string
Image string
}
func main() {
dsn := "host=localhost user=postgres password=postgres dbname=postgres port=5432 sslmode=disable TimeZone=UTC"
db, err := gorm.Open(postgres.Open(dsn), &gorm.Config{})
if err != nil {
panic("failed to connect database")
}
goose.SetDialect("postgres")
goose.SetTableName("schema_migrations")
err = goose.Run("up", db.DB(), "migrations")
if err != nil {
log.Fatal(err)
}
patient := Patient{
Name: "John Smith",
Image: "https://example.com/image.png",
}
result := db.Create(&patient)
if result.Error != nil {
panic(result.Error)
}
fmt.Println(patient)
}
where your migrations directory looks like this:
migrations/
00001_create_patients.up.sql
00001_create_patients.down.sql
and your migrations look like this:
-- 00001_create_patients.up.sql
CREATE TABLE patients (
id SERIAL PRIMARY KEY,
name TEXT,
image TEXT
);
-- 00001_create_patients.down.sql
DROP TABLE patients;
I hope this helps! Let me know if you have any questions.
I think what you're looking for is an ORM. An ORM is a library that essentially does this, taking language structures and automatically handling the SQL logic for you.
The most popular library for this in Go is GORM. Here's a link to their home page: https://gorm.io/. I've used it heavily in production and it's been a good experience!
The docs have a good example of what it'll look like.
Hope this helps.
Related
I'm trying to query from multiple databases. Each database is connected using the following function:
func connectDB(dbEnv str) *sql.DB{
// Loading environment variables from local.env file
err1 := godotenv.Load(dbEnv)
if err1 != nil {
log.Fatalf("Some error occured. Err: %s", err1)
}
dialect := os.Getenv("DIALECT")
host := os.Getenv("HOST")
dbPort := os.Getenv("DBPORT")
user := os.Getenv("USER")
dbName := os.Getenv("NAME")
password := os.Getenv("PASSWORD")
// Database connection string
dbURI := fmt.Sprintf("port=%s host=%s user=%s "+"password=%s dbname=%s sslmode=disable", dbPort, host, user, password, dbName)
// Create database object
db, err := sql.Open(dialect,dbURI)
if err != nil {
log.Fatal(err)
}
return db
}
type order struct{
OrderID string `json:"orderID"`
Name string `json:"name"`
}
type book struct{
OrderID string `json:"orderID"`
Name string `json:"name"`
}
func getOrders(db *sql.DB) []order {
var (
orderID string
name string
)
var allRows = []order{}
query := `
SELECT orderID, name
FROM orders.orders;
`
//Get rows using the query
rows, err := db.Query(query)
if err != nil { //Log if error
log.Fatal(err)
}
defer rows.Close()
// Add each row into the "allRows" slice
for rows.Next() {
err := rows.Scan(&orderID, &name, &date)
if err != nil {
log.Fatal(err)
}
//Create new order struct with the received data
row := order{
OrderID: orderID,
Name: name,
}
allRows = append(allRows, row)
}
//Log if error
err = rows.Err()
if err != nil {
log.Fatal(err)
}
return allRows
}
func getBooks(db *sql.DB) []book{
var (
bookID string
name string
)
var allRows = []book{}
query := `
SELECT bookID, name
FROM books.books;
`
//Get rows using the query
rows, err := db.Query(query)
if err != nil { //Log if error
log.Fatal(err)
}
defer rows.Close()
// Add each row into the "allRows" slice
for rows.Next() {
err := rows.Scan(&bookID, &name)
if err != nil {
log.Fatal(err)
}
//Create new book struct with the received data
row := book{
BookID: bookID,
Name: name,
}
allRows = append(allRows, row)
}
//Log if error
err = rows.Err()
if err != nil {
log.Fatal(err)
}
return allRows
}
func main() {
ordersDB:= connectDB("ordersDB.env")
booksDB:= connectDB("booksDB.env")
orders := getOrders(ordersDB)
books := getBooks(booksDB)
}
The issue is that when I use ordersDB first, the program only recognizes the table in ordersDB. And when I use booksDB first, the program only recognizes the table in booksDB.
When I try to query a table in booksDB after using ordersDB, it is giving me "relation "books.books" does not exist" error. When I try to query a table in ordersDB after using booksDB, it gives "relation "orders.orders" does not exist"
Is there a better way to connect to multiple databases?
You are using github.com/joho/godotenv to load the database configuration from the environment. Summarising (and cutting out a lot of detail) what you are doing is:
godotenv.Load("ordersDB.env")
host := os.Getenv("HOST")
// Connect to DB
godotenv.Load("booksDB.env")
host := os.Getenv("HOST")
// Connect to DB 2
However as stated in the docs "Existing envs take precedence of envs that are loaded later". This is also stated more clearly here "It's important to note that it WILL NOT OVERRIDE an env variable that already exists".
So your code will load in the first .env file, populate the environment variables, and connect to the database. You will then load the second .env file but, because the environmental variables are already set, they will not be changed and you will connect to the same database a second time.
As a work around you could use Overload. However it's probably better to reconsider your use of environmental variables (and perhaps use different variables for the second connection).
I am trying to update a record in a postgres table with an array (slice) of values. The table has the following DDL:
CREATE TABLE slm_files (
id uuid DEFAULT gen_random_uuid() PRIMARY KEY,
filename character varying NOT NULL,
status character varying NOT NULL,
original_headers text[]
);
and the Go code I have is as follows:
package main
import (
"context"
"database/sql"
"encoding/json"
"fmt"
"os"
"strings"
"time"
"github.com/aws/aws-lambda-go/events"
"github.com/aws/aws-lambda-go/lambda"
"github.com/lib/pq"
)
type message struct {
ID string `json:"id"`
Filename string `json:"filename"`
Status string `json:"status"`
OriginalHeaders []string `json:"OriginalHeaders"`
}
func main() {
host := os.Getenv("PGhost")
port := 5432
user := os.Getenv("PGuser")
password := os.Getenv("PGpassword")
dbname := os.Getenv("PGdbname")
pgConString := fmt.Sprintf("port=%d host=%s user=%s "+
"password=%s dbname=%s sslmode=disable",
port, host, user, password, dbname)
msgBody := `update_headers___
{
"id": "76b67119-d8c1-4a20-b53e-49e4972e2f19",
"filename": "SLM1171_inputData_preNCOA-5babc88b-1d14-468d-bf6e-c3b36ce90d95.csv",
"status": "Submitted",
"OriginalHeaders": [
"city",
"state",
"zipcode",
"full_name",
"individual_id"
]
}`
fmt.Println("Processing file", msgBody)
queryMethod := strings.Split(msgBody, "___")[0]
fieldDict := strings.Split(msgBody, "___")[1]
db, err := sql.Open("postgres", pgConString)
if err != nil {
panic(err)
}
fmt.Println("Connected Successfully")
defer db.Close()
body := message{}
json.Unmarshal([]byte(fieldDict), &body)
fmt.Println(queryMethod)
fmt.Println(body)
var sqlStatement string
switch queryMethod {
case "update_ncoa":
sqlStatement = fmt.Sprintf(`UPDATE slm_files SET status = '%s', updated_at = '%s' where id = '%s';`,
body.Status,
body.UpdatedAt,
body.ID,
)
case "update_headers":
sqlStatement = fmt.Sprintf(`UPDATE slm_files SET original_headers = '%s', updated_at = '%s' where id = '%s';`,
pq.Array(body.OriginalHeaders),
body.UpdatedAt,
body.ID,
)
}
fmt.Println(sqlStatement)
_, err = db.Query(sqlStatement)
if err != nil {
fmt.Println("Failed to run query", err)
return
}
}
fmt.Println("Query executed!")
return
}
but I keep getting the error
pq: malformed array literal: "&[first_name last_name city state zipcode full_name individual_id]": Error
null
I have read a few things on the internet that lead me to using pq.Array() but that doesnt seem to work.
I have read about the difference in format between Go arrays and Postgres arrays, so I had hoped that letting the pq.Array function would sort it out but apparently not.
As Peter advised, there's a lot to fix up with that database handling. And it's definitely worth redoing those SQL statements to not use Sprintf to make the query.
But in terms of just getting something working with postgres arrays and the pq library, you need to use the Value() method of pq.Array to get the postgres format. Change your update statement for the headers to something like this:
arrayVal, _ := pq.Array(body.OriginalHeaders).Value()
sqlStatement = fmt.Sprintf(`UPDATE slm_files SET original_headers = '%s', updated_at = '%s' where id = '%s';`,
arrayVal,
body.UpdatedAt,
body.ID,
)
And it's worth checking the return from the Value() method to make sure there are no errors, I just ignored it for the sake of a simple example.
Introduction
database/sql
In the Go standard sql library, the *Stmt type has methods defined like:
func (s *Stmt) Exec(args ...interface{}) (Result, error)
func (s *Stmt) Query(args ...interface{}) (*Rows, error)
The a new (unnamed) statement is prepared by:
func (db *DB) Prepare(query string) (*Stmt, error)
Connection pool is abstracted and not directly accessible
A transaction is prepared on a single connection
If the connection is not available at statment execution time, it will be re-prepared on a new connection.
pgx
The PreparedStatement type doesn't have any methods defined. A new named prepared statement is prepared by:
func (p *ConnPool) Prepare(name, sql string) (*PreparedStatement, error)
Operations are directly on the connection pool
The transaction gets prepared on all connections of the pool
There is no clear way how to execute the prepared statement
In a Github comment, the author explains better the differences of architecture between pgx and database/sql. The documentation on Prepare also states (emphasis mine):
Prepare is idempotent; i.e. it is safe to call Prepare multiple times with the same name and sql arguments. This allows a code path to Prepare and Query/Exec/PrepareEx without concern for if the statement has already been prepared.
Small example
package main
import (
"github.com/jackc/pgx"
)
func main() {
conf := pgx.ConnPoolConfig{
ConnConfig: pgx.ConnConfig{
Host: "/run/postgresql",
User: "postgres",
Database: "test",
},
MaxConnections: 5,
}
db, err := pgx.NewConnPool(conf)
if err != nil {
panic(err)
}
_, err = db.Prepare("my-query", "select $1")
if err != nil {
panic(err)
}
// What to do with the prepared statement?
}
Question(s)
The name argument gives me the impression it can be executed by calling it by name, but how?
The documentation gives the impression that Query/Exec methods somehow leverage the prepared statements. However, those methods don't take a name argument. How does it match them?
Presumably, matching is done by the query content. Then what's the whole point of naming statements?
Possible answers
This is how far I got myself:
There are no methods that refer to the queries by name (assumption)
Matching is done on the query body in conn.ExecEx(). If it is not yet prepared, it will be done:
ps, ok := c.preparedStatements[sql]
if !ok {
var err error
ps, err = c.prepareEx("", sql, nil)
if err != nil {
return "", err
}
}
PosgreSQL itself needs it for something (assumption).
#mkopriva pointed out that the sql text was misleading me. It has a double function here. If the sql variable does not match to a key in the c.preparedStatements[sql] map, the query contained in the sql gets prepared and a new *PreparedStatement struct is appointed to ps. If it did match a key, the ps variable will point to an entry of the map.
So effectively you can do something like:
package main
import (
"fmt"
"github.com/jackc/pgx"
)
func main() {
conf := pgx.ConnPoolConfig{
ConnConfig: pgx.ConnConfig{
Host: "/run/postgresql",
User: "postgres",
Database: "test",
},
MaxConnections: 5,
}
db, err := pgx.NewConnPool(conf)
if err != nil {
panic(err)
}
if _, err := db.Prepare("my-query", "select $1::int"); err != nil {
panic(err)
}
row := db.QueryRow("my-query", 10)
var i int
if err := row.Scan(&i); err != nil {
panic(err)
}
fmt.Println(i)
}
I'm using go and the package uuid to generate a uuid of type [16]byte. However when I try to insert that uuid into my postgres column of type uuid I get the error converting argument $1 type: unsupported type [16]uint8, a array. So apparently I should convert the uuid on the client before I insert it into the db. How should I do that? What type should I convert it to?
In short: What go data type will work with uuid in postgres?
Thanks to the link from #sberry, I found success. Here are snippets of the code for your benefit (with a PostgreSQL 9.5 database):
import (
"database/sql"
"net/http"
"github.com/google/uuid"
)
type Thing struct {
ID uuid.UUID `json:"-" sql:",type:uuid"`
Name string `json:"name"`
}
// For a database table created as such:
// CREATE TABLE things (
// id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
// name TEXT DEFAULT ''::text
// )
func selectThingssSQL() ([]Thing, error) {
things := make([]Thing, 0)
rows, err := db.Query("SELECT id, name FROM things")
if err != nil {
return nil, err
}
defer rows.Close()
for rows.Next() {
t := &Thing{}
if err := rows.Scan(&t.ID, &t.Name); err != nil {
return nil, err
}
things = append(things, *t)
}
return things, nil
}
I'm brand new to Go, and I've started working on some postgres queries, and I'm having very little luck.
I have a package that's just going to have some database queries in it. Here's my code.
main.go
package main
import (
"fmt"
)
func main() {
fmt.Println("Querying data")
myqueries.SelectAll("mytable")
}
myqueries.go
package myqueries
import (
"database/sql"
"fmt"
)
func SelectAll (table string) {
db, err := sql.Open("postgres","user=postgres dbname=mydb sslmode=disable")
if err != nil {
fmt.Println(err)
}
defer db.Close()
rows, err := db.Query("SELECT * FROM $1", table)
if err != nil {
fmt.Println(err)
} else {
PrintRows(rows)
}
}
func PrintRows(rows *sql.Rows) {
for rows.Next() {
var firstname string
var lastname string
err := rows.Scan(&firstname, &lastname)
if err != nil {
fmt.Println(err)
}
fmt.Println("first name | last name")
fmt.Println("%v | %v\n", firstname, lastname)
}
}
The error I get is pq: syntax error at or near "$1"
which is from myqueries.go file in the db.Query.
I've tried several variations of this, but nothing has worked yet. Any help is appreciated.
It looks like you are using https://github.com/lib/pq based on the error message and it's docs say that
pq uses the Postgres-native ordinal markers, as shown above
I've never known a database engine that allows the parameterized values in anything other than values. I think you are going to have to resort to string concatenation. I don't have a Go compiler available to me right now, but try something like this. Because you are inserting the table name by concatination, you need it sanitized. pq.QuoteIdentifier should be able to help with that.
func SelectAll (table string) {
db, err := sql.Open("postgres","user=postgres dbname=mydb sslmode=disable")
if err != nil {
fmt.Println(err)
}
defer db.Close()
table = pq.QuoteIdentifier(table)
rows, err := db.Query(fmt.Sprintf("SELECT * FROM %v", table))
if err != nil {
fmt.Println(err)
} else {
PrintRows(rows)
}
}
EDIT: Thanks to hobbs to pointing out pq.QuoteIdentifier