I'm brand new to Go, and I've started working on some postgres queries, and I'm having very little luck.
I have a package that's just going to have some database queries in it. Here's my code.
main.go
package main
import (
"fmt"
)
func main() {
fmt.Println("Querying data")
myqueries.SelectAll("mytable")
}
myqueries.go
package myqueries
import (
"database/sql"
"fmt"
)
func SelectAll (table string) {
db, err := sql.Open("postgres","user=postgres dbname=mydb sslmode=disable")
if err != nil {
fmt.Println(err)
}
defer db.Close()
rows, err := db.Query("SELECT * FROM $1", table)
if err != nil {
fmt.Println(err)
} else {
PrintRows(rows)
}
}
func PrintRows(rows *sql.Rows) {
for rows.Next() {
var firstname string
var lastname string
err := rows.Scan(&firstname, &lastname)
if err != nil {
fmt.Println(err)
}
fmt.Println("first name | last name")
fmt.Println("%v | %v\n", firstname, lastname)
}
}
The error I get is pq: syntax error at or near "$1"
which is from myqueries.go file in the db.Query.
I've tried several variations of this, but nothing has worked yet. Any help is appreciated.
It looks like you are using https://github.com/lib/pq based on the error message and it's docs say that
pq uses the Postgres-native ordinal markers, as shown above
I've never known a database engine that allows the parameterized values in anything other than values. I think you are going to have to resort to string concatenation. I don't have a Go compiler available to me right now, but try something like this. Because you are inserting the table name by concatination, you need it sanitized. pq.QuoteIdentifier should be able to help with that.
func SelectAll (table string) {
db, err := sql.Open("postgres","user=postgres dbname=mydb sslmode=disable")
if err != nil {
fmt.Println(err)
}
defer db.Close()
table = pq.QuoteIdentifier(table)
rows, err := db.Query(fmt.Sprintf("SELECT * FROM %v", table))
if err != nil {
fmt.Println(err)
} else {
PrintRows(rows)
}
}
EDIT: Thanks to hobbs to pointing out pq.QuoteIdentifier
Related
I'm trying to query from multiple databases. Each database is connected using the following function:
func connectDB(dbEnv str) *sql.DB{
// Loading environment variables from local.env file
err1 := godotenv.Load(dbEnv)
if err1 != nil {
log.Fatalf("Some error occured. Err: %s", err1)
}
dialect := os.Getenv("DIALECT")
host := os.Getenv("HOST")
dbPort := os.Getenv("DBPORT")
user := os.Getenv("USER")
dbName := os.Getenv("NAME")
password := os.Getenv("PASSWORD")
// Database connection string
dbURI := fmt.Sprintf("port=%s host=%s user=%s "+"password=%s dbname=%s sslmode=disable", dbPort, host, user, password, dbName)
// Create database object
db, err := sql.Open(dialect,dbURI)
if err != nil {
log.Fatal(err)
}
return db
}
type order struct{
OrderID string `json:"orderID"`
Name string `json:"name"`
}
type book struct{
OrderID string `json:"orderID"`
Name string `json:"name"`
}
func getOrders(db *sql.DB) []order {
var (
orderID string
name string
)
var allRows = []order{}
query := `
SELECT orderID, name
FROM orders.orders;
`
//Get rows using the query
rows, err := db.Query(query)
if err != nil { //Log if error
log.Fatal(err)
}
defer rows.Close()
// Add each row into the "allRows" slice
for rows.Next() {
err := rows.Scan(&orderID, &name, &date)
if err != nil {
log.Fatal(err)
}
//Create new order struct with the received data
row := order{
OrderID: orderID,
Name: name,
}
allRows = append(allRows, row)
}
//Log if error
err = rows.Err()
if err != nil {
log.Fatal(err)
}
return allRows
}
func getBooks(db *sql.DB) []book{
var (
bookID string
name string
)
var allRows = []book{}
query := `
SELECT bookID, name
FROM books.books;
`
//Get rows using the query
rows, err := db.Query(query)
if err != nil { //Log if error
log.Fatal(err)
}
defer rows.Close()
// Add each row into the "allRows" slice
for rows.Next() {
err := rows.Scan(&bookID, &name)
if err != nil {
log.Fatal(err)
}
//Create new book struct with the received data
row := book{
BookID: bookID,
Name: name,
}
allRows = append(allRows, row)
}
//Log if error
err = rows.Err()
if err != nil {
log.Fatal(err)
}
return allRows
}
func main() {
ordersDB:= connectDB("ordersDB.env")
booksDB:= connectDB("booksDB.env")
orders := getOrders(ordersDB)
books := getBooks(booksDB)
}
The issue is that when I use ordersDB first, the program only recognizes the table in ordersDB. And when I use booksDB first, the program only recognizes the table in booksDB.
When I try to query a table in booksDB after using ordersDB, it is giving me "relation "books.books" does not exist" error. When I try to query a table in ordersDB after using booksDB, it gives "relation "orders.orders" does not exist"
Is there a better way to connect to multiple databases?
You are using github.com/joho/godotenv to load the database configuration from the environment. Summarising (and cutting out a lot of detail) what you are doing is:
godotenv.Load("ordersDB.env")
host := os.Getenv("HOST")
// Connect to DB
godotenv.Load("booksDB.env")
host := os.Getenv("HOST")
// Connect to DB 2
However as stated in the docs "Existing envs take precedence of envs that are loaded later". This is also stated more clearly here "It's important to note that it WILL NOT OVERRIDE an env variable that already exists".
So your code will load in the first .env file, populate the environment variables, and connect to the database. You will then load the second .env file but, because the environmental variables are already set, they will not be changed and you will connect to the same database a second time.
As a work around you could use Overload. However it's probably better to reconsider your use of environmental variables (and perhaps use different variables for the second connection).
I am expected to have 80% test coverage even for pushing the basic project structure. I am a bit confused how do I write unit tests for the following code to Connect to postgres db and ping postgres for health check. Can someone help me please.
var postgres *sql.DB
// ConnectToPostgres func to connect to postgres
func ConnectToPostgres(connStr string) (*sql.DB, error) {
db, err := sql.Open("postgres", connStr)
if err != nil {
log.Println("postgres-client ", err)
return nil, err
}
postgres = db
return db, nil
}
// PostgresHealthCheck to ping database and check for errors
func PostgresHealthCheck() error {
if err := postgres.Ping(); err != nil {
return err
}
return nil
}
type PostgresRepo struct {
db *sql.DB
}
// NewPostgresRepo constructor
func NewPostgresRepo(database *sql.DB) *PostgresRepo {
return &PostgresRepo{
db: database,
}
}
You need to use this : https://github.com/DATA-DOG/go-sqlmock
Its very easy to use. Here is an example where a controller is getting tested using a mocked SQL :
Implementation
func (up UserProvider) GetUsers() ([]models.User, error) {
var users = make([]models.User, 0, 10)
rows, err := up.DatabaseProvider.Query("SELECT firstname, lastname, email, age FROM Users;")
if err != nil {
return nil, err
}
for rows.Next() {
var u models.User = models.User{}
err := rows.Scan(&u.Name, &u.Lastname, &u.Email, &u.Age)
if err != nil {
return nil, err
}
users = append(users, u)
}
if err := rows.Err(); err != nil {
return nil, err
}
return users, nil
}
Test
func TestGetUsersOk(t *testing.T) {
db, mock := NewMock()
mock.ExpectQuery("SELECT firstname, lastname, email, age FROM Users;").
WillReturnRows(sqlmock.NewRows([]string{"firstname", "lastname", "email", "age"}).
AddRow("pepe", "guerra", "pepe#gmail.com", 34))
subject := UserProvider{
DatabaseProvider: repositories.NewMockDBProvider(db, nil),
}
resp, err := subject.GetUsers()
assert.Nil(t, err)
assert.NotNil(t, resp)
assert.Equal(t, 1, len(resp))
}
func NewMock() (*sql.DB, sqlmock.Sqlmock) {
db, mock, err := sqlmock.New()
if err != nil {
log.Fatalf("an error '%s' was not expected when opening a stub database connection", err)
}
return db, mock
}
I find that writing tests against a live database makes for more high quality tests. The challenge with Postgres is that there's no good in-memory fake that you can substitute in.
What I came up with is standing up the postgres Docker container and creating temporary databases in there. The PostgresContainer type in the github.com/bitcomplete/sqltestutil package does exactly this:
# Postgres version is "12"
pg, _ := sqltestutil.StartPostgresContainer(context.Background(), "12")
defer pg.Shutdown(ctx)
db, err := sql.Open("postgres", pg.ConnectionString())
// ... execute SQL
Per the docs, it's a good idea to set up your tests so that the container is only started once, as it can take a few seconds to start up (more if the image needs to be downloaded). It suggests some approaches for mitigating that problem.
I am using Golang and Postgres, Postgres has an advance feature where it can return your queries in Json format. What I want to do is get that Json query results and return it but I am having trouble since it has to be a String in order to return it. This is my code
package main
import(
"fmt"
"database/sql"
_ "github.com/lib/pq"
"log"
)
func HelloServer(w http.ResponseWriter, req *http.Request) {
db, err := sql.Open("postgres", "user=postgres password=password dbname=name sslmode=disable")
if err != nil {
log.Fatal(err)
}
defer db.Close()
rows, err := db.Query("select To_Json(t) (SELECT * from cars)t")
io.WriteString(w, "hello, world!\n")
}
func main() {
http.HandleFunc("/hello", HelloServer)
log.Fatal(http.ListenAndServe(":12345", nil))
}
The Rows element returns a Json array how can I turn that Rows element into a String ? For C# and Java I would just append the .ToString() method to it and it would make it a string . As you can see from the code above the io.WriteString takes a String as a second parameter so I want to make the Rows variable a String after it has the Json returned so that I can display it in the browser by passing it to the method. I want to replace the Hello World with the String Rows.
Rows is a sql.Rows type. In order to use the data returned by your database query you will have to iterated over the "rows".
An example from the docs
age := 27
rows, err := db.Query("SELECT name FROM users WHERE age=?", age)
if err != nil {
log.Fatal(err)
}
defer rows.Close()
for rows.Next() {
var name string
if err := rows.Scan(&name); err != nil {
log.Fatal(err)
}
fmt.Printf("%s is %d\n", name, age)
}
if err := rows.Err(); err != nil {
log.Fatal(err)
}
You should instead use QueryRow because you are expecting the database to return one result. In either case once you have used "Scan" to put the data into your own variable then you can either parse the JSON or print it out.
I am doing attempting to build a basic API using Go which returns the results of a SQL query using the PostgreSQL library.
At the moment I can make the program return the values, but I can't get it to return a failed message to the user i.e. some JSON with an error message.
I have an error function as follows :
func handleError(w http.ResponseWriter, err error) {
if err != nil {
log.Print(err.Error() + "\r\n") // Logging
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
}
However the http.Error method doesn't appear to ever return anything. The error thrown is a table that doesn't exist in the database (which gets logged to a text file: i.e. 2016/01/11 23:28:19 pq: relation "building_roof" does not exist
My programmes query code looks like this:
table := pq.QuoteIdentifier(table)
identifier := pq.QuoteIdentifier("ID")
rows, err := db.Query( fmt.Sprintf("SELECT %s, ST_AsText(geom) FROM %s WHERE %s = $1", identifier, table, identifier), feature)
handleError(w, err)
Causing an error just gives a Chrome error:
No data received
ERR_EMPTY_RESPONSE
EDIT Full Code:
package main
import (
"fmt"
"encoding/json"
"os"
"log"
"net/http"
"database/sql"
"strings"
"time"
"github.com/lib/pq"
)
func handler(w http.ResponseWriter, r *http.Request) {
f, err := os.OpenFile("pgdump_errorlog.txt", os.O_RDWR | os.O_CREATE | os.O_APPEND, 0666)
log.Print("Couldn't open file")
defer f.Close()
log.SetOutput(f)
// Timing
start := time.Now()
// Postgres Credentials
const (
DB_USER = "postgres"
DB_PASSWORD = "OMITTED" // Removed details !
DB_PORT = "OMITTED"
DB_NAME = "OMITTED"
)
// Postgres Connect
dbinfo := fmt.Sprintf("user=%s password=%s dbname=%s port=%s sslmode=disable",
DB_USER, DB_PASSWORD, DB_NAME, DB_PORT)
db, err := sql.Open("postgres", dbinfo)
handleError(w, err)
defer db.Close()
table := r.FormValue("table")
feature := r.FormValue("id")
if table != "" {
//Postgres Query
var (
id int
geom string
)
table := pq.QuoteIdentifier(table)
identifier := pq.QuoteIdentifier("ID")
rows, qerr := db.Query( fmt.Sprintf("SELECT %s, ST_AsText(geom) FROM %s WHERE %s = $1", identifier, table, identifier), feature)
handleError(w, err)
defer rows.Close()
for rows.Next() {
err := rows.Scan(&id, &geom)
handleError(w, err)
}
err = rows.Err()
handleError(w, err)
// Maniplate Strings
returngeom := strings.Replace(geom, "1.#QNAN", "", -1)
i := strings.Index(returngeom, "(")
wkt := strings.TrimSpace(returngeom[:i])
returngeom = returngeom[i:]
type WTKJSON struct {
WTKType string
Geometry string
Elapsed time.Duration
}
returnjson := WTKJSON{Geometry: returngeom, WTKType: wkt , Elapsed: time.Since(start)/1000000.0}
json.NewEncoder(w).Encode(returnjson)
}
}
func handleError(w http.ResponseWriter, err error) {
if err != nil {
log.Print(err.Error() + "\r\n") // Logging
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
}
func main() {
http.HandleFunc("/", handler)
http.ListenAndServe(":8080", nil)
}
The following appeared to allow me to return JSON Errors:
func handleError(w http.ResponseWriter, err string) {
type APIError struct {
Error string
}
re, _ := json.Marshal(APIError{Error: err})
io.WriteString(w, string(re))
}
Used like so:
rows, err := db.Query( fmt.Sprintf("SELECT %s, ST_AsText(geom) FROM %s WHERE %s = $1", identifier, table, identifier), feature)
if err != nil {
handleError(w, err.Error())
return
}
Not suggesting this is the best method, but worked in my case.
When you are using the http.Error function the error message should a string. For simple testing I would suggest take this line
http.Error(w, err.Error(), http.StatusInternalServerError)
and change it to something like
http.Error(w,"there was an error", http.StatusInternalServerError)
and see if that response comes through. If it does its likely that you are trying to pass something that isn't a string in http.Error()
Many codes taken from Martini example, but this using Echo framework.
I can make it works in Martini but not in Echo.
server.go:
package main
import (
"database/sql"
"github.com/labstack/echo"
_ "github.com/lib/pq"
"html/template"
"io"
"log"
"net/http"
)
type Book struct {
Title, Author, Description string
}
type (
Template struct {
templates *template.Template
}
)
func (t *Template) Render(w io.Writer, name string, data interface{}) error {
return t.templates.ExecuteTemplate(w, name, data)
}
func main() {
e := echo.New()
db, err := sql.Open("postgres", "user=postgres password=apassword dbname=lesson4 sslmode=disable")
if err != nil {
log.Fatal(err)
}
t := &Template{
templates: template.Must(template.ParseFiles("public/views/testhere.html")),
}
e.Renderer(t)
e.Get("/post/:idnumber", func(c *echo.Context) {
rows, err := db.Query("SELECT title, author, description FROM books WHERE id=$1", c.Param("idnumber"))
if err != nil {
log.Fatal(err)
}
books := []Book{}
for rows.Next() {
b := Book{}
err := rows.Scan(&b.Title, &b.Author, &b.Description)
if err != nil {
log.Fatal(err)
}
books = append(books, b)
}
c.Render(http.StatusOK, "onlytestingtpl", books)
})
e.Run(":4444")
}
public/views/testhere.html:
{{define "onlytestingtpl"}}Book title is {{.Title}}. Written by {{.Author}}. The book is about {{.Description}}.{{end}}
I cannot figure out since no error message, and no SQL documentation of this framework. When running, it gives:
Book title is
(The variable not output value)
As I mentioned in a comment, you shouldn't execute a template that takes a struct with a slice of structs. Either use {{range}} in your template, or do
c.Render(http.StatusOK, "onlytestingtpl", books[0])