How could I create a new struct from an unpredictable db.Query()? - postgresql

I'm using SELECT * in a db.query() to return columns from a table. Typically, I would fmt.Scan() the rows into a pre-declared struct{} for further manipulation, but in this case, the table columns change frequently so I'm not able to use a declared struct{} as part of my Scan().
I've been struggling to figure out how I might dynamically build a struct{} based on the column results of the db.query() which I could subsequently call on the use of for Scan(). I've read a little about reflect but I'm struggling to determine if this is right for my use-case or if I might have to think about something else.
Any pointers would be greatly appreciated.

you can get column names from resulting rowset and prepare a slice for the scan.
Example (https://go.dev/play/p/ilYmEIWBG5S) :
package main
import (
"database/sql"
"fmt"
"log"
"github.com/DATA-DOG/go-sqlmock"
)
func main() {
// mock db
db, mock, err := sqlmock.New()
if err != nil {
log.Fatal(err)
}
columns := []string{"id", "status"}
mock.ExpectQuery("SELECT \\* FROM table").
WillReturnRows(sqlmock.NewRows(columns).AddRow(1, "ok"))
// actual code
rows, err := db.Query("SELECT * FROM table")
if err != nil {
log.Fatal(err)
}
cols, err := rows.Columns()
if err != nil {
log.Fatal(err)
}
data := make([]interface{}, len(cols))
strs := make([]sql.NullString, len(cols))
for i := range data {
data[i] = &strs[i]
}
for rows.Next() {
if err := rows.Scan(data...); err != nil {
log.Fatal(err)
}
for i, d := range data {
fmt.Printf("%s = %+v\n", cols[i], d)
}
}
}
This example reads all columns into strings. To detect column type one can use rows.ColumnTypes method.

Related

How to retrieve values in GoLang database/sql, when my structure in db(here postgres) is unknown?

I use Go with PostgreSQL using github.com/lib/pq and able to successfully fetch the records when my structure is known.
Now my query is how to fetch records when my structure changes dynamically?
By rows.columns I am able to fetch the column names, but could you help me with fetching the values of these columns for all the rows. I referred this link answered by #Luke, still, here the person has a structure defined.
Is it possible to retrieve a column value by name using GoLang database/sql
type Person struct {
Id int
Name string
}
Meanwhile I do not have a structure that is fixed, so how will I iterate through all the columns that too again for all rows. My approach would be a pointer to loop through all columns at first, then another one for going to next row.
Still not able to code this, Could you please help me with this, like how to proceed and get the values.
Since you don't know the structure up front you can return the rows as a two dimensional slice of empty interfaces. However for the row scan to work you'll need to pre-allocate the values to the appropriate type and to do this you can use the ColumnTypes method and the reflect package. Keep in mind that not every driver provides access to the columns' types so make sure the one you use does.
rows, err := db.Query("select * from foobar")
if err != nil {
return err
}
defer rows.Close()
// get column type info
columnTypes, err := rows.ColumnTypes()
if err != nil {
return err
}
// used for allocation & dereferencing
rowValues := make([]reflect.Value, len(columnTypes))
for i := 0; i < len(columnTypes); i++ {
// allocate reflect.Value representing a **T value
rowValues[i] = reflect.New(reflect.PtrTo(columnTypes[i].ScanType()))
}
resultList := [][]interface{}{}
for rows.Next() {
// initially will hold pointers for Scan, after scanning the
// pointers will be dereferenced so that the slice holds actual values
rowResult := make([]interface{}, len(columnTypes))
for i := 0; i < len(columnTypes); i++ {
// get the **T value from the reflect.Value
rowResult[i] = rowValues[i].Interface()
}
// scan each column value into the corresponding **T value
if err := rows.Scan(rowResult...); err != nil {
return err
}
// dereference pointers
for i := 0; i < len(rowValues); i++ {
// first pointer deref to get reflect.Value representing a *T value,
// if rv.IsNil it means column value was NULL
if rv := rowValues[i].Elem(); rv.IsNil() {
rowResult[i] = nil
} else {
// second deref to get reflect.Value representing the T value
// and call Interface to get T value from the reflect.Value
rowResult[i] = rv.Elem().Interface()
}
}
resultList = append(resultList, rowResult)
}
if err := rows.Err(); err != nil {
return err
}
fmt.Println(resultList)
This function prints the result of a query without knowing anything about the column types and count. It is a variant of the previous answer without using the reflect package.
func printQueryResult(db *sql.DB, query string) error {
rows, err := db.Query(query)
if err != nil {
return fmt.Errorf("canot run query %s: %w", query, err)
}
defer rows.Close()
cols, _ := rows.Columns()
row := make([]interface{}, len(cols))
rowPtr := make([]interface{}, len(cols))
for i := range row {
rowPtr[i] = &row[i]
}
fmt.Println(cols)
for rows.Next() {
err = rows.Scan(rowPtr...)
if err != nil {
fmt.Println("cannot scan row:", err)
}
fmt.Println(row...)
}
return rows.Err()
}
The trick is that rows.Scan can scan values into *interface{} but you have to wrap it in interface{} to be able to pass it to Scan using ....

Fetching all records from Postgres using Golang webform when no filter is applied i.e empty 'Where' clause

I have a Golang function to fetch all the records from Postgres database, this function is simply using :
SELECT * from stock_transactions
I want to apply filter to this function to fetch records with some conditions, in-short I want to use :
SELECT * from stock_transactions WHERE symbol = $symb
The problem is to handle the case where if $symb = null the query should act as SELECT * from stock_transactions. I can write an if-else clause for the same but if the number of parameters are more than 2 it could be messy. Is there a better way to handle this?
My function:
func showstocks (w http.ResponseWriter, r *http.Request){
var err error
if r.Method != "GET" {
http.Error(w, http.StatusText(405), http.StatusMethodNotAllowed)
return
}
rows, err := db.Query("SELECT * FROM stock_transaction ORDER BY id DESC")
if err != nil {
http.Error(w, http.StatusText(500), 500)
return
}
defer rows.Close()
sks := make([]stockdata, 0)
for rows.Next() {
sk := stockdata{}
err := rows.Scan(&sk.Sname, &sk.Ttype, &sk.Uprice, &sk.Qty, &sk.Bfee, &sk.Ddate)
if err != nil {
http.Error(w, http.StatusText(500), 500)
return
}
sks = append(sks, sk)
}
if err = rows.Err(); err != nil {
http.Error(w, http.StatusText(500), 500)
return
}
tpl.ExecuteTemplate(w, "dashboard.gohtml", sks)
}
//Suggested by #mkopriva. Tried and tested.
package main
import (
"fmt"
)
func main() {
// ...
// use interface{} because string types cannot be nil
var stock_symbol interface{} // the "zero-value of interface types is nil, so stock_symbol here is nil
// if empty then stock_symbol will be left as nil
if val := r.FormValue("stock_symbol"); len(val) > 0 {
stock_symbol = val // set to the provided value
}
// ...
// now stock_symbol is either the provided string value or nil
db.Query("SELECT ... WHERE (stock_symbol = $1 OR $1 IS NULL)", stock_symbol)
}
You can use the (stock_symbol = $1 OR $1 IS NULL) trick as you have already seen, which is programmatically convenient. However this can often lead to inefficient queries, as the query planner may not be smart enough to optimize them correctly. It might be better to write some code which removes the tautological phrases and also removes the corresponding bind variables, rather than passing each of them to the database. It is slightly messy, but it should be encapsulated into a function, not rewritten each time.

Serialising a form from a request and putting in in mongo

I have tried to put a form from a request (I don't know the structure of the data that I get for the moment) in a mongo database.
Here is my code :
fmt.Println(r.Form)
for key, values := range r.Form { // range over map
for _, value := range values { // range over []string
fmt.Println(key, value)
}
}
fmt.Println(r.Form)
decoder := json.NewDecoder(r.Body)
session, err := mgo.Dial("127.0.0.1")
if err != nil {
panic(err)
}
defer session.Close()
// Optional. Switch the session to a monotonic behavior.
session.SetMode(mgo.Monotonic, true)
c2 := session.DB("finger_bag").C("finger")
data, err := bson.Marshal(decoder)
err2 := c2.Insert(data)
if (err2 != nil){
Info.Println("error")
Info.Println(err2)
}
If anyone have any idea how to do it.
If you want to store the contents of r.Form, then store r.Form, rather than trying to unmarshal and remarshal the request body:
c2.Insert(r.Form)

Golang how can I make sql row a string

I am using Golang and Postgres, Postgres has an advance feature where it can return your queries in Json format. What I want to do is get that Json query results and return it but I am having trouble since it has to be a String in order to return it. This is my code
package main
import(
"fmt"
"database/sql"
_ "github.com/lib/pq"
"log"
)
func HelloServer(w http.ResponseWriter, req *http.Request) {
db, err := sql.Open("postgres", "user=postgres password=password dbname=name sslmode=disable")
if err != nil {
log.Fatal(err)
}
defer db.Close()
rows, err := db.Query("select To_Json(t) (SELECT * from cars)t")
io.WriteString(w, "hello, world!\n")
}
func main() {
http.HandleFunc("/hello", HelloServer)
log.Fatal(http.ListenAndServe(":12345", nil))
}
The Rows element returns a Json array how can I turn that Rows element into a String ? For C# and Java I would just append the .ToString() method to it and it would make it a string . As you can see from the code above the io.WriteString takes a String as a second parameter so I want to make the Rows variable a String after it has the Json returned so that I can display it in the browser by passing it to the method. I want to replace the Hello World with the String Rows.
Rows is a sql.Rows type. In order to use the data returned by your database query you will have to iterated over the "rows".
An example from the docs
age := 27
rows, err := db.Query("SELECT name FROM users WHERE age=?", age)
if err != nil {
log.Fatal(err)
}
defer rows.Close()
for rows.Next() {
var name string
if err := rows.Scan(&name); err != nil {
log.Fatal(err)
}
fmt.Printf("%s is %d\n", name, age)
}
if err := rows.Err(); err != nil {
log.Fatal(err)
}
You should instead use QueryRow because you are expecting the database to return one result. In either case once you have used "Scan" to put the data into your own variable then you can either parse the JSON or print it out.

What am I not getting about Go sql query with variables?

I'm brand new to Go, and I've started working on some postgres queries, and I'm having very little luck.
I have a package that's just going to have some database queries in it. Here's my code.
main.go
package main
import (
"fmt"
)
func main() {
fmt.Println("Querying data")
myqueries.SelectAll("mytable")
}
myqueries.go
package myqueries
import (
"database/sql"
"fmt"
)
func SelectAll (table string) {
db, err := sql.Open("postgres","user=postgres dbname=mydb sslmode=disable")
if err != nil {
fmt.Println(err)
}
defer db.Close()
rows, err := db.Query("SELECT * FROM $1", table)
if err != nil {
fmt.Println(err)
} else {
PrintRows(rows)
}
}
func PrintRows(rows *sql.Rows) {
for rows.Next() {
var firstname string
var lastname string
err := rows.Scan(&firstname, &lastname)
if err != nil {
fmt.Println(err)
}
fmt.Println("first name | last name")
fmt.Println("%v | %v\n", firstname, lastname)
}
}
The error I get is pq: syntax error at or near "$1"
which is from myqueries.go file in the db.Query.
I've tried several variations of this, but nothing has worked yet. Any help is appreciated.
It looks like you are using https://github.com/lib/pq based on the error message and it's docs say that
pq uses the Postgres-native ordinal markers, as shown above
I've never known a database engine that allows the parameterized values in anything other than values. I think you are going to have to resort to string concatenation. I don't have a Go compiler available to me right now, but try something like this. Because you are inserting the table name by concatination, you need it sanitized. pq.QuoteIdentifier should be able to help with that.
func SelectAll (table string) {
db, err := sql.Open("postgres","user=postgres dbname=mydb sslmode=disable")
if err != nil {
fmt.Println(err)
}
defer db.Close()
table = pq.QuoteIdentifier(table)
rows, err := db.Query(fmt.Sprintf("SELECT * FROM %v", table))
if err != nil {
fmt.Println(err)
} else {
PrintRows(rows)
}
}
EDIT: Thanks to hobbs to pointing out pq.QuoteIdentifier