I'm once again trying to push lots of csv data into a postgres database.
In the past I've created a struct to hold the data and unpacked each column into the struct before bumping the lot into the database table, and that is working fine, however, I've just found pgx.CopyFrom* and it would seem as though I should be able to make it work better.
So far I've got the column headings for the table into a slice of strings and the csv data into another slice of strings but I can't work out the syntax to push this into the database.
I've found this post which sort of does what I want but uses a [][]interface{} rather than []strings.
The code I have so far is
// loop over the lines and find the first one with a timestamp
for {
line, err := csvReader.Read()
if err == io.EOF {
break
} else if err != nil {
log.Error("Error reading csv data", "Loading Loop", err)
}
// see if we have a string starting with a timestamp
_, err := time.Parse(timeFormat, line[0])
if err == nil {
// we have a data line
_, err := db.CopyFrom(context.Background(), pgx.Identifier{"emms.scada_crwf.pwr_active"}, col_headings, pgx.CopyFromRows(line))
}
}
}
But pgx.CopyFromRows expects [][]interface{} not []string.
What should the syntax be? Am I barking up the wrong tree?
I recommend reading your CSV and creating a []interface{} for each record you read, appending the []interface{} to a collection of rows ([][]interface{}), then passing rows on to pgx.
var rows [][]interface{}
// read header outside of CSV "body" loop
header, _ := reader.Read()
// inside your CSV reader "body" loop...
row := make([]interface{}, len(record))
// use your logic/gate-keeping from here
row[0] = record[0] // timestamp
// convert the floats
for i := 1; i < len(record); i++ {
val, _ := strconv.ParseFloat(record[i], 10)
row[i] = val
}
rows = append(rows, row)
...
copyCount, err := conn.CopyFrom(
pgx.Identifier{"floaty-things"},
header,
pgx.CopyFromRows(rows),
)
I can't mock up the entire program, but here's a full demo of converting the CSV to [][]interface{}, https://go.dev/play/p/efbiFN2FJMi.
And check in with the documentation, https://pkg.go.dev/github.com/jackc/pgx/v4.
Related
I'm trying to get results from select query in a comma separated format (don't want results in a file). Following code works fine if I use stmt := fmt.Sprintf("SELECT * FROM table") but below code fails to reproduce any results because rows.Next() is empty. How to resolve this issue?
func (db *Dbq) Getresults() []interface{} {
stmt := fmt.Sprintf("COPY (select * from table) TO STDOUT WITH CSV")
var results []interface{}
rows, err := db.conn.Query(db.context, stmt)
for rows.Next() {
values, err := rows.Values()
if err != nil {
log.Fatal(err)
}
results = append(results, values)
}
rows.Close()
return results
}
I ran code with two different queries and debugged where it failed.
I use Go with PostgreSQL using github.com/lib/pq and able to successfully fetch the records when my structure is known.
Now my query is how to fetch records when my structure changes dynamically?
By rows.columns I am able to fetch the column names, but could you help me with fetching the values of these columns for all the rows. I referred this link answered by #Luke, still, here the person has a structure defined.
Is it possible to retrieve a column value by name using GoLang database/sql
type Person struct {
Id int
Name string
}
Meanwhile I do not have a structure that is fixed, so how will I iterate through all the columns that too again for all rows. My approach would be a pointer to loop through all columns at first, then another one for going to next row.
Still not able to code this, Could you please help me with this, like how to proceed and get the values.
Since you don't know the structure up front you can return the rows as a two dimensional slice of empty interfaces. However for the row scan to work you'll need to pre-allocate the values to the appropriate type and to do this you can use the ColumnTypes method and the reflect package. Keep in mind that not every driver provides access to the columns' types so make sure the one you use does.
rows, err := db.Query("select * from foobar")
if err != nil {
return err
}
defer rows.Close()
// get column type info
columnTypes, err := rows.ColumnTypes()
if err != nil {
return err
}
// used for allocation & dereferencing
rowValues := make([]reflect.Value, len(columnTypes))
for i := 0; i < len(columnTypes); i++ {
// allocate reflect.Value representing a **T value
rowValues[i] = reflect.New(reflect.PtrTo(columnTypes[i].ScanType()))
}
resultList := [][]interface{}{}
for rows.Next() {
// initially will hold pointers for Scan, after scanning the
// pointers will be dereferenced so that the slice holds actual values
rowResult := make([]interface{}, len(columnTypes))
for i := 0; i < len(columnTypes); i++ {
// get the **T value from the reflect.Value
rowResult[i] = rowValues[i].Interface()
}
// scan each column value into the corresponding **T value
if err := rows.Scan(rowResult...); err != nil {
return err
}
// dereference pointers
for i := 0; i < len(rowValues); i++ {
// first pointer deref to get reflect.Value representing a *T value,
// if rv.IsNil it means column value was NULL
if rv := rowValues[i].Elem(); rv.IsNil() {
rowResult[i] = nil
} else {
// second deref to get reflect.Value representing the T value
// and call Interface to get T value from the reflect.Value
rowResult[i] = rv.Elem().Interface()
}
}
resultList = append(resultList, rowResult)
}
if err := rows.Err(); err != nil {
return err
}
fmt.Println(resultList)
This function prints the result of a query without knowing anything about the column types and count. It is a variant of the previous answer without using the reflect package.
func printQueryResult(db *sql.DB, query string) error {
rows, err := db.Query(query)
if err != nil {
return fmt.Errorf("canot run query %s: %w", query, err)
}
defer rows.Close()
cols, _ := rows.Columns()
row := make([]interface{}, len(cols))
rowPtr := make([]interface{}, len(cols))
for i := range row {
rowPtr[i] = &row[i]
}
fmt.Println(cols)
for rows.Next() {
err = rows.Scan(rowPtr...)
if err != nil {
fmt.Println("cannot scan row:", err)
}
fmt.Println(row...)
}
return rows.Err()
}
The trick is that rows.Scan can scan values into *interface{} but you have to wrap it in interface{} to be able to pass it to Scan using ....
I'm just getting started with golang and I'm attempting to read several rows from a Postgres users table and store the result as an array of User structs that model the row.
type User struct {
Id int
Title string
}
func Find_users(db *sql.DB) {
// Query the DB
rows, err := db.Query(`SELECT u.id, u.title FROM users u;`)
if err != nil { log.Fatal(err) }
// Initialize array slice of all users. What size do I use here?
// I don't know the number of results beforehand
var users = make([]User, ????)
// Loop through each result record, creating a User struct for each row
defer rows.Close()
for i := 0; rows.Next(); i++ {
err := rows.Scan(&id, &title)
if err != nil { log.Fatal(err) }
log.Println(id, title)
users[i] = User{Id: id, Title: title}
}
// .... do stuff
}
As you can see, my problem is that I want to initialize an array or slice beforehand to store all the DB records, but I don't know ahead of time how many records there are going to be.
I was weighing a few different approaches, and wanted to find out which of the following was most used in the golang community -
Create a really large array beforehand (e.g. 10,000 elements). Seems wasteful
Count the rows explicitly beforehand. This could work, but I need to run 2 queries - one to count and one to get the results. If my query is complex (not shown here), that's duplicating that logic in 2 places. Alternatively I can run the same query twice, but first loop through it and count the rows. All this would work, but it just seems unclean.
I've seen examples of expanding slices. I don't quite understand slices well enough to see how it could be adapted here. Also if I'm constantly expanding a slice 10k times, it definitely seems wasteful.
Go has a built-in append function for exactly this purpose. It takes a slice and one or more elements and appends those elements to the slice, returning the new slice. Additionally, the zero value of a slice (nil) is a slice of length zero, so if you append to a nil slice, it will work. Thus, you can do:
type User struct {
Id int
Title string
}
func Find_users(db *sql.DB) {
// Query the DB
rows, err := db.Query(`SELECT u.id, u.title FROM users u;`)
if err != nil {
log.Fatal(err)
}
defer rows.Close()
var users []User
for rows.Next() {
err := rows.Scan(&id, &title)
if err != nil {
log.Fatal(err)
}
log.Println(id, title)
users = append(users, User{Id: id, Title: title})
}
if err := rows.Err(); err != nil {
log.Fatal(err)
}
// ...
}
User appending to a slice:
type DeviceInfo struct {
DeviceName string
DeviceID string
DeviceUsername string
Token string
}
func QueryMultiple(db *sql.DB){
var device DeviceInfo
sqlStatement := `SELECT "deviceName", "deviceID", "deviceUsername",
token FROM devices LIMIT 10`
rows, err := db.Query(sqlStatement)
if err != nil {
panic(err)
}
defer rows.Close()
var deviceSlice []DeviceInfo
for rows.Next(){
rows.Scan(&device.DeviceID, &device.DeviceUsername, &device.Token,
&device.DeviceName)
deviceSlice = append(deviceSlice, device)
}
fmt.Println(deviceSlice)
}
I think that what you are looking for is the capacity.
The following allocates an array that can hold 10,000 items:
users := make([]User, 0, 10_000)
but the array is, itself, still empty (len(users) == 0).
Now you can add up to at least 10,000 items before the array needs to be grown. For that purpose the append() works as expected:
users = append(users, User{...})
Maps are grown with a x2 of the size starting with 1. So it remains a power of two. I'm not sure whether slices are grown the same way (in powers of two). If so, then the allocated size above would be:
math.Pow(2, math.Ceil(math.Log(10_000)/math.Log(2)))
which is 2^14 which is 16,384.
Note: if your query uses a compatible INDEX, i.e. the WHERE clause matches the INDEX one to one, then an extra SELECT COUNT(*) ... is free since the number of elements is known and it will return that number without the need to scan all the rows of your tables.
I'm fairly new to both Go and MongoDB. Trying to select a single field from the DB and save it in an int slice without any avail.
userIDs := []int64{}
coll.Find(bson.M{"isdeleted": false}).Select(bson.M{"userid": 1}).All(&userIDs)
The above prints out an empty slice. However, if I create a struct with a single ID field that is int64 with marshalling then it works fine.
All I am trying to do is work with a simple slice containing IDs that I need instead of a struct with a single field. All help is appreciated.
Because mgo queries return documents, a few lines of code is required to accomplish the goal:
var result []struct{ UserID int64 `bson:"userid"` }
err := coll.Find(bson.M{"isdeleted": false}).Select(bson.M{"userid": 1}).All(&result)
if err != nil {
// handle error
}
userIDs := make([]int64, len(result))
for i := range result {
userIDs[i] = result.UserID
}
I am using postgresql as my backend database.
Tried to scan a field languagespoken which is an array of text
var user userprofile
row := core.db.QueryRow(
"SELECT languagespoken FROM \"user\" WHERE id = $1",
userId,
)
err := row.Scan(&user.Languages)
if err != nil {
return user, err
}
My structure looks like this
type userprofile struct {
Languages []string `json:languages`
}
But getting the error
2014/06/30 15:27:17 PANIC: reflect.Set: **value of type []uint8 is not assignable to type []string**
/usr/lib/go/src/pkg/reflect/value.go:2198 (0x56c152)
Value.assignTo: panic(context + ": value of type " + v.typ.String() + " is not assignable to type " + dst.String())
/usr/lib/go/src/pkg/reflect/value.go:1385 (0x56966b)
Value.Set: x = x.assignTo("reflect.Set", v.typ, target)
/usr/lib/go/src/pkg/database/sql/convert.go:215 (0x492d70)
convertAssign: dv.Set(sv)
/usr/lib/go/src/pkg/database/sql/sql.go:1562 (0x49c0e5)
(*Rows).Scan: err := convertAssign(dest[i], sv)
/usr/lib/go/src/pkg/database/sql/sql.go:1630 (0x49c560)
(*Row).Scan: err := r.rows.Scan(dest...)
/home/ceresti/source/gocode/src/ceresti.kilnhg.com/ceresti/server/app/databaseapi.go:144 (0x402478)
(*coreStruct).GetUserProfile: err := row.Scan(&user.Languages)
/home/ceresti/source/gocode/src/ceresti.kilnhg.com/ceresti/server/app/restfulapi.go:327 (0x40a63c)
Getuserprofile: userprofileStruct, err := core.GetUserProfile(userId)
/usr/lib/go/src/pkg/runtime/asm_amd64.s:340 (0x4309c2)
Please let me know how to resolve this issue.
Not all sql databases specify array types (e.g., sqlite3). Go doesn't support any sql implementations directly, but it supplies an implementation-agnostic interface (in the generic sense of the word) for which third-party drivers may be written. Go doesn't impose any limitations on which types its drivers may support, so if a type doesn't cooperate, it's probably the fault of the driver.
TL;DR: Try getting it as a string
// if `string` doesn't work, try `[]uint8` or `[]byte` and then convert
// that output to a string if necessary
var languages string
if err := row.Scan(&languages); err != nil {
// handle error
}
// depending on how the string is encoded, this may or may not work
// Update: since you say your list is encoded in the form:
// `{elem1, elem2, elem3}`, we'll simply ignore the first and last
// characters and split on ", "
user.Languages = strings.Split(languages[1:len(languages)-1], ", ")
It appears you're trying to scan the entire result set of that database query in one shot. You can't do that; you need to read each row, one at a time, into a byte slice, then convert the byte slice into a string.
Since you're serializing into a []string, saving the byte slice each time is not a priority. In this case, you can use sql.RawBytes instead of []byte, which will reuse the same memory.
// error checking elided
var row sql.RawBytes
myRow.Scan(&row) // note the pointer!
str := string(row)