I am trying to create some concurrent indexes using the command CRETAE INDEX CONCURRENTLY ..... through migrations in my golang project. But whenever I run that particular migration it just takes infinitely long and is never executed.
I went and checked for the logs of the POSTGRES DB and found this thing:
The weird thing is only in migrations I am not able to create concurrent indexes whereas in my main.go if i just directly write code to execute the query it is executing successfully and even on golang's DB query console it is able to create a index concurrently.
Here is my migration package code:
func NewGorm(d *gorm.DB) *GORM {
return &GORM{db: d}
}
func (g *GORM) Run(m Migrator, app, name, methods string, logger log.Logger) error {
g.txn = g.db.Begin()
ds := &datastore.DataStore{ORM: g.db}
if methods == UP {
err = m.Up(ds, logger)
} else {
err = m.Down(ds, logger)
}
if err != nil {
g.rollBack()
return &errors.Response{Reason: "error encountered in running the migration", Detail: err}
}
g.commit()
return nil
}
I know it has something to do with transactions, but i also tried disabling it by passing flag SkipDefaultTransaction: true when initializing the connection with GORM but that also didn't worked and results were the same.
Please help how can i create concurrent indexes in migrations using GORM.
Let me try to help you with the issue. First, you should update the question with all of the source code so we can understand better what's going on and help you in a more accurate way. Anyway, I'll try to help you with what we have so far. I was able to achieve your goal with the following code:
package main
import (
"fmt"
"gorm.io/driver/postgres"
"gorm.io/gorm"
)
type Post struct {
Id int
Title string `gorm:"index:idx_concurrent,option:CONCURRENTLY"`
}
type GORM struct {
db *gorm.DB
}
func main() {
dsn := "host=localhost user=postgres password=postgres dbname=postgres port=5432 sslmode=disable"
db, err := gorm.Open(postgres.Open(dsn), &gorm.Config{})
if err != nil {
panic(err)
}
db.AutoMigrate(&Post{})
m := db.Migrator()
if idxFound := m.HasIndex(&Post{}, "idx_concurrent"); !idxFound {
fmt.Println("idx missing")
return
}
fmt.Println("idx already present")
}
To achieve what you need, it should be enough to add a gorm annotation next to the field where you want to add the index (e.g. Title). In this annotation you can specify to create this index in a concurrent to avoid locking the table. Then, I used the gorm.Migrator to check for the index existence.
If you've already the table, you can simply add the annotation to the model struct definition and gorm will take care of it when you'll run the AutoMigrate method.
Thanks to this you should be able to cover all of the possible scenario you might face.
Let me know if this solves your issue or if you need something else, thanks!
Related
I am very new to Go world. I have some db functions that I need to test.
So first I have a database.go file that connects to a postgres db:
import (
"fmt"
"gorm.io/driver/postgres"
"gorm.io/gorm"
"os"
)
var DB *gorm.DB
var err error
func Open() error {
dsn := fmt.Sprintf("host=%s user=%s password=%s dbname=%s port=%s sslmode=disable", os.Getenv("HOST"), os.Getenv("USER"),
os.Getenv("PASSWORD"), os.Getenv("DB"), os.Getenv("PORT"))
DB, err = gorm.Open(postgres.Open(dsn), &gorm.Config{})
if err != nil {
return err
}
return nil
}
Then I have a customers.go file with functions that interact with that db:
import (
"customers/cmd/database"
"time"
)
type Customers struct {
ID int
CustomerName string
Active bool
Balance float32
ActiveSince time.Time
}
func Get(id int) (Customers, error) {
var customer Customers
result := database.DB.First(&customer, id)
if result.Error != nil {
return Customers{}, result.Error
} else {
return customer, nil
}
}
This is all running in docker, there is customers container and postgres container. Now the question is how do I test my Get(id int) function? I was researching dockertest but that spins up a different db and my Get function uses the one I specified in database.go. So is there a standard Go way to test these functions?
it is a docker net problem but golang:
you can create a docker net, run both container in one net.doc
or use network --network=host
export postgres container's port to localhost, and customers container link to localhost,-pxx:xx
there is standard way of unit tesing in go. pl refer testing and testify/assert. Unit tests are typically written in xxx_test.go file next to the code.
coming to unit testing of db access layer code, one option would be to have a testenv helper and use it on these lines.
customers_test.go:
package dbaccess
import "testing"
func TestDbAccessLayer(t *testing.T) {
testEnv := testUtils.NewTestEnv()
// testEnv should do the required initialization e.g.
// start any mocked services, connection to database, global logger etc.
if err := testEnv.Start(); err != nil {
t.Fatal(err)
}
// at the end of the test, stop need to reset state
// e.g. clear any entries created by test in db
defer testEnv.Stop()
// add test code
// add required assertion to validate
}
have a separate docker-compose.yml file and use it with docker compose command to start/stop services like postgresdb.
go test command may be used to run the tests. refer to the command docs for details.
I am playing around with the database/sql package trying to see how it works and understand what would happen if you don't call rows.Close() etc.
So I wrote the following piece of code for inserting a model to database:
func (db Database) Insert(m model.Model) (int32, error) {
var id int32
quotedTableName := m.TableName(true)
// Get insert query
q, values := model.InsertQuery(m)
rows, err := db.Conn.Query(q, values...)
if err != nil {
return id, err
}
for rows.Next() {
err = rows.Scan(&id)
if err != nil {
return id, err
}
}
return id, nil
}
I don't call rows.Close() on purpose to see the consequences. When setting up the database connection I set some properties such as:
conn.SetMaxOpenConns(50)
conn.SetMaxIdleConns(2)
conn.SetConnMaxLifetime(time.Second*60)
Then I attempt to insert 10000 records:
for i := 0; i < 10000; i++ {
lander := models.Lander{
// ...struct fields with random data on each iteration
}
go func() {
Insert(&lander)
}()
}
(It lacks error checking, context timeouts etc. but for the purpose of playing around it gets the job done). When I execute piece of code from above I expect to see at least some errors regarding database connections however the data gets inserted without problems (all 10000 records). When I check the Stats() I see the following:
{MaxOpenConnections:50 OpenConnections:1 InUse:0 Idle:1 WaitCount:9951 WaitDuration:3h9m33.896466243s MaxIdleClosed:48 MaxLifetimeClosed:2}
Since I didn't call rows.Close() I expected to see more OpenConnections or more InUse connections because I am never releasing the connection (maybe I might be wrong, but this is the purpose of Close() to release a Connection and return it to the pool).
So my question is simply what do these Stats() mean and why are there no errors whatsoever when doing the insertion. Also why aren't there more OpenConnections or InUse ones and what are the real consequences of not calling Close()?
According to the docs for Rows:
If Next is called and returns false and there are no further result sets, the Rows are closed automatically and it will suffice to check the result of Err.
Since you iterate all the results, the result set is closed.
I'm trying to wrap my head around test driven development with Go and having an issue testing my CRUD functions since they are written for my production database. I'm coming from Ruby on Rails so I am used to using a test database, but Go doesn't seem to be too friendly in this regard.
So, how does one go about testing CRUD with Go?
main.go
package main
import (
"database/sql"
)
type book struct {
id int `json:"id"`
isbn string `json:"isbn"`
title string `json:"title"`
author string `json:"author"`
price float32 `json:"price"`
}
// type Books []*Book
// CRUD functions for Book
func (b *book) getBook(db *sql.DB) error {
return db.QueryRow("SELECT * FROM books WHERE id=$1", b.id).Scan(&b)
}
app.go
func (a *App) Initialize(dbname string) {
var err error
a.DB, err = sql.Open("postgres", "postgresql://localhost:5432/bookstore?sslmode=disable")
if err != nil {
log.Fatal(err)
}
}
my test
func TestGetBook(t *testing.T) {
clearTable()
addBook(1)
req, _ := http.NewRequest("GET", "/book/1", nil)
response := executeRequest(req)
checkResponseCode(t, http.StatusOK, response.Code)
}
The problem is that this keeps on looking at the books table in my DB, not the books_test table I'd like to use for testing. How can I go about making ONLY the tests use the books_test DB?
You should create a dev/test database which should be a complete copy of your production database. You will never want to run test directly against your production database since too many unexpected issues could happen.
A workaround would be starting up your app first, which creates the connection to your database, then run the test. You can use IntelliJ to achieve this.
TDD in my opinion is great for developing business logic layer code since new models and business processes can have unexpected impacts on existing ones.
#Godzilla74, there'are 2 solutions: enable SSL for test DB (try to check database settings or ask your system administrator) of have completely different setting for test:
func (a *App) Initialize(dbname string) {
var err error
pgsettings := os.Getenv("PGSETTINGS")
if pgsettins == "" {
// default options if not overridden
pgsettins := "postgresql://localhost:5432/bookstore?sslmode=disable"
}
a.DB, err = sql.Open("postgres", pgsettins)
if err != nil {
log.Fatal(err)
}
}
So you can run set environment setting to any required value and run app, like so:
export PGSETTINGS="postgresql://localhost:5432/bookstore_test?sslmode=disable"
go run main.go
Using go (golang):
Is there a way to create a logger that outputs to a database?
Or more precisely, can I implement some kind of io.Writer interface that I can pass as the first argument to log.New()?
EG: (dbLogger would receive the output of the log and write it to the database)
logger := log.New(dbLogger, "dbLog: ", log.Lshortfile)
logger.Print("This message will be stored in the database")
I would assume that I should just create my own database logging function, but I was curious to see if there is already a way of doing this using the existing tools in the language.
For some context, I'm using mgo.v2 to handle my mongodb database, but I don't see any io.Writer interfaces there other than in GridFS, which I think solves a different problem.
I'm also still getting my head around the language, so I may have used some terms above incorrecly. Any corrections are very welcome.
This is easily doable, because the log.Logger type guarantees that each log message is delivered to the destination io.Writer with a single Writer.Write() call:
Each logging operation makes a single call to the Writer's Write method. A Logger can be used simultaneously from multiple goroutines; it guarantees to serialize access to the Writer.
So basically you just need to create a type which implements io.Writer, and whose Write() method creates a new document with the contents of the byte slice, and saves it in the MongoDB.
Here's a simple implementation which does that:
type MongoWriter struct {
sess *mgo.Session
}
func (mw *MongoWriter) Write(p []byte) (n int, err error) {
c := mw.sess.DB("").C("log")
err = c.Insert(bson.M{
"created": time.Now(),
"msg": string(p),
})
if err != nil {
return
}
return len(p), nil
}
Using it:
sess := ... // Get a MongoDB session
mw := &MongoWriter{sess}
log.SetOutput(mw)
// Now the default Logger of the log package uses our MongoWriter.
// Generate a log message that will be inserted into MongoDB:
log.Println("I'm the first log message.")
log.Println("I'm multi-line,\nbut will still be in a single log message.")
Obviously if you're using another log.Logger instance, set the MongoWriter to that, e.g.:
mylogger := log.New(mw, "", 0)
mylogger.Println("Custom logger")
Note that the log messages end with newline as log.Logger appends it even if the log message itself does not end with newline. If you don't want to log the ending newline, you may simply cut it, e.g.:
func (mw *MongoWriter) Write(p []byte) (n int, err error) {
origLen := len(p)
if len(p) > 0 && p[len(p)-1] == '\n' {
p = p[:len(p)-1] // Cut terminating newline
}
c := mw.sess.DB("").C("log")
// ... the rest is the same
return origLen, nil // Must return original length (we resliced p)
}
I have the following bit of code where I'm using the RESTful framework for Go called sleepy.
I can successfully start the service at: http://localhost:3000, however when I try to access http://localhost:3000/temperature I'm expecting my SparkCore function dht to execute.
I'm using the Gobot.io Spark platform to execute this function based on this example, which I've implemented in my own code.
The problem is that the code doesn't get past the gobot.Start() method inside the Get() function so I can't actually return the result data.
I'm setting the data value hoping that I can do the:
return 200, data, http.Header{"Content-type": {"application/json"}}
But it never gets called becuase of the gobot.Start().
I'm very new to Go so any help would be greatly appreciated.
package main
import (
"net/url"
"net/http"
"fmt"
"github.com/dougblack/sleepy"
"github.com/hybridgroup/gobot"
"github.com/hybridgroup/gobot/platforms/spark"
)
var gbot = gobot.NewGobot()
var sparkCore = spark.NewSparkCoreAdaptor("spark", "device_id", "auth_token")
type Temperature struct {}
func (temperature Temperature) Get(values url.Values, headers http.Header) (int, interface{}, http.Header) {
work := func() {
if result, err := sparkCore.Function("dht", ""); err != nil {
fmt.Println(err)
} else {
data := map[string]string{"Temperature": result}
fmt.Println("result from \"dht\":", result)
}
}
robot := gobot.NewRobot("spark",
[]gobot.Connection{sparkCore},
work,
)
gbot.AddRobot(robot)
gbot.Start()
return 200, data, http.Header{"Content-type": {"application/json"}}
}
func main() {
api := sleepy.NewAPI()
temperatureResource := new(Temperature)
api.AddResource(temperatureResource, "/temperature")
fmt.Println("Listening on http://localhost:3000/")
api.Start(3000)
}
gbot.Start() is a blocking call.
In this context, you are expected to call it as:
go gbot.Start()
This will launch it in a goroutine (think thread) and then let your app continue.
When you look at the gobot example app, they don't run in the background since it is the main function. If main runs everything in the background and doesn't wait for anything, the app exits immediately with no apparent effect.