I'm writting small app in go. And now I need to log postgres "raise notice" message in app console.
I was not found where postgresql raised messages stored in *sql.Conn (I'm use go/sql with lib/pq driver)
example,
create function show_mes() returns int as $$
begin
raise notice 'test message from pg';
return 1;
end
$$ language plpgsql;
from go app I call this function and can get the result.
But how can i access to that messages "test message from pg" from go app?
In current version written in Node we log messages in console for debbugging:
thank you!
If you're using lib/pq you can use its LISTEN/NOTIFY support to achieve what you want.
On the Go side you can use something like this:
package main
import (
"log"
"time"
"github.com/lib/pq"
)
func main() {
dburl := "postgres:///?sslmode=disable"
li := pq.NewListener(dburl, 10*time.Second, time.Minute, func(event pq.ListenerEventType, err error) {
if err != nil {
log.Fatal(err)
}
})
if err := li.Listen("raise_notice"); err != nil {
panic(err)
}
for {
select {
case n := <-li.Notify:
// n.Extra contains the payload from the notification
log.Println("notification:", n.Extra)
case <-time.After(5 * time.Minute):
if err := li.Ping(); err != nil {
panic(err)
}
}
}
}
Then your postgres function would look like this:
CREATE FUNCTION show_mes() RETURNS int AS $$
BEGIN
PERFORM pg_notify('raise_notice', 'test message from pg');
RETURN 1;
END
$$ LANGUAGE plpgsql;
I have created example:
https://github.com/jackc/pgx/issues/838
with pgx driver
Call back function in connection for postgres messages
Related
I am trying to create some concurrent indexes using the command CRETAE INDEX CONCURRENTLY ..... through migrations in my golang project. But whenever I run that particular migration it just takes infinitely long and is never executed.
I went and checked for the logs of the POSTGRES DB and found this thing:
The weird thing is only in migrations I am not able to create concurrent indexes whereas in my main.go if i just directly write code to execute the query it is executing successfully and even on golang's DB query console it is able to create a index concurrently.
Here is my migration package code:
func NewGorm(d *gorm.DB) *GORM {
return &GORM{db: d}
}
func (g *GORM) Run(m Migrator, app, name, methods string, logger log.Logger) error {
g.txn = g.db.Begin()
ds := &datastore.DataStore{ORM: g.db}
if methods == UP {
err = m.Up(ds, logger)
} else {
err = m.Down(ds, logger)
}
if err != nil {
g.rollBack()
return &errors.Response{Reason: "error encountered in running the migration", Detail: err}
}
g.commit()
return nil
}
I know it has something to do with transactions, but i also tried disabling it by passing flag SkipDefaultTransaction: true when initializing the connection with GORM but that also didn't worked and results were the same.
Please help how can i create concurrent indexes in migrations using GORM.
Let me try to help you with the issue. First, you should update the question with all of the source code so we can understand better what's going on and help you in a more accurate way. Anyway, I'll try to help you with what we have so far. I was able to achieve your goal with the following code:
package main
import (
"fmt"
"gorm.io/driver/postgres"
"gorm.io/gorm"
)
type Post struct {
Id int
Title string `gorm:"index:idx_concurrent,option:CONCURRENTLY"`
}
type GORM struct {
db *gorm.DB
}
func main() {
dsn := "host=localhost user=postgres password=postgres dbname=postgres port=5432 sslmode=disable"
db, err := gorm.Open(postgres.Open(dsn), &gorm.Config{})
if err != nil {
panic(err)
}
db.AutoMigrate(&Post{})
m := db.Migrator()
if idxFound := m.HasIndex(&Post{}, "idx_concurrent"); !idxFound {
fmt.Println("idx missing")
return
}
fmt.Println("idx already present")
}
To achieve what you need, it should be enough to add a gorm annotation next to the field where you want to add the index (e.g. Title). In this annotation you can specify to create this index in a concurrent to avoid locking the table. Then, I used the gorm.Migrator to check for the index existence.
If you've already the table, you can simply add the annotation to the model struct definition and gorm will take care of it when you'll run the AutoMigrate method.
Thanks to this you should be able to cover all of the possible scenario you might face.
Let me know if this solves your issue or if you need something else, thanks!
I am very new to Go world. I have some db functions that I need to test.
So first I have a database.go file that connects to a postgres db:
import (
"fmt"
"gorm.io/driver/postgres"
"gorm.io/gorm"
"os"
)
var DB *gorm.DB
var err error
func Open() error {
dsn := fmt.Sprintf("host=%s user=%s password=%s dbname=%s port=%s sslmode=disable", os.Getenv("HOST"), os.Getenv("USER"),
os.Getenv("PASSWORD"), os.Getenv("DB"), os.Getenv("PORT"))
DB, err = gorm.Open(postgres.Open(dsn), &gorm.Config{})
if err != nil {
return err
}
return nil
}
Then I have a customers.go file with functions that interact with that db:
import (
"customers/cmd/database"
"time"
)
type Customers struct {
ID int
CustomerName string
Active bool
Balance float32
ActiveSince time.Time
}
func Get(id int) (Customers, error) {
var customer Customers
result := database.DB.First(&customer, id)
if result.Error != nil {
return Customers{}, result.Error
} else {
return customer, nil
}
}
This is all running in docker, there is customers container and postgres container. Now the question is how do I test my Get(id int) function? I was researching dockertest but that spins up a different db and my Get function uses the one I specified in database.go. So is there a standard Go way to test these functions?
it is a docker net problem but golang:
you can create a docker net, run both container in one net.doc
or use network --network=host
export postgres container's port to localhost, and customers container link to localhost,-pxx:xx
there is standard way of unit tesing in go. pl refer testing and testify/assert. Unit tests are typically written in xxx_test.go file next to the code.
coming to unit testing of db access layer code, one option would be to have a testenv helper and use it on these lines.
customers_test.go:
package dbaccess
import "testing"
func TestDbAccessLayer(t *testing.T) {
testEnv := testUtils.NewTestEnv()
// testEnv should do the required initialization e.g.
// start any mocked services, connection to database, global logger etc.
if err := testEnv.Start(); err != nil {
t.Fatal(err)
}
// at the end of the test, stop need to reset state
// e.g. clear any entries created by test in db
defer testEnv.Stop()
// add test code
// add required assertion to validate
}
have a separate docker-compose.yml file and use it with docker compose command to start/stop services like postgresdb.
go test command may be used to run the tests. refer to the command docs for details.
I am using Newrelic to get insights on my golang app. I am trying to test a middleware that will log whenever a request comes with a proper new relic header. ( "Newrelic":"eyXXXXXXX" ).
This is my test :
func TestGetNewRelicTraceID(t *testing.T) {
w := httptest.NewRecorder()
req := httptest.NewRequest("GET", "/test", nil)
req.Header.Add("Newrelic", "eyJ2IjpbMCwxXSwiZCI6eyJ0eSI6IkFwcCIsImFwIjoiNDk1Njg4OTcwIiwiYWMiOiIxMzA5OTAiLCJ0eCI6IjE3MGNmYjRiNTBiMTQ2MGIiLCJpZCI6IjQ1NGY0MTFmOWNjYjA1MDgiLCJ0ciI6IjE3MGNmYjRiNTBiMTQ2MGI0MmQ0N2ZkZmQ3MTg2NzM3IiwicHIiOjEuMTI3NTUxLCJzYSI6dHJ1ZSwidGkiOjE2MjEwMTAwMjcwMjIsInRrIjoiMzQ2MDgwIn19")
app, _ := newrelic.NewApplication(
newrelic.ConfigAppName("test"),
newrelic.ConfigLicense("1TI35kweH5xJjYLvDgp6gX1LGbYvJ130n0E5Jecs"),
newrelic.ConfigDistributedTracerEnabled(true),
func(cfg *newrelic.Config) {
cfg.ErrorCollector.RecordPanics = true
},
)
_, fn := newrelic.WrapHandleFunc(app, "/test", func(w http.ResponseWriter, r *http.Request) {
txn2 := newrelic.FromContext(r.Context())
nrTraceID := fmt.Sprintf("%s", txn2.GetTraceMetadata().TraceID)
w.Write([]byte(nrTraceID))
})
fn(w, req)
assert.Equal(t, http.StatusOK, w.Code)
assert.Equal(t, "170cfb4b50b1460b42d47fdfd7186737", string(w.Body.Bytes()))
}
No matter what I do, the test never passes as every run creates a new trace id, instead of using the one coming with the header.
What am I doing incorrectly?
Well, I found my problem. So it seems that I have to wait for the connection to the New Relic server to actually take place before running the test. Which means I had to replace 1TI35kweH5xJjYLvDgp6gX1LGbYvJ130n0E5Jecs with a REAL key. This is very important it seems. Additionally, I had to add this too to the test:
err := app.WaitForConnection(time.Second * 10)
require.Nil(t, err)
And now it works as expected.
I am playing around with the database/sql package trying to see how it works and understand what would happen if you don't call rows.Close() etc.
So I wrote the following piece of code for inserting a model to database:
func (db Database) Insert(m model.Model) (int32, error) {
var id int32
quotedTableName := m.TableName(true)
// Get insert query
q, values := model.InsertQuery(m)
rows, err := db.Conn.Query(q, values...)
if err != nil {
return id, err
}
for rows.Next() {
err = rows.Scan(&id)
if err != nil {
return id, err
}
}
return id, nil
}
I don't call rows.Close() on purpose to see the consequences. When setting up the database connection I set some properties such as:
conn.SetMaxOpenConns(50)
conn.SetMaxIdleConns(2)
conn.SetConnMaxLifetime(time.Second*60)
Then I attempt to insert 10000 records:
for i := 0; i < 10000; i++ {
lander := models.Lander{
// ...struct fields with random data on each iteration
}
go func() {
Insert(&lander)
}()
}
(It lacks error checking, context timeouts etc. but for the purpose of playing around it gets the job done). When I execute piece of code from above I expect to see at least some errors regarding database connections however the data gets inserted without problems (all 10000 records). When I check the Stats() I see the following:
{MaxOpenConnections:50 OpenConnections:1 InUse:0 Idle:1 WaitCount:9951 WaitDuration:3h9m33.896466243s MaxIdleClosed:48 MaxLifetimeClosed:2}
Since I didn't call rows.Close() I expected to see more OpenConnections or more InUse connections because I am never releasing the connection (maybe I might be wrong, but this is the purpose of Close() to release a Connection and return it to the pool).
So my question is simply what do these Stats() mean and why are there no errors whatsoever when doing the insertion. Also why aren't there more OpenConnections or InUse ones and what are the real consequences of not calling Close()?
According to the docs for Rows:
If Next is called and returns false and there are no further result sets, the Rows are closed automatically and it will suffice to check the result of Err.
Since you iterate all the results, the result set is closed.
I have the following bit of code where I'm using the RESTful framework for Go called sleepy.
I can successfully start the service at: http://localhost:3000, however when I try to access http://localhost:3000/temperature I'm expecting my SparkCore function dht to execute.
I'm using the Gobot.io Spark platform to execute this function based on this example, which I've implemented in my own code.
The problem is that the code doesn't get past the gobot.Start() method inside the Get() function so I can't actually return the result data.
I'm setting the data value hoping that I can do the:
return 200, data, http.Header{"Content-type": {"application/json"}}
But it never gets called becuase of the gobot.Start().
I'm very new to Go so any help would be greatly appreciated.
package main
import (
"net/url"
"net/http"
"fmt"
"github.com/dougblack/sleepy"
"github.com/hybridgroup/gobot"
"github.com/hybridgroup/gobot/platforms/spark"
)
var gbot = gobot.NewGobot()
var sparkCore = spark.NewSparkCoreAdaptor("spark", "device_id", "auth_token")
type Temperature struct {}
func (temperature Temperature) Get(values url.Values, headers http.Header) (int, interface{}, http.Header) {
work := func() {
if result, err := sparkCore.Function("dht", ""); err != nil {
fmt.Println(err)
} else {
data := map[string]string{"Temperature": result}
fmt.Println("result from \"dht\":", result)
}
}
robot := gobot.NewRobot("spark",
[]gobot.Connection{sparkCore},
work,
)
gbot.AddRobot(robot)
gbot.Start()
return 200, data, http.Header{"Content-type": {"application/json"}}
}
func main() {
api := sleepy.NewAPI()
temperatureResource := new(Temperature)
api.AddResource(temperatureResource, "/temperature")
fmt.Println("Listening on http://localhost:3000/")
api.Start(3000)
}
gbot.Start() is a blocking call.
In this context, you are expected to call it as:
go gbot.Start()
This will launch it in a goroutine (think thread) and then let your app continue.
When you look at the gobot example app, they don't run in the background since it is the main function. If main runs everything in the background and doesn't wait for anything, the app exits immediately with no apparent effect.