Run cron in Golang while having different databases - mongodb

I am working on a SaaS based project on which Merchants can subscribe to set up their online store.
Project Overview
I am using Golang (backend), Mongodb database service and Angular4 (frontend) to build the system. I have multiple merchants that can set up their store. Each merchant has its own url (having its business name as subdomain in the url) to connect to his database.
For Routing, I am using Golang's Gin framework at back end.
Problem
I want to run the cron jobs for the merchant-specific database. In these cron jobs there are some operations that need to connect to the database. But in my routing, until a route for an API is called, the database won't be set. And ultimately, the cron does not run with proper data.
Code
cron.go
package cron
import (
"gopkg.in/robfig/cron.v2"
"controllers"
)
func RunCron(){
c := cron.New()
c.AddFunc("#every 0h1m0s", controllers.ExpireProviderInvitation)
c.Start()
}
Controller function
func ExpireProviderInvitation() {
bookingAcceptTimeSetting, _ := models.GetMerchantSetting(bson.M{"section": "providers", "option_name": "bookings_accept_time"})
if bookingAcceptTimeSetting.OptionValue != nil{
allInvitations, _ := models.GetAllBookingInvitations(bson.M{ "status": 0, "send_type": "invitation", "datetime": bson.M{"$le": float64(time.Now().Unix()) - bookingAcceptTimeSetting.OptionValue.(float64)} })
if len(allInvitations) > 0 {
for _, invitationData := range allInvitations {
_ = GetNextAvailableProvider(invitationData.Bid, invitationData.Pid)
}
}
}
}
router.go
func NewRouter() {
router := gin.Default()
router.Use(gin.Recovery())
router.Use(SetMerchantDatabase)
public := router.Group("/api/v1")
for _, route := range publicRoutes{
switch route.Method {
case "GET" : public.GET(route.Pattern, route.HandlerFunc)
case "POST" : public.POST(route.Pattern, route.HandlerFunc)
case "PUT" : public.PUT(route.Pattern, route.HandlerFunc)
case "DELETE": public.DELETE(route.Pattern, route.HandlerFunc)
default : public.GET(route.Pattern, func(c *gin.Context){
c.JSON(200, gin.H{
"result": "Specify a valid http method with this route.",
})
})
}
}
router.NoRoute(controllers.UnauthorizedAccessResponse)
router.Run(":8080")
}
func SetMerchantDatabase(c *gin.Context){
subdomain := strings.Split(c.Request.Host, ".")
if len(subdomain) > 0{
config.Database = subdomain[0]
config.CurrentBusinessName = subdomain[0]
}else{
errMsg := "Failed: Invalid domain in headers."
response := controllers.ResponseController{
config.FailureCode,
config.FailureFlag,
errMsg,
nil,
}
controllers.GetResponse(c, response)
c.Abort()
}
c.Next()
}
main.go
package main
import (
"cron"
)
func main(){
cron.RunCron()
NewRouter()
}
Explanation of above code
An example route can be:
Route{ "AddCustomer", "POST", "/customer", controllers.SaveCustomer },
An example API url can be:
http://business-name.main-domain.com/api/v1/customer
Where "business-name" is the database which is set whenever an API is called.
I want to run my cron without calling an API route.
Alternative approach
In Shell script, we can run cron by hitting url as a command. For this, I can create a url to run it as a command. But this is my theoratical approach. Also I don't know how will I get different merchant databases.
I am not sure if this approach will work. Any kind of help will be greatly appreciated.

You need to adapt SetMerchantDatabase to work independently of your router. Then you can have it set things for Cron just as well.

Related

I need advice on Go integration testing using Postgres db in Docker

I am very new to Go world. I have some db functions that I need to test.
So first I have a database.go file that connects to a postgres db:
import (
"fmt"
"gorm.io/driver/postgres"
"gorm.io/gorm"
"os"
)
var DB *gorm.DB
var err error
func Open() error {
dsn := fmt.Sprintf("host=%s user=%s password=%s dbname=%s port=%s sslmode=disable", os.Getenv("HOST"), os.Getenv("USER"),
os.Getenv("PASSWORD"), os.Getenv("DB"), os.Getenv("PORT"))
DB, err = gorm.Open(postgres.Open(dsn), &gorm.Config{})
if err != nil {
return err
}
return nil
}
Then I have a customers.go file with functions that interact with that db:
import (
"customers/cmd/database"
"time"
)
type Customers struct {
ID int
CustomerName string
Active bool
Balance float32
ActiveSince time.Time
}
func Get(id int) (Customers, error) {
var customer Customers
result := database.DB.First(&customer, id)
if result.Error != nil {
return Customers{}, result.Error
} else {
return customer, nil
}
}
This is all running in docker, there is customers container and postgres container. Now the question is how do I test my Get(id int) function? I was researching dockertest but that spins up a different db and my Get function uses the one I specified in database.go. So is there a standard Go way to test these functions?
it is a docker net problem but golang:
you can create a docker net, run both container in one net.doc
or use network --network=host
export postgres container's port to localhost, and customers container link to localhost,-pxx:xx
there is standard way of unit tesing in go. pl refer testing and testify/assert. Unit tests are typically written in xxx_test.go file next to the code.
coming to unit testing of db access layer code, one option would be to have a testenv helper and use it on these lines.
customers_test.go:
package dbaccess
import "testing"
func TestDbAccessLayer(t *testing.T) {
testEnv := testUtils.NewTestEnv()
// testEnv should do the required initialization e.g.
// start any mocked services, connection to database, global logger etc.
if err := testEnv.Start(); err != nil {
t.Fatal(err)
}
// at the end of the test, stop need to reset state
// e.g. clear any entries created by test in db
defer testEnv.Stop()
// add test code
// add required assertion to validate
}
have a separate docker-compose.yml file and use it with docker compose command to start/stop services like postgresdb.
go test command may be used to run the tests. refer to the command docs for details.

Amazon RDS PostgreSQL Performance input/output

I am facing this complex challenge with an RDS PostgreSQL instance. I am almost out of any idea how to handle it. I am launching an app (React+Go+PostreSQL) for which I expect to have around 250-300 users simultaneously making the same API GET request for how long the users wish to use it.
It is a questionnaire kind of app and users will be able to retrieve one question from the database and answer it, the server will save the answer in the DB, and then the user will be able to press next to fetch the next question. I tested my API endpoint with k6 using 500 virtual users for 2 minutes and the database returns dial: i/o timeout or even connection rejected sometimes, usually when it reaches 6000 requests and I get around 93% success. I tried to fine-tune the rds instance with tcp_keep_alive parameters but without any luck, I still cannot manage to get 100% of the request pass. I also tried to increase the general storage from 20gb min to 100gb in rds and switch from the free db.t3.micro to db.t3.medium size.
Any hint would be much appreciated. It should be possible for a normal golang server with postgres to handle this requests at the same time, shouldn't it? It is just a regular select * from x where y statement.
EDIT (CODE SAMPLE):
I use a dependency injection pattern and so I have only one instance of the DB passed to all the other repositories including the API package. The db repo looks like this:
func NewRepository() (DBRepository, error) {
dbname := getenv("POSTGRES_DATABASE", "")
username := getenv("POSTGRES_ROOT_USERNAME", "")
password := getenv("POSTGRES_ROOT_PASSWORD", "")
host := getenv("POSTGRES_HOST", "")
port := getenv("POSTGRES_PORT", "")
dsn := fmt.Sprintf("host=%s user=%s password=%s"+
" dbname=%s port=%s sslmode=disable
TimeZone=Europe/Bucharest", host, username, password, dbname,
port)
db, err := gorm.Open(postgres.Open(dsn), &gorm.Config{})
if err != nil {
return nil, err
}
if err != nil {
return nil, err
}
db.AutoMigrate(
//migrate tables are here
)
return &dbRepository{
db: db,
}, nil
}
Currently the parameters use in RDS for TCP keepalive are:
tcp_keepalives_count 30
tcp_keepalives_idle 1000
tcp_keepalives_interval 1000
and I also tried with different numbers.
The query I am doing is a simple .Find() statement from gorm package but it seems like this is not the issue since it gets blocked whenever hits the first query/connection with the db. There are 2 query executed in this endpoint I am testing but it gets stuck on the first. If more info is needed I will update but this issue it gets so frustrating.
My k6 test if the following:
import http from 'k6/http';
import { check } from 'k6';
import { sleep } from 'k6';
export const options = {
insecureSkipTLSVerify: true,
stages: [
{ target: 225, duration: '2m' },
],
};
const access_tokens = []
let random_token = access_tokens[Math.floor(Math.random()*access_tokens.length)];
const params = {
headers: {'Authorization': `Bearer ${random_token}`}
};
export default function () {
let res = http.get('endpoint here', params);
check(res, {'Message': (r)=> r.status === 202});
sleep(1);
}
The DB tables are also indexed and tested with the explain statement.

Get transaction id from new relic during tests

I am using Newrelic to get insights on my golang app. I am trying to test a middleware that will log whenever a request comes with a proper new relic header. ( "Newrelic":"eyXXXXXXX" ).
This is my test :
func TestGetNewRelicTraceID(t *testing.T) {
w := httptest.NewRecorder()
req := httptest.NewRequest("GET", "/test", nil)
req.Header.Add("Newrelic", "eyJ2IjpbMCwxXSwiZCI6eyJ0eSI6IkFwcCIsImFwIjoiNDk1Njg4OTcwIiwiYWMiOiIxMzA5OTAiLCJ0eCI6IjE3MGNmYjRiNTBiMTQ2MGIiLCJpZCI6IjQ1NGY0MTFmOWNjYjA1MDgiLCJ0ciI6IjE3MGNmYjRiNTBiMTQ2MGI0MmQ0N2ZkZmQ3MTg2NzM3IiwicHIiOjEuMTI3NTUxLCJzYSI6dHJ1ZSwidGkiOjE2MjEwMTAwMjcwMjIsInRrIjoiMzQ2MDgwIn19")
app, _ := newrelic.NewApplication(
newrelic.ConfigAppName("test"),
newrelic.ConfigLicense("1TI35kweH5xJjYLvDgp6gX1LGbYvJ130n0E5Jecs"),
newrelic.ConfigDistributedTracerEnabled(true),
func(cfg *newrelic.Config) {
cfg.ErrorCollector.RecordPanics = true
},
)
_, fn := newrelic.WrapHandleFunc(app, "/test", func(w http.ResponseWriter, r *http.Request) {
txn2 := newrelic.FromContext(r.Context())
nrTraceID := fmt.Sprintf("%s", txn2.GetTraceMetadata().TraceID)
w.Write([]byte(nrTraceID))
})
fn(w, req)
assert.Equal(t, http.StatusOK, w.Code)
assert.Equal(t, "170cfb4b50b1460b42d47fdfd7186737", string(w.Body.Bytes()))
}
No matter what I do, the test never passes as every run creates a new trace id, instead of using the one coming with the header.
What am I doing incorrectly?
Well, I found my problem. So it seems that I have to wait for the connection to the New Relic server to actually take place before running the test. Which means I had to replace 1TI35kweH5xJjYLvDgp6gX1LGbYvJ130n0E5Jecs with a REAL key. This is very important it seems. Additionally, I had to add this too to the test:
err := app.WaitForConnection(time.Second * 10)
require.Nil(t, err)
And now it works as expected.

TDD with database and Go

I'm trying to wrap my head around test driven development with Go and having an issue testing my CRUD functions since they are written for my production database. I'm coming from Ruby on Rails so I am used to using a test database, but Go doesn't seem to be too friendly in this regard.
So, how does one go about testing CRUD with Go?
main.go
package main
import (
"database/sql"
)
type book struct {
id int `json:"id"`
isbn string `json:"isbn"`
title string `json:"title"`
author string `json:"author"`
price float32 `json:"price"`
}
// type Books []*Book
// CRUD functions for Book
func (b *book) getBook(db *sql.DB) error {
return db.QueryRow("SELECT * FROM books WHERE id=$1", b.id).Scan(&b)
}
app.go
func (a *App) Initialize(dbname string) {
var err error
a.DB, err = sql.Open("postgres", "postgresql://localhost:5432/bookstore?sslmode=disable")
if err != nil {
log.Fatal(err)
}
}
my test
func TestGetBook(t *testing.T) {
clearTable()
addBook(1)
req, _ := http.NewRequest("GET", "/book/1", nil)
response := executeRequest(req)
checkResponseCode(t, http.StatusOK, response.Code)
}
The problem is that this keeps on looking at the books table in my DB, not the books_test table I'd like to use for testing. How can I go about making ONLY the tests use the books_test DB?
You should create a dev/test database which should be a complete copy of your production database. You will never want to run test directly against your production database since too many unexpected issues could happen.
A workaround would be starting up your app first, which creates the connection to your database, then run the test. You can use IntelliJ to achieve this.
TDD in my opinion is great for developing business logic layer code since new models and business processes can have unexpected impacts on existing ones.
#Godzilla74, there'are 2 solutions: enable SSL for test DB (try to check database settings or ask your system administrator) of have completely different setting for test:
func (a *App) Initialize(dbname string) {
var err error
pgsettings := os.Getenv("PGSETTINGS")
if pgsettins == "" {
// default options if not overridden
pgsettins := "postgresql://localhost:5432/bookstore?sslmode=disable"
}
a.DB, err = sql.Open("postgres", pgsettins)
if err != nil {
log.Fatal(err)
}
}
So you can run set environment setting to any required value and run app, like so:
export PGSETTINGS="postgresql://localhost:5432/bookstore_test?sslmode=disable"
go run main.go

Execute SparkCore function using Gobot.io and sleepy RESTful Framework for Go

I have the following bit of code where I'm using the RESTful framework for Go called sleepy.
I can successfully start the service at: http://localhost:3000, however when I try to access http://localhost:3000/temperature I'm expecting my SparkCore function dht to execute.
I'm using the Gobot.io Spark platform to execute this function based on this example, which I've implemented in my own code.
The problem is that the code doesn't get past the gobot.Start() method inside the Get() function so I can't actually return the result data.
I'm setting the data value hoping that I can do the:
return 200, data, http.Header{"Content-type": {"application/json"}}
But it never gets called becuase of the gobot.Start().
I'm very new to Go so any help would be greatly appreciated.
package main
import (
"net/url"
"net/http"
"fmt"
"github.com/dougblack/sleepy"
"github.com/hybridgroup/gobot"
"github.com/hybridgroup/gobot/platforms/spark"
)
var gbot = gobot.NewGobot()
var sparkCore = spark.NewSparkCoreAdaptor("spark", "device_id", "auth_token")
type Temperature struct {}
func (temperature Temperature) Get(values url.Values, headers http.Header) (int, interface{}, http.Header) {
work := func() {
if result, err := sparkCore.Function("dht", ""); err != nil {
fmt.Println(err)
} else {
data := map[string]string{"Temperature": result}
fmt.Println("result from \"dht\":", result)
}
}
robot := gobot.NewRobot("spark",
[]gobot.Connection{sparkCore},
work,
)
gbot.AddRobot(robot)
gbot.Start()
return 200, data, http.Header{"Content-type": {"application/json"}}
}
func main() {
api := sleepy.NewAPI()
temperatureResource := new(Temperature)
api.AddResource(temperatureResource, "/temperature")
fmt.Println("Listening on http://localhost:3000/")
api.Start(3000)
}
gbot.Start() is a blocking call.
In this context, you are expected to call it as:
go gbot.Start()
This will launch it in a goroutine (think thread) and then let your app continue.
When you look at the gobot example app, they don't run in the background since it is the main function. If main runs everything in the background and doesn't wait for anything, the app exits immediately with no apparent effect.