My application mostly consists of CRUDs to/from MongoDB using mongo-go-drive package. This function is one of gRPC server services and all it does is calling database method action.GetProducts(ctx) and it returns *mongo.cursor. Then the result is decoded. For each document, I put the document content into a singular product struct, then append it to products slices (the GetProductsResponse struct is made using gRPC proto repeated GetProductResponse type). After appending all product into GetProductsResponse, I return the response to gRPC client.
I am also new to testing in general, how should I break down the function and do the mocking (how to mock the cursor?) for unit testing? Is it even necessary in the first place to do unit test on the function even though all it does is appending the result, or should I just go straight for the integration test and skip the unit test since it involves database I/O?
func (s *Server) GetProducts(ctx context.Context, in *pb.EmptyRequest) (*pb.GetProductsResponse, error) {
cursor, err := action.GetProducts(ctx)
if err != nil {
return nil, err
}
products := pb.GetProductsResponse{}
res := model.Product{}
for cursor.Next(ctx) {
// Convert document to above struct
err := cursor.Decode(&res)
if err != nil {
return nil, fmt.Errorf("failed to decode document: %v", err)
}
product := &pb.GetProductResponse{ProductId: res.Product_id.Hex(), Name: res.Name, Price: res.Price, Qty: int32(res.Qty)}
products.Products = append(products.Products, product)
}
return &products, nil
}
If you interact with the DB is not unit testing anymore, because you're integrating with another external system.
Anyway, I use to define my "repository" layer function this way:
package repo
var FetchUserById = func(id string) (*model.User, error){
// here the real logic
return user, err
}
and then, when I have to test my "service" layer logic, I would mock the entire "repository" layer this way:
repo.FetchUserById = func(id string) (*model.User, err) {
return myMockedUser, nil
}
Related
I'm new in Go and unit test. I build a samll side projecy called "urlshortener" using Go with Gorm, mux and postgresql.
There is a qeustion annoying me after search many articles.
To make the question clean, I delete some irrelevant code like connect db, .env, etc
My code is below(main.go):
package main
type Url struct {
ID uint `gorm:"primaryKey"` // used for shortUrl index
Url string `gorm:"unique"` // prevent duplicate url
ExpireAt string
ShortUrl string
}
var db *gorm.DB
var err error
func main() {
// gain access to database by getting .env
...
// database connection string
...
// make migrations to the dbif they have not already been created
db.AutoMigrate(&Url{})
// API routes
router := mux.NewRouter()
router.HandleFunc("/{id}", getURL).Methods("GET")
router.HandleFunc("/api/v1/urls", createURL).Methods("POST")
router.HandleFunc("/create/urls", createURLs).Methods("POST")
// Listener
http.ListenAndServe(":80", router)
// close connection to db when main func finishes
defer db.Close()
}
Now I'm building unit test for getURL function, which is a GET method to get data from my postgresql database called urlshortener and the table name is urls.
Here is getURL function code:
func getURL(w http.ResponseWriter, r *http.Request) {
params := mux.Vars(r)
var url Url
err := db.Find(&url, params["id"]).Error
if err != nil {
w.WriteHeader(http.StatusNotFound)
} else {
w.WriteHeader(http.StatusOK)
json.NewEncoder(w).Encode(url.Url)
}
}
This is work fine with my database. See curl command below:
I know that the unit test is not for mock data, and it aim to test a function/method is stable or not. Although I import mux and net/http for conncetion, but I think the unit test on it should be "SQL syntax". So I decide to focus on testing if gorm return the right value to the test function.
In this case, db.Find will return a *gorm.DB struct which should be exactly same with second line. (see docs https://gorm.io/docs/query.html)
db.Find(&url, params["id"])
SELECT * FROM urls WHICH id=<input_number>
My question is how to write a unit test on it for check the SQL syntax is correct or not in this case (gorm+mux)? I've check some articles, but most of them are testing the http connect status but not for SQL.
And my function do not have the return value, or I need to rewrite the function to have a return value before I can test it?
below is the test structure in my mind:
func TestGetURL(t *testing.T) {
//set const answer for this test
//set up the mock sql connection
//call getURL()
//check if equal with answer using assert
}
Update
According to #Emin Laletovic answer
Now I have a prototype of my testGetURL. Now I have new questions on it.
func TestGetURL(t *testing.T) {
//set const answer for this test
testQuery := `SELECT * FROM "urls" WHERE id=1`
id := 1
//set up the mock sql connection
testDB, mock, err := sqlmock.New()
if err != nil {
panic("sqlmock.New() occurs an error")
}
// uses "gorm.io/driver/postgres" library
dialector := postgres.New(postgres.Config{
DSN: "sqlmock_db_0",
DriverName: "postgres",
Conn: testDB,
PreferSimpleProtocol: true,
})
db, err = gorm.Open(dialector, &gorm.Config{})
if err != nil {
panic("Cannot open stub database")
}
//mock the db.Find function
rows := sqlmock.NewRows([]string{"id", "url", "expire_at", "short_url"}).
AddRow(1, "http://somelongurl.com", "some_date", "http://shorturl.com")
mock.ExpectQuery(regexp.QuoteMeta(testQuery)).
WillReturnRows(rows).WithArgs(id)
//create response writer and request for testing
mockedRequest, _ := http.NewRequest("GET", "/1", nil)
mockedWriter := httptest.NewRecorder()
//call getURL()
getURL(mockedWriter, mockedRequest)
//check values in mockedWriter using assert
}
In the code, I mock the request and respone with http, httptest libs.
I run the test, but it seems that the getURL function in main.go cannot receive the args I pass in, see the pic below.
when db.find called, mock.ExpectQuery receive it and start to compare it, so far so good.
db.Find(&url, params["id"])
mock.ExpectQuery(regexp.QuoteMeta(testQuery)).WillReturnRows(rows).WithArgs(id)
According to the testing log, it shows that when db.Find triggerd, it only excute SELECT * FROM "urls" but not I expected SELECT * FROM "urls" WHERE "urls"."id" = $1.
But when I test db.Find on local with postman and log the SQL syntax out, it can be excute properly. see pic below.
In summary, I think the problem is the responeWriter/request I put in getURL(mockedWriter, mockedRequest) are wrong, and it leads that getURL(w http.ResponseWriter, r *http.Request) cannot work as we expect.
Please let me know if I missing anything~
Any idea or way to rewrite the code would be help, thank you!
If you just want to test the SQL string that db.Find returns, you can use the DryRun feature (per documentation).
stmt := db.Session(&Session{DryRun: true}).Find(&url, params["id"]).Statement
stmt.SQL.String() //returns SQL query string without the param value
stmt.Vars // contains an array of input params
However, to write a test for the getURL function, you could use sqlmock to mock the results that would be returned when executing the db.Find call.
func TestGetURL(t *testing.T) {
//set const answer for this test
testQuery := "SELECT * FROM `urls` WHERE `id` = $1"
id := 1
//create response writer and request for testing
//set up the mock sql connection
testDB, mock, err := sqlmock.New()
//handle error
// uses "gorm.io/driver/postgres" library
dialector := postgres.New(postgres.Config{
DSN: "sqlmock_db_0",
DriverName: "postgres",
Conn: testDB,
PreferSimpleProtocol: true,
})
db, err = gorm.Open(dialector, &gorm.Config{})
//handle error
//mock the db.Find function
rows := sqlmock.NewRows([]string{"id", "url", "expire_at", "short_url"}).
AddRow(1, "http://somelongurl.com", "some_date", "http://shorturl.com")
mock.ExpectQuery(regexp.QuoteMeta(testQuery)).
WillReturnRows(rows).WithArgs(id)
//call getURL()
getUrl(mockedWriter, &mockedRequest)
//check values in mockedWriter using assert
}
This Post and Emin Laletovic are really helps me alot.
I think I get the answer to this qeustion.
Let's recap this questioon. First, I'm using gorm for postgresql and mux for http services and build a CRUD service.
I need to write a unit test to check if my database syntax is correct (we assuming that the connection is statusOK), so we focus on how to write a unit test for SQL syntax.
But the handler function in main.go don't have return value, so we need to use mock-sql/ ExpectQuery(), this function will be triggered when the db.Find() inside getURL(). By doing this, we dont have to return a value to check if it match our target or not.
The problem I met in Update is fixed by This Post, building an unit test with mux, but that post is focusing on status check and return value.
I set the const answer for this test, the id variable is what we expect to get. Noticed that $1 I don't know how to change it, and I've try many times to rewrite but SQL syntax is still return $1, maybe it is some kind of constraint I dont know.
//set const answer for this test
testQuery := `SELECT * FROM "urls" WHERE "urls"."id" = $1`
id := "1"
I set the value pass into the getURL() by doint this
//set the value send into the function
vars := map[string]string{
"id": "1",
}
//create response writer and request for testing
mockedWriter := httptest.NewRecorder()
mockedRequest := httptest.NewRequest("GET", "/{id}", nil)
mockedRequest = mux.SetURLVars(mockedRequest, vars)
Finally, we call mock.ExpectationsWereMet() to check if anything went wrong.
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("SQL syntax is not match: %s", err)
}
Below is my test code:
func TestGetURL(t *testing.T) {
//set const answer for this test
testQuery := `SELECT * FROM "urls" WHERE "urls"."id" = $1`
id := "1"
//set up the mock sql connection
testDB, mock, err := sqlmock.New()
if err != nil {
panic("sqlmock.New() occurs an error")
}
// uses "gorm.io/driver/postgres" library
dialector := postgres.New(postgres.Config{
DSN: "sqlmock_db_0",
DriverName: "postgres",
Conn: testDB,
PreferSimpleProtocol: true,
})
db, err = gorm.Open(dialector, &gorm.Config{})
if err != nil {
panic("Cannot open stub database")
}
//mock the db.Find function
rows := sqlmock.NewRows([]string{"id", "url", "expire_at", "short_url"}).
AddRow(1, "url", "date", "shorurl")
//try to match the real SQL syntax we get and testQuery
mock.ExpectQuery(regexp.QuoteMeta(testQuery)).WillReturnRows(rows).WithArgs(id)
//set the value send into the function
vars := map[string]string{
"id": "1",
}
//create response writer and request for testing
mockedWriter := httptest.NewRecorder()
mockedRequest := httptest.NewRequest("GET", "/{id}", nil)
mockedRequest = mux.SetURLVars(mockedRequest, vars)
//call getURL()
getURL(mockedWriter, mockedRequest)
//check result in mockedWriter mocksql built function
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("SQL syntax is not match: %s", err)
}
}
And I run two tests with args(1, 1) and args(1, 2), and it works fine. see pic below(please ignore the chinese words)
I am trying to do db transaction with mongo-db in golang but getting cannot create namespace error
// For a replica set, include the replica set name and a seedlist of the members in the URI string; e.g.
// uri := "mongodb://mongodb0.example.com:27017,mongodb1.example.com:27017/?replicaSet=myRepl"
// For a sharded cluster, connect to the mongos instances; e.g.
// uri := "mongodb://mongos0.example.com:27017,mongos1.example.com:27017/"
uri := "mongodb://mongo-0/block-recorder?replicaSet=rs0"
// var uri string
clientOpts := options.Client().ApplyURI(uri)
client, err := mongo.Connect(ctx, clientOpts)
if err != nil {
panic(err)
}
defer func() { _ = client.Disconnect(ctx) }()
// Prereq: Create collections.
wcMajority := writeconcern.New(writeconcern.WMajority(), writeconcern.WTimeout(1*time.Second))
wcMajorityCollectionOpts := options.Collection().SetWriteConcern(wcMajority)
blockCollection := client.Database("block-recorder").Collection("Block", wcMajorityCollectionOpts)
// Step 1: Define the callback that specifies the sequence of operations to perform inside the transaction.
callback := func(sessCtx mongo.SessionContext) (interface{}, error) {
// Important: You must pass sessCtx as the Context parameter to the operations for them to be executed in the
// transaction.
dbBlock := model.TransferBlockData(block)
if _, err := blockCollection.InsertOne(sessCtx, dbBlock); err != nil {
return nil, err
}
return nil, nil
}
// Step 2: Start a session and run the callback using WithTransaction.
session, err := client.StartSession()
if err != nil {
panic(err)
}
defer session.EndSession(ctx)
result, err := session.WithTransaction(ctx, callback)
if err != nil {
panic(err)
}
fmt.Printf("result: %v\n", result)
This is the sample code I am using, Getting below error
panic: multiple write errors: [{write errors: [{Cannot create namespace block-recorder.Block in multi-document transaction.}]}, {<nil>}]
I uses gorm most of the times, This is my first time using mongo with golang, It says create collection first which I am already doing ? Is there any other method to create collection for mongo in golang ?
How to do a proper transaction with struct data example ?
MongoDB server cannot currently create collections in a transaction.
If your application is, say, inserting data into a collection that doesn't exist, the server transparently creates the collection in most cases. But this does not work currently if a transaction is active.
Create the collection ahead of time so that it exists by the time the transaction is executing.
Instantiating a collection object in your application does not actually create a collection. To create a collection, try the equivalent of createCollection in the go driver.
I'm fetching a document from MongoDB and passing it into function transform, e.g.
var doc map[string]interface{}
err := collection.FindOne(context.TODO(), filter).Decode(&doc)
result := transform(doc)
I want to write unit tests for transform, but I'm not sure how to mock a response from MongoDB. Ideally I want to set something like this up:
func TestTransform(t *testing.T) {
byt := []byte(`
{"hello": "world",
"message": "apple"}
`)
var doc map[string]interface{}
>>> Some method here to Decode byt into doc like the code above <<<
out := transform(doc)
expected := ...
if diff := deep.Equal(expected, out); diff != nil {
t.Error(diff)
}
}
One way would be to json.Unmarshal into doc, but this sometimes gives different results. For example, if the document in MongoDB has an array in it, then that array is decoded into doc as a bson.A type not []interface{} type.
A member from my team recently found out there is a hidden gem inside the official MongoDB driver for GO: https://pkg.go.dev/go.mongodb.org/mongo-driver#v1.9.1/mongo/integration/mtest. Although the package is in experimental mode and there is no backward compatibility guaranteed for it, it can help you to perform unit testing, at least with this version of the driver.
You can check this cool article with plenty of examples of how to use it: https://medium.com/#victor.neuret/mocking-the-official-mongo-golang-driver-5aad5b226a78. Additionally, here is the repository with the code samples for this article: https://github.com/victorneuret/mongo-go-driver-mock.
So, based in your example and the samples from the article I think you could try something like the following (of course, you might need to tweak and experiment with this):
func TestTransform(t *testing.T) {
mt := mtest.New(t, mtest.NewOptions().ClientType(mtest.Mock))
defer mt.Close()
mt.Run("find & transform", func(mt *mtest.T) {
myollection = mt.Coll
expected := myStructure{...}
mt.AddMockResponses(mtest.CreateCursorResponse(1, "foo.bar", mtest.FirstBatch, bson.D{
{"_id", expected.ID},
{"field-1", expected.Field1},
{"field-2", expected.Field2},
}))
response, err := myFindFunction(expected.ID)
if err != nil {
t.Error(err)
}
out := transform(response)
if diff := deep.Equal(expected, out); diff != nil {
t.Error(diff)
}
})
}
Alternatively, you can perform a more real testing and in an automated way via integration testing with Docker containers. There are a few good packages that could help you with this:
https://github.com/ory/dockertest
https://github.com/testcontainers/testcontainers-go
I have followed this approach with dockertest library to automate a full integration testing environment that could be setUp and tearDown via the go test -v -run Integration command. See a full example here: https://github.com/AnhellO/learn-dockertest/tree/master/mongo.
Hope this helps.
The best solution to write testable could would be to extract your code to a DAO or Data-Repository. You would define an interface which would return what you need. This way, you can just used a Mocked Version for testing.
// repository.go
type ISomeRepository interface {
Get(string) (*SomeModel, error)
}
type SomeRepository struct { ... }
func (r *SomeRepository) Get(id string) (*SomeModel, error) {
// Handling a real repository access and returning your Object
}
When you need to mock it, just create a Mock-Struct and implement the interface:
// repository_test.go
type SomeMockRepository struct { ... }
func (r *SomeRepository) Get(id string) (*SomeModel, error) {
return &SomeModel{...}, nil
}
func TestSomething() {
// You can use your mock as ISomeRepository
var repo *ISomeRepository
repo = &SomeMockRepository{}
someModel, err := repo.Get("123")
}
This is best used with some kind of dependency-injection, so passing this repository as ISomeRepository into the function.
Using monkey library to hook any function from mongo driver.
For example:
func insert(collection *mongo.Collection) (int, error) {
ctx, _ := context.WithTimeout(context.Background(), 10*time.Second)
u := User{
Name: "kevin",
Age: 20,
}
res, err := collection.InsertOne(ctx, u)
if err != nil {
log.Printf("error: %v", err)
return 0, err
}
id := res.InsertedID.(int)
return id, nil
}
func TestInsert(t *testing.T) {
var c *mongo.Collection
var guard *monkey.PatchGuard
guard = monkey.PatchInstanceMethod(reflect.TypeOf(c), "InsertOne",
func(c *mongo.Collection, ctx context.Context, document interface{}, opts ...*options.InsertOneOptions) (*mongo.InsertOneResult, error) {
guard.Unpatch()
defer guard.Restore()
log.Printf("record: %+v, collection: %s, database: %s", document, c.Name(), c.Database().Name())
res := &mongo.InsertOneResult{
InsertedID: 100,
}
return res, nil
})
collection := client.Database("db").Collection("person")
id, err := insert(collection)
require.NoError(t, err)
assert.Equal(t, id, 100)
}
I writing a client that connects to a server with REST endpoints. The client needs to make a chain of 11 different requests to complete an action (it's a rococo backup system).
I'm writing my client in Go and I also want to write my mocks/tests in Go. What I'm unclear about is how a test called func TestMain would call into the client's func main(), to test completion of the chain of 11 requests.
My client's binary would be run from the shell in the following way:
$ client_id=12345 region=apac3 backup
How would I call func main() from the tests, with environment variables set? Or is there another approach? (I'm comfortable writing tests, so that's not the issue)
I'm looking at the Advanced Example in jarcoal/httpmock (but I could use another library). At the end the example says // do stuff that adds and checks articles, is that where I would call main()?
I've pasted the Advanced Example below, for future reference.
func TestFetchArticles(t *testing.T) {
httpmock.Activate()
defer httpmock.DeactivateAndReset()
// our database of articles
articles := make([]map[string]interface{}, 0)
// mock to list out the articles
httpmock.RegisterResponder("GET", "https://api.mybiz.com/articles.json",
func(req *http.Request) (*http.Response, error) {
resp, err := httpmock.NewJsonResponse(200, articles)
if err != nil {
return httpmock.NewStringResponse(500, ""), nil
}
return resp, nil
},
)
// mock to add a new article
httpmock.RegisterResponder("POST", "https://api.mybiz.com/articles.json",
func(req *http.Request) (*http.Response, error) {
article := make(map[string]interface{})
if err := json.NewDecoder(req.Body).Decode(&article); err != nil {
return httpmock.NewStringResponse(400, ""), nil
}
articles = append(articles, article)
resp, err := httpmock.NewJsonResponse(200, article)
if err != nil {
return httpmock.NewStringResponse(500, ""), nil
}
return resp, nil
},
)
// do stuff that adds and checks articles
}
Writing this out helped me answer my own question.
main() would read in environment variables and then call a function like doBackup(client_id, region). My test would mock the endpoints and then call doBackup(client_id, region).
Newbie Go programmer here. I'm writing a package that reads a JSON configuration file. It uses the built-in JSON decoding, of course. But I want it to be able to include other JSON files as well, by looking for an array of filenames with the key of 'Includes'. I got it working as just a function and passing in a struct for the JSON data that includes a slice of strings labeled 'Includes', but I don't know how to specify this as a package.
Here's the function:
func ReadConfig(filename string, configuration *Configuration) error {
log.Println("reading file", filename)
file, err := os.Open(filename)
if err != nil {
log.Println("Can't read", filename)
return err
}
decoder := json.NewDecoder(file)
if err := decoder.Decode(&configuration); err != nil {
log.Println(err)
return err
}
includes := make([]string, len(configuration.Includes))
copy(includes, configuration.Includes)
config.Includes = configuration.Includes[0:0]
for _, inc := range includes {
log.Println(inc)
if err := ReadConfig(inc, configuration); err != nil {
return err
}
}
return nil
}
Which works with:
type Configuration struct {
Includes []string
.... other defs
}
But, in a package, I want ReadConfig to take any kind Configuration struct, as long as one of its members is 'Includes []string'.
I believe I need to change the ReadConfig def to:
func ReadConfig(filename string, configuration interface{})
But what I don't know is how to access the Includes slice within that.
Just create an interface for it
type Configurable interface {
Configuration() []string
}
And then provide a Configuration method instead of a field for your structs, and change the signature of your function to func ReadConfig(filename string, configuration Configurable).
It'd be much easier to just pass in the slice instead of the struct though.