I am writing a web app in go and using the GORM for my ORM. I need to be able to retrieve all the metrics of a certain user and return it via JSON to be displayed on the front end. The query seems to run successfully but I only see a memory address when printing the results and receive an error when trying to cast the results the standard way.
Here is my current code
func DisplayData(w http.ResponseWriter, r *http.Request) {
//Get the data from the database
var metric models.Metric
results := db.Where("user_id = ?", "1").Find(&metric)
//Write a json response
w.WriteHeader(http.StatusCreated)
w.Header().Set("Content-Type", "application/json")
resp := make(map[string]string)
resp["message"] = results
jsonResp, err := json.Marshal(resp)
if err != nil {
log.Fatalf("Error happened in JSON marshal. Err: %s", err)
}
w.Write(jsonResp)
return
}
This results in the error
controllers/statsCont.go:116:18: cannot use results (type *gorm.DB) as type string in assignment
note: module requires Go 1.17
When I try to cast by surrounding result in string() it gives the following error.
controllers/statsCont.go:116:26: cannot convert results (type *gorm.DB) to type string
note: module requires Go 1.17
As stated by #BaytaDarell the query result is added in the variable passed inside Find method
The return value of Find is different in the context it is called in case when it is called with db type the return type is (tx *DB) and when called with associations type the return type is Error
To solve the issue remove below lines
resp := make(map[string]string)
resp["message"] = results
And update it has
resp := map[string]interface{}{"message": metric}
Related
I'm new in Go and unit test. I build a samll side projecy called "urlshortener" using Go with Gorm, mux and postgresql.
There is a qeustion annoying me after search many articles.
To make the question clean, I delete some irrelevant code like connect db, .env, etc
My code is below(main.go):
package main
type Url struct {
ID uint `gorm:"primaryKey"` // used for shortUrl index
Url string `gorm:"unique"` // prevent duplicate url
ExpireAt string
ShortUrl string
}
var db *gorm.DB
var err error
func main() {
// gain access to database by getting .env
...
// database connection string
...
// make migrations to the dbif they have not already been created
db.AutoMigrate(&Url{})
// API routes
router := mux.NewRouter()
router.HandleFunc("/{id}", getURL).Methods("GET")
router.HandleFunc("/api/v1/urls", createURL).Methods("POST")
router.HandleFunc("/create/urls", createURLs).Methods("POST")
// Listener
http.ListenAndServe(":80", router)
// close connection to db when main func finishes
defer db.Close()
}
Now I'm building unit test for getURL function, which is a GET method to get data from my postgresql database called urlshortener and the table name is urls.
Here is getURL function code:
func getURL(w http.ResponseWriter, r *http.Request) {
params := mux.Vars(r)
var url Url
err := db.Find(&url, params["id"]).Error
if err != nil {
w.WriteHeader(http.StatusNotFound)
} else {
w.WriteHeader(http.StatusOK)
json.NewEncoder(w).Encode(url.Url)
}
}
This is work fine with my database. See curl command below:
I know that the unit test is not for mock data, and it aim to test a function/method is stable or not. Although I import mux and net/http for conncetion, but I think the unit test on it should be "SQL syntax". So I decide to focus on testing if gorm return the right value to the test function.
In this case, db.Find will return a *gorm.DB struct which should be exactly same with second line. (see docs https://gorm.io/docs/query.html)
db.Find(&url, params["id"])
SELECT * FROM urls WHICH id=<input_number>
My question is how to write a unit test on it for check the SQL syntax is correct or not in this case (gorm+mux)? I've check some articles, but most of them are testing the http connect status but not for SQL.
And my function do not have the return value, or I need to rewrite the function to have a return value before I can test it?
below is the test structure in my mind:
func TestGetURL(t *testing.T) {
//set const answer for this test
//set up the mock sql connection
//call getURL()
//check if equal with answer using assert
}
Update
According to #Emin Laletovic answer
Now I have a prototype of my testGetURL. Now I have new questions on it.
func TestGetURL(t *testing.T) {
//set const answer for this test
testQuery := `SELECT * FROM "urls" WHERE id=1`
id := 1
//set up the mock sql connection
testDB, mock, err := sqlmock.New()
if err != nil {
panic("sqlmock.New() occurs an error")
}
// uses "gorm.io/driver/postgres" library
dialector := postgres.New(postgres.Config{
DSN: "sqlmock_db_0",
DriverName: "postgres",
Conn: testDB,
PreferSimpleProtocol: true,
})
db, err = gorm.Open(dialector, &gorm.Config{})
if err != nil {
panic("Cannot open stub database")
}
//mock the db.Find function
rows := sqlmock.NewRows([]string{"id", "url", "expire_at", "short_url"}).
AddRow(1, "http://somelongurl.com", "some_date", "http://shorturl.com")
mock.ExpectQuery(regexp.QuoteMeta(testQuery)).
WillReturnRows(rows).WithArgs(id)
//create response writer and request for testing
mockedRequest, _ := http.NewRequest("GET", "/1", nil)
mockedWriter := httptest.NewRecorder()
//call getURL()
getURL(mockedWriter, mockedRequest)
//check values in mockedWriter using assert
}
In the code, I mock the request and respone with http, httptest libs.
I run the test, but it seems that the getURL function in main.go cannot receive the args I pass in, see the pic below.
when db.find called, mock.ExpectQuery receive it and start to compare it, so far so good.
db.Find(&url, params["id"])
mock.ExpectQuery(regexp.QuoteMeta(testQuery)).WillReturnRows(rows).WithArgs(id)
According to the testing log, it shows that when db.Find triggerd, it only excute SELECT * FROM "urls" but not I expected SELECT * FROM "urls" WHERE "urls"."id" = $1.
But when I test db.Find on local with postman and log the SQL syntax out, it can be excute properly. see pic below.
In summary, I think the problem is the responeWriter/request I put in getURL(mockedWriter, mockedRequest) are wrong, and it leads that getURL(w http.ResponseWriter, r *http.Request) cannot work as we expect.
Please let me know if I missing anything~
Any idea or way to rewrite the code would be help, thank you!
If you just want to test the SQL string that db.Find returns, you can use the DryRun feature (per documentation).
stmt := db.Session(&Session{DryRun: true}).Find(&url, params["id"]).Statement
stmt.SQL.String() //returns SQL query string without the param value
stmt.Vars // contains an array of input params
However, to write a test for the getURL function, you could use sqlmock to mock the results that would be returned when executing the db.Find call.
func TestGetURL(t *testing.T) {
//set const answer for this test
testQuery := "SELECT * FROM `urls` WHERE `id` = $1"
id := 1
//create response writer and request for testing
//set up the mock sql connection
testDB, mock, err := sqlmock.New()
//handle error
// uses "gorm.io/driver/postgres" library
dialector := postgres.New(postgres.Config{
DSN: "sqlmock_db_0",
DriverName: "postgres",
Conn: testDB,
PreferSimpleProtocol: true,
})
db, err = gorm.Open(dialector, &gorm.Config{})
//handle error
//mock the db.Find function
rows := sqlmock.NewRows([]string{"id", "url", "expire_at", "short_url"}).
AddRow(1, "http://somelongurl.com", "some_date", "http://shorturl.com")
mock.ExpectQuery(regexp.QuoteMeta(testQuery)).
WillReturnRows(rows).WithArgs(id)
//call getURL()
getUrl(mockedWriter, &mockedRequest)
//check values in mockedWriter using assert
}
This Post and Emin Laletovic are really helps me alot.
I think I get the answer to this qeustion.
Let's recap this questioon. First, I'm using gorm for postgresql and mux for http services and build a CRUD service.
I need to write a unit test to check if my database syntax is correct (we assuming that the connection is statusOK), so we focus on how to write a unit test for SQL syntax.
But the handler function in main.go don't have return value, so we need to use mock-sql/ ExpectQuery(), this function will be triggered when the db.Find() inside getURL(). By doing this, we dont have to return a value to check if it match our target or not.
The problem I met in Update is fixed by This Post, building an unit test with mux, but that post is focusing on status check and return value.
I set the const answer for this test, the id variable is what we expect to get. Noticed that $1 I don't know how to change it, and I've try many times to rewrite but SQL syntax is still return $1, maybe it is some kind of constraint I dont know.
//set const answer for this test
testQuery := `SELECT * FROM "urls" WHERE "urls"."id" = $1`
id := "1"
I set the value pass into the getURL() by doint this
//set the value send into the function
vars := map[string]string{
"id": "1",
}
//create response writer and request for testing
mockedWriter := httptest.NewRecorder()
mockedRequest := httptest.NewRequest("GET", "/{id}", nil)
mockedRequest = mux.SetURLVars(mockedRequest, vars)
Finally, we call mock.ExpectationsWereMet() to check if anything went wrong.
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("SQL syntax is not match: %s", err)
}
Below is my test code:
func TestGetURL(t *testing.T) {
//set const answer for this test
testQuery := `SELECT * FROM "urls" WHERE "urls"."id" = $1`
id := "1"
//set up the mock sql connection
testDB, mock, err := sqlmock.New()
if err != nil {
panic("sqlmock.New() occurs an error")
}
// uses "gorm.io/driver/postgres" library
dialector := postgres.New(postgres.Config{
DSN: "sqlmock_db_0",
DriverName: "postgres",
Conn: testDB,
PreferSimpleProtocol: true,
})
db, err = gorm.Open(dialector, &gorm.Config{})
if err != nil {
panic("Cannot open stub database")
}
//mock the db.Find function
rows := sqlmock.NewRows([]string{"id", "url", "expire_at", "short_url"}).
AddRow(1, "url", "date", "shorurl")
//try to match the real SQL syntax we get and testQuery
mock.ExpectQuery(regexp.QuoteMeta(testQuery)).WillReturnRows(rows).WithArgs(id)
//set the value send into the function
vars := map[string]string{
"id": "1",
}
//create response writer and request for testing
mockedWriter := httptest.NewRecorder()
mockedRequest := httptest.NewRequest("GET", "/{id}", nil)
mockedRequest = mux.SetURLVars(mockedRequest, vars)
//call getURL()
getURL(mockedWriter, mockedRequest)
//check result in mockedWriter mocksql built function
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("SQL syntax is not match: %s", err)
}
}
And I run two tests with args(1, 1) and args(1, 2), and it works fine. see pic below(please ignore the chinese words)
My application mostly consists of CRUDs to/from MongoDB using mongo-go-drive package. This function is one of gRPC server services and all it does is calling database method action.GetProducts(ctx) and it returns *mongo.cursor. Then the result is decoded. For each document, I put the document content into a singular product struct, then append it to products slices (the GetProductsResponse struct is made using gRPC proto repeated GetProductResponse type). After appending all product into GetProductsResponse, I return the response to gRPC client.
I am also new to testing in general, how should I break down the function and do the mocking (how to mock the cursor?) for unit testing? Is it even necessary in the first place to do unit test on the function even though all it does is appending the result, or should I just go straight for the integration test and skip the unit test since it involves database I/O?
func (s *Server) GetProducts(ctx context.Context, in *pb.EmptyRequest) (*pb.GetProductsResponse, error) {
cursor, err := action.GetProducts(ctx)
if err != nil {
return nil, err
}
products := pb.GetProductsResponse{}
res := model.Product{}
for cursor.Next(ctx) {
// Convert document to above struct
err := cursor.Decode(&res)
if err != nil {
return nil, fmt.Errorf("failed to decode document: %v", err)
}
product := &pb.GetProductResponse{ProductId: res.Product_id.Hex(), Name: res.Name, Price: res.Price, Qty: int32(res.Qty)}
products.Products = append(products.Products, product)
}
return &products, nil
}
If you interact with the DB is not unit testing anymore, because you're integrating with another external system.
Anyway, I use to define my "repository" layer function this way:
package repo
var FetchUserById = func(id string) (*model.User, error){
// here the real logic
return user, err
}
and then, when I have to test my "service" layer logic, I would mock the entire "repository" layer this way:
repo.FetchUserById = func(id string) (*model.User, err) {
return myMockedUser, nil
}
I have two simple api methods in my code. Method with endpoind /api/user/create creates user. Field username is unique. When i trying to create user with same username that already exists in database, i have an error in console:
(/home/andrej/go/src/go_contacts/models/users.go:19)
[2020-12-23 22:03:10] pq: duplicate key value violates unique constraint "users_username_key"
I want to show this error in response to user, or somehow identify type of error in my code, to show different error messages for user. I know only that if i have error user returns me id=0. But it doesnt seems like a good message for user.
main.go
package main
import (
"fmt"
"go_contacts/controllers"
"net/http"
"os"
"github.com/gorilla/mux"
"github.com/joho/godotenv"
)
func main() {
godotenv.Load(".env")
router := mux.NewRouter()
router.HandleFunc("/", controllers.ReturnHello).Methods("GET")
router.HandleFunc("/api/user/create", controllers.CreateUser).Methods("POST")
port := os.Getenv("PORT")
if port == "" {
port = "8000"
}
err := http.ListenAndServe(":"+port, router)
if err != nil {
fmt.Print(err)
}
}
models.go with user struct:
package models
import (
u "go_contacts/utils"
"github.com/jinzhu/gorm"
)
// User base model
type User struct {
gorm.Model
Username string `json:"username" gorm:"unique"`
Password string `json:"password"`
Email string `json:"email"`
}
// Create new user
func (user *User) Create() map[string]interface{} {
GetDB().Create(user)
if user.ID <= 0 {
return u.Message(false, "Failed to create user, connection error.")
}
response := u.Message(true, "Account has been created")
response["user"] = user
return response
}
As for pq v1.5.2 and gorm v1.9.12
First, you need to identify whether create method call returns error or not. Like so
err := GetDB().Create(user).Error
if err != nil {
// code to handle error
}
Then the pq package has special type which maps to postgres server error. It contains fields which can help you to identify the error severity, it's code, table which related to error etc.
Full list of psql fields identifiers can be found here in
Error and Notice Message Fields
https://www.postgresql.org/docs/current/protocol-error-fields.html
To resolve your issue as an option we can use field
Code
Which is a string type representation of error code. But firstly, we need to cast an error and check type cast was successful. Like so
err := GetDB().Create(user).Error
if err != nil {
pqErr, ok := err.(pq.Error)
if !ok {
log.Fatal(err)
}
// code to handle specific error code
}
List of error codes can be found both in the official postgresql docs https://www.postgresql.org/docs/9.3/errcodes-appendix.html
And actual go pq specific mappings to the error in the github.com/lib/pq/error.go as for pq library
And finally we can handle error as duplicate by code
err := GetDB().Create(user).Error
if err != nil {
pqErr, ok := err.(pq.Error)
if !ok {
log.Fatal(err)
}
// note that type casting here is redundant and used for showing specific
// string type from pq library
if pqErr.Code == pq.ErrorCode("23505") { // 23505 is unique_violation error code for psql
// now you can create error and write it to a message to return it for
// a client
}
}
As per the comments gorm provides access to database errors from the Create record function as follows:
result := GetDB().Create(user)
if result.Error != nil {
// Do something with the error
}
Please note that checking the error string is likely to make your application database specific (but this may not be an issue for you). The answers to this question may provide further assistance.
I am getting the following error while deleting from key from array of JSON objects using Go.
Error:
repository/orderRepository.go:394:11: first argument to delete must be map; have interface {}
repository/orderRepository.go:395:11: first argument to delete must be map; have interface {}
repository/orderRepository.go:396:11: first argument to delete must be map; have interface {}
repository/orderRepository.go:397:11: first argument to delete must be map; have interface {}
repository/orderRepository.go:398:11: first argument to delete must be map; have interface {}
I am explaining my code below.
func SyncOrders() map[string]interface{} {
logger.Log.Println("OrderRepository SyncOrders Begin")
resourceManager := resources.ResourceManager{}
session, error := driver.Connect()
db := session.DB(config.Configuration.Database)
var resp map[string]interface{}
if error != nil {
resp := utils.Message(resourceManager.GetProperty(constants.ERROR), resourceManager.GetProperty(constants.DB_SERVER_NOT_REACHABLE_CODE), resourceManager.GetProperty(constants.DB_SERVER_NOT_REACHABLE_DESC))
return resp
} else {
var result []interface{}
//filter := bson.M{"Customer.CustomerID": id, "PaymentDetails.PaymentStatus": "Payment Received"}
//fmt.Println(filter)
err := db.C(ORDERCOLLECTION).Find(nil).All(&result)
if err == nil {
resp = utils.Message(resourceManager.GetProperty(constants.SUCCESS), resourceManager.GetProperty(constants.PRODUCT_GETBYID_CODE), resourceManager.GetProperty(constants.PRODUCT_GETBYID_DESC))
for i := 1; i < len(result); i++ {
delete(result[i],"_id");
delete(result[i],"CreatedAt");
delete(result[i],"CreatedBy");
delete(result[i],"UpdatedAt");
delete(result[i],"UpdatedBy");
}
resp["data"] = result
} else {
//fmt.Println(err)
resp = utils.Message(resourceManager.GetProperty(constants.ERROR), resourceManager.GetProperty(constants.PRODUCT_GETBYID_NOTFOUND_CODE), resourceManager.GetProperty(constants.PRODUCT_GETBYID_NOTFOUND_DESC))
}
defer session.Close()
return resp
}
}
Here I am fetching some record from MongoDB and delete some key value from each record but when I am running the server I am getting these errors. As I am beginner to Go. Can anybody help me to resolve these errors?
The error message says it all: the first argument to the builtin delete() must be a value of static type map.
Your result variable is of type []interface{}, so indexing it like result[i] will result in a value of type interface{}.
If it holds a map, you may use type assertion to obtain the map value from it. Since you use the mgo driver, it is of type bson.M (which is a map[string]interface{}), so you may do it like this:
delete(result[i].(bson.M), "_id")
But it would be better if you would declare result to be a slice of maps in the first place:
var result []bson.M
So then no type assertion will be needed, and the following will be valid code:
delete(result[i], "_id")
Also note that if you want to remove these properties from the results, it would be best if you would tell MongoDB you don't need these fields and so the server wouldn't even send these (saving network traffic) and then you wouldn't have to remove them (saving time and memory).
Use projection to tell you don't need these fields. In mgo you can set a projection using the Query.Select() method.
For example:
err := db.C(ORDERCOLLECTION).Find(nil).Select(bson.M{
"_id": 0,
"CreatedAt": 0,
"CreatedBy": 0,
"UpdatedAt": 0,
"UpdatedBy": 0,
}).All(&result)
The above query will result in documents where the listed fields will not be present, so you don't have to manually remove them using delete().
You are trying to delete key from variable of type interface{}.
#icza gives a good solution.
You can use .Select() to select which fields should be retrieved for the results.
Then you don't need to delete those fields from every object.
For example, the following query would only retrieve the name and age field:
err := db.C(ORDERCOLLECTION).Find(nil).Select(bson.M{"name": 1, "age": 1}).All(&result)
I am using postgresql as my backend database.
Tried to scan a field languagespoken which is an array of text
var user userprofile
row := core.db.QueryRow(
"SELECT languagespoken FROM \"user\" WHERE id = $1",
userId,
)
err := row.Scan(&user.Languages)
if err != nil {
return user, err
}
My structure looks like this
type userprofile struct {
Languages []string `json:languages`
}
But getting the error
2014/06/30 15:27:17 PANIC: reflect.Set: **value of type []uint8 is not assignable to type []string**
/usr/lib/go/src/pkg/reflect/value.go:2198 (0x56c152)
Value.assignTo: panic(context + ": value of type " + v.typ.String() + " is not assignable to type " + dst.String())
/usr/lib/go/src/pkg/reflect/value.go:1385 (0x56966b)
Value.Set: x = x.assignTo("reflect.Set", v.typ, target)
/usr/lib/go/src/pkg/database/sql/convert.go:215 (0x492d70)
convertAssign: dv.Set(sv)
/usr/lib/go/src/pkg/database/sql/sql.go:1562 (0x49c0e5)
(*Rows).Scan: err := convertAssign(dest[i], sv)
/usr/lib/go/src/pkg/database/sql/sql.go:1630 (0x49c560)
(*Row).Scan: err := r.rows.Scan(dest...)
/home/ceresti/source/gocode/src/ceresti.kilnhg.com/ceresti/server/app/databaseapi.go:144 (0x402478)
(*coreStruct).GetUserProfile: err := row.Scan(&user.Languages)
/home/ceresti/source/gocode/src/ceresti.kilnhg.com/ceresti/server/app/restfulapi.go:327 (0x40a63c)
Getuserprofile: userprofileStruct, err := core.GetUserProfile(userId)
/usr/lib/go/src/pkg/runtime/asm_amd64.s:340 (0x4309c2)
Please let me know how to resolve this issue.
Not all sql databases specify array types (e.g., sqlite3). Go doesn't support any sql implementations directly, but it supplies an implementation-agnostic interface (in the generic sense of the word) for which third-party drivers may be written. Go doesn't impose any limitations on which types its drivers may support, so if a type doesn't cooperate, it's probably the fault of the driver.
TL;DR: Try getting it as a string
// if `string` doesn't work, try `[]uint8` or `[]byte` and then convert
// that output to a string if necessary
var languages string
if err := row.Scan(&languages); err != nil {
// handle error
}
// depending on how the string is encoded, this may or may not work
// Update: since you say your list is encoded in the form:
// `{elem1, elem2, elem3}`, we'll simply ignore the first and last
// characters and split on ", "
user.Languages = strings.Split(languages[1:len(languages)-1], ", ")
It appears you're trying to scan the entire result set of that database query in one shot. You can't do that; you need to read each row, one at a time, into a byte slice, then convert the byte slice into a string.
Since you're serializing into a []string, saving the byte slice each time is not a priority. In this case, you can use sql.RawBytes instead of []byte, which will reuse the same memory.
// error checking elided
var row sql.RawBytes
myRow.Scan(&row) // note the pointer!
str := string(row)