I am trying to test a handler, the handler is below:
router.HandleFunc("/address/{id}", DeletePerson).Methods("DELETE")
The request that I created was:
request, _ := http.NewRequest("DELETE", "/address/2", nil)
DeletePerson(response, request)
using github.com/gorilla/mux I attempted to extract the "id" with
params = mux.Vars(request)
item := params["id"]
returns: params = map[] and item = ""
However, if I call DeletePerson with the curl command:
curl -X DELETE http://localhost:8000/address/2
I get: params = map["id"] and item = "2"
HOW Do I construct a URL request that get the results like Curl?
I think the problem is that you don't put the full URL in the request. And I guess that you ignore the error while executing the request. If you don't put the full URL it will complains something like this: panic: Delete /address/2: unsupported protocol scheme ""
The following code works OK in my machine:
package main
import "net/http"
func main() {
r, err := http.NewRequest("DELETE", "http://localhost:8080/address/2", nil)
if err != nil {
panic(err)
}
if _, err := http.DefaultClient.Do(r); err != nil {
panic(err)
}
}
Hope this helps :)
If you call DeletePerson directly, the request doesn't pass through the router, which parses the parameters in the request path.
Also, http.NewRequest returns a client request. Either add scheme and host to the URL and pass the request to http.Client.Do, or use httptest.NewRequest to create a server request directly.
NewRequest returns a new incoming server Request, suitable for passing to an http.Handler for testing.
request := httptest.NewRequest("DELETE", "/address/2", nil)
mux.ServeHTTP(response, request)
Related
I'm new in Go and unit test. I build a samll side projecy called "urlshortener" using Go with Gorm, mux and postgresql.
There is a qeustion annoying me after search many articles.
To make the question clean, I delete some irrelevant code like connect db, .env, etc
My code is below(main.go):
package main
type Url struct {
ID uint `gorm:"primaryKey"` // used for shortUrl index
Url string `gorm:"unique"` // prevent duplicate url
ExpireAt string
ShortUrl string
}
var db *gorm.DB
var err error
func main() {
// gain access to database by getting .env
...
// database connection string
...
// make migrations to the dbif they have not already been created
db.AutoMigrate(&Url{})
// API routes
router := mux.NewRouter()
router.HandleFunc("/{id}", getURL).Methods("GET")
router.HandleFunc("/api/v1/urls", createURL).Methods("POST")
router.HandleFunc("/create/urls", createURLs).Methods("POST")
// Listener
http.ListenAndServe(":80", router)
// close connection to db when main func finishes
defer db.Close()
}
Now I'm building unit test for getURL function, which is a GET method to get data from my postgresql database called urlshortener and the table name is urls.
Here is getURL function code:
func getURL(w http.ResponseWriter, r *http.Request) {
params := mux.Vars(r)
var url Url
err := db.Find(&url, params["id"]).Error
if err != nil {
w.WriteHeader(http.StatusNotFound)
} else {
w.WriteHeader(http.StatusOK)
json.NewEncoder(w).Encode(url.Url)
}
}
This is work fine with my database. See curl command below:
I know that the unit test is not for mock data, and it aim to test a function/method is stable or not. Although I import mux and net/http for conncetion, but I think the unit test on it should be "SQL syntax". So I decide to focus on testing if gorm return the right value to the test function.
In this case, db.Find will return a *gorm.DB struct which should be exactly same with second line. (see docs https://gorm.io/docs/query.html)
db.Find(&url, params["id"])
SELECT * FROM urls WHICH id=<input_number>
My question is how to write a unit test on it for check the SQL syntax is correct or not in this case (gorm+mux)? I've check some articles, but most of them are testing the http connect status but not for SQL.
And my function do not have the return value, or I need to rewrite the function to have a return value before I can test it?
below is the test structure in my mind:
func TestGetURL(t *testing.T) {
//set const answer for this test
//set up the mock sql connection
//call getURL()
//check if equal with answer using assert
}
Update
According to #Emin Laletovic answer
Now I have a prototype of my testGetURL. Now I have new questions on it.
func TestGetURL(t *testing.T) {
//set const answer for this test
testQuery := `SELECT * FROM "urls" WHERE id=1`
id := 1
//set up the mock sql connection
testDB, mock, err := sqlmock.New()
if err != nil {
panic("sqlmock.New() occurs an error")
}
// uses "gorm.io/driver/postgres" library
dialector := postgres.New(postgres.Config{
DSN: "sqlmock_db_0",
DriverName: "postgres",
Conn: testDB,
PreferSimpleProtocol: true,
})
db, err = gorm.Open(dialector, &gorm.Config{})
if err != nil {
panic("Cannot open stub database")
}
//mock the db.Find function
rows := sqlmock.NewRows([]string{"id", "url", "expire_at", "short_url"}).
AddRow(1, "http://somelongurl.com", "some_date", "http://shorturl.com")
mock.ExpectQuery(regexp.QuoteMeta(testQuery)).
WillReturnRows(rows).WithArgs(id)
//create response writer and request for testing
mockedRequest, _ := http.NewRequest("GET", "/1", nil)
mockedWriter := httptest.NewRecorder()
//call getURL()
getURL(mockedWriter, mockedRequest)
//check values in mockedWriter using assert
}
In the code, I mock the request and respone with http, httptest libs.
I run the test, but it seems that the getURL function in main.go cannot receive the args I pass in, see the pic below.
when db.find called, mock.ExpectQuery receive it and start to compare it, so far so good.
db.Find(&url, params["id"])
mock.ExpectQuery(regexp.QuoteMeta(testQuery)).WillReturnRows(rows).WithArgs(id)
According to the testing log, it shows that when db.Find triggerd, it only excute SELECT * FROM "urls" but not I expected SELECT * FROM "urls" WHERE "urls"."id" = $1.
But when I test db.Find on local with postman and log the SQL syntax out, it can be excute properly. see pic below.
In summary, I think the problem is the responeWriter/request I put in getURL(mockedWriter, mockedRequest) are wrong, and it leads that getURL(w http.ResponseWriter, r *http.Request) cannot work as we expect.
Please let me know if I missing anything~
Any idea or way to rewrite the code would be help, thank you!
If you just want to test the SQL string that db.Find returns, you can use the DryRun feature (per documentation).
stmt := db.Session(&Session{DryRun: true}).Find(&url, params["id"]).Statement
stmt.SQL.String() //returns SQL query string without the param value
stmt.Vars // contains an array of input params
However, to write a test for the getURL function, you could use sqlmock to mock the results that would be returned when executing the db.Find call.
func TestGetURL(t *testing.T) {
//set const answer for this test
testQuery := "SELECT * FROM `urls` WHERE `id` = $1"
id := 1
//create response writer and request for testing
//set up the mock sql connection
testDB, mock, err := sqlmock.New()
//handle error
// uses "gorm.io/driver/postgres" library
dialector := postgres.New(postgres.Config{
DSN: "sqlmock_db_0",
DriverName: "postgres",
Conn: testDB,
PreferSimpleProtocol: true,
})
db, err = gorm.Open(dialector, &gorm.Config{})
//handle error
//mock the db.Find function
rows := sqlmock.NewRows([]string{"id", "url", "expire_at", "short_url"}).
AddRow(1, "http://somelongurl.com", "some_date", "http://shorturl.com")
mock.ExpectQuery(regexp.QuoteMeta(testQuery)).
WillReturnRows(rows).WithArgs(id)
//call getURL()
getUrl(mockedWriter, &mockedRequest)
//check values in mockedWriter using assert
}
This Post and Emin Laletovic are really helps me alot.
I think I get the answer to this qeustion.
Let's recap this questioon. First, I'm using gorm for postgresql and mux for http services and build a CRUD service.
I need to write a unit test to check if my database syntax is correct (we assuming that the connection is statusOK), so we focus on how to write a unit test for SQL syntax.
But the handler function in main.go don't have return value, so we need to use mock-sql/ ExpectQuery(), this function will be triggered when the db.Find() inside getURL(). By doing this, we dont have to return a value to check if it match our target or not.
The problem I met in Update is fixed by This Post, building an unit test with mux, but that post is focusing on status check and return value.
I set the const answer for this test, the id variable is what we expect to get. Noticed that $1 I don't know how to change it, and I've try many times to rewrite but SQL syntax is still return $1, maybe it is some kind of constraint I dont know.
//set const answer for this test
testQuery := `SELECT * FROM "urls" WHERE "urls"."id" = $1`
id := "1"
I set the value pass into the getURL() by doint this
//set the value send into the function
vars := map[string]string{
"id": "1",
}
//create response writer and request for testing
mockedWriter := httptest.NewRecorder()
mockedRequest := httptest.NewRequest("GET", "/{id}", nil)
mockedRequest = mux.SetURLVars(mockedRequest, vars)
Finally, we call mock.ExpectationsWereMet() to check if anything went wrong.
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("SQL syntax is not match: %s", err)
}
Below is my test code:
func TestGetURL(t *testing.T) {
//set const answer for this test
testQuery := `SELECT * FROM "urls" WHERE "urls"."id" = $1`
id := "1"
//set up the mock sql connection
testDB, mock, err := sqlmock.New()
if err != nil {
panic("sqlmock.New() occurs an error")
}
// uses "gorm.io/driver/postgres" library
dialector := postgres.New(postgres.Config{
DSN: "sqlmock_db_0",
DriverName: "postgres",
Conn: testDB,
PreferSimpleProtocol: true,
})
db, err = gorm.Open(dialector, &gorm.Config{})
if err != nil {
panic("Cannot open stub database")
}
//mock the db.Find function
rows := sqlmock.NewRows([]string{"id", "url", "expire_at", "short_url"}).
AddRow(1, "url", "date", "shorurl")
//try to match the real SQL syntax we get and testQuery
mock.ExpectQuery(regexp.QuoteMeta(testQuery)).WillReturnRows(rows).WithArgs(id)
//set the value send into the function
vars := map[string]string{
"id": "1",
}
//create response writer and request for testing
mockedWriter := httptest.NewRecorder()
mockedRequest := httptest.NewRequest("GET", "/{id}", nil)
mockedRequest = mux.SetURLVars(mockedRequest, vars)
//call getURL()
getURL(mockedWriter, mockedRequest)
//check result in mockedWriter mocksql built function
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("SQL syntax is not match: %s", err)
}
}
And I run two tests with args(1, 1) and args(1, 2), and it works fine. see pic below(please ignore the chinese words)
In program bellow I have two routers. One is working at localhost:3000 and acts like a public access point. It also may send requests with data to another local address which is localhost:8000 where data is being processed. Second router is working at localhost:8000 and handles processing requests for the first router.
Problem
The first router sends a request with context to the second using http.NewRequestWithContext() function. The value is being added to the context and the context is added to request. When request arrives to the second router it does not have value that was added previously.
Some things like error handling are not being written to not post a wall of code here.
package main
import (
"bytes"
"context"
"net/http"
"github.com/go-chi/chi"
"github.com/go-chi/chi/middleware"
)
func main() {
go func() {
err := http.ListenAndServe(
"localhost:3000",
GetDataAndSolve(),
)
if err != nil {
panic(err)
}
}()
go func() {
err := http.ListenAndServe( // in GetDataAndSolve() we send requests
"localhost:8000", // with data for processing
InternalService(),
)
if err != nil {
panic(err)
}
}()
// interrupt := make(chan os.Signal, 1)
// signal.Notify(interrupt, syscall.SIGTERM, syscall.SIGINT)
// <-interrupt // just a cool way to close the program, uncomment if you need it
}
func GetDataAndSolve() http.Handler {
r := chi.NewRouter()
r.Use(middleware.Logger)
r.Get("/tasks/str", func(rw http.ResponseWriter, r *http.Request) {
// receiving data for processing...
taskCtx := context.WithValue(r.Context(), "str", "strVar") // the value is being
postReq, err := http.NewRequestWithContext( // stored to context
taskCtx, // context is being given to request
"POST",
"http://localhost:8000/tasks/solution",
bytes.NewBuffer([]byte("something")),
)
postReq.Header.Set("Content-Type", "application/json") // specifying for endpoint
if err != nil { // what we are sending
return
}
resp, err := http.DefaultClient.Do(postReq) // running actual request
// pls, proceed to Solver()
// do stuff to resp
// also despite arriving to middleware without right context
// here resp contains a request with correct context
})
return r
}
func Solver(next http.Handler) http.Handler { // here we end up after sending postReq
return http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {
if r.Context().Value("str").(string) == "" {
return // the request arrive without "str" in its context
}
ctxWithResult := context.WithValue(r.Context(), "result", mockFunc(r.Context()))
next.ServeHTTP(rw, r.Clone(ctxWithResult))
})
}
func InternalService() http.Handler {
r := chi.NewRouter()
r.Use(middleware.Logger)
r.With(Solver).Post("/tasks/solution", emptyHandlerFunc)
return r
}
Your understanding of context is not correct.
Context (simplifying to an extent and in reference to NewRequestWithContext API), is just an in-memory object using which you can control the lifetime of the request (Handling/Triggering cancellations).
However your code is making a HTTP call, which goes over the wire (marshaled) using HTTP protocol. This protocol doesn't understand golang's context or its values.
In your scenario, both /tasks/str and /tasks/solution are being run on the same server. What if they were on different servers, probably different languages and application servers as well, So the context cannot be sent across.
Since the APIs are within the same server, maybe you can avoid making a full blown HTTP call and resort to directly invoking the API/Method. It might turn out to be faster as well.
If you still want to send additional values from context, then you'll have to make use of other attributes like HTTP Headers, Params, Body to send across the required information. This can provide more info on how to serialize data from context over HTTP.
I would like to send a search/query to Splunk REST API, and return a search id to later consume the results.
I can achieve the desired behavior with the below curl:
#!/bin/bash
user='my_user'
pass='my_pass'
search='search index=short sourcetype=src | head 5'
curl -u $user:$pass -k https://111.22.33.44:8089/services/search/jobs -d search="$search"
which returns:
<?xml version="1.0" encoding="UTF-8"?>
<response>
<sid>234523452435.6556_234234-3J3J-34J4-2345-123456678E3</sid>
</response>
Here are the relevant Go snippets in which I am trying to achieve the same:
Main:
//main.go
sid, err := conn.Query()
if err != nil {
fmt.Println("err creating search: %s", err)
} else {
fmt.Println("sid:", sid)
}
Query:
// query.go
func (conn SplunkConnection) Query() (string, error) {
data := make(url.Values)
data.Add("output_mode", "json")
data.Add("search%20index%3Dshort%20sourcetype%3Dsrc%20%7C%20head%205", "search")
data.Add("-60m%40m", "earliest")
data.Add("-10m%40m", "latest")
// try httpGet() here
sid, err := conn.httpPost(fmt.Sprintf("%s/services/search/jobs", conn.BaseURL), &data)
if err != nil {
return "", err
}
return string(sid), err
}
Helper:
// http.go
func (conn SplunkConnection) httpPost(url string, data *url.Values) (string, error) {
return conn.httpCall(url, "POST", data)
}
What I expect is a response containing just a JSON blob with my SID. Instead, it returns a huge JSON, which appears to be contain all current jobs at the /services/search/jobs endpoint.
How can I adjust my code to return just the SID? (I intend to poll it for completion and retrieve the results later, but don't need help with this...yet).
You seem to have reversed your postdata parameters.
data.Add("search%20index%3Dshort%20sourcetype%3Dsrc%20%7C%20head%205", "search")
This becomes search index=short sourcetype=src | head 5 = search, which is the reverse of what you want.
The key comes first, then the value, but you have specified the value first, then the key.
I think this should instead be:
data.Add("search", "search%20index%3Dshort%20sourcetype%3Dsrc%20%7C%20head%205")
I suspect the same is true of some of your other calls to url.Values.Add(), so you should check them all and be sure.
I need to implement web service in go that processes tar.gz files and I wonder what is the correct way, what content type I need to define, etc.
plus, I found that a lot of things are handled automatically - on the client side I just post a gzip reader as request body and Accept-Encoding: gzip header is added automatically, and on the server side - I do not need to gunzip the request body, it is already extracted to tar. does that make sense?
Can I rely that it would be like this with any client?
Server:
func main() {
router := mux.NewRouter().StrictSlash(true)
router.Handle("/results", dataupload.NewUploadHandler()).Methods("POST")
log.Fatal(http.ListenAndServe(*address, router))
}
Uploader:
package dataupload
import (
"errors"
log "github.com/Sirupsen/logrus"
"io"
"net/http"
)
// UploadHandler responds to /results http request, which is the result-service rest API for uploading results
type UploadHandler struct {
uploader Uploader
}
// NewUploadHandler creates UploadHandler instance
func NewUploadHandler() *UploadHandler {
return &UploadHandler{
uploader: TarUploader{},
}
}
func (uh UploadHandler) ServeHTTP(writer http.ResponseWriter, request *http.Request) {
retStatus := http.StatusOK
body, err := getBody(request)
if err != nil {
retStatus = http.StatusBadRequest
log.Error("Error fetching request body. ", err)
} else {
_, err := uh.uploader.Upload(body)
}
writer.WriteHeader(retStatus)
}
func getBody(request *http.Request) (io.ReadCloser, error) {
requestBody := request.Body
if requestBody == nil {
return nil, errors.New("Empty request body")
}
var err error
// this part is commented out since somehow the body is already gunzipped - no need to extract it.
/*if strings.Contains(request.Header.Get("Accept-Encoding"), "gzip") {
requestBody, err = gzip.NewReader(requestBody)
}*/
return requestBody, err
}
Client
func main() {
f, err := os.Open("test.tar.gz")
if err != nil {
log.Fatalf("error openning file %s", err)
}
defer f.Close()
client := new(http.Client)
reader, err := gzip.NewReader(f)
if err != nil {
log.Fatalf("error gzip file %s", err)
}
request, err := http.NewRequest("POST", "http://localhost:8080/results", reader)
_, err = client.Do(request)
if err != nil {
log.Fatalf("error uploading file %s", err)
}
}
The code you've written for the client is just sending the tarfile directly because of this code:
reader, err := gzip.NewReader(f)
...
request, err := http.NewRequest("POST", "http://localhost:8080/results", reader)
If you sent the .tar.gz file content directly, then you would need to gunzip it on the server. E.g.:
request, err := http.NewRequest(..., f)
I think that's closer to the behavior you should expect third-party clients to exhibit.
Claerly not, but maybe...
Golang provides a very good support for the http client (and server). This is one of the first language to support http2 and the design of the API clearly shows their concern on having a fast http.
This is why they add Accept-Econding: gzip automatically. That will dramatically reduce the size of the server response and then optimize the transfer.
But the gzip remains an option in http 1 and not all of the client will push this header to your server.
Note that the Content-Type describes the type of data you are sending (here a tar.gz but could be application/json, test/javascript, ...), when the Accept-Encoding describes the way the data has been encoded for the transport
Go will take care of transparently handling the Accept-Encoding for you because it is responsible of the transport of the data. Then it will be up to you to handle the Content-Type because only you know how to give a sense to the content you received
Oke, I'm currently trying to login in to my school website, with my own Crawler. Altough they have some protection against login. First I do a Get request to the Website so I get the token from the hidden Input field. That token I use in my next Post request to login to the url! But for some reason the http response is that I cannot resubmit the form. But with doing the same in Postman rest client (chrome plugin) I can login!
When I try to submit a form to this url:
postLoginUrl = "?username=%s&password=%s&submit=inloggen&_eventId=submit&credentialsType=ldap<=%s"
loginUrl = "https://login.hro.nl/v1/login"
where %s are filled in credentials
req, err := client.Post(loginUrl, "application/x-www-form-urlencoded", strings.NewReader(uri))
I'm getting as response that the Form cannot be resubmitted.
But when I try it with Postman rest client, I'm allowed to login.
code for Csrf token:
func getCSRFtoken() (key string) {
doc, err := goquery.NewDocument(loginUrl)
if err != nil {
log.Fatal(err)
}
types := doc.Find("input")
for node := range types.Nodes {
singlething := types.Eq(node)
hidden_input, _ := singlething.Attr("type")
if hidden_input == "hidden" {
key, _ := singlething.Attr("value")
return key
}
}
return ""
}
goquery.NewDocument is a http.Get()
My question now is, how does the URL get's formatted from the library
Maybe you would be better off using:
(c *Client)PostForm(url string, data url.Values) (resp *Response, err error)
from net/http like http://play.golang.org/p/8D6XI6arkz
With the params in url.Values (instead of concatenating the strings, like you are doing now.)