Request created with http.NewRequestWithContext() looses context when passed to middleware - rest

In program bellow I have two routers. One is working at localhost:3000 and acts like a public access point. It also may send requests with data to another local address which is localhost:8000 where data is being processed. Second router is working at localhost:8000 and handles processing requests for the first router.
Problem
The first router sends a request with context to the second using http.NewRequestWithContext() function. The value is being added to the context and the context is added to request. When request arrives to the second router it does not have value that was added previously.
Some things like error handling are not being written to not post a wall of code here.
package main
import (
"bytes"
"context"
"net/http"
"github.com/go-chi/chi"
"github.com/go-chi/chi/middleware"
)
func main() {
go func() {
err := http.ListenAndServe(
"localhost:3000",
GetDataAndSolve(),
)
if err != nil {
panic(err)
}
}()
go func() {
err := http.ListenAndServe( // in GetDataAndSolve() we send requests
"localhost:8000", // with data for processing
InternalService(),
)
if err != nil {
panic(err)
}
}()
// interrupt := make(chan os.Signal, 1)
// signal.Notify(interrupt, syscall.SIGTERM, syscall.SIGINT)
// <-interrupt // just a cool way to close the program, uncomment if you need it
}
func GetDataAndSolve() http.Handler {
r := chi.NewRouter()
r.Use(middleware.Logger)
r.Get("/tasks/str", func(rw http.ResponseWriter, r *http.Request) {
// receiving data for processing...
taskCtx := context.WithValue(r.Context(), "str", "strVar") // the value is being
postReq, err := http.NewRequestWithContext( // stored to context
taskCtx, // context is being given to request
"POST",
"http://localhost:8000/tasks/solution",
bytes.NewBuffer([]byte("something")),
)
postReq.Header.Set("Content-Type", "application/json") // specifying for endpoint
if err != nil { // what we are sending
return
}
resp, err := http.DefaultClient.Do(postReq) // running actual request
// pls, proceed to Solver()
// do stuff to resp
// also despite arriving to middleware without right context
// here resp contains a request with correct context
})
return r
}
func Solver(next http.Handler) http.Handler { // here we end up after sending postReq
return http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {
if r.Context().Value("str").(string) == "" {
return // the request arrive without "str" in its context
}
ctxWithResult := context.WithValue(r.Context(), "result", mockFunc(r.Context()))
next.ServeHTTP(rw, r.Clone(ctxWithResult))
})
}
func InternalService() http.Handler {
r := chi.NewRouter()
r.Use(middleware.Logger)
r.With(Solver).Post("/tasks/solution", emptyHandlerFunc)
return r
}

Your understanding of context is not correct.
Context (simplifying to an extent and in reference to NewRequestWithContext API), is just an in-memory object using which you can control the lifetime of the request (Handling/Triggering cancellations).
However your code is making a HTTP call, which goes over the wire (marshaled) using HTTP protocol. This protocol doesn't understand golang's context or its values.
In your scenario, both /tasks/str and /tasks/solution are being run on the same server. What if they were on different servers, probably different languages and application servers as well, So the context cannot be sent across.
Since the APIs are within the same server, maybe you can avoid making a full blown HTTP call and resort to directly invoking the API/Method. It might turn out to be faster as well.
If you still want to send additional values from context, then you'll have to make use of other attributes like HTTP Headers, Params, Body to send across the required information. This can provide more info on how to serialize data from context over HTTP.

Related

How should Go Context be declared?

Suppose I have a file that modifies database, should the functions share a single context or should each function have their own context?
Sharing context
var (
ctx = context.Background()
)
func test1() {
res, err := Collection.InsertOne(ctx, data)
}
func test2() {
res, err := Collection.InsertOne(ctx, data)
}
Or should it be like this?
func test1() {
res, err := Collection.InsertOne(context.Background(), data)
}
func test2() {
res, err := Collection.InsertOne(context.Background(), data)
}
Neither. Your context should be request-scoped (for whatever "request" means in your application) and passed down the call chain to each function.
func test1(ctx context.Context) {
res, err := Collection.InsertOne(ctx, data)
}
func test2(ctx context.Context) {
res, err := Collection.InsertOne(ctx, data)
}
If you're building a web server, as indicated in comments, you'll usually get your context from the HTTP request in your handler:
func myHandler(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
// now pass ctx to any fuction that needs it. This way, if the HTTP
// client/browser cancels the request, your downstream functions will
// be able to abort immediately, without wasting time finishing their
// work.
}
You should not use the first approach. Context is something to be passed down to functions, it should not be declared as a global variable.
The second approach can be used at times, especially if there is no request context. However, if multiple calls are handled within a single server context, you should pass down the context for that call to all the other calls getting a context, so when the context is canceled or expired, all calls would fail.

Sending email onward that was parsed using Golang net/mail.ReadMessage

I'm looking for the cleanest way in Golang to transfer a message through (i.e. act as an SMTP proxy) while performing some manipulation on the message body html (e.g. adding an open tracking pixel - not yet coded).
The net/mail package includes a method ReadMessage that parses mail headers into a map, and gives you an io.Reader for the body. This is necessary to determine the MIME parts of the body for processing, rather than just io.Copying them through. (the simple stub version of this function, shown in the block comment, does just that).
The following function copies an incoming mail "src" to an outgoing mail stream "dest". (The calling code sets these up as DotReader and DotWriter which takes care of most of the "dot" processing needed for RFC5321.
// Processing of email body via IO stream functions
package main
import (
"bufio"
"io"
"log"
"net/mail"
"strings"
)
/* If you just want to pass through the entire mail headers and body, you can just use
the following alernative:
func MailCopy(dst io.Writer, src io.Reader) (int64, error) {
return io.Copy(dst, src)
}
*/
// MailCopy transfers the mail body from downstream (client) to upstream (server)
// The writer will be closed by the parent function, no need to close it here.
func MailCopy(dst io.Writer, src io.Reader) (int64, error) {
var totalWritten int64
const smtpCRLF = "\r\n"
message, err := mail.ReadMessage(bufio.NewReader(src))
if err != nil {
return totalWritten, err
}
// Pass through headers. The m.Header map does not preserve order, but that should not matter.
for hdrType, hdrList := range message.Header {
for _, hdrVal := range hdrList {
hdrLine := hdrType + ": " + hdrVal + smtpCRLF
log.Print("\t", hdrLine)
bytesWritten, err := dst.Write([]byte(hdrLine))
totalWritten += int64(bytesWritten)
if err != nil {
return totalWritten, err
}
}
}
// Blank line denotes end of headers
bytesWritten, err := io.Copy(dst, strings.NewReader(smtpCRLF))
totalWritten += int64(bytesWritten)
if err != nil {
return totalWritten, err
}
// Copy the body
bytesWritten, err = io.Copy(dst, message.Body)
totalWritten += int64(bytesWritten)
if err != nil {
return totalWritten, err
}
return totalWritten, err
}
It does seem necessary to build this, because there is no net/mail.WriteMessage() method.
the header order is always randomised by Golang's map functionality. This seems harmless in my tests
A forced CRLF needs to be put in between the end of the headers and the body, as per RFCs. DotWriter takes care of the terminating dot.
The function shown above works, I was wondering if there is a better way to do this?

Change HTTP routing by server status

I'm building a REST server in Go and using mongoDB as my database (but this question is actually related to any other external resource).
I want my server to start and respond, even if the database is not up yet (not when the database is down after the server started - this is a different issue, much easier one).
So my dao package includes a connection go-routine that receives a boolean channel, and write 'true' to the channel when it successfully connected to the database. If the connection failed, the go-routine will keep trying every X seconds.
When I use this package with another software I wrote, that is a just a command-line program, I'm using select with timeout:
dbConnected := make(chan bool)
storage.Connect(dbConnected)
timeout := time.After(time.Minute)
select {
case <-dbConnected:
createReport()
case <-timeout:
log.Fatalln("Can't connect to the database")
}
I want to use the same packge in a server, but I don't want to fail the whole server. Instead, I want to start the server with handler that returns 503 SERVER BUSY, until the server is connected to the database, and then start serve requests normally. Is there a simple way to implement this logic in go standard library? Using solutions like gorilla is an option, but the server is simple with very few APIs, and gorilla is a bit overkill.
== edited: ==
I know I can use a middleware but I don't know how to do that without sharing data between the main method and the handlers. That why I'm using the channel in the first place.
I have something working, but it does based on common data. However, the data is a single boolean, so I guess it's not so dramatic. I would love to get comments for this solution:
In the dao package, I have this Connect method, that return a boolean channel. The private connect go routine, writes 'true' and exit when succeed:
func Connect() chan bool {
connected := make(chan bool)
go connect(mongoUrl, connected)
return connected
}
I also added the Ping() method to the dao package; it run forever and monitor the database status. It reports the status to a new channel and try to reconnect if needed:
func Ping() chan bool {
status := make(chan bool)
go func() {
for {
if err := session.Ping(); err != nil {
session.Close()
status <- false
<- Connect()
status <- true
}
time.Sleep(time.Second)
}
}()
return status
}
In the main package, I have this simple type:
type Connected struct {
isConnected bool
}
// this one is called as go-routine
func (c *Connected) check(dbConnected chan bool) {
// first connection, on server boot
c.isConnected = <- dbConnected
// monitor the database status
status := dao.Ping()
for {
c.isConnected = <- status
}
}
// the middleware
func (c *Connected) checkDbHandleFunc(next http.HandlerFunc) http.HandlerFunc {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if !c.isConnected {
w.Header().Add("Retry-After", "10")
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(503)
respBody := `{"error":"The server is busy; Try again soon"}`
w.Write([]byte(respBody))
} else {
next.ServeHTTP(w, r)
}
})
}
Middleware usage:
...
connected := Connected{
isConnected: false,
}
dbConnected := dao.Connect()
go connected.check(dbConnected)
mux := http.NewServeMux()
mux.HandleFunc("/", mainPage)
mux.HandleFunc("/some-db-required-path/", connected.checkDbHandleFunc(someDbRequiredHandler))
...
log.Fatal(http.ListenAndServe(addr, mux))
...
Does it make sense?

Golang HTTP REST mocking

I writing a client that connects to a server with REST endpoints. The client needs to make a chain of 11 different requests to complete an action (it's a rococo backup system).
I'm writing my client in Go and I also want to write my mocks/tests in Go. What I'm unclear about is how a test called func TestMain would call into the client's func main(), to test completion of the chain of 11 requests.
My client's binary would be run from the shell in the following way:
$ client_id=12345 region=apac3 backup
How would I call func main() from the tests, with environment variables set? Or is there another approach? (I'm comfortable writing tests, so that's not the issue)
I'm looking at the Advanced Example in jarcoal/httpmock (but I could use another library). At the end the example says // do stuff that adds and checks articles, is that where I would call main()?
I've pasted the Advanced Example below, for future reference.
func TestFetchArticles(t *testing.T) {
httpmock.Activate()
defer httpmock.DeactivateAndReset()
// our database of articles
articles := make([]map[string]interface{}, 0)
// mock to list out the articles
httpmock.RegisterResponder("GET", "https://api.mybiz.com/articles.json",
func(req *http.Request) (*http.Response, error) {
resp, err := httpmock.NewJsonResponse(200, articles)
if err != nil {
return httpmock.NewStringResponse(500, ""), nil
}
return resp, nil
},
)
// mock to add a new article
httpmock.RegisterResponder("POST", "https://api.mybiz.com/articles.json",
func(req *http.Request) (*http.Response, error) {
article := make(map[string]interface{})
if err := json.NewDecoder(req.Body).Decode(&article); err != nil {
return httpmock.NewStringResponse(400, ""), nil
}
articles = append(articles, article)
resp, err := httpmock.NewJsonResponse(200, article)
if err != nil {
return httpmock.NewStringResponse(500, ""), nil
}
return resp, nil
},
)
// do stuff that adds and checks articles
}
Writing this out helped me answer my own question.
main() would read in environment variables and then call a function like doBackup(client_id, region). My test would mock the endpoints and then call doBackup(client_id, region).

Go web service - POST tar.gz file as request body

I need to implement web service in go that processes tar.gz files and I wonder what is the correct way, what content type I need to define, etc.
plus, I found that a lot of things are handled automatically - on the client side I just post a gzip reader as request body and Accept-Encoding: gzip header is added automatically, and on the server side - I do not need to gunzip the request body, it is already extracted to tar. does that make sense?
Can I rely that it would be like this with any client?
Server:
func main() {
router := mux.NewRouter().StrictSlash(true)
router.Handle("/results", dataupload.NewUploadHandler()).Methods("POST")
log.Fatal(http.ListenAndServe(*address, router))
}
Uploader:
package dataupload
import (
"errors"
log "github.com/Sirupsen/logrus"
"io"
"net/http"
)
// UploadHandler responds to /results http request, which is the result-service rest API for uploading results
type UploadHandler struct {
uploader Uploader
}
// NewUploadHandler creates UploadHandler instance
func NewUploadHandler() *UploadHandler {
return &UploadHandler{
uploader: TarUploader{},
}
}
func (uh UploadHandler) ServeHTTP(writer http.ResponseWriter, request *http.Request) {
retStatus := http.StatusOK
body, err := getBody(request)
if err != nil {
retStatus = http.StatusBadRequest
log.Error("Error fetching request body. ", err)
} else {
_, err := uh.uploader.Upload(body)
}
writer.WriteHeader(retStatus)
}
func getBody(request *http.Request) (io.ReadCloser, error) {
requestBody := request.Body
if requestBody == nil {
return nil, errors.New("Empty request body")
}
var err error
// this part is commented out since somehow the body is already gunzipped - no need to extract it.
/*if strings.Contains(request.Header.Get("Accept-Encoding"), "gzip") {
requestBody, err = gzip.NewReader(requestBody)
}*/
return requestBody, err
}
Client
func main() {
f, err := os.Open("test.tar.gz")
if err != nil {
log.Fatalf("error openning file %s", err)
}
defer f.Close()
client := new(http.Client)
reader, err := gzip.NewReader(f)
if err != nil {
log.Fatalf("error gzip file %s", err)
}
request, err := http.NewRequest("POST", "http://localhost:8080/results", reader)
_, err = client.Do(request)
if err != nil {
log.Fatalf("error uploading file %s", err)
}
}
The code you've written for the client is just sending the tarfile directly because of this code:
reader, err := gzip.NewReader(f)
...
request, err := http.NewRequest("POST", "http://localhost:8080/results", reader)
If you sent the .tar.gz file content directly, then you would need to gunzip it on the server. E.g.:
request, err := http.NewRequest(..., f)
I think that's closer to the behavior you should expect third-party clients to exhibit.
Claerly not, but maybe...
Golang provides a very good support for the http client (and server). This is one of the first language to support http2 and the design of the API clearly shows their concern on having a fast http.
This is why they add Accept-Econding: gzip automatically. That will dramatically reduce the size of the server response and then optimize the transfer.
But the gzip remains an option in http 1 and not all of the client will push this header to your server.
Note that the Content-Type describes the type of data you are sending (here a tar.gz but could be application/json, test/javascript, ...), when the Accept-Encoding describes the way the data has been encoded for the transport
Go will take care of transparently handling the Accept-Encoding for you because it is responsible of the transport of the data. Then it will be up to you to handle the Content-Type because only you know how to give a sense to the content you received