I wrote Kanali which is an open source Kubernetes Ingress/API management tool and for about 1/200k requests I receive the following fatal error:
2017/08/16 12:40:57 http: multiple response.WriteHeader calls
{"level":"error","method":"GET","msg":"unknown error","time":"2017-08-16T12:40:57Z","uri":"/ommitted/path"}
{"level":"fatal","msg":"write tcp4 192.168.2.160:8443-\u003e192.168.4.0:54554: write: broken pipe","time":"2017-08-16T12:40:57Z"}
I'm having a really hard time reproducing it but here is the relevant code. Kanali is a large project but the td;lr is that after this first code snippet is executed, the second code snipped is executed which handles errors.
func (step WriteResponseStep) Do(ctx context.Context, m *metrics.Metrics, c *controller.Controller, w http.ResponseWriter, r *http.Request, resp *http.Response, trace opentracing.Span) error {
for k, v := range resp.Header {
for _, value := range v {
w.Header().Set(k, value)
}
}
closer, str, err := utils.DupReaderAndString(resp.Body)
if err != nil {
logrus.Errorf("error copying response body, response may not be as expected: %s", err.Error())
}
trace.SetTag("http.status_code", resp.StatusCode)
trace.SetTag("http.response_body", str)
w.WriteHeader(resp.StatusCode)
if _, err := io.Copy(w, closer); err != nil {
return err
}
return nil
}
later in the code...
if err != nil {
w.Header().Set("Content-Type", "application/json")
switch e := err.(type) {
case utils.Error:
logrus.WithFields(logrus.Fields{
"method": r.Method,
"uri": r.RequestURI,
}).Error(e.Error())
errStatus, err := json.Marshal(utils.JSONErr{Code: e.Status(), Msg: e.Error()})
if err != nil {
logrus.Warnf("could not marsah request headers into JSON - tracing data maybe not be as expected")
}
w.WriteHeader(e.Status())
if err := json.NewEncoder(w).Encode(utils.JSONErr{Code: e.Status(), Msg: e.Error()}); err != nil {
logrus.Fatal(err.Error())
}
default:
logrus.WithFields(logrus.Fields{
"method": r.Method,
"uri": r.RequestURI,
}).Error("unknown error")
errStatus, err := json.Marshal(utils.JSONErr{Code: http.StatusInternalServerError, Msg: "unknown error"})
if err != nil {
logrus.Warnf("could not marsah request headers into JSON - tracing data maybe not be as expected")
}
w.WriteHeader(http.StatusInternalServerError)
if err := json.NewEncoder(w).Encode(utils.JSONErr{Code: http.StatusInternalServerError, Msg: "unknown error"}); err != nil {
logrus.Fatal(err.Error())
}
}
}
The error message is telling you that WriteHeader is called more than once (either directly or indirectly by a call to Write). The header can only be written to the network once. Both snippets have a call to WriteHeader.
Related
I'm trying to implement a FindOne method in my Golang REST API. The trouble comes where i have to search by ID. I have to convert the ID into something readable by the database, so i use primitive.ObjectIDFromHex(id)
The problem is that this method throws an error :
2021/06/19 06:56:15 encoding/hex: invalid byte: U+000A
ONLY when i call it with the id that comes from my URL GET params.
I did two versions : one with hard-coded ID, and one with GET ID. See code below.
func Admin(id string) (bson.M, error) {
coll, err := db.ConnectToCollection("admin")
if err != nil {
log.Fatal(err)
}
var admin bson.M
HardCoded := "60cb275c074ab46a1aeda45e"
fmt.Println(HardCoded) // Just to be sure : the two strings seem identical
fmt.Println(id)
objetId, err := primitive.ObjectIDFromHex(id) // throws encoding error
// objetId, err := primitive.ObjectIDFromHex(HardCoded) // Doesnt throw encoding err
if err != nil {
log.Fatal(err)
}
var ctx = context.TODO()
if err := coll.FindOne(ctx, bson.M{"_id": objetId}).Decode(&admin); err != nil {
log.Fatal(err)
}
return admin, nil
}
Of course, you'll want to know where the param id comes from.
Here you go :
func GetAdmin(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
params := mux.Vars(r)
admin, err := Admin(params["id"]) // Calling the Admin function above
if err != nil {
fmt.Println(err)
http.Error(w, err.Error(), http.StatusUnauthorized)
} else {
JSON, err := json.Marshal(admin)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
w.Write(JSON)
}
}
Trim the line feed from the end of id:
id = strings.TrimSpace(id)
Use the %q format verb when debugging issues like this. The line feed is clearly visible in this output:
fmt.Printf("%q\n", HardCoded) // prints "60cb275c074ab46a1aeda45e"
fmt.Printf("%q\n", id) // prints "60cb275c074ab46a1aeda45e\n"
I am trying to talk to server via Golang
and I am unable to get it to work:
However, the following CURL Command works:
"curl http://localhost:8888/auth"
I know the form is:
<subject> <payload length> "\r\n" <payload>
but everything I try fails.
The publish code is as follows:
// Publish takes a subject as an immutable string and payload inbytes,
// then sends the message to the server.
func (c *Client) Publish(subject string, payload []byte) error {
c.Lock()
pub := fmt.Sprintf("%s %d\r\n", subject, len(payload))
_, err := c.w.WriteString(pub)
if err == nil {
_, err = c.w.Write(payload)
}
if err == nil {
_, err = c.w.WriteString("\r\n")
}
if err == nil {
err = c.w.Flush()
}
c.Unlock()
if err != nil {
return err
}
return nil
}
I would appreciate any help that can be offered.
Let's say we try to communicate with a server (XMPP) which sends back XML data. We can use
conn, err := net.Dial("tcp", s.Addr+":5222")
//...
r := bufio.NewReader(conn)
//...
s, err := s.R.ReadString(10) // to read a string
But there is one problem that the server doesn't send the \10 (newline) symbol. I also tried 12 but without any luck. Same goes for readLine function as it also relies on \10. So how do I read the data sent by server?
I tried using '>' as a delimiter and succeeded to receive only parts of the messages (predictable). I had an idea to loop while error is nil and use delimiter of '>' but it also didn't work.
My research shown that the last symbol of the message is really '>' (62) and there is not any anything else at the end.
Use an xml.Decoder to read stanzas from an XMPP stream.
conn, err := net.Dial("tcp", s.Addr+":5222")
if err != nil {
// handle error
}
dec := xml.NewDecoder(conn)
Use the decoder Token method to read the root document element and to skip over character data between stanzas:
func readStartElement(dec *xml.Decoder) (xml.StartElement, error) {
for {
t, err := dec.Token()
if err != nil {
return xml.StartElement{}, err
}
switch t := t.(type) {
case xml.StartElement:
return t, nil
}
}
}
Use the decoder DecodeElement method to read a stanza:
func readStanza(dec *xml.Decoder) (interface{}, error) {
se, err := readStartElement(dec)
if err != nil {
return nil, err
}
var v interface{}
switch se.Name.Space + " " + se.Name.Local {
case "jabber:client message":
v = &jabberMessage{} // jabberMessage is struct type defined by app for messages
// Add other stanza types here.
default:
v = &struct{}{}
}
if err := dec.DecodeElement(v, &se); err != nil {
return nil, err
}
return v, nil
}
Type switch on the return value from readStanza to handle the different types of received stanzas.
A client reads stanzas synchronously. Here's rough outline (ignoring authentication, etc).
conn, err := net.Dial("tcp", s.Addr+":5222")
if err != nil {
// handle error
}
dec := xml.NewDecoder(conn)
// read and discard root element
_, err := readStartElement(dec)
if err != nil {
// handle error
}
// read stanzas
for {
v, err := readStanza(dec)
if err != nil {
// handle error
// must break out of loop on error
}
switch v := v.(type) {
case *jabberMessage:
// handle message
case *someOtherStanzaType:
// handle other stanza types
// ... and so on
}
}
I'm trying to read from request then use that result to do POST request to another endpoint then process its results then return its results in JSON.
I have below code so far:
// POST
func (u *UserResource) authenticate(request *restful.Request, response *restful.Response) {
Api := Api{url: "http://api.com/api"}
usr := new(User)
err := request.ReadEntity(&usr)
if err != nil {
response.WriteErrorString(http.StatusInternalServerError, err.Error())
return
}
api_resp, err := http.Post(Api.url, "text/plain", bytes.NewBuffer(usr))
if err != nil {
response.WriteErrorString(http.StatusInternalServerError, err.Error())
return
}
defer api_resp.Body.Close()
body, err := ioutil.ReadAll(api_resp.Body)
response.WriteHeader(http.StatusCreated)
err = xml.Unmarshal(body, usr)
if err != nil {
fmt.Printf("error: %v", err)
return
}
// result, err := json.Marshal(usr)
// response.Write(result)
response.WriteEntity(&usr)
fmt.Printf("Name: %q\n", usr.UserName)
}
I'm using Go Restful package for Writes and Reads.
I'm getting this error when I compile the file:
src\login.go:59: cannot use usr (type *User) as type []byte in argument to bytes.NewBuffer
What would be the best way to solve this issue so I can do a POST with payload correctly?
You need to marshal your data structure to slice of bytes. Something like this:
usrXmlBytes, err := xml.Marshal(usr)
if err != nil {
response.WriteErrorString(http.StatusInternalServerError, err.Error())
return
}
api_resp, err := http.Post(Api.url, "text/plain", bytes.NewReader(usrXmlBytes))
http.Post takes an io.Reader as the third argument. You could implement io.Reader on your User type or more simply serialize your data and use the bytes pkg to to implement io.Reader
b, err := json.Marshal(usr)
if err != nil {
response.WriteErrorString(http.StatusInternalServerError, err.Error())
return
}
api_resp, err := http.Post(Api.url, "text/plain", bytes.NewReader(b))
My GO version is 1.1.1
the sever recieved messages after connection close, but NoDelay was setted.
Is there something wrong
addr, _ := net.ResolveTCPAddr("tcp", "localhost:5432")
conn, err := net.DialTCP("tcp", nil, addr)
defer conn.Close()
if err != nil {
fmt.Println("connect fail")
return
}
err = conn.SetNoDelay(true)
if err != nil {
fmt.Println(err.Error())
}
for {
var message string
_, err := fmt.Scanln(&message)
if err != nil && err.Error() != "unexpected newline" {
fmt.Println("input finished", err)
break
}
if message == "" {
fmt.Println("no input, end")
break
}
// message = fmt.Sprintf("%s\n",message)
//fmt.Fprintf(conn, message) // send immediately but following message won't send any more
conn.Write([]byte(message)) // won't send until connection close
}
There doesn't seem to be anything vitally wrong with your code so I'm guessing the error is on the server end.
If you create a local TCP server on port 5432 you can test this.
Try running the below server code and then test your client code against it. It just echos all received data to stdout.
package main
import (
"io"
"log"
"net"
"os"
)
func main() {
l, err := net.Listen("tcp", "localhost:5432")
if err != nil {
log.Fatal(err)
}
defer l.Close()
for {
conn, err := l.Accept()
if err != nil {
log.Fatal(err)
}
go func(c net.Conn) {
defer c.Close()
io.Copy(os.Stdout, c)
}(conn)
}
}
You should see each line sent to the client printed (without the newline) as soon as you hit enter.
the problem is on the server end.
func handleConnection(conn net.Conn) {
// I didn't put it in for loop
message, err := bufio.NewReader(conn).ReadString('\n')
}