How to read data (xml) sent by server if it doesn't send new line - sockets

Let's say we try to communicate with a server (XMPP) which sends back XML data. We can use
conn, err := net.Dial("tcp", s.Addr+":5222")
//...
r := bufio.NewReader(conn)
//...
s, err := s.R.ReadString(10) // to read a string
But there is one problem that the server doesn't send the \10 (newline) symbol. I also tried 12 but without any luck. Same goes for readLine function as it also relies on \10. So how do I read the data sent by server?
I tried using '>' as a delimiter and succeeded to receive only parts of the messages (predictable). I had an idea to loop while error is nil and use delimiter of '>' but it also didn't work.
My research shown that the last symbol of the message is really '>' (62) and there is not any anything else at the end.

Use an xml.Decoder to read stanzas from an XMPP stream.
conn, err := net.Dial("tcp", s.Addr+":5222")
if err != nil {
// handle error
}
dec := xml.NewDecoder(conn)
Use the decoder Token method to read the root document element and to skip over character data between stanzas:
func readStartElement(dec *xml.Decoder) (xml.StartElement, error) {
for {
t, err := dec.Token()
if err != nil {
return xml.StartElement{}, err
}
switch t := t.(type) {
case xml.StartElement:
return t, nil
}
}
}
Use the decoder DecodeElement method to read a stanza:
func readStanza(dec *xml.Decoder) (interface{}, error) {
se, err := readStartElement(dec)
if err != nil {
return nil, err
}
var v interface{}
switch se.Name.Space + " " + se.Name.Local {
case "jabber:client message":
v = &jabberMessage{} // jabberMessage is struct type defined by app for messages
// Add other stanza types here.
default:
v = &struct{}{}
}
if err := dec.DecodeElement(v, &se); err != nil {
return nil, err
}
return v, nil
}
Type switch on the return value from readStanza to handle the different types of received stanzas.
A client reads stanzas synchronously. Here's rough outline (ignoring authentication, etc).
conn, err := net.Dial("tcp", s.Addr+":5222")
if err != nil {
// handle error
}
dec := xml.NewDecoder(conn)
// read and discard root element
_, err := readStartElement(dec)
if err != nil {
// handle error
}
// read stanzas
for {
v, err := readStanza(dec)
if err != nil {
// handle error
// must break out of loop on error
}
switch v := v.(type) {
case *jabberMessage:
// handle message
case *someOtherStanzaType:
// handle other stanza types
// ... and so on
}
}

Related

ObjectIdFromHex invalid byte error on identical strings

I'm trying to implement a FindOne method in my Golang REST API. The trouble comes where i have to search by ID. I have to convert the ID into something readable by the database, so i use primitive.ObjectIDFromHex(id)
The problem is that this method throws an error :
2021/06/19 06:56:15 encoding/hex: invalid byte: U+000A
ONLY when i call it with the id that comes from my URL GET params.
I did two versions : one with hard-coded ID, and one with GET ID. See code below.
func Admin(id string) (bson.M, error) {
coll, err := db.ConnectToCollection("admin")
if err != nil {
log.Fatal(err)
}
var admin bson.M
HardCoded := "60cb275c074ab46a1aeda45e"
fmt.Println(HardCoded) // Just to be sure : the two strings seem identical
fmt.Println(id)
objetId, err := primitive.ObjectIDFromHex(id) // throws encoding error
// objetId, err := primitive.ObjectIDFromHex(HardCoded) // Doesnt throw encoding err
if err != nil {
log.Fatal(err)
}
var ctx = context.TODO()
if err := coll.FindOne(ctx, bson.M{"_id": objetId}).Decode(&admin); err != nil {
log.Fatal(err)
}
return admin, nil
}
Of course, you'll want to know where the param id comes from.
Here you go :
func GetAdmin(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
params := mux.Vars(r)
admin, err := Admin(params["id"]) // Calling the Admin function above
if err != nil {
fmt.Println(err)
http.Error(w, err.Error(), http.StatusUnauthorized)
} else {
JSON, err := json.Marshal(admin)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
w.Write(JSON)
}
}
Trim the line feed from the end of id:
id = strings.TrimSpace(id)
Use the %q format verb when debugging issues like this. The line feed is clearly visible in this output:
fmt.Printf("%q\n", HardCoded) // prints "60cb275c074ab46a1aeda45e"
fmt.Printf("%q\n", id) // prints "60cb275c074ab46a1aeda45e\n"

How to set ToS field in IP header for a TCP connection using Golang

I am trying to create a TCP server and client using Golang where I am able to set the Type of Service field in the IP header in order to prioritise different traffic flows.
The client and servers are able to communicate but I can not figure out how to set the ToS field.
I have tried using the ipv4 Golang package with the method described here: https://godoc.org/golang.org/x/net/ipv4#NewConn
A simplified server example:
func main () {
ln, err := net.Listen("tcp4", "192.168.0.20:1024")
if err != nil {
// error handling
}
defer ln.Close()
for {
c, err := ln.Accept()
if err != nil {
// error handling
}
go func(c net.Conn) {
defer c.Close()
if err := ipv4.NewConn(c).SetTOS(0x28); err != nil {
fmt.Println("Error: ", err.Error())
}
}(c)
}
And the corresponding client (also simplified)
func main () {
conn, err := net.Dial("tcp4", "192.168.0.20:1024")
if err != nil {
fmt.Println(err)
}
for {
writer := bufio.NewWriter(conn)
// Create "packet"
Data := make([]byte, 1200)
endLine := "\r\n"
//Set packetLength
length := strconv.FormatInt(int64(1200), 10)
copy(Data[0:], length)
//Set ID
idString := strconv.FormatInt(int64(1), 10)
if strings.Contains(idString, "\r") || strings.Contains(idString, "\n") || strings.Contains(idString, "\r\n") {
fmt.Println("This is gonna result in an error in the id string.")
}
idbuf := []byte(idString)
copy(Data[15:], idbuf)
//Set timestamp
timestamp0 := time.Now().UnixNano()
timestampString := strconv.FormatInt(timestamp0, 10)
if strings.Contains(timestampString, "\r") || strings.Contains(timestampString, "\n") || strings.Contains(timestampString, "\r\n") {
fmt.Println("This is gonna result in an error in the timestamp string.")
}
buf := []byte(timestampString)
copy(Data[50:], buf)
copy(Data[int(1200)-2:], endLine)
if len(Data) != int(1200) {
fmt.Println("This is also gonna be an error. Length is: ", len(Data))
}
//Send the data and flush the writer
writer.Write(Data)
writer.Flush()
}
//time.Sleep(1*time.Nanosecond)
}
I have also tried creating my own dialer with a control function that passes a syscall in order to set the socket like this:
dialer := &net.Dialer{
Timeout: 5 * time.Second,
Deadline: time.Time{},
LocalAddr: tcpAddr,
DualStack: false,
FallbackDelay: 0,
KeepAlive: 0,
Resolver: nil,
Control: highPrio,
}
func highPrio(network, address string, c syscall.RawConn) error {
return c.Control(func(fd uintptr) {
// set the socket options
err := syscall.SetsockoptInt(syscall.Handle(fd), syscall.IPPROTO_IP, syscall.IP_TOS, 128)
if err != nil {
log.Println("setsocketopt: ", err)
}
})
I am verifying that it does not work by inspecting the traffic with Wireshark and am using Windows 10 Pro as my OS.
I am try you ToS set method at Dial() with golang 1.15.5 and its worked:
dialer := net.Dialer{
Timeout: this.TcpWaitConnectTimeout,
}
dialer.Control = func(network, address string, c syscall.RawConn) error {
var err error
c.Control(func(fd uintptr) {
err = syscall.SetsockoptInt(int(fd), syscall.IPPROTO_IP, syscall.IP_TOS, 0x80)
})
return err
}
c, err := dialer.Dial("tcp", this.serverAddr)
tcpdump show me right ToS

Write on a closed net.Conn but returned nil error

Talk is cheap, so here we go the simple code:
package main
import (
"fmt"
"time"
"net"
)
func main() {
addr := "127.0.0.1:8999"
// Server
go func() {
tcpaddr, err := net.ResolveTCPAddr("tcp4", addr)
if err != nil {
panic(err)
}
listen, err := net.ListenTCP("tcp", tcpaddr)
if err != nil {
panic(err)
}
for {
if conn, err := listen.Accept(); err != nil {
panic(err)
} else if conn != nil {
go func(conn net.Conn) {
buffer := make([]byte, 1024)
n, err := conn.Read(buffer)
if err != nil {
fmt.Println(err)
} else {
fmt.Println(">", string(buffer[0 : n]))
}
conn.Close()
}(conn)
}
}
}()
time.Sleep(time.Second)
// Client
if conn, err := net.Dial("tcp", addr); err == nil {
for i := 0; i < 2; i++ {
_, err := conn.Write([]byte("hello"))
if err != nil {
fmt.Println(err)
conn.Close()
break
} else {
fmt.Println("ok")
}
// sleep 10 seconds and re-send
time.Sleep(10*time.Second)
}
} else {
panic(err)
}
}
Ouput:
> hello
ok
ok
The Client writes to the Server twice. After the first read, the Server closes the connection immediately, but the Client sleeps 10 seconds and then re-writes to the Server with the same already closed connection object(conn).
Why can the second write succeed (returned error is nil)?
Can anyone help?
PS:
In order to check if the buffering feature of the system affects the result of the second write, I edited the Client like this, but it still succeeds:
// Client
if conn, err := net.Dial("tcp", addr); err == nil {
_, err := conn.Write([]byte("hello"))
if err != nil {
fmt.Println(err)
conn.Close()
return
} else {
fmt.Println("ok")
}
// sleep 10 seconds and re-send
time.Sleep(10*time.Second)
b := make([]byte, 400000)
for i := range b {
b[i] = 'x'
}
n, err := conn.Write(b)
if err != nil {
fmt.Println(err)
conn.Close()
return
} else {
fmt.Println("ok", n)
}
// sleep 10 seconds and re-send
time.Sleep(10*time.Second)
} else {
panic(err)
}
And here is the screenshot:
attachment
There are several problems with your approach.
Sort-of a preface
The first one is that you do not wait for the server goroutine
to complete.
In Go, once main() exits for whatever reason,
all the other goroutines still running, if any, are simply
teared down forcibly.
You're trying to "synchronize" things using timers,
but this only works in toy situations, and even then it
does so only from time to time.
Hence let's fix your code first:
package main
import (
"fmt"
"log"
"net"
"time"
)
func main() {
addr := "127.0.0.1:8999"
tcpaddr, err := net.ResolveTCPAddr("tcp4", addr)
if err != nil {
log.Fatal(err)
}
listener, err := net.ListenTCP("tcp", tcpaddr)
if err != nil {
log.Fatal(err)
}
// Server
done := make(chan error)
go func(listener net.Listener, done chan<- error) {
for {
conn, err := listener.Accept()
if err != nil {
done <- err
return
}
go func(conn net.Conn) {
var buffer [1024]byte
n, err := conn.Read(buffer[:])
if err != nil {
log.Println(err)
} else {
log.Println(">", string(buffer[0:n]))
}
if err := conn.Close(); err != nil {
log.Println("error closing server conn:", err)
}
}(conn)
}
}(listener, done)
// Client
conn, err := net.Dial("tcp", addr)
if err != nil {
log.Fatal(err)
}
for i := 0; i < 2; i++ {
_, err := conn.Write([]byte("hello"))
if err != nil {
log.Println(err)
err = conn.Close()
if err != nil {
log.Println("error closing client conn:", err)
}
break
}
fmt.Println("ok")
time.Sleep(2 * time.Second)
}
// Shut the server down and wait for it to report back
err = listener.Close()
if err != nil {
log.Fatal("error closing listener:", err)
}
err = <-done
if err != nil {
log.Println("server returned:", err)
}
}
I've spilled a couple of minor fixes
like using log.Fatal (which is
log.Print + os.Exit(1)) instead of panicking,
removed useless else clauses to adhere to the coding standard of keeping the main
flow where it belongs, and lowered the client's timeout.
I have also added checking for possible errors Close on sockets may return.
The interesting part is that we now properly shut the server down by closing the listener and then waiting for the server goroutine to report back (unfortunately Go does not return an error of a custom type from net.Listener.Accept in this case so we can't really check that Accept exited because we've closed the listener).
Anyway, our goroutines are now properly synchronized, and there is
no undefined behaviour, so we can reason about how the code works.
Remaining problems
Some problems still remain.
The more glaring is you making wrong assumption that TCP preserves
message boundaries—that is, if you write "hello" to the client
end of the socket, the server reads back "hello".
This is not true: TCP considers both ends of the connection
as producing and consuming opaque streams of bytes.
This means, when the client writes "hello", the client's
TCP stack is free to deliver "he" and postpone sending "llo",
and the server's stack is free to yield "hell" to the read
call on the socket and only return "o" (and possibly some other
data) in a later read.
So, to make the code "real" you'd need to somehow introduce these
message boundaries into the protocol above TCP.
In this particular case the simplest approach would be either
using "messages" consisting of a fixed-length and agreed-upon
endianness prefix indicating the length of the following
data and then the string data itself.
The server would then use a sequence like
var msg [4100]byte
_, err := io.ReadFull(sock, msg[:4])
if err != nil { ... }
mlen := int(binary.BigEndian.Uint32(msg[:4]))
if mlen < 0 {
// handle error
}
if mlen == 0 {
// empty message; goto 1
}
_, err = io.ReadFull(sock, msg[5:5+mlen])
if err != nil { ... }
s := string(msg[5:5+mlen])
Another approach is to agree on that the messages do not contain
newlines and terminate each message with a newline
(ASCII LF, \n, 0x0a).
The server side would then use something like
a usual bufio.Scanner loop to get
full lines from the socket.
The remaining problem with your approach is to not dealing with
what Read on a socket returns: note that io.Reader.Read
(that's what sockets implement, among other things) is allowed
to return an error while having had read some data from the
underlying stream. In your toy example this might rightfully
be unimportant, but suppose that you're writing a wget-like
tool which is able to resume downloading of a file: even if
reading from the server returned some data and an error, you
have to deal with that returned chunk first and only then
handle the error.
Back to the problem at hand
The problem presented in the question, I beleive, happens simply because in your setup you hit some TCP buffering problem due to the tiny length of your messages.
On my box which runs Linux 4.9/amd64 two things reliably "fix"
the problem:
Sending messages of 4000 bytes in length: the second call
to Write "sees" the problem immediately.
Doing more Write calls.
For the former, try something like
msg := make([]byte, 4000)
for i := range msg {
msg[i] = 'x'
}
for {
_, err := conn.Write(msg)
...
and for the latter—something like
for {
_, err := conn.Write([]byte("hello"))
...
fmt.Println("ok")
time.Sleep(time.Second / 2)
}
(it's sensible to lower the pause between sending stuff in
both cases).
It's interesting to note that the former example hits the
write: connection reset by peer (ECONNRESET in POSIX)
error while the second one hits write: broken pipe
(EPIPE in POSIX).
This is because when we're sending in chunks worth 4k bytes,
some of the packets generated for the stream manage to become
"in flight" before the server's side of the connection manages
to propagate the information on its closure to the client,
and those packets hit an already closed socket and get rejected
with the RST TCP flag set.
In the second example an attempt to send another chunk of data
sees that the client side already knows that the connection
has been teared down and fails the sending without "touching
the wire".
TL;DR, the bottom line
Welcome to the wonderful world of networking. ;-)
I'd recommend buying a copy of "TCP/IP Illustrated",
read it and experiment.
TCP (and IP and other protocols above IP)
sometimes works not like people expect them to by applying
their "common sense".

Very Sporadic Go HTTP Error: multiple response.WriteHeader calls

I wrote Kanali which is an open source Kubernetes Ingress/API management tool and for about 1/200k requests I receive the following fatal error:
2017/08/16 12:40:57 http: multiple response.WriteHeader calls
{"level":"error","method":"GET","msg":"unknown error","time":"2017-08-16T12:40:57Z","uri":"/ommitted/path"}
{"level":"fatal","msg":"write tcp4 192.168.2.160:8443-\u003e192.168.4.0:54554: write: broken pipe","time":"2017-08-16T12:40:57Z"}
I'm having a really hard time reproducing it but here is the relevant code. Kanali is a large project but the td;lr is that after this first code snippet is executed, the second code snipped is executed which handles errors.
func (step WriteResponseStep) Do(ctx context.Context, m *metrics.Metrics, c *controller.Controller, w http.ResponseWriter, r *http.Request, resp *http.Response, trace opentracing.Span) error {
for k, v := range resp.Header {
for _, value := range v {
w.Header().Set(k, value)
}
}
closer, str, err := utils.DupReaderAndString(resp.Body)
if err != nil {
logrus.Errorf("error copying response body, response may not be as expected: %s", err.Error())
}
trace.SetTag("http.status_code", resp.StatusCode)
trace.SetTag("http.response_body", str)
w.WriteHeader(resp.StatusCode)
if _, err := io.Copy(w, closer); err != nil {
return err
}
return nil
}
later in the code...
if err != nil {
w.Header().Set("Content-Type", "application/json")
switch e := err.(type) {
case utils.Error:
logrus.WithFields(logrus.Fields{
"method": r.Method,
"uri": r.RequestURI,
}).Error(e.Error())
errStatus, err := json.Marshal(utils.JSONErr{Code: e.Status(), Msg: e.Error()})
if err != nil {
logrus.Warnf("could not marsah request headers into JSON - tracing data maybe not be as expected")
}
w.WriteHeader(e.Status())
if err := json.NewEncoder(w).Encode(utils.JSONErr{Code: e.Status(), Msg: e.Error()}); err != nil {
logrus.Fatal(err.Error())
}
default:
logrus.WithFields(logrus.Fields{
"method": r.Method,
"uri": r.RequestURI,
}).Error("unknown error")
errStatus, err := json.Marshal(utils.JSONErr{Code: http.StatusInternalServerError, Msg: "unknown error"})
if err != nil {
logrus.Warnf("could not marsah request headers into JSON - tracing data maybe not be as expected")
}
w.WriteHeader(http.StatusInternalServerError)
if err := json.NewEncoder(w).Encode(utils.JSONErr{Code: http.StatusInternalServerError, Msg: "unknown error"}); err != nil {
logrus.Fatal(err.Error())
}
}
}
The error message is telling you that WriteHeader is called more than once (either directly or indirectly by a call to Write). The header can only be written to the network once. Both snippets have a call to WriteHeader.

How to read from request then use that result to do POST request then process its results

I'm trying to read from request then use that result to do POST request to another endpoint then process its results then return its results in JSON.
I have below code so far:
// POST
func (u *UserResource) authenticate(request *restful.Request, response *restful.Response) {
Api := Api{url: "http://api.com/api"}
usr := new(User)
err := request.ReadEntity(&usr)
if err != nil {
response.WriteErrorString(http.StatusInternalServerError, err.Error())
return
}
api_resp, err := http.Post(Api.url, "text/plain", bytes.NewBuffer(usr))
if err != nil {
response.WriteErrorString(http.StatusInternalServerError, err.Error())
return
}
defer api_resp.Body.Close()
body, err := ioutil.ReadAll(api_resp.Body)
response.WriteHeader(http.StatusCreated)
err = xml.Unmarshal(body, usr)
if err != nil {
fmt.Printf("error: %v", err)
return
}
// result, err := json.Marshal(usr)
// response.Write(result)
response.WriteEntity(&usr)
fmt.Printf("Name: %q\n", usr.UserName)
}
I'm using Go Restful package for Writes and Reads.
I'm getting this error when I compile the file:
src\login.go:59: cannot use usr (type *User) as type []byte in argument to bytes.NewBuffer
What would be the best way to solve this issue so I can do a POST with payload correctly?
You need to marshal your data structure to slice of bytes. Something like this:
usrXmlBytes, err := xml.Marshal(usr)
if err != nil {
response.WriteErrorString(http.StatusInternalServerError, err.Error())
return
}
api_resp, err := http.Post(Api.url, "text/plain", bytes.NewReader(usrXmlBytes))
http.Post takes an io.Reader as the third argument. You could implement io.Reader on your User type or more simply serialize your data and use the bytes pkg to to implement io.Reader
b, err := json.Marshal(usr)
if err != nil {
response.WriteErrorString(http.StatusInternalServerError, err.Error())
return
}
api_resp, err := http.Post(Api.url, "text/plain", bytes.NewReader(b))