Reading Data from Socket Golang - sockets

I'm trying to read data from a telnet session in golang. I wrote the following functions in an attempt to accomplish this.
Initially I was having an issue where I was reading from a socket with no data so it would lock and never return. BufferSocketData is my attempt to work around this issue as I can't know if there is data to read. The idea is it will wait 1 second before determining there is not data in the socket and return an empty string.
GetData seems to work the first time there is new data in the buffer, but beyond that it gets no new data. I'm sure this has something to do with my use of goroutines and channels, I'm new to go and I'm sure I'm not using them correctly.
Any ideas as to why my subsequent reads return no data?
/*
ReadDataFromSocket - Attempts to read any data in the socket.
*/
func ReadDataFromSocket(sock io.Reader, c chan string) {
var recvData = make([]byte, 1024)
var numBytes, _ = sock.Read(recvData)
c <- string(recvData[:numBytes])
}
/*
BufferSocketData - Read information from the socket and store it in the buffer.
*/
func (tn *TelnetLib) BufferSocketData(inp chan string, out chan string) {
var data string
var timeout int64 = 1000 // 1 second timeout.
var start = utils.GetTimestamp()
for utils.GetTimestamp()-start < timeout {
select {
case data = <-inp:
default:
}
if data != "" {
break
}
}
out <- data
}
/*
GetData - Start goroutines to get and buffer data.
*/
func (tn *TelnetLib) GetData() {
var sockCh = make(chan string)
var buffCh = make(chan string)
go ReadDataFromSocket(tn.Conn, sockCh)
go tn.BufferSocketData(sockCh, buffCh)
var data = <-buffCh
if data != "" {
tn.Buffer += data
}
}
Please let me know if you need any additional information.

Use SetReadDeadline to read data with a time limit:
func (tn *TelnetLib) GetData() {
tn.Conn.SetReadDeadline(time.Second)
recvData := make([]byte, 1024)
n, err := tn.Conn.Read(recvData)
if n > 0 {
// do something with recvData[:n]
}
if e, ok := err.(interface{ Timeout() bool }); ok && e.Timeout() {
// handle timeout
} else if err != nil {
// handle error
}
}
Note that a single call Read may not read all data sent by the peer. You may want to accumulate data by calling Read in a loop or call io.ReadFull.

Related

Rewriting looping blocking code to SwiftNIO style non-blocking code

I'm working on a driver that will read data from the network. It doesn't know how much is in a response, other than that when it tries to read and gets 0 bytes back, it is done. So my blocking Swift code looks naively like this:
func readAllBlocking() -> [Byte] {
var buffer: [Byte] = []
var fullBuffer: [Byte] = []
repeat {
buffer = read() // synchronous, blocking
fullBuffer.append(buffer)
} while buffer.count > 0
return fullBuffer
}
How can I rewrite this as a promise that will keep on running until the entire result is read? After trying to wrap my brain around it, I'm still stuck here:
func readAllNonBlocking() -> EventLoopFuture<[Byte]> {
///...?
}
I should add that I can rewrite read() to instead of returning a [Byte] return an EventLoopFuture<[Byte]>
Generally, loops in synchronous programming are turned into recursion to get the same effect with asynchronous programming that uses futures (and also in functional programming).
So your function could look like this:
func readAllNonBlocking(on eventLoop: EventLoop) -> EventLoopFuture<[Byte]> {
// The accumulated chunks
var accumulatedChunks: [Byte] = []
// The promise that will hold the overall result
let promise = eventLoop.makePromise(of: [Byte].self)
// We turn the loop into recursion:
func loop() {
// First, we call `read` to read in the next chunk and hop
// over to `eventLoop` so we can safely write to `accumulatedChunks`
// without a lock.
read().hop(to: eventLoop).map { nextChunk in
// Next, we just append the chunk to the accumulation
accumulatedChunks.append(contentsOf: nextChunk)
guard nextChunk.count > 0 else {
promise.succeed(accumulatedChunks)
return
}
// and if it wasn't empty, we loop again.
loop()
}.cascadeFailure(to: promise) // if anything goes wrong, we fail the whole thing.
}
loop() // Let's kick everything off.
return promise.futureResult
}
I would like to add two things however:
First, what you're implementing above is to simply read in everything until you see EOF, if that piece of software is exposed to the internet, you should definitely add a limit on how many bytes to hold in memory maximally.
Secondly, SwiftNIO is an event driven system so if you were to read these bytes with SwiftNIO, the program would actually look slightly differently. If you're interested what it looks like to simply accumulate all bytes until EOF in SwiftNIO, it's this:
struct AccumulateUntilEOF: ByteToMessageDecoder {
typealias InboundOut = ByteBuffer
func decode(context: ChannelHandlerContext, buffer: inout ByteBuffer) throws -> DecodingState {
// `decode` will be called if new data is coming in.
// We simply return `.needMoreData` because always need more data because our message end is EOF.
// ByteToMessageHandler will automatically accumulate everything for us because we tell it that we need more
// data to decode a message.
return .needMoreData
}
func decodeLast(context: ChannelHandlerContext, buffer: inout ByteBuffer, seenEOF: Bool) throws -> DecodingState {
// `decodeLast` will be called if NIO knows that this is the _last_ time a decode function is called. Usually,
// this is because of EOF or an error.
if seenEOF {
// This is what we've been waiting for, `buffer` should contain all bytes, let's fire them through
// the pipeline.
context.fireChannelRead(self.wrapInboundOut(buffer))
} else {
// Odd, something else happened, probably an error or we were just removed from the pipeline. `buffer`
// will now contain what we received so far but maybe we should just drop it on the floor.
}
buffer.clear()
return .needMoreData
}
}
If you wanted to make a whole program out of this with SwiftNIO, here's an example that is a server which accepts all data until it sees EOF and then literally just writes back the number of received bytes :). Of course, in the real world you would never hold on to all the received bytes to count them (you could just add each individual piece) but I guess it serves as an example.
import NIO
let group = MultiThreadedEventLoopGroup(numberOfThreads: 1)
defer {
try! group.syncShutdownGracefully()
}
struct AccumulateUntilEOF: ByteToMessageDecoder {
typealias InboundOut = ByteBuffer
func decode(context: ChannelHandlerContext, buffer: inout ByteBuffer) throws -> DecodingState {
// `decode` will be called if new data is coming in.
// We simply return `.needMoreData` because always need more data because our message end is EOF.
// ByteToMessageHandler will automatically accumulate everything for us because we tell it that we need more
// data to decode a message.
return .needMoreData
}
func decodeLast(context: ChannelHandlerContext, buffer: inout ByteBuffer, seenEOF: Bool) throws -> DecodingState {
// `decodeLast` will be called if NIO knows that this is the _last_ time a decode function is called. Usually,
// this is because of EOF or an error.
if seenEOF {
// This is what we've been waiting for, `buffer` should contain all bytes, let's fire them through
// the pipeline.
context.fireChannelRead(self.wrapInboundOut(buffer))
} else {
// Odd, something else happened, probably an error or we were just removed from the pipeline. `buffer`
// will now contain what we received so far but maybe we should just drop it on the floor.
}
buffer.clear()
return .needMoreData
}
}
// Just an example "business logic" handler. It will wait for one message
// and just write back the length.
final class SendBackLengthOfFirstInput: ChannelInboundHandler {
typealias InboundIn = ByteBuffer
typealias OutboundOut = ByteBuffer
func channelRead(context: ChannelHandlerContext, data: NIOAny) {
// Once we receive the message, we allocate a response buffer and just write the length of the received
// message in there. We then also close the channel.
let allData = self.unwrapInboundIn(data)
var response = context.channel.allocator.buffer(capacity: 10)
response.writeString("\(allData.readableBytes)\n")
context.writeAndFlush(self.wrapOutboundOut(response)).flatMap {
context.close(mode: .output)
}.whenSuccess {
context.close(promise: nil)
}
}
func errorCaught(context: ChannelHandlerContext, error: Error) {
print("ERROR: \(error)")
context.channel.close(promise: nil)
}
}
let server = try ServerBootstrap(group: group)
// Allow us to reuse the port after the process quits.
.serverChannelOption(ChannelOptions.socket(.init(SOL_SOCKET), .init(SO_REUSEADDR)), value: 1)
// We should allow half-closure because we want to write back after having received an EOF on the input
.childChannelOption(ChannelOptions.allowRemoteHalfClosure, value: true)
// Our program consists of two parts:
.childChannelInitializer { channel in
channel.pipeline.addHandlers([
// 1: The accumulate everything until EOF handler
ByteToMessageHandler(AccumulateUntilEOF(),
// We want 1 MB of buffering max. If you remove this parameter, it'll also
// buffer indefinitely.
maximumBufferSize: 1024 * 1024),
// 2: Our "business logic"
SendBackLengthOfFirstInput()
])
}
// Let's bind port 9999
.bind(to: SocketAddress(ipAddress: "127.0.0.1", port: 9999))
.wait()
// This will never return.
try server.closeFuture.wait()
Demo:
$ echo -n "hello world" | nc localhost 9999
11

Golang - net.Conn infinite loop on the same message

I'm pretty news in Golang and only use sockets in this langage for 2days. However, I'm not sure to understand something. I know in C, I used select() to know who wrote etc, but here, no one is writing until one send a message. After this message sent, my dialTCP uses it endlessly.
I think I missunderstood something about close() but I'm not sure it comes from here.. there is my code:
package dial
import (
"errors"
"encoding/json"
m "models"
"net"
"net/http"
"time"
"server"
)
type DialTCP struct {}
// tcpKeepAliveListener sets TCP keep-alive timeouts on accepted
// connections. It's used by ListenAndServe and ListenAndServeTLS so
// dead TCP connections (e.g. closing laptop mid-download) eventually
// go away.
type tcpKeepAliveListener struct {
*net.TCPListener
}
func (ln tcpKeepAliveListener) Accept() (c net.Conn, err error) {
tc, err := ln.AcceptTCP()
if err != nil {
return
}
tc.SetKeepAlive(true)
tc.SetKeepAlivePeriod(3 * time.Minute)
return tc, nil
}
func (dialTCP *DialTCP) ListenAndServe(addr string) error {
ln, err := net.Listen("tcp", addr)
if err != nil {
return err
}
return dialTCP.Serve(tcpKeepAliveListener{ln.(*net.TCPListener)})
}
func (dialTCP *DialTCP) Serve(l net.Listener) error {
defer l.Close()
for {
conn, e := l.Accept()
if e != nil {
return e
}
// you want to create server_conn here with buffers, channels and stuff
// to use async thread safe read/write from it
go dialTCP.serve_conn(conn)
}
}
func (dialTCP *DialTCP) serve_conn(conn net.Conn) error {
// var buf [512]byte
dec := json.NewDecoder(conn)
//read 1st message he sent, should be token to connect
var auth m.DialAuthentication
dec.Decode(&auth)
user := m.User{
UUID: auth.UUID,
}
ok, sb := server.IsConnected(user)
if ok == false {
json.NewEncoder(conn).Encode(sb)
return errors.New("User isn't connected.")
} else {
user.Conn = conn
}
//defer conn.Close()
var message m.DialMessageContainer
for {
dec.Decode(&message)
switch message.Type {
case ".....":
/* ....(message, user)
case "....":
....(message, user)
// case "...":*/
default:
json.NewEncoder(conn).Encode(m.StatusBack{Description: "Bad entry.", StatusId: http.StatusNotAcceptable})
}
//defer conn.Close()
}
}
I think everything is good before serv_conn(), but the error should comes from inside the for. I tried lot of things, but this for{} in Golang... Why does it have not any params/var such as C/C++/C#/Java?
for (int i = 0; i < 10; i++) {}
I'm lost about the closing of the Conn as well, so I continue to read tutorial, post and doc about how to use it.. Days pass whitout find anything

Get unseen messages via IMAP

I need to extract some data from new (unseen) messages; I'm trying to make range of that data:
c, _ = imap.Dial("someserverr")
defer c.Logout(30 * time.Second)
fmt.Println("Server says hello:", c.Data[0].Info)
c.Data = nil
if c.State() == imap.Login {
c.Login("somedata", "somedata")
}
c.Select("INBOX", false)
set, _ := imap.NewSeqSet("")
fmt.Println("unseen", c.Mailbox.Unseen)
fmt.Println(c.Mailbox)
if c.Mailbox.Unseen >= 1 {
set.AddRange(1, c.Mailbox.Unseen)
} else {
set.Add("0:0")
}
The main problem here is command c.Mailbox showing wrong number of unseen messages.
For example, if I have 5 unread messages in INBOX it shows 1. If I mark that one as read, it will show 4, and so on.

How to read all lines in socket (swift)

I have problem to read lines. When server (server code in java) sent more than one line code, my secket can not read all lines.
I used this library https://github.com/swiftsocket/SwiftSocket
and socket read method code:
private func sendRequest(data: String, client: TCPClient?) -> (String?) {
// It use ufter connection
if client != nil {
// Send data (WE MUST ADD TO SENDING MESSAGE '\n' )
let (isSuccess, errorMessage) = client!.send(str: "\(data)\n")
if isSuccess {
// Read response data
let data = client!.read(1024*10)
if let d = data {
// Create String from response data
if let str = NSString(bytes: d, length: d.count, encoding: NSUTF8StringEncoding) as? String {
return (data: str)
} else {
return (data: nil)
}
} else {
return (data: nil)
}
} else {
print(errorMessage)
return (data: nil)
}
} else {
return (data: nil)
}
}
I have changed like this : let data = client!.read(1024*40) but again can not read (received data large). it read some received data like this:
338C6EAAD0740ED101860EFBA286A653F42793CF1FC7FB55D244C010941BDE3DCA540223B291639D1CD7285B4240B330EBC7C002003F957790749256D54EAC4DB7FC0AD5E2970FE951DFA0A1635A93DFCB031DBA642BA928B8327B661F9F4F22CA657AE803B25A208C23F72D6F934B95558108C1187F90E6D8DE13F9367534E7EC28DBA5AC6C8597800154033B63A3B2185DF68ABF67BB76AC6B593C52E5D7B0A5D7674BEC7C6AA9B4C57343C32C944FEB5D1C2D6C2A400D454196A0029C23DB30F1B6049423DC5BEE728E03C275F207639C25E226A38A23EAE04AD673132336D9E113FC6CA32DAD1D75191BCE40A281D40549C1D6FDD23BC5A38B472ED1EEB6BA1D80D00EF2A08F4729FD05329993A6AE58C34253660A77C20139C73FE0E4A68D9857E02C3F8589A61B22C4E26B3DA00098158FB6CE0C43F271ABC823BF9AD7DBEB32A4E9BE4E90C284E1FD633956F82EB5387DE2E8D7626C88900D183C7E1F683C27B8A654EE75017E897D11F2431A9AB4C0662CC91B3897D52E630A788A1A8D552ABB04916B5E52A1382DB2803A796A96ACC50C00913C650393AA919100E6477B7D88066FEB6C78D4F5853122AB6D7309540053B9DB98BEC0D518CD8BF41620506E1224FB0F8B240B7E9FD60649E703FA9E6A21E785BB0F646DB028F5C5E64697E41A857F6A2A459B2C1B070C244F7B6FD252FFF016CC0A4F0457711213C025E6165706DF6C35C6C38F53610373C3DE99DFE426102860E2D4CD1BF5F6B97346655F0E26103CD6BF37AFCE705D4D2F78F7F40C5316143725D0D7FB6647A92F98A42570F423D646DC2CA726ABB16ED6C62C
but not all. How to fix this. Please, advise me. Thanks
there is your responsibility to check what did you received and how to process all data received from your socket. try to check this simple example how to read line delimited text from network. you can easily adopt this code for use with your library, or use it as is if you have working socket handle ...
import Darwin
private var line = [UInt8]()
private var excess = [UInt8]()
private func readLine(inout lineBufer: [UInt8], inout excessBufer: [UInt8], var sockHandle: Int32, wait: Bool = true) -> Int{
// (1) received bytes in one cycle and numbers of cycles to receive
// whole line (line delimited text)
var received = 0
var i = 0
// (2) clear the buffer for the extra bytes readed after the \n
excessBufer.removeAll()
var buffer = [UInt8](count: 64, repeatedValue: 0)
// (3) read from network until at least one \n is found
repeat {
received = read(sockHandle, &buffer, buffer.count)
if received == 0 {
close(sockHandle)
sockHandle = -1
print("connection lost")
return -1
}
if received < 0 {
print("\treceived failed:", String.fromCString(strerror(errno))!)
return -1
}
i = 0
while i < received && buffer[i] != UInt8(ascii: "\n") { i++ }
lineBufer.appendContentsOf(buffer[0..<i])
// (4) now consume '\n'
i++
// (5) we have extra bytes for next line received -i > 0
if i < received {
excessBufer.appendContentsOf(buffer[i..<received])
break // from loop
}
// (6) if no extra bytes and no whole line, than received - i < 0 (-1)
// if whole line and no extra bytes, than received - i = 0
} while i != received
return received
}
// (1) prepare buffers
line.removeAll()
line.appendContentsOf(excess)
// (2) read line delimited text ( socket is you uderlying socket handle )
readLine(&line, excessBufer: &excess, sockHandle: socket)
// now your line buffer consist of one line ot text
// do something with the line and repeate the proces as you need .....
// (1) prepare buffers
line.removeAll()
line.appendContentsOf(excess)
// (2) read line delimited text
readLine(&line, excessBufer: &excess, sockHandle: socket)
Looking at the GitHub repository you mentioned, I'm under the impression that this is a relatively new and not well-supported project. Yes, it may provide a somewhat simplified interface to sockets, but it is unknown how buggy it is and what its future is. Instead of using it, one might be better off using well-supported Apple Objective-C frameworks. Please see
https://developer.apple.com/library/ios/documentation/NetworkingInternet/Conceptual/NetworkingTopics/Articles/UsingSocketsandSocketStreams.html
https://developer.apple.com/library/ios/documentation/NetworkingInternetWeb/Conceptual/NetworkingOverview/CommonPitfalls/CommonPitfalls.html
It is fairly easy to use the Foundation framework with Swift. Yes, you can definitely call POSIX C networking API from Swift, but that is more work and is not recommended in iOS, as it doesn't activate the cellular radio in the device, as explained in the aforementioned Apple articles.

golang: goroute with select doesn't stop unless I added a fmt.Print()

I tried the Go Tour exercise #71
If it is run like go run 71_hang.go ok, it works fine.
However, if you use go run 71_hang.go nogood, it will run forever.
The only difference is the extra fmt.Print("") in the default in the select statement.
I'm not sure, but I suspect some sort of infinite loop and race-condition? And here is my solution.
Note: It's not deadlock as Go didn't throw: all goroutines are asleep - deadlock!
package main
import (
"fmt"
"os"
)
type Fetcher interface {
// Fetch returns the body of URL and
// a slice of URLs found on that page.
Fetch(url string) (body string, urls []string, err error)
}
func crawl(todo Todo, fetcher Fetcher,
todoList chan Todo, done chan bool) {
body, urls, err := fetcher.Fetch(todo.url)
if err != nil {
fmt.Println(err)
} else {
fmt.Printf("found: %s %q\n", todo.url, body)
for _, u := range urls {
todoList <- Todo{u, todo.depth - 1}
}
}
done <- true
return
}
type Todo struct {
url string
depth int
}
// Crawl uses fetcher to recursively crawl
// pages starting with url, to a maximum of depth.
func Crawl(url string, depth int, fetcher Fetcher) {
visited := make(map[string]bool)
doneCrawling := make(chan bool, 100)
toDoList := make(chan Todo, 100)
toDoList <- Todo{url, depth}
crawling := 0
for {
select {
case todo := <-toDoList:
if todo.depth > 0 && !visited[todo.url] {
crawling++
visited[todo.url] = true
go crawl(todo, fetcher, toDoList, doneCrawling)
}
case <-doneCrawling:
crawling--
default:
if os.Args[1]=="ok" { // *
fmt.Print("")
}
if crawling == 0 {
goto END
}
}
}
END:
return
}
func main() {
Crawl("http://golang.org/", 4, fetcher)
}
// fakeFetcher is Fetcher that returns canned results.
type fakeFetcher map[string]*fakeResult
type fakeResult struct {
body string
urls []string
}
func (f *fakeFetcher) Fetch(url string) (string, []string, error) {
if res, ok := (*f)[url]; ok {
return res.body, res.urls, nil
}
return "", nil, fmt.Errorf("not found: %s", url)
}
// fetcher is a populated fakeFetcher.
var fetcher = &fakeFetcher{
"http://golang.org/": &fakeResult{
"The Go Programming Language",
[]string{
"http://golang.org/pkg/",
"http://golang.org/cmd/",
},
},
"http://golang.org/pkg/": &fakeResult{
"Packages",
[]string{
"http://golang.org/",
"http://golang.org/cmd/",
"http://golang.org/pkg/fmt/",
"http://golang.org/pkg/os/",
},
},
"http://golang.org/pkg/fmt/": &fakeResult{
"Package fmt",
[]string{
"http://golang.org/",
"http://golang.org/pkg/",
},
},
"http://golang.org/pkg/os/": &fakeResult{
"Package os",
[]string{
"http://golang.org/",
"http://golang.org/pkg/",
},
},
}
Putting a default statement in your select changes the way select works. Without a default statement select will block waiting for any messages on the channels. With a default statement select will run the default statement every time there is nothing to read from the channels. In your code I think this makes an infinite loop. Putting the fmt.Print statement in is allowing the scheduler to schedule other goroutines.
If you change your code like this then it works properly, using select in a non blocking way which allows the other goroutines to run properly.
for {
select {
case todo := <-toDoList:
if todo.depth > 0 && !visited[todo.url] {
crawling++
visited[todo.url] = true
go crawl(todo, fetcher, toDoList, doneCrawling)
}
case <-doneCrawling:
crawling--
}
if crawling == 0 {
break
}
}
You can make your original code work if you use GOMAXPROCS=2 which is another hint that the scheduler is busy in an infinite loop.
Note that goroutines are co-operatively scheduled. What I don't fully understand about your problem is that select is a point where the goroutine should yield - I hope someone else can explain why it isn't in your example.
You have 100% CPU load because almost all times the default case will be executed, resulting effectively in an infinite loop because it's executed over and over again. In this situation the Go scheduler does not hand control to another goroutine, by design. So any other goroutine will never have the opportunity to set crawling != 0 and you have your infinite loop.
In my opinion you should remove the default case and instead create another channel if you want to play with the select statement.
Otherwise the runtime package helps you to go the dirty way:
runtime.GOMAXPROCS(2) will work (or export GOMAXPROCS=2), this way you will have more than one OS thread of execution
call runtime.Gosched() inside Crawl from time to time. Eventhough CPU load is 100%, this will explicitely pass control to another Goroutine.
Edit: Yes, and the reason why fmt.Printf makes a difference: because it explicitely passes control to some syscall stuff... ;)