Send and read data through socket from fiber - sockets

Trying to figure out how to send/read data through socket. On remote server I create new netcat -l 4444 and from local send text data echo "test" | netcat remote.host 4444. This is always works fine.
Trying to reproduce:
require "socket"
HOST = "remote.host"
PORT = 4444
ch_request = Channel(String).new
ch_response = Channel(String).new
spawn do
socket = TCPSocket.new(HOST, PORT)
loop do
select
when request = ch_request.receive
socket << request
socket.flush
end
if response = socket.gets
ch_response.send response
end
end
end
sleep 0.1
ch_request.send "hello"
loop do
select
when response = ch_response.receive
pp response
end
end
In my dream I send data to channel, read it from first loop then send to socket. The same way but reverse order need for read it from second loop.
On practice this is not happens. On local after connect I got "test" and can't send back anything. On remote I can send to local but on local got only empty string once and nothing more after.
What mean this behavior and how to achieve planned?

You didn't show this, but I suppose you have a second implementation using TCPServer for the netcat -l equivalent.
You need to use separate fibers for reading/writing to the socket and the channel. It's a bit hard to judge what exactly happens without seeing the server, but I suppose you end up in a deadlock where both sides are waiting on input of the other side or user, but cannot proceed to actually send or read anything. In other words you interlocked the sending/receiving part, requiring the other side to carefully react and interplay so to not lock up the client. This is obviously a brittle approach.
Instead you should make sure any fiber does not do more than one operation in a loop. One receives from the socket and forwards that to a channel, the second one receives from a channel and forwards that to the socket, the third one receives from the reader side channel and prints or does whatever you want to do the data and the last one fills the sender channel. This way no operation can block one of the others. Of course one of those fibers should simply be the main program one.
In the server you additionally need one fiber that accepts the client connections and spawns the sender and receiver loops for each.
Finally note that a select statement with a single when branch has no effect, you can make the call directly. select is useful if you need to read or write to multiple channels concurrently in the same fiber, so for example if you would have multiple channels providing data to be send out to a socket, you would use select to not have the messages be corrupted by two fibers writing to the same socket at the same time. An additional usecase for select is to send or receive from a channel with a timeout.

Who is looking for an answer to similar questions. The final result what I wanted looks like this:
# Open socket for simulate remote server: `netcat -v -4 -l 4444`
require "socket"
HOST = "remote.host"
PORT = 4444
# JFYI: In real life this packed into class and I use class variable instead consts.
TUBE_REQUEST = Channel(String).new
TUBE_RESPONSE = Channel(String).new
SOCKET = TCPSocket.new(HOST, PORT)
spawn do
until SOCKET.closed?
if request = TUBE_REQUEST.receive
SOCKET << request
SOCKET.flush
end
end
end
spawn do
until SOCKET.closed?
if response = SOCKET.gets
TUBE_RESPONSE.send response
end
end
end
sleep 0.1
def receive_response
TUBE_RESPONSE.receive
end
def send(message, wait_for_response = true)
TUBE_REQUEST.send message
receive_response if wait_for_response
end
send("command with response")
send("command with new line and response\n")
send("command without new line and response", false)
It will send each command and wait for answer (except the last) from remote and then call the next command.

Related

Can RemoteEvents be two way?

For example, would these scripts work, hang, or possibly give an error?
(In a script)
RemoteEvent = game:GetService("ReplicatedStorage"):WaitForChild("RemoteEvent")
RemoteEvent:FireClient()
RemoteEvent.onServerEvent:Wait()
print("Hello World")
(In a LocalScript)
RemoteEvent = game:GetService("ReplicatedStorage"):WaitForChild("RemoteEvent")
RemoteEvent.onClientEvent:Wait()
RemoteEvent:FireServer()
To simply answer your question, RemoteEvents do work both ways. The same event can be used from client-to-server as server-to-client.
Your example, as-is, will likely have some timing issues.
The server will start, fire on all zero clients, then wait for a client to send the signal back.
At some time later, a client will join and then wait for the server signal (which has already happened) and get stuck.
Depending on your desired logic, you could wait to execute the code when a player joins the game :
-- server Script
local PlayerService = game:GetService("Players")
local ReplicatedStorage = game:GetService("ReplicatedStorage")
local RemoteEvent = ReplicatedStorage.RemoteEvent
PlayerService.PlayerAdded:Connect( function(player)
-- Wait() will fire for any player that sends up the signal, so make sure that we know which player sent it
-- instead, connect to the signal to make sure we're listening to the right player
RemoteEvent.OnServerEvent:Connect( function(clientPlayer)
if clientPlayer.Name == player.Name then
print("Hello World", player.Name)
-- pass some data from the server to the client
RemoteEvent:FireClient( player, 1, 2, 3)
end)
end)
end)
Then in your client...
-- LocalScript in PlayerScripts
local ReplicatedStorage = game:GetService("ReplicatedStorage")
local RemoteEvent = ReplicatedStorage.RemoteEvent
-- since there's no telling when this event will fire, tell the server that we've loaded!
RemoteEvent:FireServer()
-- wait for the server to respond and give us some data
local connection
connection = RemoteEvent.OnClientEvent:Connect( function(a, b, c)
print("got data from server : ", a, b, c)
-- disconnect so this event only fires once
connection:Disconnect()
end)
I would like to add to Kylaaa's answer and in response to your code example that if your intention is to wait for a response from the client, you can also invoke a RemoteFunction instead of firing an event and waiting for an event to come back to you. Something like this:
Script:
game.Players.PlayerAdded:Connect(function(plr)
local response = game.ReplicatedStorage.RemoteFunction:InvokeClient(plr, "Hello")
print("Client said: " .. response)
end)
LocalScript:
game.ReplicatedStorage.RemoteFunction.OnClientInvoke = function(text)
print ("Server said: " .. text)
return "Hi"
end
PS. A RemoteFunction also works both ways.

What is meant by record or data boundaries in the sense of TCP & UDP protocol?

I am learning to sockets and found the word Data OR Record Boundaries in SOCK_SEQPACKET communication protocol? Can anyone explain in simple words what is Data boundary and how the SOCK_SEQPACKET is different from SOCK_STREAM & SOCK_DGRAM ?
This answer https://stackoverflow.com/a/9563694/1076479 has a good succinct explanation of message boundaries (a different name for "record boundaries").
Extending that answer to SOCK_SEQPACKET:
SOCK_STREAM provides reliable, sequenced communication of streams of data between two peers. It does not maintain message (record) boundaries, which means the application must manage its own boundaries on top of the stream provided.
SOCK_DGRAM provides unreliable transmission of datagrams. Datagrams are self-contained capsules and their boundaries are maintained. That means if you send a 20 byte buffer on peer A, peer B will receive a 20 byte message. However, they can be dropped, or received out of order, and it's up to the application to figure that out and handle it.
SOCK_SEQPACKET is a newer technology that is not yet widely used, but tries to marry the benefits of both of the above. That is, it provides reliable, sequenced communication that also transmits entire "datagrams" as a unit (and hence maintains message boundaries).
It's easiest to demonstrate the concept of message boundaries by showing what happens when they're neglected. Beginners often post client code like this here on SO (using python for convenience):
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('192.168.4.122', 9000))
s.send(b'FOO') # Send string 1
s.send(b'BAR') # Send string 2
reply = s.recv(128) # Receive reply
And server code similar to this:
lsock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
lsock.bind(('', 9000))
lsock.listen(5)
csock, caddr = lsock.accept()
string1 = csock.recv(128) # Receive first string
string2 = csock.recv(128) # Receive second string <== XXXXXXX
csock.send(b'Got your messages') # Send reply
They don't understand then why the server hangs on the second recv call, while the client is hung on its own recv call. That happens because both strings the client sent (may) get bundled together and received as a single unit in the first recv on the server side. That is, the message boundary between the two logical messages was not preserved, and so string1 will often contain both chunks run together: 'FOOBAR'
(Often there are other timing-related aspects to the code that influence when/whether that actually happens or not.)

Monitor TCP/IP stream

I am interested in learning Vapor, so I decided to work on a website that displays government issued weather alerts. Alert distribution is done via a TCP/IP data stream (streaming1.naad-adna.pelmorex.com port 8080).
What I have in mind is to use IBM's BlueSocket (https://github.com/IBM-Swift/BlueSocket) to create a socket, though after this point, I gave it a bit of thought but was unable to come to a conclusion on what the next steps would be.
Alerts are streamed over the data stream, so I am aware the socket would need to be opened and listened on but wasn't able to get to much past that.
A few things with the data stream are that the start and end of an alert is detected using the start and end tags of the XML document (alert and /alert). There are no special or proprietary headers added to the data, it's only raw XML. I know some alerts also include an XML declaration so I assume the encoding should be taken into account if the declaration is available.
I was then thinking of using XMLParser to parse the XML and use the data I am interested in from the alert.
So really, the main thing I am struggling with is, when the socket is open, what would be the method to listen to it, determine the start and end of the alert and then pass that XML alert for processing.
I would appreciate any input, I am also not restricted to BlueSocket so if there is a better option for what I am trying to achieve, I would be more than open to it.
So really, the main thing I am struggling with is, when the socket is
open, what would be the method to listen to it, determine the start
and end of the alert and then pass that XML alert for processing.
The method that you should use is read(into data: inout Data). It stores any available data that the server has sent into data. There are a few reasons for this method to fail, such as the connection disconnecting.
Here's an example of how to use it:
import Foundation
import Socket
let s = try Socket.create()
try s.connect(to: "streaming1.naad-adna.pelmorex.com", port: 8080)
while true {
if try Socket.wait(for: [s], timeout: 0, waitForever: true) != nil {
var alert = Data()
try s.read(into: &alert)
if let message = String(data: alert, encoding: .ascii) {
print(message)
}
}
}
s.close()
First create the socket. The default is what we want, a IPv4 TCP Stream.
Second connect() to the server using the hostname and port. Without this step, the socket isn't connected and cannot receive or send any data.
wait() until hostname has sent us some data. It returns a list of sockets that have data available to read.
read() the data, decode it and print it. By default this call will block if there is no data available on the socket.
close() the socket. This is good practice.
You might also like to consider thinking about:
non blocking sockets
error handling
streaming (a single call to read() might not give a complete alert).
I hope this answers your question.

Epoll events for connecting sockets

I create epoll and register some non-blocking sockets which try connect to closed ports on localhost. Why epoll tells me, that i can write to this socket (it give event for one of created socket with eventmask contain EPOLLOUT)? But this socket doesn't open and if i try send something to it i get an error Connection refused.
Another question - what does mean even EPOLLHUP? I thought that this is event for refused connection. But how in this case event can have simultaneously EPOLLHUP and EPOLLOUT events?
Sample code on Python:
import socket
import select
poll = select.epoll()
fd_to_sock = {}
for i in range(1, 3):
s = socket.socket()
s.setblocking(0)
s.connect_ex(('localhost', i))
poll.register(s, select.EPOLLOUT)
fd_to_sock[s.fileno()] = s
print(poll.poll(0.1))
# prints '[(4, 28), (5, 28)]'
All that poll guarantees is that your application won't block after calling corresponding function. So you are getting what you've paid for - you can now rest assured writing to this socket won't block - and it didn't block, did it?
Poll never guarantees that corresponding operation will succeed.
poll/select/epoll return when the file descriptor is "ready" but that just means that the operation will not block (not that you will necessarily be able to write to it successfully).
Likewise for EPOLLIN: for example, it will return ready when a socket is closed; in that case, you won't actually be able to read data from it.
EPOLLHUP means that there was a "hang up" on the connection. That would really only occur once you actually had a connection. Also, the documentation (http://linux.die.net/man/2/epoll_ctl) says that you don't need to include it anyway:
EPOLLHUP
Hang up happened on the associated file descriptor. epoll_wait(2) will always wait for this event; it is not necessary to set it in events.

How to write a proxy in go (golang) using tcp connections

I apologize before hand if some of these questions might be obvious for expert network programmers. I have researched and read about coding in networking and it is still not clear to me how to do this.
Assume that I want to write a tcp proxy (in go) with the connection between some TCP client and some TCP server. Something like this:
First assume that these connection are semi-permanent (will be closed after a long long while) and I need the data to arrive in order.
The idea that I want to implement is the following: whenever I get a request from the client, I want to forward that request to the backend server and wait (and do nothing) until the backend server responds to me (the proxy) and then forward that response to the client (assume that both TCP connection will be maintained in the common case).
There is one main problem that I am not sure how to solve. When I forward the request from the proxy to the server, and get the response, how do I know when the server has sent me all the information that I need if I do not know beforehand the format of the data being sent from the server to the proxy (i.e. I don't know if the response from the server is of the form of type-length-value scheme nor do I know if `\r\n\ indicates the end of the message form the server). I was told that I should assume that I get all the data from the server connection whenever my read size from the tcp connection is zero or smaller than the read size that I expected. However, this does not seem correct to me. The reason it might not be correct in general is the following:
Assume that the server for some reason is only writing to its socket one byte at a time but the total length of the response to the "real" client is much much much longer. Therefore, isn't it possible that when the proxy reads the tcp socket connected to the server, that the proxy only reads one byte and if it loops fast enough (to do a read before it receives more data), then read zero and incorrectly concludes that It got all the message that the client intended to receive?
One way to fix this might be to wait after each read from the socket, so that the proxy doesn't loop faster than it gets bytes. The reason that I am worried is, assume there is a network partition and i can't talk to the server anymore. However, it is not disconnected from me long enough to timeout the TCP connection. Thus, isn't it possible that I try to read from the tcp socket to the server again (faster than I get data) and read zero and incorrectly conclude that its all the data and then send it pack to the client? (remember, the promise I want to keep is that I only send whole messages to the client when i write to the client connection. Thus, its illegal to consider correct behaviour if the proxy goes, reads the connection again at a later time after it already wrote to the client, and sends the missing chunk at a later time, maybe during the response of a different request).
The code that I have written is in go-playground.
The analogy that I like to use to explain why I think this method doesn't work is the following:
Say we have a cup and the proxy is drinking half the cup every time it does a read from the server, but the server only puts 1 teaspoon at a time. Thus, if the proxy drinks faster than it gets teaspoons it might reach zero too soon and conclude that its socket is empty and that its ok to move on! Which is wrong if we want to guarantee we are sending full messages every time. Either, this analogy is wrong and some "magic" from TCP makes it work or the algorithm that assumes until the socket is empty is just plain wrong.
A question that deals with a similar problems here suggests to read until EOF. However, I am unsure why that would be correct. Does reading EOF mean that I got the indented message? Is an EOF sent each time someone writes a chunk of bytes to a tcp socket (i.e. I am worried that if the server writes one byte at a time, that it sends 1 EOF per bytes)? However, EOF might be some of the "magic" of how a TCP connection really works? Does sending EOF's close the connection? If it does its not a method that I want to use. Also, I have no control of what the server might be doing (i.e. I do not know how often it wants to write to the socket to send data to the proxy, however, its reasonable to assume it writes to the socket with some "standard/normal writing algorithm to sockets"). I am just not convinced that reading till EOF from the socket from server is correct. Why would it? When can I even read to EOF? Are EOFs part of the data or are they in the TCP header?
Also, the idea that I wrote about putting a wait just epsilon bellow the time-out, would that work in the worst-case or only on average? I was also thinking, I realized that if the Wait() call is longer than the time-out, then if you return to the tcp connection and it doesn't have anything, then its safe to move on. However, if it doesn't have anything and we don't know what happened to the server, then we would time out. So its safe to close the connection (because the timeout would have done that anyway). Thus, I think if the Wait call is at least as long as the timeout, this procedure does work! What do people think?
I am also interested in an answer that can justify maybe why this algorithm work on some cases. For example, I was thinking, even if the server only write a byte at a time, if the scenario of deployment is a tight data centre, then on average, because delays are really small and the wait call is almost certainly enough, then wouldn't this algorithm be fine?
Also, are there any risks of the code I wrote getting into a "deadlock"?
package main
import (
"fmt"
"net"
)
type Proxy struct {
ServerConnection *net.TCPConn
ClientConnection *net.TCPConn
}
func (p *Proxy) Proxy() {
fmt.Println("Running proxy...")
for {
request := p.receiveRequestClient()
p.sendClientRequestToServer(request)
response := p.receiveResponseFromServer() //<--worried about this one.
p.sendServerResponseToClient(response)
}
}
func (p *Proxy) receiveRequestClient() (request []byte) {
//assume this function is a black box and that it works.
//maybe we know that the messages from the client always end in \r\n or they
//they are length prefixed.
return
}
func (p *Proxy) sendClientRequestToServer(request []byte) {
//do
bytesSent := 0
bytesToSend := len(request)
for bytesSent < bytesToSend {
n, _ := p.ServerConnection.Write(request)
bytesSent += n
}
return
}
// Intended behaviour: waits until ALL of the response from backend server is obtained.
// What it does though, assumes that if it reads zero, that the server has not yet
// written to the proxy and therefore waits. However, once the first byte has been read,
// keeps writting until it extracts all the data from the server and the socket is "empty".
// (Signaled by reading zero from the second loop)
func (p *Proxy) receiveResponseFromServer() (response []byte) {
bytesRead, _ := p.ServerConnection.Read(response)
for bytesRead == 0 {
bytesRead, _ = p.ServerConnection.Read(response)
}
for bytesRead != 0 {
n, _ := p.ServerConnection.Read(response)
bytesRead += n
//Wait(n) could solve it here?
}
return
}
func (p *Proxy) sendServerResponseToClient(response []byte) {
bytesSent := 0
bytesToSend := len(request)
for bytesSent < bytesToSend {
n, _ := p.ServerConnection.Write(request)
bytesSent += n
}
return
}
func main() {
proxy := &Proxy{}
proxy.Proxy()
}
Unless you're working with a specific higher-level protocol, there is no "message" to read from the client to relay to the server. TCP is a stream protocol, and all you can do is shuttle bytes back and forth.
The good news is that this is amazingly easy in go, and the core part of this proxy will be:
go io.Copy(server, client)
io.Copy(client, server)
This is obviously missing error handling, and doesn't shut down cleanly, but clearly shows how the core data transfer is handled.