Yo, I've written a server with a simple protocol: the client sends a line, the server sends a line back in response, repeat. To prevent a client from filling Tcl's output buffer by sending lots of lines but not accepting data back, can I just check chan pending output instead of using the writable fileevent?
proc respond {stream msg} {
if {[chan pending output $stream] <= 1024} {
puts $stream $msg
} else {
#close $stream
}
}
For output, chan pending output will correctly describe the number of bytes waiting in the output queue. Normally, that value will be bounded by the -buffersize value that you chan configure (or fconfigure) it to have.
That value will only be exceeded when the channel is non-blocking; with a blocking channel, when the value would go over it, instead there's a blocking write to the underlying device (socket, pipe, file, serial line, whatever) so by the time you could see that it went over, it's back under the limit again.
But if you're using non-blocking channels, you really should use chan event (or fileevent). Luckily for the actual writes, Tcl will actually do this for you automatically; the single most useful thing you could want from a writable event is already there. In practice, the most common actual use of a writable event is in detecting when an async socket connection becomes ready for service.
So what you are doing will work, but you'll have to think carefully about what to do if the output buffer is “getting full”; the idea that a message can need to be delayed is a place where a simple abstraction tends to become leaky. With 8.6's coroutines, you could (probably) do a transparent suspend or something like that, but getting that sort of thing right can take a little thought. (For example, a GUI client might need to show a busy indicator and put things into a state where the user can't enter more requests.)
Related
I'm very new to Tcl/Tk and have been dealing with an issue for the last couple of days. Basically I have a server written in C and a client GUI written in Tcl/Tk. So far it doesn't do a ton. To test it, I start up the server so that it's listening for connections, then run my GUI. When I click one of the buttons, the GUI should open up a separate toplevel window with a text widget embedded in it. (This part works.) Then, my client connects to the server and gives it a couple of settings, and through this the server decides what info to send back. The server's response is what gets printed to that second window's text widget.
What I'm trying to add in now is a Stop button. Right now, my server is set up to wait a couple of seconds, then write the same message to the client. This is set up inside a loop that is waiting to hear a "Stop" command from my client. I have a Stop button in the GUI with a command set up to write that command to the server when clicked. However, all of my buttons get frozen as soon as I hit the begin button and messages are written to the client.
Basically, how can I keep allowing my server to write to my client while still keeping the rest of my GUI usable? I want my client to write a new line to the text widget on my separate window whenever it receives a new message from the server, but I still want the main GUI window that has all my command buttons to behave independently.
In general, it depends on whether what you are doing is CPU-intensive (where reading from a plain file counts as CPU-intensive) or I/O-intensive (where running things in another process counts as I/O-intensive; database calls often count as CPU-intensive here despite not really needing to). I'm only going to mention summaries of what's going on as you aren't quite providing enough information.
For I/O-intensive code, you want to structure your code to be event-driven. Tcl has good tools for this, in that fileevent works nicely on sockets, terminals and pipelines on all supported platforms. The coroutine system of Tcl 8.6 can help a lot with preventing the callbacks required from turning your code into a tangled mess!
For CPU-intensive code, the main option is to run in another thread. That thread won't be able to touch the GUI directly (which in turn will be free to be responsive), but will be able to do all the work and send messages back to the main thread with whatever UI updates it wants done. (Technically you can do this with I/O-intensive code too, but it's more irritating than using a coroutine.) Farming things out to a subprocess is just another variation on this where the communications are more expensive (but much isolation is enforced by the OS).
If you're dealing with sockets, you're probably I/O-intensive. Assume that until you show otherwise. Here's a simple example:
proc gets_async {sock} {
set sock [lindex $args end]
fileevent $sock readable [info coroutine]
while {[gets $sock data] < 0 && [fblocked $sock]} {
yield
}
fileevent $sock readable {}
return $data
}
proc handler {socket} {
set n 0
while {![eof $socket]} {
# Write to the server
puts $socket "this is message [incr n] to the server"
# Read from the server
puts [gets_async $socket]
}
close $socket
}
proc launchCommunications {host port} {
set sock [socket $host $port]
fconfigure $sock -blocking 0 -encoding utf-8
coroutine comms($host:$port) handler $socket
}
Note that gets_async is much like coroutine::util gets in Tcllib.
Imagine inter-processes communication
+--------------+ +------------+
| main_process | ==produces data somewhat to=> | monitoring |
+--------------+ +------------+
where:
the main_process is running non-stop and produces some data for monitoring
the monitoring is running only sometimes, and when it is running it should read the data produced by main_process
and when the monitoring is not running the data produced by main_process should be not saved.
The question is: How to write like "on demand" IPC?
The code for main_process is basically the next (the real one is more complicated):
use 5.014;
use warnings;
my $box = new BlackBox( callback => sub {
my ($self, $jref) = #_;
#
# processing of $jref
#
});
$box->run();
The callback is called every 2-5 seconds and as i told above, this process should run non-stop. I can't change the BlackBox.
I need:
write the processing of $jref part - what should send $jref to somewhere
and the monitoring process itself, what should read the data, when it is running...
Don't need any code, need only some pointers to the right direction, or idea how to do this, without filling up my memory or HDD, so the simplest way:
write the $jref to the file is not suitable because it will fill my HDD when the monitoring is not running.
If someone care, the $jref is a reference to json string, so i can do:
use JSON::XS qw(decode_json):
my $perlref = decode_json($$jref);
My first thought was "UDP to localhost?"
Variations on that idea include AF_UNIX or a named pipe. With a stream socket you'd do a non-blocking connect, and with the pipe you'd do O_WRONLY|O_NONBLOCK, and if you get EAGAIN just return without writing.
You can save your file handle and reuse it across multiple calls, just close it and reopen if you get EPIPE. You'll want $SIG{PIPE}='IGNORE'; hopefully the black box doesn't object to that.
The reading side is as simple as cat $path_to_fifo or nc -l -u -p $udpport, slightly harder if you do an AF_UNIX socket.
It sounds like using a socket will be enough. Choose a port number P and try to connect to the monitor from the main application (localhost:P). If the port is open, send the actual data, not the reference. If the port is closed then just ignore the data. Your monitor will listen on P and process any data received through this port.
I'm writing a little VOIP app like Skype, which works quite good right now, but I've run into a very strange problem.
In one thread, I'm calling within a while(true) loop the winsock recv() function twice per run to get data from a socket.
The first call gets 2 bytes which will be casted into a (short) while the second call gets the rest of the message which looks like:
Complete Message: [2 Byte Header | Message, length determined by the 2Byte Header]
These packets are round about 49/sec which will be round about 3000bytes/sec.
The content of these packets is audio-data that gets converted into wave.
With ioctlsocket() I determine wether there is some data on the socket or not at each "message" I receive (2byte+data). If there's something on the socket right after I received a message within the while(true) loop of the thread, the message will be received, but thrown away to work against upstacking latency.
This concept works very well, but here's the problem:
While my VOIP program is running and when I parallely download (e.g. via browser) a file, there always gets too much data stacked on the socket, because while downloading, the recv() loop seems actually to slow down. This happens in every download/upload situation besides the actual voip up/download.
I don't know where this behaviour comes from, but when I actually cancel every up/download besides the voip traffic of my application, my apps works again perfectly.
If the program runs perfectly, the ioctlsocket() function writes 0 into the bytesLeft var, defined within the class where the receive function comes from.
Does somebody know where this comes from? I'll attach my receive function down below:
std::string D_SOCKETS::receive_message(){
recv(ClientSocket,(char*)&val,sizeof(val),MSG_WAITALL);
receivedBytes = recv(ClientSocket,buffer,val,MSG_WAITALL);
if (receivedBytes != val){
printf("SHORT: %d PAKET: %d ERROR: %d",val,receivedBytes,WSAGetLastError());
exit(128);
}
ioctlsocket(ClientSocket,FIONREAD,&bytesLeft);
cout<<"Bytes left on the Socket:"<<bytesLeft<<endl;
if(bytesLeft>20)
{
// message gets received, but ignored/thrown away to throw away
return std::string();
}
else
return std::string(buffer,receivedBytes);}
There is no need to use ioctlsocket() to discard data. That would indicate a bug in your protocol design. Assuming you are using TCP (you did not say), there should not be any left over data if your 2byte header is always accurate. After reading the 2byte header and then reading the specified number of bytes, the next bytes you receive after that constitute your next message and should not be discarded simply because it exists.
The fact that ioctlsocket() reports more bytes available means that you are receiving messages faster than you are reading them from the socket. Make your reading code run faster, don't throw away good data due to your slowness.
Your reading model is not efficient. Instead of reading 2 bytes, then X bytes, then 2 bytes, and so on, you should instead use a larger buffer to read more raw data from the socket at one time (use ioctlsocket() to know how many bytes are available, and then read at least that many bytes at one time and append them to the end of your buffer), and then parse as many complete messages are in the buffer before then reading more raw data from the socket again. The more data you can read at a time, the faster you can receive data.
To help speed up the code even more, don't process the messages inside the loop directly, either. Do the processing in another thread instead. Have the reading loop put complete messages in a queue and go back to reading, and then have a processing thread pull from the queue whenever messages are available for processing.
I have an intermittent problem with a telnet based server on Unix (the problem crops up on both AIX and Linux).
The server opens two sockets, one to a client telnet session, and one to a program running on the same machine as the server. The idea is that the data is passed through the server to and from this program.
The current setup has a loop using select to wait for a "read" file descriptor to become available, then uses select to wait for a "write" file descriptor to become available.
Then the program reads from the incoming file descriptor, then processes the data before writing to the outgoing descriptor.
The snippet below shows what is going on. The problem is that very occasionally the read fails, with errno being set to ECONNRESET or ETIMEDOUT. Neither of these are codes documented by read, so where are they coming from?
The real question is, how can I either stop this happening, or handle it gracefully?
Could doing two selects in a row be the problem?
The current handling behaviour is to shut down and restart. One point to note is that once this happens it normally happens three or four times, then clears up. The system load doesn't really seem to be that high (it's a big server).
if (select(8, &readset, NULL, NULL, NULL) < 0)
{
break;
}
if (select(8, NULL, &writeset, NULL, NULL) < 0)
{
break;
}
if (FD_ISSET(STDIN_FILENO, &readset)
&& FD_ISSET(fdout, &writeset))
{
if ((nread = read(STDIN_FILENO, buff, BUFFSIZE)) < 0)
{
/* This sometimes fails with errno =
ECONNRESET or ETIMEDOUT */
break;
}
}
Look at the comments in http://lxr.free-electrons.com/source/arch/mips/include/asm/errno.h on lines 85 and 98: these basically say there was a network connection reset or time out. Check and see if there are timeouts you can adjust on the remote network program, or send some periodic filler bytes to ensure that the connection stays awake consistently. You may just be victim of an error in the network transit path between the remote client and your local server (this happens to me when my DSL line hiccups).
EDIT: not sure what the downvote is for. The man page for read explicitly says:
Other errors may occur, depending on the object connected to fd.
The error is probably occuring in the select, not in the read: you're not checking errors after the select, you're just proceeding to the read, which will fail if the select returned an error. I'm betting if you check the errno value after the select call you'll see the errors: you don't need to wait for the read to see the errors.
I have a loop that reads from a socket in Lua:
socket = nmap.new_socket()
socket:connect(host, port)
socket:set_timeout(15000)
socket:send(command)
repeat
response,data = socket:receive_buf("\n", true)
output = output..data
until data == nil
Basically, the last line of the data does not contain a "\n" character, so is never read from the socket. But this loop just hangs and never completes. I basically need it to return whenever the "\n" delimeter is not recognised. Does anyone know a way to do this?
Cheers
Updated
to include socket code
Update2
OK I have got around the initial problem of waiting for a "\n" character by using the "receive_bytes" method.
New code:
--socket set as above
repeat
data = nil
response,data = socket:receive_bytes(5000)
output = output..data
until data == nil
return output
This works and I get the large complete block of data back. But I need to reduce the buffer size from 5000 bytes, as this is used in a recursive function and memory usage could get very high. I'm still having problems with my "until" condition however, and if I reduce the buffer size to a size that will require the method to loop, it just hangs after one iteration.
Update3
I have gotten around this problem using string.match and receive_bytes. I take in at least 80 bytes at a time. Then string.match checks to see if the data variable conatins a certain pattern. If so it exits. Its not the cleanest solution, but it works for what I need it to do. Here is the code:
repeat
response,data = socket:receive_bytes(80)
output = output..data
until string.match(data, "pattern")
return output
I believe the only way to deal with this situation in a socket is to set a timeout.
The following link has a little bit of info, but it's on http socket: lua http socket timeout
There is also this one (9.4 - Non-Preemptive Multithreading): http://www.lua.org/pil/9.4.html
And this question: http://lua-list.2524044.n2.nabble.com/luasocket-howto-read-write-Non-blocking-TPC-socket-td5792021.html
A good discussion on Socket can be found on this link:
http://nitoprograms.blogspot.com/2009/04/tcpip-net-sockets-faq.html
It's .NET but the concepts are general.
See update 3. Because the last part of the data is always the same pattern, I can read in a block of bytes and each time check if that block has the pattern. If it has the pattern it will mean that it is the end of the data, append to the output variable and exit.