I am trying to flush a pipe handle obtained from open using either autoflush() and flush() methods from the IO::Handle module, but I think it is not working. Here is an example:
host.pl:
use feature qw(say);
use strict;
use warnings;
my $client_pid = open ( my $fh, '|-', 'client.pl' )
or die "Could not open client: $!";
#$fh->autoflush(1); # adding this line does not help
sleep 2;
say "Host: sent message";
print $fh "Hello";
#print $fh "Hello\n"; # adding a newline works fine
$fh->flush() or warn "$!"; # this does not work
sleep 2;
say "Host exits.";
close $fh;
client.pl:
use feature qw(say);
use strict;
use warnings;
say "Client running..";
chomp (my $line = <STDIN>);
say "Client got line: '$line'";
sleep 1;
say "Client exits..";
The output of running host.pl is:
Client running..
Host: sent message
Host exits.
Client got line: 'Hello'
Client exits..
Expected output would be:
Client running..
Host: sent message
Client got line: 'Hello'
Client exits..
Host exits.
I know I can fix this by adding a newline at the end of string to be printed:
print $fh "Hello\n";
but I am curious why $fh->flush() is not working here?
The data is being sent to the client immediately, but the client waits for a newline to arrive.
readline (for which <> is a shortcut in your program) reads until a newline is encountered before returning (although changing $/ can change that behaviour. If you want a call that returns as soon as data is available, use sysread.
use BLOCK_SIZE => 64*1024;
say "Client running..";
while (1) {
my $rv = sysread(\*STDIN, my $buf, BLOCK_SIZE);
die($!) if !defined($rv);
last if !$rv;
say "Got: $buf";
}
Note a single print can result in data being received in multiple chunks. In practice, especially with a socket instead of a pipe, you'd need some way of framing your messages in order to reliably identify them. For example, the following client expects sentinel-terminated messages (the sentinel being a newline):
use BLOCK_SIZE => 64*1024;
say "Client running..";
my $buf = '';
while (1) {
my $rv = sysread(\*STDIN, $buf, BLOCK_SIZE, length($buf));
die($!) if !defined($rv);
last if !$rv;
while ($buf =~ s/^([^\n]*)\n//) {
my $msg = $1;
say "Got: $msg";
}
say "Got a partial message" if length($buf);
}
die("Premature EOF\n") if length($buf);
Try sending:
$fh->autoflush();
print($fh "abc");
sleep(1);
print($fh "def\n");
sleep(1);
print($fh "ghi\njkl\nmno");
sleep(1);
print($fh "pqr\n");
This can be adapted to handle length-prefixed messages or any other message format.
Related
I have 2 scripts written in perl. First one takes a file and send it via socket to server. The server is my second script - and it saves to a file.
Server save file as a specified name - fixed in code. How to take the name of sending file, and send it to the server, before sending a file?
My code below:
Client:
my $socket = IO::Socket::INET->new(
PeerAddr => $local_host,
PeerPort => $local_port,
Proto => 'tcp',
)or die "Alert!";
my $bandwidth = 1024*5 ; # 5Kb/s -
open my $fh, '<', "$direc/$my_data"
or die "couldn't open the file";
my $buffer ;
while( sysread($fh, $buffer , $bandwidth) ) {
print $socket $buffer ;
sleep(1) ;
}
print "Data send.End \n" ;
close ($fh) ;
close($socket) ;
My server:
my $my_socket = new IO::Socket::INET(
LocalHost => $local_host,
LocalPort => $local_port,
Proto => 'tcp',
Listen => 5,
Reuse => 1
);
die "Couldn't open my_socket $!n " unless $my_socket;
print "You can send the data now \n";
my $accepter = $my_socket->accept();
my $count=0;
#print "$directory.$save_dir/$my_data";
open my $fh, '>', "$direc/$save_dir/$my_data" #my data is the name, and it's "fixed", how to take it from client?
or die "Couldn't open the file";
while(<$accepter>){
chomp;
last if $count++ ==10;
say $fh $_;
}
print "End \n";
close $fh;
close $my_socket;
Having the server write a filename specified by the client is a security risk. The client could tell the server to overwrite files, including the server itself.
Instead, use a UUID for the real filename. Store the client filename / real filename pair elsewhere.
You need to come up with a protocol so the server can distinguish between the filename and content. We could use an existing format such as JSON or YAML, but they require slurping the whole file into memory and encoding the content. You could make something up, like "the first line is the filename", but we can do a little better.
If you want to stream, we can use a stripped down HTTP protocol. Send headers as Key: Value lines. A blank line ends headers and begins sending content. For just a little extra effort we have a simple protocol that's extensible.
Here's the main loop of the server using UUID::Tiny and also autodie.
# Read Key: Value headers until we see a blank line.
my %headers;
while(my $line = <$accepter>) {
chomp $line;
last if $line eq "";
my($key, $value) = split /\s*:\s*/, $line;
$headers{$key} = $value;
}
# Store the file in a random filename. Do not use the client's filename
# to avoid a host of security issues.
my $filename = create_uuid_as_string(UUID_V4);
open my $fh, ">", "incoming/$filename";
# Read the content and save it to the file.
my $buf;
while( read($accepter, $buf, 1024) ) {
print $fh $buf;
}
say "$headers{Filename} was stored in incoming/$filename";
close $my_socket;
And the client simply sends a Filename header before sending the file's content.
open my $fh, '<', $filename;
print $socket "Filename: $filename\n\n";
my $buffer ;
while( sysread($fh, $buffer , $bandwidth) ) {
print $socket $buffer ;
}
I am trying to improve the warning message issued by Encode::decode(). Instead of printing the name of the module and the line number in the module, I would like it to print the name of the file being read and the line number in that file where the malformed data was found. To a developer, the origial message can be useful, but to an end user not familiar with Perl, it is probably quite meaningless. The end user would probably rather like to know which file is giving the problem.
I first tried to solve this using a $SIG{__WARN__} handler (which is probably not a good idea), but I get a segfault. Probably a silly mistake, but I could not figure it out:
#! /usr/bin/env perl
use feature qw(say);
use strict;
use warnings;
use Encode ();
binmode STDOUT, ':utf8';
binmode STDERR, ':utf8';
my $fn = 'test.txt';
write_test_file( $fn );
# Try to improve the Encode::FB_WARN fallback warning message :
#
# utf8 "\xE5" does not map to Unicode at <module_name> line xx
#
# Rather we would like the warning to print the filename and the line number:
#
# utf8 "\xE5" does not map to Unicode at line xx of file <filename>.
my $str = '';
open ( my $fh, "<:encoding(utf-8)", $fn ) or die "Could not open file '$fn': $!";
{
local $SIG{__WARN__} = sub { my_warn_handler( $fn, $_[0] ) };
$str = do { local $/; <$fh> };
}
close $fh;
say "Read string: '$str'";
sub my_warn_handler {
my ( $fn, $msg ) = #_;
if ( $msg =~ /\Qdoes not map to Unicode\E/ ) {
recover_line_number_and_char_pos( $fn, $msg );
}
else {
warn $msg;
}
}
sub recover_line_number_and_char_pos {
my ( $fn, $err_msg ) = #_;
chomp $err_msg;
$err_msg =~ s/(line \d+)\.$/$1/; # Remove period at end of sentence.
open ( $fh, "<:raw", $fn ) or die "Could not open file '$fn': $!";
my $raw_data = do { local $/; <$fh> };
close $fh;
my $str = Encode::decode( 'utf-8', $raw_data, Encode::FB_QUIET );
my ($header, $last_line) = $str =~ /^(.*\n)([^\n]*)$/s;
my $line_no = $str =~ tr/\n//;
++$line_no;
my $pos = ( length $last_line ) + 1;
warn "$err_msg, in file '$fn' (line: $line_no, pos: $pos)\n";
}
sub write_test_file {
my ( $fn ) = #_;
my $bytes = "Hello\nA\x{E5}\x{61}"; # 2 lines ending in iso 8859-1: åa
open ( my $fh, '>:raw', $fn ) or die "Could not open file '$fn': $!";
print $fh $bytes;
close $fh;
}
Output:
utf8 "\xE5" does not map to Unicode at ./p.pl line 27
, in file 'test.txt' (line: 2, pos: 2)
Segmentation fault (core dumped)
Here is another way to locate where the warning fires, with un-buffered sysread
use warnings;
use strict;
binmode STDOUT, ':utf8';
binmode STDERR, ':utf8';
my $file = 'test.txt';
open my $fh, "<:encoding(UTF-8)", $file or die "Can't open $file: $!";
$SIG{__WARN__} = sub { print "\t==> WARN: #_" };
my $char_cnt = 0;
my $char;
while (sysread($fh, $char, 1)) {
++$char_cnt;
print "$char ($char_cnt)\n";
}
The file test.txt was written by the posted program, except that I had to add to it to reproduce the behavior -- it runs without warnings on v5.10 and v5.16. I added \x{234234} to the end. The line number can be tracked with $char =~ /\n/.
The sysread returns undef on error. It can be moved into the body of while (1) to allow reads to continue and catch all warnings, breaking out on 0 (returned on EOF).
This prints
H (1)
e (2)
l (3)
l (4)
o (5)
(6)
A (7)
å (8)
a (9)
==> WARN: Code point 0x234234 is not Unicode, may not be portable at ...
(10)
While this does catch the character warned about, re-reading the file using Encode may well be better than reaching for sysread, in particular if sysread uses Encode.
However, Perl is utf8 internally and I am not sure that sysread needs Encode.
Note. The page for sysread supports its use on data with encoding layers
Note that if the filehandle has been marked as :utf8 Unicode
characters are read instead of bytes (the LENGTH, OFFSET, and the
return value of sysread are in Unicode characters). The
:encoding(...) layer implicitly introduces the :utf8 layer.
See binmode, open, and the open pragma.
Note Apparently, things have moved on and after a certain version sysread does not support encoding layers. The link above, while for an older version (v5.10 for one) indeed shows what is quoted, with a newer version tells us that there'll be an exception.
I have my connection working and data transfer working.... to an extent. I am designing a small program that sets up a client and server socket. Once connected, the client can send files to the server.
My problem is, when I begin to send my "test" file, the server never ends it's while loop. It keeps concatenating data into the output file. Even stranger, the data in the output file is correct, except there is extra white space between the lines.
I know this is because I am not chomping the \n character, and I add another \n character on the server. But if I chomp on the client, then everything is on one line. Therefore the server (regardless of it adding a newline character) outputs it all on one line, because it only received one line. If I chomp on the server side, I get an empty text file... which confuses me.
Furthermore, the server never stops concatenating... it makes an infinite loop even after the client disconnected. The terminal outputs this indefinitely:
Use of uninitialized value $data in concatenation (.) or string at ./tcp_server.pl line 51, <GEN2> line 14.
Here is my code for the server:
#!/usr/bin/perl
# Flushing to STDOUT after each write
$| = 1;
use warnings;
use strict;
use IO::Socket::INET;
use v5.10;
# Server side information
my $listen_port = '7070';
my $protocal = 'tcp';
# Finds IP address of host machine
# Connects to example site on HTTP
my $ip_finder = IO::Socket::INET->new(
PeerAddr=> "www.google.com",
PeerPort=> 80,
Proto => "tcp"
) or die "The IP can not be resolved: $!\n";
# The found IP address of Host
my $ip_address = $ip_finder->sockhost;
# Creating socket for server
my $server = IO::Socket::INET->new (
LocalPort => $listen_port,
Proto => $protocal,
Listen => 5,
Reuse => 1,
) or die "Socket could not be created, failed with error: $!\n"; # Prints error code
print "Socket created using IP: $ip_address\n";
print "Waiting for client connection on port $listen_port\n";
# Accept connection
my $client_socket = $server->accept();
open(my $fh, ">out")
or die "File can not be opened: $!";
while($client_socket) {
# Retrieve client information
my $client_address = $client_socket->peerhost();
my $client_port = $client_socket->peerport();
print "Client accepted: $client_address, $client_port\n";
my $data = <$client_socket>;
print $fh "$data\n";
}
close $fh;
$server->close();
and the client:
#!/usr/bin/perl
# Flushing to STDOUT after each write
$| = 1;
use warnings;
use strict;
use IO::Socket::INET;
use v5.10;
# Client side information
# Works by setting $dest to server address, needs to be on same domain
# my $dest = '<IP goes here>';
# my $dest = '<IP goes here>';
my $dest = '<host>.cselabs.umn.edu';
my $port = '7070';
my $protocal = 'tcp';
my $client = IO::Socket::INET->new (
PeerHost => $dest,
PeerPort => $port,
Proto => $protocol,
) or die "Socket could not be created, failed with error: $!\n"; # Prints error code
print "TCP connection established!\n";
open(my $fh, "<test")
or die "Can't open: $!";
while(my $line = <$fh>) {
print $client $line;
}
close $fh;
# sleep(10);
$client->close();
You don't add newline at the client. The client just reads a line from the file (including newline) and prints it to the socket, but does not add another newline. But at the server you read the line from the socket (including the newline) and add another newline. Also, the loop condition in the server is not correct. You should not check if the socket exists because it will exists even after the connection got closed. Instead you should check for error while reading (that's where you undefined warnings come from). And, it may be better not to read line by line but instead chunks independent from the lines, e.g.
while (read($client_socket, my $data, 8192)) {
print $fh $data;
}
I have this script to read a devices list and send a command. but currently it reads only the first device and sends it the command, ignoring the rest. What have I missed?
#!\usr\bin\Perl\bin\perl
use warnings;
use strict;
use NET::SSH2;
use MIME::Base64;
my $host = "C:/temp/devices.txt"; # input file
my $user = "XXX"; # your account
my $pass = "XXXXXX"; # your password 64 bit mime
my $ssh2 = Net::SSH2->new();
my $result = "C:/temp/result.txt"; # output file
$ssh2->debug(1); # debug on/off
open(List, '<', "$host") or die "$!";
while(<List>) {
chomp $_;
$ssh2->connect("$_") or die "Unable to connect host $# \n";
my $dp=decode_base64("$pass");
$ssh2->auth_password("$user","$dp");
my $chan = $ssh2->channel();
$chan->exec('sh run');
my $buflen =100000;
my $buf = '0' x $buflen;
my $read = $chan->read($buf, $buflen );
warn 'More than ', $buflen, ' characters in listing' if $read >= $buflen;
open OUTPUT, ">", "$result";
print OUTPUT "HOST: $_\n\n";
print OUTPUT "$buf\n";
print OUTPUT "\n\n\n";
print OUTPUT
close (List);
$chan->close();
}
You shouldn't be closing your List filehandle inside of your while loop. Move the close (List); line to after the close brace:
open(List, '<', "$host") or die "$!";
while(<List>) {
# ⋮
}
close (List);
close(List);
should be after the closing bracket.
You're closing your filehandle inside the while() loop. Move close(List) so it's outside of the while():
while(<List>) {
...
}
close(List);
Edit: I just noticed that you're also doing this within your while() loop:
open OUTPUT, ">", "$result";
This will cause your output file to be overwritten each time through the loop, so it will only have the last command's results in it. You could either move the open() / close() outside of the loop, or open the file in append mode:
open OUTPUT, '>>', $result;
You're also not checking to see if the open() succeeds; you should put or die $! at the end of your open() statement.
Right now, I read one character at a time in a loop, until I reach the \0 character. Is there a better way to do this?
Set your line ending to \x{00} (\0), be sure to localise it, and getline on the handle, like so:
{
local $/ = "\x{00}";
while (my $line = $sock->getline) {
print "$line\n"; # do whatever with your data here
}
}
You could use FIONREAD with ioctl. The program below connects to the SSH server on localhost and waits on its greeting:
#! /usr/bin/perl
use warnings;
use strict;
use subs 'FIONREAD';
require "sys/ioctl.ph";
use Socket;
socket my $s, PF_INET, SOCK_STREAM, getprotobyname "tcp"
or die "$0: socket: $!";
connect $s, sockaddr_in 22, inet_aton "localhost"
or die "$0: connect: $!";
my $rin = "";
vec($rin, fileno($s), 1) = 1;
my $nfound = select my$rout=$rin, "", "", undef;
die "$0: select: $!" if $nfound < 0;
if ($nfound) {
my $size = pack "L", 0;
ioctl $s, FIONREAD, $size
or die "$0: ioctl: $!";
print unpack("L", $size), "\n";
sysread $s, my $buf, unpack "L", $size
or die "$0: sysread: $!";
my $length = length $buf;
$buf =~ s/\r/\\r/g;
$buf =~ s/\n/\\n/g;
print "got: [$buf], length=$length\n";
}
Sample run:
$ ./howmuch
39
got: [SSH-2.0-OpenSSH_5.3p1 Debian-3ubuntu4\r\n], length=39
But you'll probably prefer using the IO::Socket::INET and IO::Select modules as in the code below that talks to Google:
#! /usr/bin/perl
use warnings;
use strict;
use subs "FIONREAD";
require "sys/ioctl.ph";
use IO::Select;
use IO::Socket::INET;
my $s = IO::Socket::INET->new(PeerAddr => "google.com:80")
or die "$0: can't connect: $#";
my $CRLF = "\015\012";
print $s "HEAD / HTTP/1.0$CRLF$CRLF" or warn "$0: print: $!";
my #ready = IO::Select->new($s)->can_read;
die "$0: umm..." unless $s == $ready[0];
my $size = pack "L", 0;
ioctl $s, FIONREAD, $size
or die "$0: ioctl: $!";
print unpack("L", $size), "\n";
sysread $s, my $buf, unpack "L", $size
or die "$0: sysread: $!";
my $length = length $buf;
$buf =~ s/\r/\\r/g;
$buf =~ s/\n/\\n/g;
print "got: [$buf], length=$length\n";
Output:
573
got: [HTTP/1.0 200 OK\r\nDate: Sun, 18 Jul 2010 12:03:48 GMT\r\nExpires: -1\r\nCache-Control: private, max-age=0\r\nContent-Type: text/html; charset=ISO-8859-1\r\nSet-Cookie: PREF=ID=6742ab80dd810a95:TM=1279454628:LM=1279454628:S=ewNg64020FbnGzHR; expires=Tue, 17-Jul-2012 12:03:48 GMT; path=/; domain=.google.com\r\nSet-Cookie: NID=36=kn2wtTD4UJ3MYYQ5uvA4iAsrS2wcrb_W781pZ1hrVUhUDHrIJTMg_kOgVKhjQnO5SM6MdC_jrRdxFRyXwyyv5N3Xja1ydhVLWWaYqpMHQOmGVi2K5qRWAKwDhCVRd8WS; expires=Mon, 17-Jan-2011 12:03:48 GMT; path=/; domain=.google.com; HttpOnly\r\nServer: gws\r\nX-XSS-Protection: 1; mode=block\r\n\r\n], length=573
What is the best way to receive data from a socket in Perl, when the data length is unknown?
A sound solution to this is impossible, in any language. If you don't know how long the data length is, then you can't possibly know when you've finished receiving all of it from the socket.
Your only hope is to use some kind of a metric to determine if it's been "long enough" since data started coming in, to make the decision that data flow has stopped. But it won't be perfect.
The answer depends on the protocol. Since your protocol uses '\0' as a separator, you're doing the right thing. I'm pretty sure Perl handles buffering for you, so reading one character at a time is not inefficient.
Many network oriented protocols precede strings with a length. To read a protocol like this, you read the length (usually one or two bytes, depending on the protocol spec), then read that many bytes into a string.
You can use sysread to read whatever data is available:
my $data;
my $max_length = 1000000;
sysread $sock, $data, $max_length;
Perl's read function waits for the full number of bytes that you requested, or EOF.
This is similar to libc stdio fread(3).
Perl's sysread function returns as soon as it receives any data.
This is similar to UNIX read(2).
Note that sysread bypasses buffered IO, so don't mix it with the buffered read.
Check perldoc -f read and perldoc -f sysread for more info.
For this specific question, it would be better to follow the top answer, and use getline with a line-ending of \0, but we can use sysread if there is no terminating character.
Here's a little example. It requests a web page, and prints the first chunk of data received.
#!/usr/bin/perl -w
use strict; use warnings;
use IO::Socket;
my $host = $ARGV[0] || 'google.com';
my $port = $ARGV[1] || 80;
my $sock = IO::Socket::INET->new(Proto => 'tcp', PeerAddr => $host, PeerPort => $port)
or die "connect failed: $!";
$sock->autoflush(1);
# use HTTP/1.1, which keeps the socket open by default
$sock->print("GET / HTTP/1.1\r\nHost: $host\r\n\r\n");
my $reply;
my $max_length = 1000000;
# $sock->read($reply, $max_length); # read would hang waiting for 1000000 bytes
my $count = $sock->sysread($reply, $max_length);
if (!defined $count) {
die "read failed: $!";
}
print $reply;