<> operator buffer size - perl

Question

I have a file with very long lines that I need to process and I found that the process gets stuck/'really slow' because the buffer is not big enough or due to the fact that handling a very long line might take a while. Here's a code sample:
open FH, "<$fname" or die "...";
while (<FH>) {
my #arr = split //, $_;
pop #arr;
pop #arr;
... for some "limited small portion of the string length" number of times ...
pop #arr;
if ($arr[-1] eq '0') {
print "done!\n";
last;
}
push #big_arr, join('', #arr);
}
The line processing is not "heavy".
I looked for something to solve it and came across PerlIO::buffersize but it looks like it wasn't maintained for a while now and I don't want to use a module with version 0.001. How can I modify the <> operator buffer size? Or alternatively, is there any way to know the line length before reading it with <>?

Answer 1

It may be that what you need is this:
$/ - can be set to a numeric value, for a number of bytes to read from a file.
Setting $/ to a reference to an integer, scalar containing an integer, or scalar that's convertible to an integer will attempt to read records instead of lines, with the maximum record size being the referenced integer number of characters.
Source: perlvar

Answer 2

How can I modify the <> operator buffer size?
<> reads into a scalar that can grow to any size, so I think you are referring to the size of the buffer passed to the read system call.
Before 5.14, Perl read from file handles in 4 KiB chunks. 5.14 made this configurable, with a default of 8 KiB.
$ perl -e'print("x" x 9_999, "\n") for 1..2' >large_lines
$ strace 5.10.1t/bin/perl -e'my $line = <>' large_lines 2>&1 | grep read.*xxx
read(3, "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"..., 4096) = 4096
read(3, "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"..., 4096) = 4096
read(3, "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"..., 4096) = 4096
$ strace 5.14.2t/bin/perl -e'my $line = <>' large_lines 2>&1 | grep read.*xxx
read(3, "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"..., 8192) = 8192
read(3, "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"..., 8192) = 8192
It can only be configured when perl is built, using the following command
./Configure -Accflags=-DPERLIOBUF_DEFAULT_BUFSIZ=8192
This applies to all buffered reading functions, including read, readline (for which <> is an alias), readpipe and eof, but not sysread.
Note that setting $/ to a reference to a number will cause readline (<>) to act as read, which is still buffered.
$ strace perl -e'$/ = \8193; my $block = <>' large_lines 2>&1 | grep read.*xxx
read(3, "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"..., 8192) = 8192
read(3, "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"..., 8192) = 8192
If you actually want to perform a single read system call, you need to use sysread.
$ strace perl -e'sysread(STDIN, $buf, 8193)' <large_lines 2>&1 | grep read.*xxx
read(0, "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"..., 8193) = 8193

Answer 3

Altering Perl's read buffer size is unlikely to make any significant difference to the speed of your program, and the impact you are seeing is much more likely to be a result of the longer read time from the disk drive itself. Take a look at Perl Read-Ahead I/O Buffering on perlmonks.org
Furthermore, implementing your own buffering by using read or setting the record separator $/ to a fixed size, is more than likely to slow down your program, as you still have to separate what you have read into lines of data but now have to do it in Perl code instead of letting perl do it for you in C
Note also that the measure of changing $/ to a fixed record size will still use Perl's standard, probably 8KB, buffer. The only difference is that the amount of data handed back to you will be determined according to a byte count instead of the position of a separator string