Read and write a text file - perl

How to read and write a text file and put it in the same file? I tried but I got empty files.
My code is :
#!/opt/lampp/bin/perl
print "Content-Type : text/html\n\n";
$ref = '/opt/lampp/htdocs/perl/Filehandling/data_1.txt';
open(REF, "+<", $ref) or die "Uable to open the file $ref: $!";
while (<REF>) {
if($_=~ m/Others/gi) {
$_ =~ s/Others/Other/gi;
print "$_\n";
}
}
$ref = '/opt/lampp/htdocs/perl/Filehandling/data_1.txt';
open(REF, "+>", $ref) or die "Uable to open the file $ref: $!";
print REF "$_\n";
close REF;
close REF;

perl -pi -e 's/Others/Other/gi' /opt/lampp/htdocs/perl/Filehandling/data_1.txt
The -i option edits a file in-place, i.e. overwrites the file with the new contents.
The s/// operator doesn't substitute anything if there is no match, so you don't have to check with m// if you have a match before attempting a substitution.
The flow control in your script is flawed; you open for reading, then print to standard output (not back to the file), then open for writing, but don't write anything useful (or, well, only the last line you read in the input loop).
What's with the Content-Type: header, do you want to run this as a CGI script? That sounds like a security problem.
Do you only want to print if there is a substitution? If so, try this:
perl -ni -e 's/Others/Other/ig and print' /opt/lampp/htdocs/perl/Filehandling/data_1.txt
Notice the change from -p which adds a read-and-print loop around your script, to -n which only reads, but doesn't print your file.

It seems that all you want is to s/Others/Other/gi in your data_1.txt.
You can make it a lot of easier with one line: perl -pi.bak -e's/Others/Other/gi' data_1.txt.
If you still want to read and write into the same file you can check the IO::InSitu module at CPAN.

Fᴍᴛᴇʏᴇᴡᴛᴋ: Learning How to Learn How to Update a Text File in Place
If only you were to Rᴇᴄᴏɴɴᴏɪᴛʀᴇ Tʜᴇ Fᴀʙᴜʟᴏᴜs perlfaq5 Mᴀɴᴘᴀɢᴇ — which is available on every Perl installation worldwide including your own — you would easily learn the answer to your question under that Fᴏʀᴍɪᴅᴀʙʟʏ Aᴘᴏᴛʜᴇᴏsɪᴢᴇᴅ Qᴜᴀɴᴅᴀʀʏ entitled “How do I change, delete, or insert a line in a file, or append to the beginning of a file?”.
Somehow that key first step to problem solving failed to execute.
Learning how to efficiently search the standard Perl documentation set is one of the three keys steps in becoming a proficient Perl programmer. (For the record, the other two are to have a rolodex full of working examples and to have a particular itch you need scratched which you can use Perl to scratch it with.)
If you for whatever reason you either cannot or will not search the standard documentation, and so find yourself coming online begging help in doing something that was long ago answered in that documentation set, you will never be self‐enabling. You may also be perceived as something as a pest.
The Recipe Rolodex 📖
Looking beyond the standard documentation included with every Perl installation whose contents you should have searched before coming here, we find that O’Reilly’s Perl Cookbook chapter 7 on “File Access” and chapter 8 on “File Contents” contain several recipes particularly relevant to your current question, especially but not only those marked in bold below. The Perl Cookbook is a compendium of example code meant as the paired companion volume to Programming Perl. Arguably, the Cookbook is even more useful for the beginning Perl programmer than the Camel itself.
However, you can’t grep dead trees. So I will show you what you should look for, and how to grep it no matter whether you happen to possess the ungreppable dead‐tree version of that particular primer or not.
NB: While I sure that the publisher would be delighted if you were to purchase this essential Perl tome, whether as a dead tree or electronically, I recognize that this is not possible for everyone, so I below will show you how to access the relevant portions completely free of charge. In fact, I include them for your convenience.
First, the table of contents for chapter 7 of the Perl Cookbook:
Recipe 7.0: Introduction to File Access
Recipe 7.1: Opening a File
Recipe 7.2: Opening Files with Unusual Filenames
Recipe 7.3: Expanding Tildes in Filenames
Recipe 7.4: Making Perl Report Filenames in Error Messages
Recipe 7.5: Storing Filehandles into Variables
Recipe 7.6: Writing a Subroutine That Takes Filehandles as Built-ins Do
Recipe 7.7: Caching Open Output Filehandles
Recipe 7.8: Printing to Many Filehandles Simultaneously
Recipe 7.9: Opening and Closing File Descriptors by Number
Recipe 7.10: Copying Filehandles
Recipe 7.11: Creating Temporary Files
Recipe 7.12: Storing a File Inside Your Program Text
Recipe 7.13: Storing Multiple Files in the DATA Area
Recipe 7.14: Writing a Unix-Style Filter Program
☞ Recipe 7.15: Modifying a File in Place with Temporary File
☞ Recipe 7.16: Modifying a File in Place with -i Switch
☞ Recipe 7.17: Modifying a File in Place Without a Temporary File
Recipe 7.18: Locking a File
Recipe 7.19: Flushing Output
Recipe 7.20: Doing Non‐Blocking I/O
Recipe 7.21: Determining the Number of Unread Bytes
Recipe 7.22: Reading from Many Filehandles Without Blocking
Recipe 7.23: Reading an Entire Line Without Blocking
Recipe 7.24: Program: netlock
Recipe 7.25: Program: lockarea
And here is Chapter 8’s ᴛᴏᴄ:
☞ Recipe 8.0: Introduction to File Contents
Recipe 8.1: Reading Lines with Continuation Characters
Recipe 8.2: Counting Lines (or Paragraphs or Records) in a File
Recipe 8.3: Processing Every Word in a File
☞ Recipe 8.4: Reading a File Backwards by Line or Paragraph
Recipe 8.5: Trailing a Growing File
Recipe 8.6: Picking a Random Line from a File
Recipe 8.7: Randomizing All Lines
☞ Recipe 8.8: Reading a Particular Line in a File
Recipe 8.9: Processing Variable‐Length Text Fields
☞ Recipe 8.10: Removing the Last Line of a File
Recipe 8.11: Processing Binary Files
Recipe 8.12: Using Random‐Access I/O
☞ Recipe 8.13: Updating a Random‐Access File
Recipe 8.14: Reading a String from a Binary File
Recipe 8.15: Reading Fixed‐Length Records
Recipe 8.16: Reading Configuration Files
Recipe 8.17: Testing a File for Trustworthiness
Recipe 8.18: Treating a File as an Array
Recipe 8.19: Setting the Default I/O Layers
Recipe 8.20: Reading or Writing Unicode from a Filehandle
Recipe 8.21: Converting Microsoft®‐Proprietary Text Files into Standard Unicode
Recipe 8.22: Comparing the Contents of Two Files
Recipe 8.23: Pretending a String Is a File
Recipe 8.24: Program: tailwtmp
Recipe 8.25: Program: tctee
Recipe 8.26: Program: laston
Recipe 8.27: Program: Flat file indexes
Muddled Mental Models 😞
Chapter 8’s section 8.0 “Introduction to File Contents” is especially poignant, so much so that its copyrighted text I here reproduce by kind permission of its author:
Treating files as unstructured streams necessarily governs what you can do with them. You can read and write sequential, fixed‐size blocks of data at any location in the file, increasing its size if you write past the current end. Perl uses an I/O library that emulates C’s stdio(3) to implement reading and writing of variable‐length records like lines, paragraphs, and words.
What can’t you do to an unstructured file? Because you can’t insert or delete bytes anywhere but at end of file, you can't change easily the length of, insert, or delete records. An exception is the last record, which you can delete by truncating the file to the end of the previous record. For other modifications, you need to use a temporary file or work with a copy of the file in memory. If you need to do this a lot, a database system may be a better solution than a raw file. Standard with Perl as of v5.8 is the Tie::File module, which offers an array interface to files of records.
The last reference may be the most useful, because it offers a model of a text file that may better accord with the non‐programmer’s notion of what a text file is. To the operating system, a file is merely an ordered set of bytes, and the only operations that you can use on a file are read, write, seek, and truncate. There is on insert, there is no go‐to‐line‐number‐N, and there certainly is no search and replace. The operating system model is more that of a paper tape reader than it is a deck of cards. You can no more insert new data in the middle of a file than you can wedge a new sector between existing ones on a hard disk. They don’t call it a hard disk for nothing, you know.
The problem is that the non‐programmer has no experience with the operating system’s model of a fixed file filled with bytes. Her only model for a text file is one whose operations match that of the text editor she is used to, one which in these post–pre‐Internet [sic] days of computing often more resembles a child’s video game than it does a serious tool for getting real work done.
Whether you have a video‐game version or a power‐tool version, all text editors are almost always line based, and allow one to move lines around, search by line, change lines including their lengths, insert into the middle of a file or delete from the middle shortening everything up. Even her notion of a character is very different from the operating system, since a user‐visible grapheme can easily comprise multiple programmer-visible code points, and each of these code points can easily comprise multiple operating‐system–visible code units.
This non‐programmer model does not work with real operating system files. The Tie::File module provides an abstraction layer that can help present a higher level and perhaps more non‐programmer–friendly model of a text file.
ᴘʟᴇᴀᴄ: The Programming Language Examples Alike Cookbook
Have you ever wanted a Rosetta Stone for programming languages? If so, then today is your lucky day. 😹
Although the published Perl Cookbook’s complete code is readly available for free and easy download, you might also find interesting the ᴘʟᴇᴀᴄ project, the multilingual Programming Language Examples Alike Cookbook. It includes not just the full Perl Cookbook code, but also translations in varying states of completeness into other popular programming languages like Ruby and Python, old languages like Rexx and TCL, nascent languages like Go and Groovy, and exotic languages like Haskell and OCaml (no relation to 🐪 :).
I quote below from the ᴘʟᴇᴀᴄ project’s source for the three relevant recipes from the Perl Cookbook.
Recipe 7.15: Modifying a File in Place with a Temporary File
# ^^PLEAC^^_7.8
#-----------------------------
open(OLD, "< $old") or die "can't open $old: $!";
open(NEW, "> $new") or die "can't open $new: $!";
while (<OLD>) {
# change $_, then...
print NEW $_ or die "can't write $new: $!";
}
close(OLD) or die "can't close $old: $!";
close(NEW) or die "can't close $new: $!";
rename($old, "$old.orig") or die "can't rename $old to $old.orig: $!";
rename($new, $old) or die "can't rename $new to $old: $!";
#-----------------------------
while (<OLD>) {
if ($. == 20) {
print NEW "Extra line 1\n";
print NEW "Extra line 2\n";
}
print NEW $_;
}
#-----------------------------
while (<OLD>) {
next if 20 .. 30;
print NEW $_;
}
#-----------------------------
Recipe 7.16: Modifying a File in Place with the -i Switch
# ^^PLEAC^^_7.9
#-----------------------------
#% perl -i.orig -p -e 'FILTER COMMAND' file1 file2 file3 ...
#-----------------------------
#!/usr/bin/perl -i.orig -p
# filter commands go here
#-----------------------------
#% perl -pi.orig -e 's/DATE/localtime/e'
#-----------------------------
while (<>) {
if ($ARGV ne $oldargv) { # are we at the next file?
rename($ARGV, $ARGV . '.orig');
open(ARGVOUT, ">$ARGV"); # plus error check
select(ARGVOUT);
$oldargv = $ARGV;
}
s/DATE/localtime/e;
}
continue{
print;
}
select (STDOUT); # restore default output
#-----------------------------
#Dear Sir/Madam/Ravenous Beast,
# As of DATE, our records show your account
#is overdue. Please settle by the end of the month.
#Yours in cheerful usury,
# --A. Moneylender
#-----------------------------
#Dear Sir/Madam/Ravenous Beast,
# As of Sat Apr 25 12:28:33 1998, our records show your account
#is overdue. Please settle by the end of the month.
#Yours in cheerful usury,
# --A. Moneylender
#-----------------------------
#% perl -i.old -pe 's{\bhisvar\b}{hervar}g' *.[Cchy]
#-----------------------------
# set up to iterate over the *.c files in the current directory,
# editing in place and saving the old file with a .orig extension
local $^I = '.orig'; # emulate -i.orig
local #ARGV = glob("*.c"); # initialize list of files
while (<>) {
if ($. == 1) {
print "This line should appear at the top of each file\n";
}
s/\b(p)earl\b/${1}erl/ig; # Correct typos, preserving case
print;
} continue {close ARGV if eof}
#-----------------------------
Recipe 7.17: Modifying a File in Place Without a Temporary File
# ^^PLEAC^^_7.10
#-----------------------------
open(FH, "+< FILE") or die "Opening: $!";
#ARRAY = <FH>;
# change ARRAY here
seek(FH,0,0) or die "Seeking: $!";
print FH #ARRAY or die "Printing: $!";
truncate(FH,tell(FH)) or die "Truncating: $!";
close(FH) or die "Closing: $!";
#-----------------------------
open(F, "+< $infile") or die "can't read $infile: $!";
$out = '';
while (<F>) {
s/DATE/localtime/eg;
$out .= $_;
}
seek(F, 0, 0) or die "can't seek to start of $infile: $!";
print F $out or die "can't print to $infile: $!";
truncate(F, tell(F)) or die "can't truncate $infile: $!";
close(F) or die "can't close $infile: $!";
#----------------------------
Conclusion
There. If that doesn’t point the way to answering your question, I don’t know what will. But I bet it did. Congrats! 👏

Related

How to pipe to and read from the same tempfile handle without race conditions?

Was debugging a perl script for the first time in my life and came over this:
$my_temp_file = File::Temp->tmpnam();
system("cmd $blah | cmd2 > $my_temp_file");
open(FIL, "$my_temp_file");
...
unlink $my_temp_file;
This works pretty much like I want, except the obvious race conditions in lines 1-3. Even if using proper tempfile() there is no way (I can think of) to ensure that the file streamed to at line 2 is the same opened at line 3. One solution might be pipes, but the errors during cmd might occur late because of limited pipe buffering, and that would complicate my error handling (I think).
How do I:
Write all output from cmd $blah | cmd2 into a tempfile opened file handle?
Read the output without re-opening the file (risking race condition)?
You can open a pipe to a command and read its contents directly with no intermediate file:
open my $fh, '-|', 'cmd', $blah;
while( <$fh> ) {
...
}
With short output, backticks might do the job, although in this case you have to be more careful to scrub the inputs so they aren't misinterpreted by the shell:
my $output = `cmd $blah`;
There are various modules on CPAN that handle this sort of thing, too.
Some comments on temporary files
The comments mentioned race conditions, so I thought I'd write a few things for those wondering what people are talking about.
In the original code, Andreas uses File::Temp, a module from the Perl Standard Library. However, they use the tmpnam POSIX-like call, which has this caveat in the docs:
Implementations of mktemp(), tmpnam(), and tempnam() are provided, but should be used with caution since they return only a filename that was valid when function was called, so cannot guarantee that the file will not exist by the time the caller opens the filename.
This is discouraged and was removed for Perl v5.22's POSIX.
That is, you get back the name of a file that does not exist yet. After you get the name, you don't know if that filename was made by another program. And, that unlink later can cause problems for one of the programs.
The "race condition" comes in when two programs that probably don't know about each other try to do the same thing as roughly the same time. Your program tries to make a temporary file named "foo", and so does some other program. They both might see at the same time that a file named "foo" does not exist, then try to create it. They both might succeed, and as they both write to it, they might interleave or overwrite the other's output. Then, one of those programs think it is done and calls unlink. Now the other program wonders what happened.
In the malicious exploit case, some bad actor knows a temporary file will show up, so it recognizes a new file and gets in there to read or write data.
But this can also happen within the same program. Two or more versions of the same program run at the same time and try to do the same thing. With randomized filenames, it is probably exceedingly rare that two running programs will choose the same name at the same time. However, we don't care how rare something is; we care how devastating the consequences are should it happen. And, rare is much more frequent than never.
File::Temp
Knowing all that, File::Temp handles the details of ensuring that you get a filehandle:
my( $fh, $name ) = File::Temp->tempfile;
This uses a default template to create the name. When the filehandle goes out of scope, File::Temp also cleans up the mess.
{
my( $fh, $name ) = File::Temp->tempfile;
print $fh ...;
...;
} # file cleaned up
Some systems might automatically clean up temp files, although I haven't care about that in years. Typically is was a batch thing (say once a week).
I often go one step further by giving my temporary filenames a template, where the Xs are literal characters the module recognizes and fills in with randomized characters:
my( $name, $fh ) = File::Temp->tempfile(
sprintf "$0-%d-XXXXXX", time );
I'm often doing this while I'm developing things so I can watch the program make the files (and in which order) and see what's in them. In production I probably want to obscure the source program name ($0) and the time; I don't want to make it easier to guess who's making which file.
A scratchpad
I can also open a temporary file with open by not giving it a filename. This is useful when you want to collect outside the program. Opening it read-write means you can output some stuff then move around that file (we show a fixed-length record example in Learning Perl):
open(my $tmp, "+>", undef) or die ...
print $tmp "Some stuff\n";
seek $tmp, 0, 0;
my $line = <$tmp>;
File::Temp opens the temp file in O_RDWR mode so all you have to do is use that one file handle for both reading and writing, even from external programs. The returned file handle is overloaded so that it stringifies to the temp file name so you can pass that to the external program. If that is dangerous for your purpose you can get the fileno() and redirect to /dev/fd/<fileno> instead.
All you have to do is mind your seeks and tells. :-) Just remember to always set autoflush!
use File::Temp;
use Data::Dump;
$fh = File::Temp->new;
$fh->autoflush;
system "ls /tmp/*.txt >> $fh" and die $!;
#lines = <$fh>;
printf "%s\n\n", Data::Dump::pp(\#lines);
print $fh "How now brown cow\n";
seek $fh, 0, 0 or die $!;
#lines2 = <$fh>;
printf "%s\n", Data::Dump::pp(\#lines2);
Which prints
[
"/tmp/cpan_htmlconvert_DPzx.txt\n",
"/tmp/cpan_htmlconvert_DunL.txt\n",
"/tmp/cpan_install_HfUe.txt\n",
"/tmp/cpan_install_XbD6.txt\n",
"/tmp/cpan_install_yzs9.txt\n",
]
[
"/tmp/cpan_htmlconvert_DPzx.txt\n",
"/tmp/cpan_htmlconvert_DunL.txt\n",
"/tmp/cpan_install_HfUe.txt\n",
"/tmp/cpan_install_XbD6.txt\n",
"/tmp/cpan_install_yzs9.txt\n",
"How now brown cow\n",
]
HTH

Splitting one txt file into multiple txt files based on delimiter and naming them with a specific character

I have a text file that looks like this http://www.uniprot.org/uniprot/?sort=score&desc=&compress=no&query=id:P01375%20OR%20id:P04626%20OR%20id:P08238%20OR%20id:P06213&format=txt.
This file is contained of different entries that are divided with //. I think I have almost found the way how to divide txt file into multiple txt files whenever this specific pattern appears, but I still don't know how to name them after dividing and how to print them in specific directory. I would like that each file that is divided carries specific ID which is a first line nut second column in each entry.
This is the code that I have wrote so far:
mkdir "spliced_files"; #directory where I would like to put all my splitted files
$/="//\n"; # divide them whenever //appears and new line after this
open (IDS, 'example.txt') or die "Cannot open"; #example.txt is an input file
my #ids = <IDS>;
close IDS;
my $entry = 25444; #number of entries or //\n characters
my $i=0;
while ($i eq $entry) {
print $ids[$i];
};
$i++;
I am still having problem with finding how to split all entries from 'example.txt' file whenever "//\n" and to print all this seperated files into directory spliced_files. In addition I would have to name all of these seperated files with the ID that is specific for each of these files or entries (which appears in the first row, but only a second column).
So I expect output to be number of files in spliced_files directory, and each of them are named with their ID (first row, but only second column). For example name of the first file wiould be TNFA_HUMAN, od the second would be ERBB2_HUMAN and so on..)
You still look like you're programming by guesswork. And you haven't made use of any of the advice you have been given in answers to your previous questions. I strongly recommend that you spend a week working through a good beginners book like Learning Perl and come back when you understand more about how Perl works.
But here are some comments on your new code:
open (IDS, 'example.txt') or die "Cannot open";
You have been told that using lexical variables and the three-arg version of open() is a better approach here. You should also include $! in your error message, so you know what has gone wrong.
open my $ids_fh, '<', 'example.txt'
or die "Cannot open: $!";
Then later on (I added the indentation in the while loop to make things clearer)...
my $i=0;
while ($i eq $entry) {
print $ids[$i];
};
$i++;
The first time you enter this loop, $i is 1 and $entry is 25444. You compare them (as strings! You probably want ==, not eq) to see if they are equal. Clearly they are different, so your while loop exits. Once the loop exits, you increment $i.
This code bears no relation at all to the description of your problem. I'm not going to give you the answer, but here is the structure of what you need to do:
mkdir "spliced_files";
local $/ = "//\n"; # Always localise changes to special vars
open my $ids_fh, '<', 'example.txt'
or die "Cannot open: $!";
# No need to read the whole file in one go.
# Process it a line at a time.
while (<$ids_fh>) {
# Your record (the whole thing, not just the first line) is in $_.
# You need to extract the ID value from that string. Let's assume
# you've stored in it $id
# Open a file with the right name
open my $out_fh, '>', "spliced_files/$id" or die $!;
# Print the record to the new file.
print $out_fh $_;
}
But really, you need to take the time to learn about programming before you attack this task. Or, if you don't have the time for that, pay a programmer to do it for you.

Perl: Substitute text string with value from list (text file or scalar context)

I am a perl novice, but have read the "Learning Perl" by Schwartz, foy and Phoenix and have a weak understanding of the language. I am still struggling, even after using the book and the web.
My goal is to be able to do the following:
Search a specific folder (current folder) and grab filenames with full path. Save filenames with complete path and current foldername.
Open a template file and insert the filenames with full path at a specific location (e.g. using substitution) as well as current foldername (in another location in the same text file, I have not gotten this far yet).
Save the new modified file to a new file in a specific location (current folder).
I have many files/folders that I want to process and plan to copy the perl program to each of these folders so the perl program can make new .
I have gotten so far ...:
use strict;
use warnings;
use Cwd;
use File::Spec;
use File::Basename;
my $current_dir = getcwd;
open SECONTROL_TEMPLATE, '<secontrol_template.txt' or die "Can't open SECONTROL_TEMPLATE: $!\n";
my #secontrol_template = <SECONTROL_TEMPLATE>;
close SECONTROL_TEMPLATE;
opendir(DIR, $current_dir) or die $!;
my #seq_files = grep {
/gz/
} readdir (DIR);
open FASTQFILENAMES, '> fastqfilenames.txt' or die "Can't open fastqfilenames.txt: $!\n";
my #fastqfiles;
foreach (#seq_files) {
$_ = File::Spec->catfile($current_dir, $_);
push(#fastqfiles,$_);
}
print FASTQFILENAMES #fastqfiles;
open (my ($fastqfilenames), "<", "fastqfilenames.txt") or die "Can't open fastqfilenames.txt: $!\n";
my #secontrol;
foreach (#secontrol_template) {
$_ =~ s/#/$fastqfilenames/eg;
push(#secontrol,$_);
}
open SECONTROL, '> secontrol.txt' or die "Can't open SECONTROL: $!\n";
print SECONTROL #secontrol;
close SECONTROL;
close FASTQFILENAMES;
My problem is that I cannot figure out how to use my list of files to replace the "#" in my template text file:
my #secontrol;
foreach (#secontrol_template) {
$_ =~ s/#/$fastqfilenames/eg;
push(#secontrol,$_);
}
The substitute function will not replace the "#" with the list of files listed in $fastqfilenames. I get the "#" replaced with GLOB(0x8ab1dc).
Am I doing this the wrong way? Should I not use substitute as this can not be done, and then rather insert the list of files ($fastqfilenames) in the template.txt file? Instead of the $fastqfilenames, can I substitute with content of file (e.g. s/A/{r file.txt ...). Any suggestions?
Cheers,
JamesT
EDIT:
This made it all better.
foreach (#secontrol_template) {
s/#/$fastqfilenames/g;
push #secontrol, $_;
}
And as both suggestions, the $fastqfiles is a filehandle.
replaced this: open (my ($fastqfilenames), "<", "fastqfilenames.txt") or die "Can't open fastqfilenames.txt: $!\n";
with this:
my $fastqfilenames = join "\n", #fastqfiles;
made it all good. Thanks both of you.
$fastqfilenames is a filehandle. You have to read the information out of the filehandle before you can use it.
However, you have other problems.
You are printing all of the filenames to a file, then reading them back out of the file. This is not only a questionable design (why read from the file again, since you already have what you need in an array?), it also won't even work:
Perl buffers file I/O for performance reasons. The lines you have written to the file may not actually be there yet, because Perl is waiting until it has a large chunk of data saved up, to write it all at once.
You can override this buffering behavior in a few different ways (closing the file handle being the simplest if you are done writing to it), but as I said, there is no reason to reopen the file again and read from it anyway.
Also note, the /e option in a regex replacement evaluates the replacement as Perl code. This is not necessary in your case, so you should remove it.
Solution: Instead of reopening the file and reading it, just use the #fastqfiles variable you previously created when replacing in the template. It is not clear exactly what you mean by replacing # with the filenames.
Do you want to to replace each # with a list of all filenames together? If so, you should probably need to join the filenames together in some way before doing the replacement.
Do you want to create a separate version of the template file for each filename? If so, you need an inner for loop that goes over each filename for each template. And you will need something other than a simple replacement, because the replacement will change the original string on the first time through. If you are on Perl 5.16, you could use the /r option to replace non-destructively: push(#secontrol,s/#/$file_name/gr); Otherwise, you should copy to another variable before doing the replacement.
$_ =~ s/#/$fastqfilenames/eg;
$fastqfilenames is a file handle, not the file contents.
In any case, I recommend the use of Text::Template module in order to do this kind of work (file text substitution).

Line breaks don't exist on input from FTP file (Perl)

I downloaded a csv file using Net::FTP. When I look at this file in text editor or excel or even when I cut/paste it has line breaks and looks like this:
000000000G911|06
0000000000CDR|25|123
0000000000EGP|19
When I read the file in Perl it sees the entire text as one line like this:
000000000G911|060000000000CDR|25|1230000000000EGP|19
I have tried reading it using
tie #lines, 'Tie::File', "C:/Programs/myfile.csv", autochomp=>0 or die "Can't read file: $!\n";
foreach $l (#lines1)
{print "$l\n";
}
and
open FILE, "`<`$filename" or die $!;
my #lines=`<`FILE>;
foreach $l (#lines)
{print "$l\n";
}
close FILE;
The file has line breaks in a format that Perl is not recognizing because it is coming from a different operating system. The other programs are automatically detecting the different line break format, but Perl doesn't do that.
If you have Net::FTP perform the transfer in ASCII mode (e.g. $ftp->ascii to enable this mode), this should be taken care of and corrected for you.
Alternatively, you can figure out what is being used for line breaks and then set the special $/ variable to that value.

Can someone explain this Perl code snippet?

This little piece of code has been a staple in a bunch of my scripts, but I took the syntax from another working script that someone else wrote and adapted it to fit my needs. I'm not even sure that the syntax used here is the best or most common way to open a file handler either.
The code is:
$fh = \*STAT_FILE;
open ($fh,">>".$stat_file) or die "Can't open $stat_file: $!\n";
my $print_flag = ( -z $stat_file );
I don't fully understand the first line and also the last line of the code above. Specifically, \*STAT_FILE and -z, respectively.
I know that, for the most part, the second line will open a file for appending or quit and throw an error. But again, I don't understand what purpose the $! serves in that line either.
Can someone explain this Perl code, line-by-line, to me in pseudo? Also, if the method above is not the preferred method, then what is?
Thanks in advance
Before perl 5.6, file handles could only be globs (bare words) or references to globs (which is what \*STAT_FILE is). Also, it's better to use 3-argument open (See the docs. Also see perlopentut). So you can now do:
open(my $fh, ">>", $stat_file) or die "Failed to open $stat_file: $!";
and forget about \*STAT_FILE.
-z is one of the file test functions (and takes a file name or file handle as an argument) and tests to see if the file has zero size.
$! is one of the Special Variables and contains the most recent system error message (in this case why you can not open the file, perhaps permission issues, or a directory in the path to the file does not exist, etc.).
You should learn to use perldoc, all of this is in perldoc:
perldoc perlfunc (specifically perldoc -f open and perldoc -f -X)
perldoc perlvar
The first row assign to the variable a reference (the backslash sign) to the typeglob (a fullsymbol table entry) STAT_FILE. This has been a quite idiomatic perl construct to pass filehandles as reported, just to name it, in the Larry Wall "Programming perl". The $! variable contains the error message reurned by the operating system.
So the whole meaning is:
line 1. put in the $fh variable a filehandle;
line 2. Open for append the file reporting the system message error should a fault happens;
line 3. Set a flag variable warning if the file has zero length