HI all,
i need to have this command line command executed from a Perl file:
for file in *.jpg ; do mv $file `echo $file | sed 's/\(.*\.\)jpg/\1jp4/'` ; done
So I added the folders and tried:
system "bash -c 'for file in $mov0to/*.jp4 ; do mv $file `echo $basedir/internal/0/$file | sed 's/\(.*\.\)jp4/\1jpg/'` ; done'";
But all I get is:
sh: Syntax error: "(" unexpected
No file specified
I am on Kubuntu 10.4
Thanks,
Jan
I can think of many better ways of doing this, but ideally you want pure Perl:
use File::Copy qw( move );
opendir DIR, $mov0to or die "Unable to open $mov0to: $!";
foreach my $file ( readdir DIR ) {
my $out = $file;
next unless $out =~ s/\.jp4$/.jpg/;
move "$mov0to/$file", "$basedir/internal/0/$out" or die "Unable to move $mov0to/$file->$basedir/internal/0/$out: $!";
}
closedir DIR;
If you insist on doing the lifting in Bash, you should be doing:
system qq(bash -c 'for file in $mov0to/*.jp4 ; do mv \$file $basedir/internal/0/\${file\/%.jp4\/.jpg} ; done');
All of the variables inside the double quotes will get interpolated, the $file that you're using in the bash for loop in particular. We don't have enough context to know where the other variables ($mov0to and $basedir) come from or what they contain so they might be having the same problems. The double quotes are also eating your backslashes in the sed part.
Andy White is right, you'd have less of a mess if the bash commands were in a separate script.
It's most likely a problem with nested quotes. I would recommend either moving that bash command into its own (bash) script file, or writing the loop/logic directly in Perl code, making simpler system calls if you need to.
Related
I want to make the same calculations in two similar files, but I do not want to double the code for each file nor to create two scripts for this.
my $file = "file1.txt";
my $tempfile = "file1_temp.txt";
if (not defined $file) {
die "Input file not found";
}
open(my $inputFileHandler, '<:encoding(UTF-8)', $file)
or die "Could not open file '$file' $!";
open(my $outs, '>', $tempfile) or die $!;
/*Calculations made*/
close($outs);
copy($tempfile,$file) or die "Copy failed: $!";
unlink($tempfile) or die "Could not delete the file!\n";
close($inputFileHandler);
So i want to do the exact calculations for file2.txt_temp and copy it in file2.txt is there a way to do it without writing the code again for file2?
Thank you very much.
Write your code as a Unix filter. Read the data from STDIN and write it to STDOUT. Your code will be simpler and your program will be more flexible.
#!/usr/bin/perl
use strict;
use warnings;
while (<STDIN>) {
# do a calculation using the data that is in $_
print $output_data
}
The cleverness is in how you call the program:
$ ./my_clever_filter < file1.txt > file1_out.txt
$ ./my_clever_filter < file2.txt > file2_out.txt
See The Unix Filter Model: What, Why and How? for far more information.
Assuming your code is well written (not manipulating any globals, ...) you could use a for-loop
foreach my $prefix ('file1', 'file2') {
my $file = $prefix . ".txt";
my $tempfile = $prefix . "_temp.txt";
...
}
There is a certain Perl feature that is designed especially for cases like this, and that is this:
$ perl -pi -e'/*Calculations made*/' file1.txt file2.txt ... fileN.txt
Loosely referred to as "in-place edit", which basically does what your code does: It writes to a temp file and then overwrites the original.
Which will apply your calculations to the files named as arguments. If you have complex calculations you can put them in a file and skip the -e'....' part
$ perl -pi foo.pl file1.txt ...
Say for example that your "calculations" consist of incrementing each pair of numbers by 1:
s/(\d+) (\d+)/($1 + 1) . ($2 + 1)/ge
You would do either
$ perl -pi -e's/(\d+) (\d+)/($1 + 1) . ($2 + 1)/ge' file1.txt file2.txt
$ perl -pi foo.pl file1.txt file2.txt
Where foo.pl contains the code.
Be aware that the -i switch is destructive, so make backups before running the command. You can supply a backup extension to save a backup, but that backup is overwritten if you run the command again. E.g. -i.bak.
-p places a while (<>) loop around your code, followed by a print of each line,
-i.bak does the editing of the original file, and saves a backup with the extension, if it is supplied.
I'm trying to run a tail command from within a perl script using the usual backticks.
The section in my perl script is as follows:
$nexusTime += nexusUploadTime(`tail $log -n 5`);
So I'm trying to get the last 5 lines of this file but I'm getting the following error when the perl script finishes:
sh: line 1: -n: command not found
Even though when I run the command on the command line it is indeed successful and I can see the 5 lines from that particular.
Not sure what is going on here. Why it works from command line but through perl it won't recognize the -n option.
Anybody have any suggestions?
$log has an extraneous trailing newline, so you are executing
tail file.log
-n 5 # Tries to execute a program named "-n"
Fix:
chomp($log);
Note that you will run into problems if log $log contains shell meta characters (such as spaces). Fix:
use String::ShellQuote qw( shell_quote );
my $tail_cmd = shell_quote('tail', '-n', '5', '--', $log);
$nexusTime += nexusUploadTime(`$tail_cmd`);
ikegami pointed out your error, but I would recommend avoiding external commands whenever possible. They aren't portable and debugging them can be a pain, among other things. You can simulate tail with pure Perl code like this:
use strict;
use warnings;
use File::ReadBackwards;
sub tail {
my ($file, $num_lines) = #_;
my $bw = File::ReadBackwards->new($file) or die "Can't read '$file': $!";
my ($lines, $count);
while (defined(my $line = $bw->readline) && $num_lines > $count++) {
$lines .= $line;
}
$bw->close;
return $lines;
}
print tail('/usr/share/dict/words', 5);
Output
ZZZ
zZt
Zz
ZZ
zyzzyvas
Note that if you pass a file name containing a newline, this will fail with
Can't read 'foo
': No such file or directory at tail.pl line 10.
instead of the more cryptic
sh: line 1: -n: command not found
that you got from running the tail utility in backticks.
The answer to this question is to place the option -n 5 before the target file
I am writing a script that will create a new tar file containing only those files that were created after the previous tar.gz file was created.
my $path_to_logs = "/home/myscripts/";
my $FNAME= `ls -t *.tar.gz | head -n1`;
my $FILENAME = $path_to_logs.$FNAME;
chomp ($FILENAME);
if (-e $FILENAME){
my $changed= `find . -name '*.log' -newer $FILENAME`;
chomp $changed;
$command = "tar -cvzT ". $changed." -f deleteme-$(date +%Y-%m-%d-%H-%M-%S).tar.gz";
chomp $command;
print $command;
}
However, the outout for $command shows that each of the find results are on a new line, so I dont get on concatenated command for tar. Any idea why?
Thanks.
How about this to solve your immediate problem:
my $find_cmd = "find . -name '*.log' -newer $filename";
open my $in, '-|', $find_cmd or die "Couldn't run command. $!\n";
while(<$in>) {
chomp;
print "Do something with file: $_\n";
}
If you need the files in a single line you can create a variable and concatenate them or whatever, I just wanted to show you a better way to call a system command (it would be even better if you could call the command directly without the shell expansion but you are relying on the shell expansion there).
In the long run you might want to learn how to use perl's own find/wanted routine and how to do dir globbing instead of having to rely so much on the system.
Just transform the output from find into a single line:
my $changed= `find . -name '*.log' -newer $FILENAME`;
chomp $changed;
$changed =~ s/\n/ /g;
$command = "tar -cvzT -f deleteme-$(date +%Y-%m-%d-%H-%M-%S).tar.gz " . $changed;
Btw, in general, it's often better to reduce one's dependency on OS specific features. You can duplicate all of the commands that you're shelling to the OS in perl using not too much effort:
use strict;
use warnings;
use autodie;
use File::Find::Rule;
use File::stat;
use Time::Piece;
my $path_to_logs = "/home/myscripts/";
my ($FILENAME) = sort {
stat($a)->mtime <=> stat($b)->mtime
} glob('/home/myscripts/*.tar.gz');
if (-e $FILENAME){
my $modified = stat($FILENAME)->mtime;
my #files = File::Find::Rule->file()
->name('*.log')
->modified(">$modified")
->in('.');
my $datenow = localtime->strftime('%Y-%m-%d-%H-%M-%S');
my $command = "tar -cvzT -f deleteme-${datenow}.tar.gz ". join(' ', #files);
You could even use Archive::Tar instead of /bin/tar, but that would potentially have a performance hit.
However, regardless, these simple changes make your script much more portable, and didn't require that much additional code.
It doesn't make much sense to do this in Perl anyway. Regardless of the wrapper language, it would be simpler and more robust to pipe the find output straight to tar, which knows how to handle it.
Anyway, your use of tar's -T option is wrong. It expects a file name containing file names, one per line, not a list of file names.
Also, your FNAME contains the newest file in the current directory, where apparently the intent is to find the newest file in /home/myscripts.
Finally, the $(date ...) interpolation will not be interpolated by Perl, but trivially works if you convert this (back?) to a shell script.
#!/bin/sh
path_to_logs = "/home/myscripts/"
FNAME=$(cd "$path_to_logs"; ls -t *.tar.gz | head -n1)
FILENAME = "$path_to_logs/$FNAME"
if [ -e "$FILENAME" ]; then
find . -name '*.log' -newer "$FILENAME" |
tar -c -v -z -T - -f deleteme-$(date +%Y-%m-%d-%H-%M-%S).tar.gz
fi
I currently have an issue with reading files in one directory.
I need to take all the fastq files in a file and run the script for each file then put new files in an ‘Edited_sequences’ folder.
The one script I had is
perl -ne '$i++; if($i<80001){print}' BM2003_TCCCAGAACAAC_L001_R1_001.fastq > ./Edited_sequences/BM2003_TCCCAGAACAAC_L001_R1_001.fastq
It takes the first 80000 lines in one fastq file then outputs the result.
Now for example I have 2000 fastq files, then I need to copy and paste for 2000 times.
I know there is a glob command suit for this situation but I just do not know how to deal with that.
Please help me out.
You can use perl to do copy/paste for you, first argument *.fastq are all fastq files, and second ./Edited_sequences is target folder for new files,
perl -e '$d=pop; `head -8000 "$_" > "$d/$_"` for #ARGV' *.fastq ./Edited_sequences
glob gets you an array of filenames matching a particular expression. It's frequently used with <> brackets, a lot like reading input (you can think of it as reading files from a directory).
This is a simple example that will print the names of every ".fastq" file in the current directory:
print "$_\n" for <*.fastq>;
The important part is <*.fastq>, which gives us an array of filenames matching that expression (in this case, a file extension). If you need to change which directory your Perl script is working in, you can use chdir.
From there, we can process your files as needed:
while (my $filename = <*.fastq>) {
open(my $in, '<', $filename) or die $!;
open(my $out, '>', "./Edited_sequences/$filename") or die $!;
for (1..80000) {
my $line = <$in>;
print $out $line;
}
}
You have two choices:
Use Perl to read in the 2000 files and run it as part of your program
Use the Shell to pass each of those 2000 file to your command line
Here's the bash alternative:
for file in *.fastq
do
perl -ne '$i++; if($i<80001){print}' "$file" > "./Edited_sequences/$file"
done
Your same Perl script, but with the shell finding each file. This should work and not overload the command line. The for loop in bash, if handed a glob can expand them correctly.
However, I always recommend that you don't actually execute the command, but echo the resulting commands into a file:
for file in *.fastq
do
echo "perl -ne '\$i++; if(\$i<80001){print}' \
\"$file\" > \"./Edited_sequences/$file\"" >> myoutput.txt
done
Then, you can look at myoutput.txt to make sure it looks good before you actually do any real harm. Once you've determined that myoutput.txt is a good file, you can execute that as a shell script:
$ bash myoutput.txt
I have a lot of text files in a folder. The folder is 'c:\vehicles'. For every text file, I want to copy any row that includes the words: model, make, year. The file I want to write to is 'vehicles.txt' and located in 'c:\'.
I know I've written the code wrong. What should I do to correct it? Thanks for the help.
C:\vehicles $ ls -A | xargs head -qn 30 | perl -Mstrict -wne 'if( $ +_ =~ /(make)|(model)|(year)/ ) { print "$_"; }' > vehicles.txt
grep -rE "(make|model|year)" c:
Perhaps the following will help:
use strict;
use warnings;
for my $file (<*.txt>) {
open my $fh, '<', $file or die $!;
while (<$fh>) {
chomp;
print "$_\n" if /(?:\b(?:model|make|year)\b)/i;
}
close $fh;
}
Assuming the script will be in c:\vehicles, type perl scriptName.pl >vehicles.txt at the command prompt.
The <*.txt> notation returns a list of all text files in the directory. Each of these files are opened and read, line by line. If any of the words your looking for are found on a line, it's printed. The >vehicles.txt notation means to print to the file.