I have a lot of text files in a folder. The folder is 'c:\vehicles'. For every text file, I want to copy any row that includes the words: model, make, year. The file I want to write to is 'vehicles.txt' and located in 'c:\'.
I know I've written the code wrong. What should I do to correct it? Thanks for the help.
C:\vehicles $ ls -A | xargs head -qn 30 | perl -Mstrict -wne 'if( $ +_ =~ /(make)|(model)|(year)/ ) { print "$_"; }' > vehicles.txt
grep -rE "(make|model|year)" c:
Perhaps the following will help:
use strict;
use warnings;
for my $file (<*.txt>) {
open my $fh, '<', $file or die $!;
while (<$fh>) {
chomp;
print "$_\n" if /(?:\b(?:model|make|year)\b)/i;
}
close $fh;
}
Assuming the script will be in c:\vehicles, type perl scriptName.pl >vehicles.txt at the command prompt.
The <*.txt> notation returns a list of all text files in the directory. Each of these files are opened and read, line by line. If any of the words your looking for are found on a line, it's printed. The >vehicles.txt notation means to print to the file.
I am piping the directory command output to file handle, followed by the print to the file handle. I wish to append some text to the dir/ls output.
open (FH, "| dir") or die "$OS_ERROR";
print FH ("sometext") or die "$OS_ERROR";
while (<FH>){
print;
}
When I execute the Perl script, I see the directory contents, but I do not see the text printed using the print statement, in this case I do not see sometext. What am I missing?
To explain in more detail - I want pipe dir contents to FH, and following that I want to append some text to the same filehandle FH . I have referred link http://perldoc.perl.org/perlopentut.html#Pipe-Opens
You are not redirecting anything: You are piping your script's output to either the cmd.exe builtin dir or an alias to ls depending on your OS (which means, you might run into trouble if you run this script with Cygwin's ls in your path on Windows).
Writing to dir does not seem useful. If you wanted to filter dirs output, i.e. take the output from running dir and manipulate it before printing, you should pipe it into your script and you should print the processed output.
#!/usr/bin/env perl
use strict; use warnings;
my $pid = open my $dir_out, '-|', 'cmd.exe /c dir';
die "Cannot open pipe: $!\n" unless $pid;
my $output_file = 'output.txt';
open my $my_out, '>', $output_file
or die "Cannot open '$output_file': $!";
while (my $line = <$dir_out>) {
$line =~ s/bytes free/peons liberated/;
print $my_out $line;
}
close $my_out
or die "Cannot close '$output_file': $!";
close $dir_out
or die "Cannot close pipe: $!\n";
Of course, I am assuming there are other things going on in your program and this is only a small part of it. Otherwise, you don't need to write this much code for a simple filter.
You cannot write to FH with print and then expect to read from FH in the next statement. File handles are not FIFOs (by default).
The open gives you a writable file handle, the read end of which is connected to the stdin of dir. Reading from a write file handle just gives you nothing.
What do you actually want to achieve? Send some text to the dir program, or read output of the dir program?
Since in the comment you said you want to read the output of the dir command, you have the open command wrong; use "dir |" instead of "| dir" and read the Perl Open Tutorial.
Maybe this does what you want to do:
open (FH, "dir|") or die "$OS_ERROR";
while (<FH>){
print;
}
print "sometext\n";
Greetings,
I have the following code and am stuck on how I would proceed to modify it so it will ask for the directory, read all files in the directory, then extract specific strings and ouput to an html file? Thanks in advance.
#!/usr/local/bin/perl
use warnings;
use strict;
use Cwd;
print "Enter filename: "; # Should be Enter directory
my $perlfile =STDIN;
open INPUT_FILE, $perlfile || die "Could not open file: $!";
open OUTPUT, '>out.html' || die "Could not open file: $!";
# Evaluates the file and imports it into an array.
my #comment_array = ;
close(INPUT_FILE);
chomp #comment_array;
#comment_array = grep /^\s*#/g, #comment_array;
my $comment;
foreach $comment (#comment_array) {
$comment =~ /####/; #Pattern match to grab only #s
# Prints comments to screen
Print results in html format
# Writes comments to output.html
Writes results to html file
}
close (OUTPUT);
Take it one step at a time. You have a lot planned, but so far you haven't even changed your prompt string to ask for a directory.
To read the entered directory name, your:
my $perlfile =STDIN;
gives an error (under use strict;). Start by looking that error up (use diagnostics; automates this) and trying to figure out what you should be doing instead.
Once you can prompt for a directory name and print it out, then add code to open the directory and read the directory. Directories can be opened and read with opendir and readdir. Make sure you can read the directory and print out the filenames before going on to the next step.
a good starting point to learn about specific functions (from the cmd line)
perldoc -f opendir
However, your particular problem is answered as follows, you can also use command line programs and pipe them into a string to simplify file handling ('cat') and pattern matching ('grep').
#!/usr/bin/perl -w
use strict;
my $dir = "/tmp";
my $dh;
my #patterns;
my $file;
opendir($dh,$dir);
while ($file = readdir($dh)){
if (-f "$dir/$file"){
my $string = `cat $dir/$file | grep pattern123`;
push #patterns, $string;
}
}
closedir($dh);
my $html = join("<br>",#patterns);
open F, ">out.html";
print F $html;
close F;
I'm writing a script in perl strawberry. The first thing it needs to be able to do is take a path argument and get a directory listing for that path, and it needs to be able to distinguish between files and folders. I read some tutorials on the subject and wrote the script below, but it only works when I give it the path that the script is currently residing in. If I give it any other path, the -f and -d tests don't work.
EDIT: Clarification: The script DOES put all the files and folders into #thefiles if I give it a path other than it's own, it's just the -f and -d tests that don't work.
use Getopt::Long;
my $dir;
GetOptions('-d=s' => \$dir);
opendir(DIR, $dir) or die "BORKED";
#thefiles = readdir(DIR);
print DIR;
closedir(DIR);
#filez;
#dirz;
foreach $file (#thefiles){
if (-f$file){
push(#filez, $file);
}
if (-d$file){
push(#dirz, $file);
}
}
print "files: #filez \n";
print "Directories: #dirz \n";
Here's a screenshot: http://i.stack.imgur.com/RMmFz.jpg
Hope someone can help and thanks very much for your time. :)
martin clayton told you the reason your code does not work.
Here is a way to fix it using map and some more modern Perl constructs:
use strict;
use warnings;
use Getopt::Long;
my $dir;
GetOptions('-d=s' => \$dir);
opendir my $dh, $dir or die "BORKED: $!";
my #thefiles = map { "$dir/$_" } readdir $dh;
closedir $dh;
my #filez;
my #dirz;
for my $file (#thefiles) {
push #filez, $file if -f $file;
push #dirz , $file if -d $file;
}
print "files: #filez \n";
print "Directories: #dirz \n";
It's because the filetest -f and -d operators use a relative path unless you provide an absolute one. The readdir function will return the file (and subdirectory...) names found in the directory, but not the full paths.
From the docs:
If you're planning to filetest the
return values out of a readdir, you'd
better prepend the directory in
question. Otherwise, because we didn't
chdir there, it would have been
testing the wrong file.
Recently there have been a few attackers trying malicious things on my server so I've decided to somewhat "track" them even though I know they won't get very far.
Now, I have an entire directory containing the server logs and I need a way to search through every file in the directory, and return a filename if a string is found. So I thought to myself, what better of a language to use for text & file operations than Perl? So my friend is helping me with a script to scan all files for a certain IP, and return the filenames that contain the IP so I don't have to search for the attacker through every log manually. (I have hundreds)
#!/usr/bin/perl
$dir = ".";
opendir(DIR, "$dir");
#files = grep(/\.*$/,readdir(DIR));
closedir(DIR);
foreach $file(#files) {
open FILE, "$file" or die "Unable to open files";
while(<FILE>) {
print if /12.211.23.200/;
}
}
although it is giving me directory read errors. Any assistance is greatly appreciated.
EDIT: Code edited, still saying permission denied cannot open directory on line 10. I am just going to run the script from within the logs directory if you are questioning the directory change to "."
Mike.
Can you use grep instead?
To get all the lines with the IP, I would directly use grep, no need to show a list of files, it's a simple command:
grep 12\.211\.23\.200 *
I like to pipe it to another file and then open that file in an editor...
If you insist on wanting the filenames, it's also easy
grep -l 12\.211\.23\.200 *
grep is available on all Unix//Linux with the GNU tools, or on windows using one of the many implementations (unxutils, cygwin, ...etc.)
You have to concatenate $dirname with $filname when using files found through readdir, remember you haven't chdir'ed into the directory where those files resides.
open FH, "<", "$dirname/$filname" or die "Cannot open $filname:$!";
Incidentally, why not just use grep -r to recursively search all subdirectories under your log dir for your string?
EDIT: I see your edits, and two things. First, this line:
#files = grep(/\.*$/,readdir(DIR));
Is not effective, because you are searching for zero or more . characters at the end of the string. Since it's zero or more, it'll match everything in the directory. If you're trying to exclude files ending in ., try this:
#files = grep(!/\.$/,readdir(DIR));
Note the ! sign for negation if you're trying to exclude those files. Otherwise (if you only want those files and I'm misunderstanding your intent), leave the ! out.
In any case, if you're getting your die message on line 10, most likely you're hitting a file that has permissions such that you can't read it. Try putting the filename in the die output so you can see which file it's failing on:
open FILE, "$file" or die "Unable to open file: $file";
But as with other answers, and to reiterate: Why not use grep? The unix command, not the Perl function.
This will get the file names you are looking for in perl, and probably do it much faster than running and doing a perl regex.
#files = `find ~/ServerLogs -name "*.log" | xargs grep -l "<ip address>"`'
Although, this will require a *nix compliant system, or Cygwin on Windows.
Firstly get a list of files within your source directory:
opendir(DIR, "$dir");
#files = grep(/\.log$/,readdir(DIR));
closedir(DIR);
And then loop through those files
foreach $file(#files)
{
// file processing code
}
My first suggest would be to use grep instead. The right tool for the job, they say...
But to answer your question:
readdir just returns the filenames from the directory. You'll need to concatenate the directory name and filename together.
$path = "$dirname/$filname";
open FH, $path or die ...
Then you should ignore files that are actually directories, such as "." and "..". After getting the $path, check to see if it's a file.
if (-f $path) {
open FH, $path or die ...
while (<FH>)
BTW, I thought I would throw in a mention for File::Next. To iterate over all files in a directory (recursively):
use Path::Class; # always useful.
use File::Next;
my $files = File::Next::files( dir(qw/path to files/) ); # look in path/to/files
while( defined ( my $file = $files->() ) ){
$file = file( $file );
say "Examining $file";
say "found foo" if $file->slurp =~ /foo/;
}
File::Next is taint-safe.
~ doesn't auto-expand in Perl.
opendir my $fh, '~/' or die("Doin It Wrong"); # Doing It Wrong.
opendir my $fh, glob('~/') and die( "Thats right!" );
Also, if you must use readdir(), make sure you guard the expression thus:
while (defined(my $filename = readdir(DH))) {
...
}
If you don't do the defined() test, the loop will terminate if it finds a file called '0'.
Have you looked on CPAN for log parsers? I searched with 'log parse' and it yielded over 200 hits. Some (probably many) won't be relevant - some may be. It depends, in part, on which web server you are using.
Am I reading this right? Your line 10 that gives you the error is
open FILE, "$file" or die "Unable to open files";
And the $file you are trying to read, according to line 6,
#files = grep(/\.*$/,readdir(DIR));
is a file that ends with zero or more dot. Is this what you really wanted? This basically matches every file in the directory, including "." and "..". Maybe you don't have enough permission to open the parent directory for reading?
EDIT: if you only want to read all files (including hidden ones), you might want to use something like the following:
opendir(DIR, ".");
#files = readdir(DIR);
closedir(DIR);
foreach $file (#files) {
if ($file ne "." and $file ne "..") {
open FILE, "$file" or die "cannot open $file\n";
# do stuff with FILE
}
}
Note that this doesn't take care of sub directories.
I know I am way late to this discussion (ran across it while searching for grep related posts) but I am going to answer anyway:
It isn't specified clearly if these are web server logs (Apache, IIS, W3SVC, etc.) but the best tool for mining those for data is the LogParser tool from Microsoft. See logparser.com for more info.
LogParser will allow you to write SQL-like statements against the log files. It is very flexible and very fast.
Use perl from the command line, like a better grep
perl -wnl -e '/12.211.23.200/ and print;' *.log > output.txt
the benefit here is that you can chain logic far easier
perl -wnl -e '(/12.211.23.20[1-11]/ or /denied/i ) and print;' *.log
if you are feeling wacky you can also use more advanced command line options to feed perl one liner result into other perl one liners.
You really need to read "Minimal Perl: For UNIX and Linux People", awesome book on this very sort of thing.
First, use grep.
But if you don't want to, here are two small improvements you can make that I haven't seen mentioned yet:
1) Change:
#files = grep(/\.*$/,readdir(DIR));
to
#files = grep({ !-d "$dir/$_" } readdir(DIR));
This way you will exclude not just "." and ".." but also any other subdirectories that may exist in the server log directory (which the open downstream would otherwise choke on).
2) Change:
print if /12.211.23.200/;
to
print if /12\.211\.23\.200/;
"." is a regex wildcard meaning "any character". Changing it to "\." will reduce the number of false positives (unlikely to change your results in practice but it's more correct anyway).