Mass remove lines that contain certain words? - command-line

Question

I need to remove any lines that contain certain keywords in them from a huge list of text files I have in a directory.
For example, I need all lines with any of these keywords in them to be removed: test1, example4, coding9
This is the closest example to what I'm trying to do that I can find:
sed '/Unix\|Linux/d' *.txt
Note: the lines don't need to contain all the keywords to be removed, just one should remove it :)

Answer 1

It appears that you are looking for some 1 liner command to read and write back to thousands of files and millions of lines. I wouldn't do it like that personally because I would prefer to write a quick and dirty script in Perl. I very briefly tested this on very simple files and it works but since you are working with thousands of files and millions of lines, I would test whatever you write in a test directory first with some of the files so that you can verify.
#!/usr/bin/perl
# the initial directory to read from
my $directory = 'tmp';
opendir (DIR, $directory) or die $!;
my #keywords = ('woohoo', 'blah');
while (my $file = readdir(DIR)) {
# ignore files that begin with a period
next if ($file =~ m/^\./);
# open the file
open F, $directory.'/'.$file || die $!;
# initialize empty file_lines
#file_lines = ();
# role through and push the line into the new array if no keywords are found
while (<F>) {
next if checkForKeyword($_);
push #file_lines, $_;
}
close F;
# save in a temporary file for testing
# just change these 2 variables to fit your needs
$save_directory = $directory.'-save';
$save_file = $file.'-tmp.txt';
if (! -d $save_directory) {
`mkdir $save_directory`;
}
$new_file = $save_directory.'/'.$save_file;
open S, ">$new_file" || die $!;
print S for #file_lines;
close S;
}
# role through each keyword and return 1 if found, return '' if not
sub checkForKeyword()
{
$line = shift;
for (0 .. $#keywords) {
$k = $keywords[$_];
if ($line =~ m/$k/) {
return 1;
}
}
return '';
}