I'm trying to find some stuff in a large number of text files, and I want the output to be in a file so I can read it at leisure:
grep -i 'alter table' *.sql >> tables.txt
grep (this is the Windows version of the Gnu tool) complains at the >>. I've tried piping and all the rest, and there doesn't saeem to be an option to define an output file either.
Any ideas?
Reviving this old question, but it's among the first Google results.
grep outputs differently, so I needed to add this option to ouptut the results to a file:
grep --line-buffered
Source
This works here:
grep -i "other something" *.txt >> tables.txt
Related
I've made a small script trying to search through a file looking for all occurrences of specific strings like this: a0002 b0590 c0964
The script goes like this:
#!/bin/sh
#include <stdio.h>
#
while read id;
do
awk {'print $1'} test.trans | grep -e "$id"
done < test.id
To simplify things, I made a stripped down version of the file I'm searching through (test.trans):
"a0001"
"a0002"
"b0586"
"b0587"
"b0588"
"b0589"
"b0590"
"b0591"
"b0852"
"b0952"
"a0002"
"b0587"
"c0952"
"c0964"
"c1783"
"c1786"
"c1787"
I have stored all the relevant search strings in a separate file named test.id which looks like this:
a0002
b0587
b0588
b0589
b0590
b0591
b0852
b0952
c0952
c0964
c1781
The idea is to pass each search string in the test.id file as a variable which is then used by grep to filter out all occurrences in the test.trans file
However, when I run the script, grep only matches some of the strings. When I change the order of the search patterns in the test.id file, the result also changes. What am I doing wrong?
I consider myself a newbie in shell programming, and would appreciate any help.
I don't know the reason for your problem.
But here are some remarks that don't fit in a comment.
If you want to put out a file to stdout (the standard output) you can use cat test.trans instead of awk {'print $1'} test.trans.
But is you want grep to process a file you must not read it with some other tool and pipr it to grep. grep can read this file directly by using `grep -e "$id" test.trans
If you alread use awk you don't need grep. you can achiefe this wih awk by calling awk /"$id"'/ {print $1}' test.trans grep can filter more than one pattern. Instead of your for loop do
grep -f test.id test.trans
After some experimenting, I found out that it all boiled down to removing the carriage return (\r) from each line in the test.id file. This file was received from a DOS-machine, and my iMac is using UNIX-format which only contains newline (\n)
I wonder if anyone out there can assist me in trying to solve a issue with me.
I have written a set of shell scripts with the purpose of auditing remote file systems based on a GOLD build on a audit server.
As part of this, I do the following:
1) Use rsync to work out any new files or directories, any modified or removed files
2) Use find ${source_filesystem} -ls on both local & remote to work out permissions differences
Now as part of this there are certain files or directories that I am excluding, i.e. logs, trace files etc.
So in order to achieve this I use 2 methods:
1) RSYNC - I have an exclude-list that is added using --exclude-from flag
2) find -ls - I use a egrep -v statement to exclude the same as the rsync exclude-list:
e.g. find -L ${source_filesystem} -ls | egrep -v "$SEXCLUDE_supt"
So my issue is that I have to maintain 2 separate lists and this is a bit of a admin nightmare.
I am looking for some assistance or some advice on if it is possible to dynamically build a list of exlusions that can be used for both the rsync or the find -ls?
Here is the format of what the exclude lists look like::
RSYNC:
*.log
*.out
*.csv
logs
shared
tracing
jdk*
8.6_Code
rpsupport
dbarchive
inarchive
comms
PR116PICL
**/lost+found*/
dlxwhsr*
regression
tmp
working
investigation
Investigation
dcsserver_weblogic_*.ear
dcswebrdtEAR_weblogic_*.ear
FIND:
SEXCLUDE_supt="\.log|\.out|\.csv|logs|shared|PR116PICL|tracing|lost\+found|jdk|8\.6\_Code|rpsupport|dbarchive|inarchive|comms|dlxwhsr|regression|tmp|working|investigation|Investigation|dcsserver_weblogic_|dcswebrdtEAR_weblogic_"
You don't need to create a second list for your find command. grep can handle a list of patterns using the -f flag. From the manual:
-f FILE, --file=FILE
Obtain patterns from FILE, one per line. The empty file contains zero
patterns, and therefore matches nothing. (-f is specified by POSIX.)
Here's what I'd do:
find -L ${source_filesystem} -ls | grep -Evf your_rsync_exclude_file_here
This should also work for filenames containing newlines and spaces. Please let me know how it goes.
In the end the grep -Evf was a bit of a nightmare as rsync didnt support regex, it uses regex but not the same.
So I then pursued my other idea of dynamically building the exclude list for egrep by parsing the rsync exclude-list and building variable on the fly to pass into egrep.
This the method I used:
#!/bin/ksh
# Create Signature of current build
AFS=$1
#Create Signature File
crSig()
{
find -L ${SRC} -ls | egrep -v **"$SEXCLUDE"** | awk '{fws = ""; for (i = 11; i <= NF; i++) fws = fws $i " "; print $3, $6, fws}' | sort >${BASE}/${SIFI}.${AFS}
}
#Setup SRC, TRG & SCROOT
LoadAuditReqs()
{
export SRC=`grep ${AFS} ${CONF}/fileSystem.properties | awk {'print $2'}`
export TRG=`grep ${AFS} ${CONF}/fileSystem.properties | awk {'print $3'}`
export SCROOT=`grep ${AFS} ${CONF}/fileSystem.properties | awk {'print $4'}`
**export BEXCLUDE=$(sed -e 's/[*/]//g' -e 's/\([._+-]\)/\\\1/g' ${CONF}/exclude-list.${AFS} | tr "\n" "|")**
**export SEXCLUDE=$(echo ${BEXCLUDE} | sed 's/\(.*\)|/\1/')**
}
#Load Properties File
LoadProperties()
{
. /users/rpapp/rpmonit/audit_tool/conf/environment.properties
}
#Functions
LoadProperties
LoadAuditReqs
crSig
So with these new variables:
**export BEXCLUDE=$(sed -e 's/[*/]//g' -e 's/\([._+-]\)/\\\1/g' ${CONF}/exclude-list.${AFS} | tr "\n" "|")**
**export SEXCLUDE=$(echo ${BEXCLUDE} | sed 's/\(.*\)|/\1/')**
I use them to remove "*" and "/", then match my special characters and prepend with "\" to escape them.
Then it using "tr" replace a newline with "|" and then rerunning that output to remove the trailing "|" to make the variable $SEXCLUDE to use for egrep that is used in the crSig function.
What do you think?
I'm looking for help with a one-liner that I can run from the Mac OS X terminal. I use MAMP for web development on my Mac. I have a lot of CakePHP projects in my "/Applications/MAMP/htdocs" directory. For the sake of simplicity, let's just say that I had two CakePHP projects and that this was the output of the find /Applications/MAMP/htdocs -type d -iname Controller* command:
/Applications/MAMP/htdocs/my_cake1.3_project/app/controllers
/Applications/MAMP/htdocs/my_cake1.3_project/app/tests/cases/controllers
/Applications/MAMP/htdocs/my_cake1.3_project/cake/console/templates/skel/controllers
/Applications/MAMP/htdocs/my_cake1.3_project/cake/console/templates/skel/tests/cases/controllers
/Applications/MAMP/htdocs/my_cake1.3_project/cake/libs/controller
/Applications/MAMP/htdocs/my_cake1.3_project/cake/tests/cases/libs/controller
/Applications/MAMP/htdocs/my_cake1.3_project/cake/tests/test_app/controllers
/Applications/MAMP/htdocs/my_cake1.3_project/cake/tests/test_app/plugins/test_plugin/controllers
/Applications/MAMP/htdocs/my_cake2_project/app/Controller
/Applications/MAMP/htdocs/my_cake2_project/app/Test/Case/Controller
/Applications/MAMP/htdocs/my_cake2_project/lib/Cake/Console/Templates/skel/Controller
/Applications/MAMP/htdocs/my_cake2_project/lib/Cake/Console/Templates/skel/Test/Case/Controller
/Applications/MAMP/htdocs/my_cake2_project/lib/Cake/Controller
/Applications/MAMP/htdocs/my_cake2_project/lib/Cake/Test/Case/Controller
/Applications/MAMP/htdocs/my_cake2_project/lib/Cake/Test/test_app/Controller
/Applications/MAMP/htdocs/my_cake2_project/lib/Cake/Test/test_app/Plugin/TestPlugin/Controller
Now, sometimes I want to find a piece of code that I know I used in one of my CakePHP projects' controllers, but I can't remember which project it was, so I want to search all of them. I don't want to waste time searching in the "app/tests/cases/controllers" folder or any of the ones within "cake/", though. The find /Applications/MAMP/htdocs -type d -iname Controller* | grep -i /app/Controller command gives me the list of folders I want to search in:
/Applications/MAMP/htdocs/my_cake1.3_project/app/controllers
/Applications/MAMP/htdocs/my_cake2_project/app/Controller
I just need to find a way to take that output, add a slash and asterisk (/*) to the end of each line, and pipe each line to the grep -il "string to search for" command. Any help would be appreciated. Thanks!
solution 1
maybe you want to check two options of find command: (i)path and regex
with them you could narrow your find result and pass found files to your grep -il "searchString" for example by |xargs . it looks like:
find /Applications/MAMP/htdocs -type f -ipath "*/app/Controller/*.php"
| xargs grep -il 'foo'
with -regex would be more flexiable.
solution 2
however if you really really want to :
find a way to take that output, add a slash and asterisk (/*) to the
end of each line, and pipe each line to the grep -il "string to search
for" command.
(btw, here "pipe" won't work.)
you could do this:
find .(your original find).. |grep -i "/app/Controller"
|sed -r 's#^(.*)$#grep -il "foo" \1/*#g'|sh
the trick was done by the sed....|sh. the sed line will pick the result of your previous grep, add grep command and options :(grep -il "foo") and append "/*" in order to construct a complete grep command. finally pipe to sh, to execute it.
Have you tried this?
find /Applications/MAMP/htdocs -type d -iname Controller*
-exec grep -il "string to search for" {} /;
Does sed -i work on AIX?
If not, how can I edit a file "in place" on AIX?
The -i option is a GNU (non-standard) extension to the sed command. It was not part of the classic interface to sed.
You can't edit in situ directly on AIX. You have to do the equivalent of:
sed 's/this/that/' infile > tmp.$$
mv tmp.$$ infile
You can only process one file at a time like this, whereas the -i option permits you to achieve the result for each of many files in its argument list. The -i option simply packages this sequence of events. It is undoubtedly useful, but it is not standard.
If you script this, you need to consider what happens if the command is interrupted; in particular, you do not want to leave temporary files around. This leads to something like:
tmp=tmp.$$ # Or an alternative mechanism for generating a temporary file name
for file in "$#"
do
trap "rm -f $tmp; exit 1" 0 1 2 3 13 15
sed 's/this/that/' $file > $tmp
trap "" 0 1 2 3 13 15
mv $tmp $file
done
This removes the temporary file if a signal (HUP, INT, QUIT, PIPE or TERM) occurs while sed is running. Once the sed is complete, it ignores the signals while the mv occurs.
You can still enhance this by doing things such as creating the temporary file in the same directory as the source file, instead of potentially making the file in a wholly different file system.
The other enhancement is to allow the command (sed 's/this/that' in the example) to be specified on the command line. That gets trickier!
You could look up the overwrite (shell) command that Kernighan and Pike describe in their classic book 'The UNIX Programming Environment'.
#!/bin/ksh
host_name=$1
perl -pi -e "s/#workerid#/$host_name/g" test.conf
Above will replace #workerid# to $host_name inside test.conf
You can simply install GNU version of Unix commands on AIX :
http://www-03.ibm.com/systems/power/software/aix/linux/toolbox/alpha.html
You can use a here construction with vi:
vi file >/dev/null 2>&1 <<#
:1,$ s/old/new/g
:wq
#
When you want to do things in the vi-edit mode, you will need an ESC.
For an ESC press CTRL-V ESC.
When you use this in a non-interactive mode, vi can complain about the TERM not set. The solution is adding export TERM=vt100 before calling vi.
Another option is to use good old ed, like this:
ed fileToModify <<EOF
,s/^ff/gg/
w
q
EOF
you can use perl to do it :
perl -p -i.bak -e 's/old/new/g' test.txt
is going to create a .bak file.
I run the code gives me the following sample data
md5deep find * | awk '{ print $1 }'
A sample of the output
/Users/math/Documents/Articles/Number theory: Is a directory
258fe6853b1bfb2d07f512ff6bec52b1
/Users/math/Documents/Articles/Probability and statistics: Is a directory
4811bfb2ad04b9f4318049c01ebb52ef
8aae4ac3694658cf90005dbdea37b4d5
258fe6853b1bfb2d07f512ff6bec52b1
I have tried to filter the rows which contain Is a directory by SED unsuccessfully
md5deep find * | awk '{ print $1 }' | sed s/\/*//g
Its sample output is
/Users/math/Documents/Articles/Number theory: Is a directory
/Users/math/Documents/Articles/Topology: Is a directory
/Users/math/Documents/Articles/useful: Is a directory
How can I filter Out each row which contains "Is a directory" by SED/AWK?
[clarification]
I want to filter out the rows which contain Is a directory.
I have not used the md5deep tool, but I believe those lines are error messages; they would be going to standard error instead of standard out, and so they are going directly to your terminal instead of through the pipe. Thus, they won't be filtered by your sed command. You could filter them by merging your standard error and standard output streams, but
It looks like (I'm not sure because you are missing the backquotes) you are trying to call
md5deep `find *`
and find is returning all of the files and directories.
Some notes on what you might want to do:
It looks like md5deep has a -r for "recursive" option. So, you may want to try:
md5deep -r *
instead of the find command.
If you do wish to use a find command, you can limit it to only files using -type f, instead of files and directories. Also, you don't need to pass * into a find command (which may confuse find if there are files that have names that looks like the options that find understands); passing in . will search recursively through the current directory.
find . -type f
In sed if you wish to use slashes in your pattern, it can be a pain to quote them correctly with \. You can instead choose a different character to delimit your regular expression; sed will use the first character after the s command as a delimiter. Your pattern is also lacking a .; in regular expressions, to indicate one instance of any character you use ., and to indicate "zero or more of the preceding expression" you use *, so .* indicates "zero or more of any character" (this is different from glob patterns, in which * alone means "zero or more of any character").
sed "s|/.*||g"
If you really do want to be including your standard error stream in your standard output, so it will pass through the pipe, then you can run:
md5deep `find *` 2>&1 | awk ...
If you just want to ignore stderr, you can redirect that to /dev/null, which is a special file that just discards anything that goes into it:
md5deep `find *` 2>/dev/null | awk ...
In summary, I think the command below will help you with your immediate problem, and the other suggestions listed above may help you if I did not undersand what you were looking for:
md5deep -r * | awk '{ print $1 }'
To specifically answer the clarification: how to filter out lines using awk and sed:
awk '/Is a directory/ {next} {print}'
sed 'g/Is a directory/d'
Why not use grep instead?
ie,
md5deep find * | grep "Is a directory" | awk '{ print $1 }'
Edit: I just re-read your question and if you want to remove the lines with Is a directory, use the -v flag of grep, ie:
md5deep find * | grep -v "Is a directory" | awk '{ print $1 }'
I'm not intimately familiar with md5deep, but this may do something like you are tying to do.
find -type f -exec md5sum {} +