I have been trying to do a recursive grep command on files in sub folders using grep in NTemacs and Cygwin. So far the "best" results have been using grep in eshell. When I use this:
grep "t" -r *
I get a list of all file names containing the letter t, in all sub folders one layer down but notthing else. In Cygwin i get nothing. I'm working on a directroy that is not in the Cygwin install. Don't know if that mather or not.
What I want is to match the content of a more complex string in all files (and not just the file names, but the content). And in all sub directories.
I would like to use eshell from emacs but I'm open to suggestions, apart form using LINUX. This is a work PC and I don't want to do all the setup of a LINUX install.
i just wrote a very similar answer to another question, but i suspect it's the same root problem:
my first thought is that your files have windows line endings (CRLF) as opposed to unix/linux line endings (LF), and that is messing with grep's ability to parse the file. try running this:
dos2unix filename
on each file you need to search then try your grep statement again.
if you need to convert many files across several directories, i suggest using dos2unix with the -exec action of find:
find . -exec dos2unix {} \;
(add whatever other options you need to find before running that, of course)
Related
Question: In Git Bash on windows, how would you run the following in a way that it will also search folders with spaces in the name, and execute on files with spaces in the name?
$ find ./ -type f -name '*.png' -exec sh -c 'cwebp -q 75 $1 -o "${1%.png}.webp"' _ {} \;
Context I'm running Git Bash on windows, trying to execute a command on all found .png files to convert them to .webp format. It works for all files without spaces in the path, but it's failing to find files with spaces in the filename or files within folders that have spaces in the folder name.A few considerations:
I have many, many levels of folders to iterate through, and I can't run this command separately for each. I really need the recursion to work.I cannot change the folder names; it will break other dependencies (nor did I create the folder or filenames originally, so cut me some slack!)I arrived here by following the suggestions from this article: https://www.smashingmagazine.com/2018/07/converting-images-to-webp/the program, to my knowledge, doesn't ship with any built-in recursive command... golly that'd be handy
Any help you can provide will be appreciated. Thanks!
My question is in two parts :
1) Why does grep hang when I grep all files under "/" ?
for example :
grep -r 'h' ./
(note : right before the hang/crash, I note that I see some "no such device or address" messages , regarding sockets....
Of course, I know that grep shouldn't run against a socket, but I would think that since sockets are just files in Unix, it should return a negative result, rather than crashing.
2) Now, my follow up question : In any case -- how can I grep the whole filesystem? Are there certain *NIX directories which we should leave out when doing this ? In particular, I'm looking for all recently written log files.
As #ninjalj said, if you don't use -D skip, grep will try to read all your device files, socket files, and FIFO files. In particular, on a Linux system (and many Unix systems), it will try to read /dev/zero, which appears to be infinitely long.
You'll be waiting for a while.
If you're looking for a system log, starting from /var/log is probably the best approach.
If you're looking for something that really could be anywhere in your file system, you can do something like this:
find / -xdev -type f -print0 | xargs -0 grep -H pattern
The -xdev argument to find tells it to stay within a single filesystem; this will avoid /proc and /dev (as well as any mounted filesystems). -type f limits the search to ordinary files. -print0 prints the file names separated by null characters rather than newlines; this avoid problems with files having spaces or other funny characters in their names.
xargs reads a list of file names (or anything else) on its standard input and invokes the specified command on everything in the list. The -0 option works with find's -print0.
The -H option to grep tells it to prefix each match with the file name. By default, grep does this only if there are two or more file names on its command line. Since xargs splits its arguments into batches, it's possible that the last batch will have just one file, which would give you inconsistent results.
Consider using find ... -name '*.log' to limit the search to files with names ending in .log (assuming your log files have such names), and/or using grep -I ... to skip binary files.
Note that all this depends on GNU-specific features. Some of these options might not be available on MacOS (which is based on BSD) or on other Unix systems. Consult your local documentation, and consider installing GNU findutils (for find and xargs) and/or GNU grep.
Before trying any of this, use df to see just how big your root filesystem is. Mine is currently 268 gigabytes; searching all of it would probably take several hours. A few minutes spent (a) restricting the files you search and (b) making sure the command is correct will be well worth the time you spend.
By default, grep tries to read every file. Use -D skip to skip device files, socket files and FIFO files.
If you keep seeing error messages, then grep is not hanging. Keep iotop open in a second window to see how hard your system is working to pull all the contents off its storage media into main memory, piece by piece. This operation should be slow, or you have a very barebones system.
Now, my follow up question : In any case -- how can I grep the whole filesystem? Are there certain *NIX directories which we should leave out when doing this ? In particular, Im looking for all recently written log files.
Grepping the whole FS is very rarely a good idea. Try grepping the directory where the log files should have been written; likely /var/log. Even better, if you know anything about the names of the files you're looking for (say, they have the extension .log), then do a find or locate and grep the files reported by those programs.
I am back with a second no-brainer question, but I would like to get this straight in my head.
I have an assignment in which I am charged with providing a command to find a file named test in my home directory (one command using find, and one using grep). I understand that using find is just 'find ~/test', but using grep, wouldn't I have to search out a pattern within the file 'test'? Or is there a way to search for the file (using grep), even if the file is empty?
ls ~ | grep test
I understand that using find is just 'find ~/test'
No. find ~/test will also have a match for every file or directory under the directory $HOME/test/. Rather use find ~ -type f -name test.
The assignment sounds unclear. But yes, if you give any filenames to grep, it will look at the contents of the files and ignore the names of the files. Perhaps you can grep the output of another command? Maybe ls as #Reese suggested, or maybe a different find command.
ls -R ~ | grep test
Explanation: ls -R ~ will recursively list all files and directories in your home folder. grep test will narrow down that list to files (and directories) that have "test" in their name.
I'm wondering if there is a way to change a specific word in all of the files within the /www/ directory using command line. Just looking for a faster way to change out a specific word so I don't need to open all the files manually! Thanks!
find /www -type f -exec sed -i 's/foo/bar/g' \{\} \;
This line will replace foo with bar every time foo occurs in any file in /www. Be very sure you know what's under /www and what the replacement would do to those files before running it.
You might be looking for a grep-sed solution to find and replace, if you are on a Mac (and referring to the Mac's Terminal app).
I'm trying to run the following command in Windows Server 2003 but sed creates a pile of files that I can't delete from the command line inside the current directory.
for /R %f in (*.*) do "C:\Program Files\gnuwin32\bin\sed.exe" -i "s/bad/good/g" "%f"
Does anyone have any suggestions? Mysteriously enough, I'm able to delete the files using Windows Explorer.
As requested, here are some example filenames:
sed0E3WZJ
sed5miXwt
sed6fzFKh
And, more troubleshooting info...
It occurs from both the command prompt & batch files
If I just need to run sed on a single directory, then I use sed "s/bad/good/g" *.* and everything is OK. Alas, I also need it to tackle all the subdirectories.
I only have Sed installed.
Sed is creating the files
I have replicated your setup and I have the following observations.
I dont think there is a problem in the loop. The simple command "C:\Program Files\gnuwin32\bin\sed.exe" -i "s/bad/good/g" . - creates the same set of temporary files.
The files are indeed created by sed. sed creates these temporary files when the "in place" (-i) option is turned on. In the normal course, sed actually deletes the files (that is what happens in cygwin) using a call to the 'unlink' library. In case of gnuwin32, it looks like the 'unlink' fails. I have not been able to figure out why. I took a guess that maybe the unlink call is dependent on the gnuwin32 'coreutils' library and tried to download and install the coreutils library - no dice.
If you remove the 'read-only' restriction in the parent folder before executing the sed command, you can delete the temporary files from windows command prompt. So that should give you some temporary respite.
I think we now have enough information to raise a bug report. If you agree, I think it may be a good idea to bring it to the notice of the good folks responsible for gnuwin32 and ask them for help.
Meanwhile, the following version cleans up its temporary file:
https://github.com/mbuilov/sed-windows
As this is a known bug in sed with the -i option you can run attrib -R <filename> to remove the read only attribute from file after sed completes.
Alternatively do not use the -i option and redirect the output to a new file and then delete and rename the input and output.
Cygwin hoses the ACLs on files sometimes, you'll probably have to use cacls or chmod to fix it up before you can delete the file.
Here is where a bit of troubleshooting comes into play. Does this happen when you run that command from the command-line and a batch file? What if you run sed on an individual file on the command line - does it create these files for every file, or just certain files/filetypes? Does it only happen for that replacement, or all replacements in general, or just always when you run sed.exe on a file? Is it only sed creating these files, or all Gnuwin32 exe's (eg. awk, cat, etc)? Does the same thing happen on a sed.exe from a new install of Gnuwin32? What error message does it give when you try to delete the files? Can you delete the files from explorer while the command prompt is still open? What if you close the command prompt and reopen it, then try to delete the files?
you can just run sed without for loop
c:\test> sed -i.bak "s/bad/good/g" file*.*
This is a stab in the dark, but it wouldn't surprise me in the least if the gnuwin32 implementation of sed is duff (i.e., faulty in some way). Can you try to replicate the problem using the AT&T U/Win POSIX support for windows? It is easy to install and includes the Korn shell, sed, and find, so you can use find instead of the FOR /R. (I'm wondering if part of the problem is that the MS FOR and gnuwin32 sed don't play nicely together.)
I realize this is an old thread but It's still an issue. My fix is to add
"DEL sed*" to the end of a batch file after sed. Quick and dirty.
I am using this command to clean up the temporary files created by gnuwin32's sed:
FOR /f "tokens=*" %%a in ('dir /b ^| findstr /i "^sed[0-9a-zA-Z][0-9a-zA-Z][0-9a-zA-Z][0-9a-zA-Z][0-9a-zA-Z][0-9a-zA-Z]$"') DO del %%a
i know this is old. But just want to share with people what I did that cause this.
It was in fact the temp file for an already open file through gvim example .swap file that causing the sed tmp file didnt get remove completely.
So sed trying to read and append into the opened file which the user is currently viewing and having trouble doing it which causes the tmp file error.