I want to execute the following shell command in emacs-lisp:
ls -t ~/org *.txt | head -5
My attempt at the following:
(call-process "ls" nil t nil "-t" "~/org" "*.txt" "| head -5")
results in
ls: ~/org: No such file or directory
ls: *.txt: No such file or directory
ls: |head -5: No such file or directory
Any help would be greatly appreciated.
The problem is that tokens like ~, *, and | aren't processed/expanded by the ls program. Since the tokens aren't processed, ls is look for a file or directory literally called ~/org, a file or directory literally called *.txt, and a file or directory literally called | head -5. Thus the error message you received about `No such file or directory".
Those tokens are processed/expanded by the shell (like Bourne shell /bin/sh or Bash /bin/bash). Technically, interpretation of the tokens can be shell-specific, but most shell interpret at least some of the same standard tokens the same way, e.g. | means connecting programs together end-to-end to almost all shells. As a counterexample, Bourne shell (/bin/sh) does not do ~ tilde/home-directory expansion.
If you want to get the expansions, you have to get your calling program to do the expansion itself like a shell would (hard work) or run your ls command in a shell (much easier):
/bin/bash -c "ls -t ~/org *.txt | head -5"
so
(call-process "/bin/bash" nil t nil "-c" "ls -t ~/org *.txt | head -5")
Edit: Clarified some issues, like mentioning that /bin/sh doesn't do ~ expansion.
Depending on your use case, if you find yourself wanting to execute shell commands and have the output made available in a new buffer frequently, you can also make use of the shell-command feature. In your example, it would look something like this:
(shell-command "ls -t ~/org *.txt | head -5")
To have this inserted into the current buffer, however, would require that you set current-prefix-arg manually using something like (universal-argument), which is a bit of a hack. On the other hand, if you just want the output someplace you can get it and process it, shell-command will work as well as anything else.
Related
I'm using PostgreSQL on Windows 7 through the command line. I want to import the content of different CSV files into a newly created table.
After executing the command the database name appeared like:
database=#
Now appears like
database*# after executing:
type directory/*.csv | psql -c 'COPY sch.trips(value1, value2) from stdin CSV HEADER';
What does *# mean?
Thanks
This answer is for Linux and as such doesn't answer OP's question for Windows. I'll leave it up anyway for anyone that comes across this in the future.
You accidentally started a block comment with your type directory/*.csv. type doesn't do what you think it does. From the bash built-ins:
With no options, indicate how each name would be interpreted if used as a command name.
Try doing cat instead:
cat directory/*.csv | psql -c 'COPY sch.trips(value1, value2) from stdin CSV HEADER';
If this gives you issues because each CSV has its own header, you can also do:
for file in directory/*.csv; do cat "$file" | psql -c 'COPY sch.trips(value1, value2) from stdin CSV HEADER'; done
Type Command
The type built-in command in Bash is a way of viewing command interpreter results. For example, using it with ssh:
$ type ssh
ssh is /usr/bin/ssh
This indicates how ssh would be interpreted when you run ssh as a command in the current Bash environment. This is useful for things like aliases. As an example for this, ll is usually an alias to ls -l. Here's what my Bash environment had for ll:
$ type ll
ll is aliased to `ls -l --color=auto'
For you, when you pipe the result of this command to psql, it encounters the /* in the input and assumes it's a block comment, which is what the database*# prompt means (the * indicates it's waiting for the comment close pattern, */).
Cat Command
cat is for concatenating multiple files together. By default, it writes to standard out, so cat directory/*.csv will write each CSV file to standard out one after another. However, piping this means that each CSV's header will also be piped mid-stream of the copy. This may not be desirable, so:
For Loop
We can use for to loop over each file and individually import it. The version I have above, for file in directory/*.csv, will properly handle files with spaces. Properly formatted:
for file in directory/*; do
cat "$file" | psql -c 'COPY sch.trips(value1, value2) from stdin CSV HEADER'
done
References
PostgreSQL 10 Comments Documentation (postgresql.org)
type built-in Manual page (mankier.com)
cat Manual page (mankier.com)
Bash looping tutorial (tldp.org)
I have a folder full of jpg files which all end with "-x-large.jpg" I would like to rename them all using command line so that it gets rid of the -x-large and just becomes .jpg.
So for example 123-x-large.jpg will become 123.jpg
Can someone tell me how I can do this with the ren command?
Thanks.
for img in *-x-large.jpg; do mv -i -v "$img" "${img%-x-large.jpg}.jpg"; done
This loops on all matching images and moves them into a new file with a truncated name (removing -x-large.jpg from the end) with the .jpg added back to the end of the file name. I'm invoking this interactively with mv -i so you are prompted before overwriting each file. To force overwriting (always say "yes"), change that to mv (remove the -i). To prevent overwriting (always say "no"), change that to mv -n.
Remove the -v (verbose) if you don't want to see each rename happen.
If you have a very large number of these files, the command line will be too long for the above command (since *-x-large.jpg will be expanded onto a command line). You can work around that with find and xargs as follows:
sh <(find . -maxdepth 1 -name '*-x-large.jpg' \
|sed -r 's/(.*)(-x-large.jpg)$/mv -i "\1\2" "\1.jpg"/')
This creates a shell script using bash process substitution, using find to generate a list of all files we want to rename and then piping them through sed to create the mv commands.
(See above for the mv flags. I removed -v because presumably this will be a very long list.)
See the version below if you want to check the script before running it.
The above one-liner requires GNU bash or Korn shell (ksh) as well as GNU sed.
Here's how to do it with neither (in three commands):
find . -maxdepth 1 -name '*-x-large.jpg' \
|sed 's/.*/mv "&" "&/; s/-x-large.jpg$/.jpg"/' > temp.sh
sh temp.sh
rm temp.sh
Posix sed doesn't reliably support capture groups (\(…\) or sed -r to invoke ERE) and therefore we can't expect it to be able to match and recall text, so this version simply writes most of the command and then fixes the ending (the absence of a trailing double quotes in the first replacement is intentional; we add it in the second replacement). Posix shell (/bin/sh proper) doesn't support process substitution, so we dump to a temporary file, evaluate it, and then remove it.
If we're referring to Windows command-line, then SET /? is your friend. Loads of good info in there.
setlocal ENABLEDELAYEDEXPANSION
set SEARCH_SUFFIX=-x-large.jpg
set REPLACE_SUFFIX=.jpg
for %%A in ("*%SEARCH_SUFFIX%") do (
set OLD_NAME=%%~nxA
set NEW_NAME=!OLD_NAME:%SEARCH_SUFFIX%=%REPLACE_SUFFIX%!
ren "!OLD_NAME!" "!NEW_NAME!"
)
endlocal
I need to check if local file is same as remote host file.
The file locations are like below:
File1 at Local machine
./remotehostname/home/a/b/scripts/xyz.cpp
File2 at remote machine
remotehostname:/home/a/b/scripts/xyz.cpp
I intend to compare these 2 files, using the command
diff ./remotehostname/home/a/b/scripts/xyz.cpp remotehostname:/home/a/b/scripts/xyz.cpp
find . -type f | grep -v .svn |xargs -I % diff %
I need to change % to take remotehost and compare the file.
Not sure how to apply sed on %. Or is there a better way to compare such files.
One way could be to save the list of files and then apply sed on that file, but I think there should be an even better way. Also the diff doesnt work on remote hosts, maybe I need to use output of dry rsync?
This can be done with xargs, but I prefer to use while read in bash.
xargs method
find . -type f | grep -v .svn | sed 's/.*/& remotehostname:&/' | xargs -n2 diff
The sed command duplicates the input and makes whatever modifications you need. The xargs then passes the inputs to diff two at a time. This will not work if any filename contain spaces.
bash method
find . -type f | grep -v .svn | while read line; do
diff "$line" "remotehostname:$line"
done
The bash read command reads a line from stdin, places it in the name variable, $line, and returns true. You can then put whatever you like inside the loop, so you get total freedom to rewrite the filename however you need. When the input runs out, read returns false, and the loop exits.
Note that piping things into loops has some interesting side effects that are not relevant here, but might bite you one day.
If you are interested in the actual difference (and not just whether they differ - which rsync is brilliant for telling you) then you can do this using GNU Parallel:
find . -type f | grep -v .svn |
parallel diff {} '<(ssh {= s:./::;s:/.*:: =} cat {= s:([^/]+/){2,2}::;$_=::shell_quote_scalar($_) =})'
s:./::;s:/.*:: = hostname from path
s:([^/]+/){2,2}:: = rest of path
::shell_quote_scalar = \-quote special chars as needed by the shell
GNU Parallel is a general parallelizer and makes is easy to run jobs in parallel on the same machine or on multiple machines you have ssh access to. It can often replace a for loop.
If you have 32 different jobs you want to run on 4 CPUs, a straight forward way to parallelize is to run 8 jobs on each CPU:
GNU Parallel instead spawns a new process when one finishes - keeping the CPUs active and thus saving time:
Installation
If GNU Parallel is not packaged for your distribution, you can do a personal installation, which does not require root access. It can be done in 10 seconds by doing this:
(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash
For other installation options see http://git.savannah.gnu.org/cgit/parallel.git/tree/README
Learn more
See more examples: http://www.gnu.org/software/parallel/man.html
Watch the intro videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial: http://www.gnu.org/software/parallel/parallel_tutorial.html
Sign up for the email list to get support: https://lists.gnu.org/mailman/listinfo/parallel
I am trying to find executable files. Trying to use bourne shell /bin/sh for greater portability. Below script echos everything with find: at beginning of string.
#!/bin/sh
DIRS=`find / -perm -4000`
for DIR in "$DIRS"
do
case "$DIR" in
find:*);;
esac
done
QUESTION) Why is it echoing for find:*) when no commands are given?
If i add *) echo "$DIR";; clause to the case statement, it will echo the files that are executable for current user, this is all i really want, but isn't happening (i haven't scripted for /bin/sh really, but this has bewildered me)
Yeah sed, awk, cut can help immensely, but some of these commands most likely will not be available (why aren't they available. because they might not be!) so i thought a bourne shell version is more portable. Maybe there is a better way for /bin/sh substring matching, any ideas?
The lines that you are trying to get rid of presumably look like this:
find: `/root': Permission denied
That's an error message. The command substitution
`find ...`
only captures output, not errors. You need to add a redirection to include the errors:
`find ... 2>&1`
Also, -perm 4000 is the setuid bit, not an executable bit.
You can put find directly in the for loop
for DIR in `find / -perm -4000`
I am having problems with the greps in Emacs.
a) grep doesnt seem to understand the .[ch] for searching .c and .h files. This is a default option provided by Emacs with the lgrep command. The example is searching for the word "global" in .c/.h files.
grep -i -nH "global" *.[ch]
grep: *.[ch]: No such file or directory
Grep exited abnormally with code 2 at Mon Feb 16 19:34:36
Is this format not valid?
b) Using rgrep I get the following error:
find . "(" -path "*/CVS" -o -path "*/.svn" -o -path "*/{arch}" -o -path "*/.hg" -o -path "*/_darcs" -o -path "*/.git" -o -path "*/.bzr" ")" -prune -o -type f "(" -iname "*.[ch]" ")" -print0 | xargs -0 -e grep -i -nH "global"
FIND: Wrong parameter format
Grep finished (matches found) at Mon Feb 16 19:37:10
I am using Emacs 22.3.1 on Windows XP with the GNU W32 Utils (grep, find, xargs etc.). Grep v2.5.3 and find v4.2.20.
What am I missing?
UPDATE:
Too bad one can't accept multiple answers...since the solution to my problems are spread out.
grep -i -nH "global" *.c *.h
This solves the first problem. Thanks luapyad!
(setq find-program "c:\\path\\to\\gnuw32\\find.exe")
emacs was indeed using the Windows find.exe. Forcing the gnu32 find fixed the second problem. Thanks scottfrazer.
However, I still like ack best.
I found out that using:
(setq find-program "\"C:/path/to/GnuWin32/bin/find.exe\"")
(setq grep-program "\"C:/path/to/GnuWin32/bin/grep.exe\"")
Works better in windows, since you could have a space laying around the path and will screw up eventually.
Notice I used the two programs in my .emacs file.
Hope it's of some help to some other programmer in need ;)
Well, there is always Ack and Ack.el
For a) it looks like there are simply no .c or .h files in the current directory.
For b) Windows is trying to use its own find instead of the one from the GNU W32 Utils. Try:
(setq find-program "c:\\path\\to\\gnuw32\\find.exe")
Adam Rosenfield comment is worth expanding into an answer:
grep -r --include=\*.[ch] --exclude=\*{CVS,.svn,arch} -i -nH
To make the example given in this question work, use this:
grep -i -nH --include=\*.[ch] "global" *
It is also helpful to set the variable grep-command providing defaults to M-x grep:
(setq grep-command "grep -i -nH --include=\*.[ch] ")
Also here are some other useful command line parameters to grep:
-n print the line number
-s suppress error messages
-r recursive
I think the general problem is the windows cmd "shell" behaves very differently to a unix shell in respect to filename expansion regexps and wildcards.
To answer your (a) above try using:
grep -i -nH "global" *.c *.h
(You will still get an "invalid argument" if no *.c's or *.h's exist).
Or you can use command line option --include=\*.[ch] to make windows grep do "proper" filename pattern matching (see grep --help for other options)
I usually just use M-x grep and alter the command line args when prompted if I need to. But I just tried running M-x lgrep and got the same thing as you. It simply means that no files match *.[ch] in the current directory. You can customize the default options to include -r and search recursively through child directories as well:
M-x customize-group RET grep RET
Search for lgrep in that buffer to find/edit the Grep Template.
As far as M-x rgrep goes, I suspect it has something to do with the Windows version of find not liking the default options. That command works fine for me on Linux. Search for rgrep in that same customize buffer and tweak those options until the Windows find is happy.
Sorry I can't be more help with the Windows options, but I'm not familiar with them.