I am basically looking for a way to do this
list=$(command)
while read -r arg
do
...
done <<< "$list"
Using sh intead of bash. The code as it is doesn't run because of the last line:
syntax error: unexpected redirection
Any fixes?
Edit: I need to edit variables and access them outside the loop, so using | is not acceptable (as it creates a sub-shell with independent scope)
Edit 2: This question is NOT similar to Why does my Bash counter reset after while loop as I am not using | (as I just noticed in the last edit). I am asking for another way of achiving it. (The answers to the linked question only explain why the problem happens but do not provide any solutions that work with sh (no bash).
There's no purely syntactic way to do this in POSIX sh. You'll need to use either a temporary file for the output of the command, or a named pipe.
mkfifo output
command > output &
while read -r arg; do
...
done < output
rm output
Any reason you can't do this? Should work .. unless you are assigning any variables inside the loop that you want visible when it's done.
command |
while read -r arg
do
...
done
Related
I'm working with a command line utility that requires passing the name of a file to write output to, e.g.
foo -o output.txt
The only thing it writes to stdout is a message that indicates that it ran successfully. I'd like to be able to pipe everything that is written to output.txt to another command line utility. My motivation is that output.txt will end up being a 40 GB file that I don't need to keep, and I'd rather pipe the streams than work on massive files in a stepwise manner.
Is there any way in this scenario to pipe the real output (i.e. output.txt) to another command? Can I somehow magically pass stdout as the file argument?
Solution 1: Using process substitution
The most convenient way of doing this is by using process substitution. In bash the syntax looks as follows:
foo -o >(other_command)
(Note that this is a bashism. There's similar solutions for other shells, but bottom line is that it's not portable.)
Solution 2: Using named pipes explicitly
You can do the above explicitly / manually as follows:
Create a named pipe using the mkfifo command.
mkfifo my_buf
Launch your other command with that file as input
other_command < my_buf
Execute foo and let it write it's output to my_buf
foo -o my_buf
Solution 3: Using /dev/stdout
You can also use the device file /dev/stdout as follows
foo -o /dev/stdout | other_command
Named pipes work fine, but you have a nicer, more direct syntax available via bash process substitution that has the added benefit of not using a permanent named pipe that must later be deleted (process substitution uses temporary named pipes behind the scenes):
foo -o >(other command)
Also, should you want to pipe the output to your command and also save the output to a file, you can do this:
foo -o >(tee output.txt) | other command
For the sake of making stackoverflow happy let me write a long enough sentence because my proposed solution is only 18 characters long instead of the required 30+
foo -o /dev/stdout
You could use the magic of UNIX and create a named pipe :)
Create the pipe
$ mknod -p mypipe
Start the process that reads from the pipe
$ second-process < mypipe
Start the process, that writes into the pipe
$ foo -o mypipe
foo -o <(cat)
if for some reason you don't have permission to write to /dev/stdout
I use /dev/tty as the output filename, equivalent to using /dev/nul/ when you want to output nothing at all. Then | and you are done.
I'm trying to do a simple operation in ksh that I need to repeat in sh (Bourne shell)
All I want to do is append the contents of the first line of hte pay_period.txt file to the end of the new file name. This works great in ksh, but does not work in bourne. The program I'm using defaults to sh and I can't change that. Also I can't have actual shell scipts in the directories. So I have to issue commands.
How can I make the equivalent command below work in bourne
mv HEPAY.txt HE_PAY"$(/usr/bin/head -1 pay_period.txt)."txt
The results of $(/usr/bin/head -1 pay_period.txt) is 20140101.
If you are really talking about a real Bournce shell then you need to use backticks for command substitution ($() is POSIX and portable among "modern", POSIX-compliant shells but won't work in old, legacy shells), e.g.
mv HEPAY.txt HE_PAY`/usr/bin/head -1 pay_period.txt`.txt
Other than that I see no reason why this should not work.
PS: Note that head -1 isn't POSIX-compliant either (head -n 1 is).
In bash shell scripting, I would typically run :> file to empty a file.
Now using fish, things are slightly different and the above command doesn't work.
What is fish equivalent?
Although it's not as short as :, true is a command that will work everywhere and produces no output:
true > file
Probably the easiest way that will be work in both Fish and Bash is to do echo "" > file
EDIT: Commenter was absolutely right echo "" > file produces a file with a newline, the correct command I was thinking of to create an empty file is cat /dev/null > file.
There is, and always was the magic method called touch which set change time to actual or create non-existent file. For compatiblity I suggest you to use this way in all scripts that you write (even if you write bash code).
I am asked to diff two directories using Perl but I think something is wrong with my command,
$diff = system("sudo diff -r '/Volumes/$vol1' '/Volumes/$vol2\\ 1/' >> $diff.txt");
It doesn't display and output. Can someone help me with this? Thanks!
It seems that you want to store all differences in a string.
If this is the case, the command in the question is not going to work for a few reasons:
It's hard to tell whether it's intended or not, but the $diff variable is being used to set the filename storing the differences. Perhaps this should be diff.txt, not $diff.txt
The result of the diff command is saved in $diff.txt. It doesn't display anything in STDOUT. This can be remedied by omitting the >> $diff.txt part. If it also needs to be stored in file, consider the tee command:
sudo diff -r dir1/ dir2/ | tee diff.txt
When a system call is assigned to a variable, it will return 0 upon success. To quote the documentation:
The return value is the exit status of the program as returned by the wait call.
This means that $diff won't store the differences, but the command exit status. A more sensible approach would be to use backticks. Doing this will allow $diff to store whatever is output to STDOUT by the command:
my $diff = `sudo diff -r dir1/ dir2/ | tee diff.txt`; # Not $diff.txt
Is it a must to use the sudo command? Avoid using it if even remotely possible:
my $diff = `diff -r dir1/ dir2/ | tee diff.txt`; # Not $diff.txt
A final recommendation
Let a good CPAN module take care of this task, as backtick calls can only go so far. Some have already been suggested here; it may be well worth a look.
Is sudo diff being prompted for a password?
If possible, take out the sudo from the invocation of diff, and run your script with sudo.
"It doesn't display and output." -- this is becuase you are saving the differences to a file, and then (presumably) not doing anything with that resulting file.
However, I expect "diff two directories using Perl" does not mean "use system() to do it in the shell and then capture the results". Have you considered doing this in the language itself? For example, see Text::Diff. For more nuanced control over what constitutes a "difference", you can simply read in each file and craft your own algorithm to perform the comparisons and compile the similarities and differences.
You might want to check out Test::Differences for a more flexible diff implementation.
It seems to me that there are two ways to run Matlab in batch mode:
the first one:
unset DISPLAY
matlab > matlab.out 2>&1 << EOF
plot(1:10)
print file
exit
EOF
The second one uses option "-r MATLAB_command":
matlab -nojvm -nosplash -r MyCommand
Are these two equivalent?
What does "<< EOF" and the last "EOF" mean in the first method?
Thanks and regards!
The first method simply redirects the standard output > matlab.out and the standard error 2>&1 to the file matlab.out.
Then it uses the heredoc way of passing input to MATLAB (this is not specific to MATLAB, it is a method of passing multiple lines as input to command line programs in general).
The syntax is << followed by an unique identifier, then your text, finally the unique id to finish.
You can try this on the shell:
cat << END
some
text
multiple lines
END
The second method of using the -r option starts MATLAB and execute the statement passed immediately. It could be some commands or the name of a script or function found on the path.
It is equivalent to doing something like:
python -c "print 'hello world'"
Refer to this page for a list of the other start options.