The terminal transcript speaks for itself:
iMac:~$ echo -n a | md5
0cc175b9c0f1b6a831c399e269772661
iMac:~$ perl -e 'system "echo -n a | md5"'
c3392e9373ccca33629d82b17699420f
Note that the MD5 hash of a is 0cc175b9c0f1b6a831c399e269772661, the first
result. Why does it turns out to be different when the same command is called
by perl?
By the way, perl is perl 5, version 12, subversion 4 (v5.12.4) built for darwin-thread-multi-2level. And the system: Mac OS 10.8, Darwin 12.0
When in the /bin/sh shell on mac, echo -n doesn't not print out the newline like it does in /bin/bash. You can see this if you drop into /bin/sh and run echo -n a, your output should look like this:
sh-3.2$ echo -n a
-n a
so you're literally getting -n a instead of the desired a. As perl system runs /bin/sh to evaluate your command, -n a is being passed into md5 instead of your desired a
The specific question has already been answered, but I want to point out that od is useful to help understand exactly what any command outputs or file contains. This is useful especially to show otherwise non-printing characters.
$ echo -n a | od -tc
0000000 a
0000001
$ perl -e 'system "echo -n a | od -tc";'
0000000 - n a \n
0000005
Related
I'm trying to migrate this working command
docker-compose $(find docker-compose* | sed -e "s/^/-f /") up -d --remove-orphans
from bash to fish. The intention of this command is to get this
docker-compose -f docker-compose.backups.yml ... -f docker-compose.wiki.yml up -d --remove-orphans
My naive try
docker-compose (find docker-compose* | sed -e "s/^/-f /") up -d --remove-orphans
is not working, though. The error is:
ERROR: .FileNotFoundError: [Errno 2] No such file or directory: './ docker-compose.backups.yml'
What is the correct translation?
The difference in behavior is due to the fact fish, sanely, only splits the output of a command capture on line boundaries. Whereas POSIX shells like bash split it on whitespace by default. That is, POSIX shells split the output of $(...) on the value of $IFS which is space, tab, and newline by default.
There are several ways to rewrite that command so it works in fish. The one that requires the smallest change is to change the sed to insert a newline between the -f and the filename:
docker-compose (find docker-compose* | sed -e "s/^/-f\n/") up -d --remove-orphans
With Sed, I want to use "!" to exclude lines that matches "he". Here is an example.
echo "hello" |sed "/he/!s/hello/hi/"
To my surprise, my ubuntu 14 returns
"bash !s/hello/hi: event not found"
error. Any ideas? How could I exclude line ranges corresponding to a pattern with Sed?
You have history expansion enabled. You need to disable it with set +H.
Example
Let's enable history expansion and run your command:
$ set -H
$ echo "hello" |sed "/he/!s/hello/hi/"
bash: !s/hello/hi/: event not found
Now, let's disable it and observe that the command now runs correctly and without error:
$ set +H
$ echo "hello" |sed "/he/!s/hello/hi/"
hello
Alternative
If you want to keep history expansion enabled, then single-quote your string:
$ set -H
$ echo "hello" |sed '/he/!s/hello/hi/'
hello
I have the following sed command:
sed -i -E "s/\{\{(.*)\}\}/$(echo "$\1")/g" test.conf
In test.conf, I have this:
this is a {{TEST}}
and this is an {{ANSWER}} here.
And I have the follow environment variables set:
export TEST=1234
export ANSWER=5678
When I run the sed command, I end up with this result:
this is a $TEST
and this is an $ANSWER here.
I want 1234 and 5678 there respectively. Is there a reason the echo command is interpreting things literally?
Backreferences are used internally by a single sed command. The echo has no idea about sed backreferences and would have been invoked by the shell before the sed command has even run so the $(echo "$\1") is outputing $\1 so
sed -i -E "s/\{\{(.*)\}\}/$(echo "$\1")/g" test.conf
is really:
sed -i -E "s/\{\{(.*)\}\}/$\1/g" test.conf
hence the output you are seeing.
Anyway, sed is for simple subsitutions on individual lines, for anything else you should be using awk:
$ export TEST=1234 ANSWER=5678
$ awk 'match($0,/(.*)\{\{(.*)\}\}(.*)/,a){$0=a[1] ENVIRON[a[2]] a[3]} 1' file
this is a 1234
and this is an 5678 here.
The above uses GNU awk for the 3rd arg to match(), with other awks it'd be:
$ awk 'match($0,/\{\{(.*)\}\}/){$0=substr($0,1,RSTART-1) ENVIRON[substr($0,RSTART+2,RLENGTH-4)] substr($0,RSTART+RLENGTH)} 1' file
this is a 1234
and this is an 5678 here.
If anyone suggests running eval or similar on the sed output - don't do it (google eval is evil and friends), just use the awk command above for simple string operations.
You can use perl which conveniently has a %ENV hash variable, see perldoc for more info
perl -pe 's/\{\{(.*)\}\}/$ENV{$1}/' test.conf
I want to filter the output of the blkid to get the UUID.
The output of blkid looks like
CASE 1:-
$ blkid
/dev/sda2: LABEL="A" UUID="4CC9-0015"
/dev/sda3: LABEL="B" UUID="70CF-169F"
/dev/sda1: LABEL=" NTFS_partition" UUID="3830C24D30C21234"
In somecases the output of blkid looks like
CASE 2:-
$ blkid
/dev/sda1: UUID="d7ec380e-2521-4fe5-bd8e-b7c02ce41601" TYPE="ext4"
/dev/sda2: UUID="fc54f19a-8ec7-418b-8eca-fbc1af34e57f" TYPE="ext4"
/dev/sda3: UUID="6f218da5-3ba3-4647-a44d-a7be19a64e7a" TYPE="swap"
I want to filter out the UUID.
Using the combination of grep and cut it can be done as
/sbin/blkid | /bin/grep 'sda1' | /bin/grep -o -E 'UUID="[a-zA-Z|0-9|\-]*' | /bin/cut -c 7-
I have tried using awk , grep and cut as below for filtering the UUID
$ /sbin/blkid | /bin/grep 'sda1' | /usr/bin/awk '{print $2}' | /bin/sed 's/\"//g' | cut -c 7-
7ec380e-2521-4fe5-bd8e-b7c02ce41601
The above command(which uses awk) is not reliable since sometimes an extra field such as LABEL may be present in the output of the blkid program as shown in the above output.
What is the best way to create a command using awk which works reliably?
Please post if any other elegant method exits for the job using bin and core utils. I dont want to use perl or python since this has to be run on busybox.
NOTE:-I am using busybox blkid to which /dev/sda1 can not be passed as the args(the version i am using does not support it) hence the grep to filter the line.
UPDATE :- added the CASE 2: -output to show that field position can not be relied upon.
Why are you making it so complex?
Try this:
# blkid -s UUID -o value
d7ec380e-2521-4fe5-bd8e-b7c02ce41601
fc54f19a-8ec7-418b-8eca-fbc1af34e57f
6f218da5-3ba3-4647-a44d-a7be19a64e7a
Or this:
# blkid -s UUID -o value /dev/sda1
d7ec380e-2521-4fe5-bd8e-b7c02ce41601
Install proper blkid package if you don't have it:
sudo apt-get install util-linux
sudo yum install util-linux
For all the UUID's, you can do :
$ blkid | sed -n 's/.*UUID=\"\([^\"]*\)\".*/\1/p'
d7ec380e-2521-4fe5-bd8e-b7c02ce41601
fc54f19a-8ec7-418b-8eca-fbc1af34e57f
6f218da5-3ba3-4647-a44d-a7be19a64e7a
Say, only for a specific sda1:
$ blkid | sed -n '/sda1/s/.*UUID=\"\([^\"]*\)\".*/\1/p'
d7ec380e-2521-4fe5-bd8e-b7c02ce41601
The sed command tries to group the contents present within the double quotes after the UUID keyword, and replaces the entire line with the token.
Here's a short awk solution:
blkid | awk 'BEGIN{FS="[=\"]"} {print $(NF-1)}'
Output:
4CC9-0015
70CF-169F
3830C24D30C21234
Explanation:
BEGIN{FS="[=\"]"} : Use = and " as delimiters
{print $(NF-1)}: NF stands of Number of Fields; here we print the 2nd to last field
This is based on the consistent structure of blkid output: UUID in quotes is at the end of each line.
Alternatively:
blkid | awk 'BEGIN{FS="="} {print $NF}' | sed 's/"//g'
data.txt
/dev/sda2: LABEL="A" UUID="4CC9-0015"
/dev/sda3: LABEL="B" UUID="70CF-169F"
/dev/sda1: LABEL=" NTFS_partition" UUID="3830C24D30C21234"
awk and sed combination
cat data.txt | awk 'BEGIN{FS="UUID";RS="\n"} {print $2}' | sed -e 's/=//' -e 's/"//g'
Explanation:
Set the Field Separator to the string 'UUID', $2 will give the rest output
use sed then to remove the = and " as shown where -e is a switch so that you can give multiple sed commands/expression in one.
All occurrences of " are removed using the ending g option i.e. global.
The question has a "e.t.c" so I'm going to assume python is one of the options ;)
#!/usr/bin/env python3
import subprocess, re, json
# get blkid output
blkid = subprocess.check_output(["blkid"]).decode('utf-8')
devices = []
for line in [x for x in blkid.split('\n') if x]:
parameters = line.split()
for idx, parameter in enumerate(parameters):
if idx is 0:
devices.append({"DEVICE": re.sub(r':$','',parameter)})
continue
key_and_value = parameter.split('=')
devices[-1].update({
key_and_value[0]: re.sub(r'"','',key_and_value[1])
})
uuids = [{dev['DEVICE']: dev['UUID']} for dev in devices if 'UUID' in dev.keys()]
print(json.dumps(uuids, indent=4, sort_keys=True))
Although, this is probably overkill and quite a few error handling/optimization is missing from this script XD
I assume you're using busybox in an initramfs and you are waiting for your e.g. USB drive with the rootfs on it to become available.
You could use the following awk script (busybox awk compliant).
# cat get-ruuid.awk
BEGIN {
ruuid=ENVIRON["RUUID"]
}
/^\/dev\/sd[a-z]/ {
if (index($0, tolower(ruuid)) || index($0, toupper(ruuid))) {
split($1, parts, ":")
printf("%s\n", parts[1])
exit(0) # Return success and stop further scanning.
}
}
END {
exit(1) # If we reach the end, it means RUUID was not found.
}
Call it as follows from e.g. the init script; this is not the most ideal way.
# The UUID of your root partition
export RUUID="<put proper uuid value here>"
for x in 1, 2, 3, 4, 5 ; do
mdev -s
found=$(blkid | awk -f ./get-ruuid.awk)
test -z $found || break; # If no longer zero length, break the loop.
sleep 1
done
But if this is the only reason why you would want to have an initramfs, I would use the 'root=PARTUUID=... waitroot' Linux kernel command line option. Check the kernel docs and sources.
Get the proper PARTUUID (NOT UUID) of your root partition with the blkid command.
i want to trim an output of wipe command with sed.
i try to use this one:
wipe -vx7 /dev/sdb 2>&1 | sed -u 's/.*\ \([0-9]\+\).*/\1/g'
but it don't work for some reason.
when i use echo & sed to print the output of wipe command it works!
echo "/dev/sdb: 10%" | sed -u 's/.*\ \([0-9]\+\).*/\1/g'
what i'm doing wrong?
Thanks!
That looks like a progress indicator. They are often output directly to the tty instead of to stdout or stderr. You may be able to use the expect script called unbuffer (source) or some other method to create a pseudo tty. Be aware that there will probably be more junk such as \r, etc., that you may need to filter out.
Demonstration:
$ cat foo
#!/bin/sh
echo hello > /dev/tty
$ a=$(./foo)
hello
$ echo $a
$ a=$(unbuffer ./foo)
$ echo $a
hello