In yq 4.11.2, If I do this:
echo '---\nfoo: foo\n---\nfoo: foo\n' | yq eval '.foo' -
I get this:
foo
---
foo
Using yq only (not grep, awk, etc.), how can I remove the --- to get this?
foo
foo
There is a problem with your echo invocation, that it does not expand literal newline characters by default, unless you run with echo -e or enable xpg_echo shell option if you are running this on bash/zsh
mikefarah/yq implementation has a mode -N that prints the filter output with out the document separators See yq --help when tested on version 4.13.3
echo -e '---\nfoo: foo\n---\nfoo: foo\n' | yq -N e '.foo' -
foo
foo
Related
I am trying to separate out parts of a path as follows. My input path takes the following possible forms:
bucket
bucket/dir1
bucket/dir1/dir2
bucket/dir1/dir2/dir3
...
I want to separate the first part of the path (bucket) from the rest of the string if present (dir1/dir2/dir3/...), and store both in separate variables.
The following gives me something close to what I want:
❯ BUCKET=$(echo "bucket/dir1/dir2" | sed 's#\(^[^\/]*\)[\/]\(.*\)#\1#')
❯ EXTENS=$(echo "bucket/dir1/dir2" | sed 's#\(^[^\/]*\)[\/]\(.*\)#\2#')
echo $BUCKET $EXTENS
❯ bucket dir1/dir2
HOWEVER, it fails if I only have bucket as input (without a slash):
❯ BUCKET=$(echo "bucket" | sed 's#\(^[^\/]*\)[\/]\(.*\)#\1#')
❯ EXTENS=$(echo "bucket" | sed 's#\(^[^\/]*\)[\/]\(.*\)#\2#')
echo $BUCKET $EXTENS
❯ bucket bucket
... because, in the absence of the first '/', no capture happens, so no substitution takes place. When the input is just 'bucket' I would like $EXTENS to be set to the empty string "".
Thanks!
For something so simple you could use bash built-in instead of launching sed:
$ path="bucket/dir1/dir2"
$ bucket="${path%%/*}"
$ extens="${path#$bucket}"
$ printf '|%s|%s|\n' "$bucket" "$extens"
|bucket|/dir1/dir2|
$ path="bucket"
$ bucket="${path%%/*}"
$ extens="${path#$bucket}"
$ printf '|%s|%s|\n' "$bucket" "$extens"
|bucket||
But if you really want to use sed and capture groups:
$ declare -a bucket_extens
$ mapfile -td '' bucket_extens < <(printf '%s' "bucket/dir1/dir2" | sed -E 's!([^/]*)(.*)!\1\x00\2!')
$ printf '|%s|%s|\n' "${bucket_extens[#]}"
|bucket|/dir1/dir2|
$ mapfile -td '' bucket_extens < <(printf '%s' "bucket" | sed -E 's!([^/]*)(.*)!\1\x00\2!')
$ printf '|%s|%s|\n' "${bucket_extens[#]}"
|bucket||
We use the extended regex (-E) to simplify a bit, and ! as separator of the substitute command. The first capture group is simply anything not containing a slash and the second is everything else, including nothing if there's nothing else.
In the replacement string we separate the two capture groups with a NUL character (\x00). We then use mapfile to assign the result to bash array bucket_extens.
The NUL trick is a way to deal with file names containing spaces, newlines... NUL is the only character that cannot be part of a file name. The -d '' option of mapfile indicates that the lines to map are separated by NUL instead of the default newline.
Don't capture anything. Instead, just match what you don't want and replace it with nothing:
BUCKET=$(echo "bucket" | sed 's#/.*##'). # bucket
BUCKET=$(echo "bucket/dir1/dir2" | sed 's#/.*##') # bucket
EXTENS=$(echo "bucket" | sed 's#[^/]*##') # blank
EXTENS=$(echo "bucket/dir1/dir2" | sed 's#[^/]*##') # /dir1/dir2
As you are putting a slash in the regex. the string with no slashes will not
match. Let's make the slash optional as /\?. (A backslash before ?
is requires due to the sed BRE.) Then would you please try:
#!/bin/bash
#path="bucket/dir1/dir2"
path="bucket"
bucket=$(echo "$path" | sed 's#\(^[^/]*\)/\?\(.*\)#\1#')
extens=$(echo "$path" | sed 's#\(^[^/]*\)/\?\(.*\)#\2#')
echo "$bucket" "$extens"
You don't need to prepend a backslash to a slash.
By convention, it is recommended to use lower cases for user variables.
I want to replace a placeholder on a file <<string>> in the example to the contents of a var that has several special characters.
file.txt
My string: <<string>>
script.sh
STRING="something-else;device=name.of.device;key=abcd1234/wtp="
sed -i "s/<<string>>/${STRING}/g" file.txt
I get this error:
sed: -e expression #1, char 165: unknown option to `s'
I already use this sed command for other vars that do not have special characters. Any way to escape the var ${STRING} entirely?
You can't do this job easily and robustly with sed, see Is it possible to escape regex metacharacters reliably with sed. Instead just use a tool like awk that understands literal strings:
$ string='~`!##$%^&*()-_+={[}]|\:;"'\''<,>.?/\1'
$ echo "$string"
~`!##$%^&*()-_+={[}]|\:;"'<,>.?/\1
$ string="$string" awk -i inplace 'match($0,/(.*)(<<string>>)(.*)/,a){ $0=a[1] ENVIRON["string"] a[3] } 1' file.txt
$ cat file.txt
My string: ~`!##$%^&*()-_+={[}]|\:;"'<,>.?/\1
That above will work for any characters (or backreference substrings like \1) that string might contain because it's simply using a literal string operation (concatenation) for the replacement.
It's using GNU awk for -i inplace just the same as your original script used GNU sed for -i.
Don't use all upper case for non-exported variable names by the way to avoid clashes with exported and built-in variables and not obfuscate your code by making it look like you're using exported variables, see Correct Bash and shell script variable capitalization.
Note that if you have multiple <<whatever>> placeholders you can easily parameterize the above, e.g.:
$ foo='Now is the Winter'
$ bar='Of our discontent'
$ cat file.txt
My foo string: <<foo>>
My bar string: <<bar>>
$ foo="$foo" bar="$bar" awk -i inplace 'match($0,/(.*)<<(\w+)>>(.*)/,a) && (a[2] in ENVIRON){ $0=a[1] ENVIRON[a[2]] a[3] } 1' file.txt
$ cat file.txt
My foo string: Now is the Winter
My bar string: Of our discontent
If you don't want to set foo and bar on the awk command line you can export them before it, or read them from a config file or a here-doc or ... - lots of options.
Since STRING contains a /, you should use a other delimiter, for example, you can use ^ like so:
sed 's^<<string>>^'"$STRING"'^g' file.txt
The quoting logic (''""'') is explained nicely on this SO answer.
Example on my locale machine:
$
$ cat file.txt
My string: <<string>>
$
$
$ STRING="something-else;device=name.of.device;key=abcd1234/wtp="
$
$
$ sed -i 's^<<string>>^'"$STRING"'^g' file.txt
$
$ cat file.txt
My string: something-else;device=name.of.device;key=abcd1234/wtp=
$
$
I'm refactoring some code across lots of files, and am trying to do a search and replace using sed on a Mac.
The goal is to go from this:
fooActions: typeof fooActionCreators
to this:
fooActions: Partial<typeof fooActionCreators>
I've gotten it to mostly work using this sed expression:
ag -l "Actions: typeof " | xargs sed -i "" -e "s/Actions: typeof \(\w*\)/Actions: Partial<typeof \1>/g"
Basically using ag to find the list of file names containing a matching string, then use xargs to pass those as input to my sed expression.
The output I get though looks like this:
fooActions: Partial<typeof >fooActionCreators
I can't figure out why my capture group is being placed at then end instead of where I have it in the replace clause.
The sed that comes with mac doesn't support \w for word characters.
If you have gsed (gnu sed) this should work as expected.
$ echo 'Actions: typeof foo' | gsed -e "s/Actions: typeof \(\w*\)/Actions: Partial<typeof \1>/g"
Actions: Partial<typeof foo>
Otherwise you can use something like [a-zA-Z]* instead of \w*.
$ echo 'Actions: typeof foo' | sed -e "s/Actions: typeof \([a-zA-Z]*\)/Actions: Partial<typeof \1>/g"
Actions: Partial<typeof foo>
The seemingly odd behavior that you're seeing is because in the plain sed version, the \w* is matching the empty string in front of the thing you're hoping to match, as illustrated below:
Plain sed matches empty string in front of a
$ echo a | sed "s/\(\w*\)/\1x/"
xa
Gnu sed matches the a as intended
$ echo a | gsed "s/\(\w*\)/\1x/"
ax
if your data in 'd' file, by gnu sed,
sed -E 's/^(fooAction)s: (typeof \1Creators)/\1s: Partial<\2>/' d
In Bourne shell I have a string variable s which contains the character sequence foo $(ls) bar (which is not a string literal). How can I get a string where the ls command is replaced by its output?
You would have to use eval, although this isn't really recommended unless you have complete control over how the value of s is set.
# Modified slightly for this example
$ s='foo $(echo baz) bar'
$ eval "s=\"$s\""
$ echo "$s"
foo baz bar
i want to trim an output of wipe command with sed.
i try to use this one:
wipe -vx7 /dev/sdb 2>&1 | sed -u 's/.*\ \([0-9]\+\).*/\1/g'
but it don't work for some reason.
when i use echo & sed to print the output of wipe command it works!
echo "/dev/sdb: 10%" | sed -u 's/.*\ \([0-9]\+\).*/\1/g'
what i'm doing wrong?
Thanks!
That looks like a progress indicator. They are often output directly to the tty instead of to stdout or stderr. You may be able to use the expect script called unbuffer (source) or some other method to create a pseudo tty. Be aware that there will probably be more junk such as \r, etc., that you may need to filter out.
Demonstration:
$ cat foo
#!/bin/sh
echo hello > /dev/tty
$ a=$(./foo)
hello
$ echo $a
$ a=$(unbuffer ./foo)
$ echo $a
hello