How do I remove the text till the first ( and after )?
INSERT INTO `todel` VALUES (1,'akbar\'s','Mumbai, Delhi road, India');
INSERT INTO `todel` VALUES (2,'amar\"s','South Africa, ghana');
The expected output is like this...
1,'akbar\'s','Mumbai, Delhi road, India'
2,'amar\"s','South Africa, ghana'
Ruby(1.9+)
$> ruby -ne 'print $_.sub(/.*\(|\).*$/,"")' file
1,'akbar\'s','Mumbai, Delhi road, India'
2,'amar\"s','South Africa, ghana'
or the shell(bash)
$> while read -r line; do line=${line#*(}; echo ${line%)*}; done <file
1,'akbar\'s','Mumbai, Delhi road, India'
2,'amar\"s','South Africa, ghana'
or awk
$> awk '{sub(/.*\(/,"");sub(/\).*/,"")}1' file
1,'akbar\'s','Mumbai, Delhi road, India'
2,'amar\"s','South Africa, ghana'
or sed
$> sed -rn 's/.*\(//;s/\).*//p' file
1,'akbar\'s','Mumbai, Delhi road, India'
2,'amar\"s','South Africa, ghana'
awk can take a regular expression as field separator, so use either parenthesis as the field separator and just emit the 2nd field:
awk -F'[()]' '{print $2}' filename
You can remove everything from beginning of line until the first ( and from (including) the last ) till the end of line with sed:
sed -r 's/^[^(]*\(.*)\)[^)]*$/\1/'
Related
is it possible to use the pipe to redirect the output of the previous command, to sed, and let sed use this as input(pattern or string) to access a file?
I know if you only use sed, you can use something like
sed -i '1 i\anything' file
But can I do something like
head -1 file1 | sed -i '1 i\OutputFromPreviousCmd' file2
This way, I don't need to manually copy the output and change the sed command everytime
Update:
Added the files I meant
head -3 file1.txt
Side A,Age(us),mm:ss.ms_us_ns_ps
84 Vendor Specific, 0000000009096, 0349588242
84 Vendor Specific, 0000000011691, 0349591828
head -3 file2.txt
84 Vendor Specific, 0000000000418, 0349575322
83 Vendor Specific, 0000000002099, 0349575343
83 Vendor Specific, 0000000001628, 0349576662
I'd like to grab the first line of file1 and insert it to file2, so the result should be :
head -3 file2.txt
Side A,Age(us),mm:ss.ms_us_ns_ps
84 Vendor Specific, 0000000000418, 0349575322
83 Vendor Specific, 0000000002099, 0349575343
83 Vendor Specific, 0000000001628, 0349576662
head -1 file1 | sed '1s/^/1i /' | sed -i -f- file2
This takes your one line of output, prepends the sed 1i command, the pipes that sed command stream to sed using -f- to take sed commands from stdin.
For example:
$ echo bob > bob.txt
$ echo alice | sed '1s/^/1i /' | sed -i -f- bob.txt
$ more bob.txt
alice
bob
This looks like pipes and not commands ending in > temp ; mv temp file2, but sed is doing that nonetheless when -i is used.
This might work for you (GNU sed):
head -1 file1 | sed -i '1e cat /dev/stdin' file2
Insert the first line of file1 into the start of file2.
But why not use cat?:
cat <(head -1 file1) file2
I'm trying to remove a range of words in Unix command line with sed from a file and I just can't figure it out. For example, how can I remove the words at positions 2-4?
If the file contains: "This is a file created by me." I want it to be: "This created by me."
Thanks a lot!
Try this with GNU sed (to print word 1 and word 5 to last word):
echo "This is a file created by me." | sed 'y/ /\n/' | sed -n '1p;5,$p' | sed 'N;N;N;y/\n/ /'
Output:
This created by me.
You can use also use awk for this:
echo "This is a file created by me." | awk '{for (i=1;i<=NF;i++) if (i<2||i>4) printf "%s ",$i;print ""}'
This created by me.
This might work for you (GNU sed):
sed -r 's/(\s+\S+){3}//' file
I have a multi-line string, downloaded from the Web:
toast the lemonade
blend with the lemonade
add one tablespoon of the lemonade
grill the spring onions
add the lemonade
add the raisins to the saucepan
rinse the horseradish sauce
I have assigned this to $INPUT, like this:
INPUT=$(lynx --dump 'http://example.net/recipes' \
| python -m json.tool \
| awk '/steps/,/]/' \
| egrep -v "steps|]" \
| sed 's/[",]\|^ *//g; $d')
At this point, $INPUT is ready for substitution into my target file as follows:
sed -i "0,/OLDINPUT/s//$INPUT/" /home/test_file
Of course, sed complains about an unterminated s command - herein lies the problem.
The current workaround I am using is to echo $INPUT prior to giving it to sed, but then the newlines are not preserved. echo strips newlines - which is the problem.
The correct output should maintain its newlines. How can sed be instructed to preserve the newlines?
The hacky direct answer is to replace all newlines with \n, which you can do by adding
| sed ':a $!{N; ba}; s/\n/\\n/g'
to the long command above. A better answer, because substituting shell variables into code is always a bad idea and with sed you wouldn't have a choice, is to use awk instead:
awk -i inplace -v input="$INPUT" 'NR == 1, /OLDINPUT/ { sub(/OLDINPUT/, input) } 1' /home/test_file
This requires GNU awk 4.1.0 or later for the -i inplace.
If you're using Bash, you can substitute \n for the newlines:
INPUT="${INPUT//
/\\n}"
If you don't like the literal linefeed in your parameter expansion, you might prefer
INPUT="${INPUT//$'\n'/\\n}"
Side note - you probably mean to change the matched lines to your input, not substitute each of them. In which case, you don't want to quote the newlines, after all...
To clean up your code some.
This:
lynx --dump 'http://somesite.net/recipes' | python -m json.tool | awk '/steps/,/]/' | egrep -v "steps|]" | sed 's/"//g' |sed 's/,//g' | sed 's/^ *//g' | sed '$d'
Can be replaced with this:
lynx --dump 'http://somesite.net/recipes' | python -m json.tool | awk '/]/ {f=0} f {if (c--) print line} /steps/{f=1} {gsub(/[",]|^ */,"");line=$0}'
It may be shorten more, but I do not now what this does: python -m json.tool
This:
awk '/]/ {f=0} f {if (c--) print line} /steps/{f=1} {gsub(/[",]|^ */,"");line=$0}'
Does:
Print line after pattern steps to line before ] - awk '/steps/,/]/' | egrep -v "steps|]"
Removes ", , and all space in front of all lines. - sed 's/"//g' |sed 's/,//g' | sed 's/^ *//g'
Then remove last line of this group. - sed '$d'
Example:
cat file
my data
steps data
more
do not delet this
hei "you" , more data
extra line
here is end ]
this is good
awk '/]/ {f=0} f {if (c--) print line} /steps/{f=1} {gsub(/[",]|^ */,"");line=$0}' file
more
do not delet this
hei you more data
Assuming your input JSON fragment looks something like this:
{ "other": "random stuff",
"steps": [
"toast the lemonade",
"blend with the lemonade",
"add one tablespoon of the lemonade",
"grill the spring onions",
"add the lemonade",
"add the raisins to the saucepan",
"rinse the horseradish sauce"
],
"still": "yet more stuff" }
you can extract just the steps member with
jq -r .steps
To interpolate that into a sed statement, you'd need to escape any regex metacharacters in the result. A less intimidating and hopefully slightly less hacky solution would be to read static text from standard input:
lynx ... | jq ... |
sed -i -e '/OLDINPUT/{s///; r /dev/stdin' -e '}' /home/test_file
The struggle to educate practitioners to use structure-aware tools for structured data has reached epic heights and continues unabated. Before you decide to use the quick and dirty approach, at least make sure you understand the dangers (technical and mental).
You'll want to use an editor instead of sed's substitution:
$ input="toast the lemonade
blend with the lemonade
add one tablespoon of the lemonade
grill the spring onions
add the lemonade
add the raisins to the saucepan
rinse the horseradish sauce"
$ seq 10 > file
$ ed file <<END
1,/5/d
1i
$input
.
w
q
END
$ cat file
toast the lemonade
blend with the lemonade
add one tablespoon of the lemonade
grill the spring onions
add the lemonade
add the raisins to the saucepan
rinse the horseradish sauce
6
7
8
9
10
I want to use sed to do this. I have 2 files:
keys.txt:
host1
host2
test.txt
host1 abc
host2 cdf
host3 abaasdf
I want to use sed to remove any lines in test.txt that contains the keyword in keys.txt. So the result of test.txt should be
host3 abaasdf
Can somebody show me how to do that with sed?
Thanks,
I'd recommend using grep for this (especially fgrep since there are no regexps involved), so
fgrep -v -f keys.txt test.txt
does it fine. With sed quickly this works:
sed -i.ORIGKEYS.txt ^-e 's_^_/_' -e 's_$_/d_' keys.txt
sed -f keys.txt test.txt
(This modifies the original keys.txt in place - with backup - to a sourceable sed script.)
fgrep -v -f is the best solution. Here are a couple of alternatives:
A combination of comm and join
comm -13 <(join keys.txt test.txt) test.txt
or awk
awk 'NR==FNR {key[$1]; next} $1 in key {next} 1' keys.txt test.txt
This might work for you (GNU sed):
sed 's|.*|/^&\\>/d|' keys.txt | sed -f - test.txt
I have a line:
<random junk>TYPE=snp;<more random junk>
and I need to return everything between the end of TYPE= and the ; (in this case snp but it could be any of a number of text strings.
I tried various sed / awk solutions but I can't seem to get it working. I have the feeling this is a simple problem so, sorry about that.
This seems to work:
sed 's/.*TYPE=\(.*\);.*/\1/'
EDIT:
Ah, so there can be semicolons in the random junk. Try this:
sed 's/.*TYPE=\([^;]*\);.*/\1/'
requires GNU grep:
grep -Po '(?<=TYPE=)[^;]+'
meaning: preceded by "TYPE=", find some non-semicolon characters
One way using GNU sed:
sed -r 's/.*TYPE=([^;]+).*/\1/' file.txt
Since you also tagged this awk:
$ text='<random junk>TYPE=snp;<more random junk>'
$ echo "$text" | awk -FTYPE= '{sub(/;.*/,"",$2); print $2}'
snp
$ text='foo=bar;baz=fnu;TYPE=snp;XAI=0;XAM=0'
$ echo "$text" | awk -FTYPE= '{sub(/;.*/,"",$2); print $2}'
snp
(Only using the variable to keep the lines from wrapping.)
Or, to parse this as set of variable=value pairs rather than just a string of text:
$ echo "$text" | awk -vRS=";" -F= '$1=="TYPE" {print $2}'
snp
You can also do this in pure bash, if you want:
$ t="red=blue;TYPE=snp;XAI=0.0037843;XAM=0.0170293;XAS=0.013245;XRI=0;XRM=0"
$ t=${t#*TYPE=}
$ t=${t%%;*}
$ echo $t
snp