I would like to use (GNU) sed to do a simple search and replace. The issue is that I'm searching for a special character and it might be the reason it failed for me.
The input is:
^A9=139^A35=V^A34=9^A49=xxxx^A52=20140527-06:18:43.759^A5
and I want to replace the ^A with ;. I used:
sed -i '/s/^A/;/g' file.log
but I didn't get anything.
Your command should be,
sed -i 's/\^A/;/g' file
Command you tried,
sed -i '/s/^A/;/g' file.log
| |
| |______________You have to escape this special character. Because in general(regex) it means the starting point.
[No need to use `/` before s]
Example:
$ sed 's/\^A/;/g' file
;9=139;35=V;34=9;49=xxxx;52=20140527-06:18:43.759;5
^ has a special meaning with regular expressions. Use \^ (or potentially \\^, depending on how bash escapes things, I never quite remember it).
Related
Using sed I want to parse Heroku's log-runtime-metrics like this one:
2016-01-29T00:38:43.662697+00:00 heroku[worker.2]: source=worker.2 dyno=heroku.17664470.d3f28df1-e15f-3452-1234-5fd0e244d46f sample#memory_total=54.01MB sample#memory_rss=54.01MB sample#memory_cache=0.00MB sample#memory_swap=0.00MB sample#memory_pgpgin=17492pages sample#memory_pgpgout=3666pages
the desired output is:
worker.2: 54.01MB (54.01MB is being memory_total)
I could not manage although I tried several alternatives including:
sed -E 's/.+source=(.+) .+memory_total=(.+) .+/\1: \2/g'
What is wrong with my command? How can it be corrected?
The .+ after source= and memory_total= are both greedy, so they accept as much of the line as possible. Use [^ ] to mean "anything except a space" so that it knows where to stop.
sed -E 's/.+source=([^ ]+) .+memory_total=([^ ]+) .+/\1: \2/g'
Putting your content into https://regex101.com/ makes it really obvious what's going on.
I'd go for the old-fashioned, reliable, non-extended sed expressions and make sure that the patterns are not too greedy:
sed -e 's/.*source=\([^ ]*\) .*memory_total=\([^ ]*\) .*/\1: \2/'
The -e is not the opposite of -E, which is primarily a Mac OS X (BSD) sed option; the normal option for GNU sed is -r instead. The -e simply means that the next argument is an expression in the script.
This produces your desired output from the given line of data:
worker.2: 54.01MB
Bonus question: There are some odd lines within the stream, I can usually filter them out using a grep pipe like | grep memory_total. However if I try to use it along with the sed command, it does not work. No output is produced with this:
heroku logs -t -s heroku | grep memory_total | sed.......
Sometimes grep | sed is necessary, but it is often redundant (unless you are using a grep feature that isn't readily supported by sed, such as Perl regular expressions).
You should be able to use:
sed -n -e '/memory_total=/ s/.*source=\([^ ]*\) .*memory_total=\([^ ]*\) .*/\1: \2/p'
The -n means "don't print by default". The /memory_total=/ matches the lines you're after; the s/// content is the same as before. I removed the g suffix that was there previously; the regex would never match multiple times anyway. I added the p to print the line when the substitution occurs.
$ echo "haha#nihao" | sed "s/#.+$/end/"
haha#nihao
I want to match contents between character # and the end of line.why can't i get it?
:%!sed "s/#.\+$/end/"
E194:No alternate file name to substitute for '#'
problem 1:
why i can't use it in sed of vim?
problem 2:
how to see the EORROR 194 ?
problem 1: why i can't use it in sed of vim?
Because by default sed uses BRE, basic regular expressions:
/.+/ this matches any character followed by a "+"
/.\+/ this matches one or more occurrences of any character
You can tell sed to use extended regular expressions with the -r flag in GNU implementations and -E flag with BSD implementations:
$ echo "haha#nihao" | sed -r "s/#.+$/end/"
hahaend
problem 2: how to see the EORROR 194 ?
You get this error because # has a special meaning in vim when you run commands with !: # marks on the command line are replaced with the alternate file. It should work if you escape the #:
%!sed "s/\#.\+$/end/"
You can read about this error with the :help E194 command, and about alternative file with :help alternate-file.
use sed -r
echo "haha#nihao" | sed -r "s/#.+$/end/"
hahaend
from man sed
-r, --regexp-extended
use extended regular expressions in the script.
sed 's_((checksum|compressed)=\").*(\")_\1\2_' -i filename
I am using this command to replace the checksum and compressed filed with empty? But it didn't change anything?
for example, I want change this line " checksum="XXXXX" with checksum="", and also replace
compressed="XXXX" with compressed=""
What is wrong with my sed command?
It's because sed uses a funny regex dialect by default: you have to escape capturing brackets.
If you want to use "normal" regex that you're familiar with, use the -r flag (if you're on unix, GNU sed) or the -E flag (Mac OS X BSD sed):
sed -r 's_((checksum|compressed)=\").*(\")_\1\3_' -i filename
Additionally, note that you have three sets of capturing brackets in your sed, and I think you want to change the \1\2 to \1\3. (\1 contains checksum=", \2 contains checksum, and \3 contains ").
(For interest, here's how you would do it without the extended-regexp (-r/-E) flag, note that capturing brackets and the OR | are only considered in the regex sense if they are escaped:
sed 's_\(\(checksum\|compressed\)=\"\).*\(\"\)_\1\3_' -i filename
)
This might work for you:
echo 'checksum="XXXXX" compressed="YYYYYYY"' |
sed 's/\(checksum\|compressed\)="[^"]*"/\1=""/g'
checksum="" compressed=""
In sed (without the -r switch), ()|+?{}'s must have a \ prepended to give them the qualities of grouping. alternation, one or more, zero or one and intervals. .[]* work as metacharacters either way.
Try:
sed 's/\(\(checksum\|compressed\)\)="[^"]*"/\1=""/' -i filename
How can I replace ; with ;\n (semicolon followed by a newline) in sed?
I've tried building off of
sed s/;/\\n/g file
and
sed -e '/;/G' file
but I can't get either to work
You need to cheat a bit: in bash you can say
sed $'s/;/;\\\n/g'
or, portably (POSIX):
sed "s/;/;$(printf '\\\n')/g"
sed does not portably/reliably handle backslash-escapes anywhere but in the pattern, and even there it's limited (POSIX only requires that \n be handled, not \t or the others). Note that you also need a backslash before the \n so sed doesn't read it as the end of the command.
sed -ie 's/;/;\n/g' <file>
That's assuming you want to do it inline in the file, remove the "i" and just use "-e" if that's not the case.
I am trying to write a sed expression that can remove urls from a file
example
http://samgovephotography.blogspot.com/ updated my blog just a little bit ago. Take a chance to check out my latest work. Hope all is well:)
Meet Former Child Star & Author Melissa Gilbert 6/15/09 at LA's B&N https://hollywoodmomblog.com/?p=2442 Thx to HMB Contributor #kdpartak :)
But I dont get it:
sed 's/[\w \W \s]*http[s]*:\/\/\([\w \W]\)\+[\w \W \s]*/ /g' posFile
FIXED!!!!!
handles almost all cases, even malformed URLs
sed 's/[\w \W \s]*http[s]*[a-zA-Z0-9 : \. \/ ; % " \W]*/ /g' positiveTweets | grep "http" | more
The following removes http:// or https:// and everything up until the next space:
sed -e 's!http\(s\)\{0,1\}://[^[:space:]]*!!g' posFile
updated my blog just a little bit ago. Take a chance to check out my latest work. Hope all is well:)
Meet Former Child Star & Author Melissa Gilbert 6/15/09 at LA's B&N Thx to HMB Contributor #kdpartak :)
Edit:
I should have used:
sed -e 's!http[s]\?://\S*!!g' posFile
"[s]\?" is a far more readable way of writing "an optional s" compared to "\(s\)\{0,1\}"
"\S*" a more readable version of "any non-space characters" than "[^[:space:]]*"
I must have been using the sed that came installed with my Mac at the time I wrote this answer (brew install gnu-sed FTW).
There are better URL regular expressions out there (those that take into account schemes other than HTTP(S), for instance), but this will work for you, given the examples you give. Why complicate things?
The accepted answer provides the approach that I used to remove URLs, etc. from my files. However it left "blank" lines. Here is a solution.
sed -i -e 's/http[s]\?:\/\/\S*//g ; s/www\.\S*//g ; s/ftp:\S*//g' input_file
perl -i -pe 's/^'`echo "\012"`'${2,}//g' input_file
The GNU sed flags, expressions used are:
-i Edit in-place
-e [-e script] --expression=script : basically, add the commands in script
(expression) to the set of commands to be run while processing the input
^ Match start of line
$ Match end of line
? Match one or more of preceding regular expression
{2,} Match 2 or more of preceding regular expression
\S* Any non-space character; alternative to: [^[:space:]]*
However,
sed -i -e 's/http[s]\?:\/\/\S*//g ; s/www\.\S*//g ; s/ftp:\S*//g'
leaves nonprinting character(s), presumably \n (newlines). Standard sed-based approaches to remove "blank" lines, tabs and spaces, e.g.
sed -i 's/^[ \t]*//; s/[ \t]*$//'
do not work, here: if you do not use a "branch label" to process newlines, you cannot replace them using sed (which reads input one line at a time).
The solution is to use the following perl expression:
perl -i -pe 's/^'`echo "\012"`'${2,}//g'
which uses a shell substitution,
'`echo "\012"`'
to replace an octal value
\012
(i.e., a newline, \n), that occurs 2 or more times,
{2,}
(otherwise we would unwrap all lines), with something else; here:
//
i.e., nothing.
[The second reference below provides a wonderful table of these values!]
The perl flags used are:
-p Places a printing loop around your command,
so that it acts on each line of standard input
-i Edit in-place
-e Allows you to provide the program as an argument,
rather than in a file
References:
perl flags: Perl flags -pe, -pi, -p, -w, -d, -i, -t?
ASCII control codes: https://www.cyberciti.biz/faq/unix-linux-sed-ascii-control-codes-nonprintable/
remove URLs: sed to remove URLs from a file
branch labels: How can I replace a newline (\n) using sed?
GNU sed manual: https://www.gnu.org/software/sed/manual/sed.html
quick regex guide: https://www.gnu.org/software/sed/manual/html_node/Regular-Expressions.html
Example:
$ cat url_test_input.txt
Some text ...
https://stackoverflow.com/questions/4283344/sed-to-remove-urls-from-a-file
https://www.google.ca/search?dcr=0&ei=QCsyWtbYF43YjwPpzKyQAQ&q=python+remove++citations&oq=python+remove++citations&gs_l=psy-ab.3...1806.1806.0.2004.1.1.0.0.0.0.61.61.1.1.0....0...1c.1.64.psy-ab..0.0.0....0.-cxpNc6youY
http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html
https://bbengfort.github.io/tutorials/2016/05/19/text-classification-nltk-sckit-learn.html
http://datasynce.org/2017/05/sentiment-analysis-on-python-through-textblob/
https://www.google.ca/?q=halifax&gws_rd=cr&dcr=0&ei=j7UyWuGKM47SjwOq-ojgCw
http://www.google.ca/?q=halifax&gws_rd=cr&dcr=0&ei=j7UyWuGKM47SjwOq-ojgCw
www.google.ca/?q=halifax&gws_rd=cr&dcr=0&ei=j7UyWuGKM47SjwOq-ojgCw
ftp://ftp.ncbi.nlm.nih.gov/
ftp://ftp.ncbi.nlm.nih.gov/1000genomes/ftp/alignment_indices/20100804.alignment.index
Some more text.
$ sed -e 's/http[s]\?:\/\/\S*//g ; s/www\.\S*//g ; s/ftp:\S*//g' url_test_input.txt > a
$ cat a
Some text ...
Some more text.
$ perl -i -pe 's/^'`echo "\012"`'${2,}//g' a
Some text ...
Some more text.
$