I'm having some trouble getting sed to do a find/replace of some hex characters. I want to replace all instances within a file of the following hexadecimal string:
0x0D4D5348
with the following hexadecimal string:
0x0D0A4D5348
How can I do that?
EDIT: I'm trying to do a hex find/replace. The input file does not have the literal value of "0x0D4D5348" in it, but it does have the ASCII representation of that in it.
GNU sed v3.02.80, GNU sed v1.03, and HHsed v1.5 by Howard Helman
all support the notation \xNN, where "NN" are two valid hex numbers, 00-FF.
Here is how to replace a HEX sequence in your binary file:
$ sed 's/\x0D\x4D\x53\x48/\x0D\x0A\x4D\x53\x48/g' file > temp; rm file; mv temp file
As #sputnik pointed out, you can use sed's in place functionality. One caveat though, if you use it on OS/X, you'd have to add an empty set of quotes:
$ sed '' 's/\x0D\x4D\x53\x48/\x0D\x0A\x4D\x53\x48/g' file
As sed in place on OS/X takes a parameter to indicate what extension to add to the file name when making a backup, since it does create a temp file first. But then.. OS/X's sed doesn't support \x.
This worked for me on Linux and OSX.
Replacing in-place:
sed -i '.bk' 's'/`printf "\x03"`'/foo/g' index.html
(See #Ernest's comment in the answer by #tolitius)
In OS/X system's Bash, You can use command like this:
# this command will crate a variable named a which contains '\r\n' in it
a=`echo -e "hello\r\nworld\r\nthe third line\r\n"`
echo "$a" | sed $'s/\r//g' | od -c
and now you can see the output characters :
0000000 h e l l o \n w o r l d \n t h e
0000020 t h i r d l i n e \n
0000033
You should notice the difference between 's/\r//g' and $'s/\r//g'.
Based on the above practices, you can use command like this to replace hex String
echo "$a" | sed $'s/\x0d//g' | od -c
Related
sed -i 's/$/\'/g'
sed -i "s/$/\'/g"
How to escape both $ and ' by 1 command?
This might work for you (GNU sed):
sed 's/$/'\''/' file
Adds a single quote to the end of a line.
sed 's/\$/'\''/' file
Replaces a $ by a single quote.
sed 's/\$$/'\''/' file
Replaces a $ at the end of line by a single quote.
N.B. Surrounding sed commands by double quotes is fine for some interpolation but may return unexpected results.
Use octal values
sed 's/$/\o47/'
Care to use backslash + letter o minus + octal number 1 to 3 digit
Just don't use single quotes to start the sed script?
sed "s/$/'/"
The /g at the end means to apply everywhere it's found on each stream (line) - you don't need this since $ is a special character indicating end of stream.
To add a quote at the end of a line use
sed -i "s/$/'/g" file
sed -i 's/$/'"'"'/g' file
See proof.
If there are already single quotes, and you want to make sure there is single occurrence at the end of string use
sed -i "s/'*$/'/g" file
sed -i 's/'"'*"'$/'"'"'/g' file
See this proof.
To escape $ and ' chars use
sed -i "s/[\$']/\\\\&/g" file
See proof
[\$'] - matches $ (escaped as in double quotes it can be treated as a variable interpolation char) or '
\\\\& - a backslash (need 4, that is literal 2 backslashes, it is special in the replacement), and & is the whole match.
I just copied couple of files from windows to unix and they all have ^M at the end. I know how to remove them using vi, but I can only do one file at a time, is there a way I can do it for all the files in the folder?. There are like 60 files and manually doing it for all of them is time consuming!.
I'm open to using other tools as well!
PS: The OS is Solaris
Thanks
For posterity, let's post the solution from within VI. You can remove the Ctrl-M at the end of every line like this:
:%s/^V^M$//
Note that this is what you type, wnere ^V means Ctrl-V and ^M means Ctrl-M. The idea here is that ^V will "escape" the following ^M, so that you can match it in the substitution regex.
And the % expression means "do this on every line".
Note that this may or may not work in vim, depending on your settings.
But your question asks how to do this in vi, in which you can't easily make a change to multiple files. If you're open to using other tools, please indicate so in your question.
You can use sed on a single file or stream:
$ printf 'one\r\ntwo\r\n' > /tmp/test.txt
$ od -c < /tmp/test.txt
0000000 o n e \r \n t w o \r \n
0000012
$ sed -i'' -e 's/^M$//' /tmp/test.txt
$ od -c < /tmp/test.txt
0000000 o n e \n t w o \n
0000010
$
In this case, in /bin/sh in FreeBSD, I escaped the ^M by ... you guessed it ... using ^V.
When using sed's -i option, you can specify multiple files and they will all be modified in place, perhaps eliminating the need to wrap this in a script. If you want to put this into a script anyway, I recommend you try to do so, and then ask for help if it doesn't work. That's the StackOverflow Way. :-)
Or just use Jonathan's for loop example. You don't need temp files.
UPDATE
If your sed does not have a -i option, then you can still do this pretty easily in a for loop:
[ghoti#pc ~]$ od -c /tmp/test1.txt
0000000 o n e \r \n t w o \r \n
0000012
[ghoti#pc ~]$ for f in /tmp/test*.txt; do sed -e 's/^M$//' "$f" > /tmp/temp.$$ && mv -v /tmp/temp.$$ "$f"; done
/tmp/temp.26687 -> /tmp/test1.txt
/tmp/temp.26687 -> /tmp/test2.txt
[ghoti#pc ~]$ od -c /tmp/test1.txt
0000000 o n e \n t w o \n
0000010
If you don't have a dos2unix or dtou command on your machine, you can use tr instead:
for file in "$#" # LIst of files passed as argument to script
do
tr -d '\015' < "$file" > tmp.$$
cp tmp.$$ "$file"
done
rm tmp.$$
You can add trap commands around that to clean up if you interrupt. Using cp instead of mv preserves owner, permissions, symlinks, hard links.
use the command dos2ux.
dos2ux file >file2
We have a process which can use a file containing sed commands to alter piped input.
I need to replace a placeholder in the input with a variable value, e.g. in a single -e type of command I can run;
$ echo "Today is XX" | sed -e "s/XX/$(date +%F)/"
Today is 2012-10-11
However I can only specify the sed aspects in a file (and then point the process at the file), E.g. a file called replacements.sed might contain;
s/XX/Thursday/
So obviously;
$ echo "Today is XX" | sed -f replacements.sed
Today is Thursday
If I want to use an environment variable or shell value, though, I can't find a way to make it expand, e.g. if replacements.txt contains;
s/XX/$(date +%F)/
Then;
$ echo "Today is XX" | sed -f replacements.sed
Today is $(date +%F)
Including double quotes in the text of the file just prints the double quotes.
Does anyone know a way to be able to use variables in a sed file?
This might work for you (GNU sed):
cat <<\! > replacements.sed
/XX/{s//'"$(date +%F)"'/;s/.*/echo '&'/e}
!
echo "Today is XX" | sed -f replacements.sed
If you don't have GNU sed, try:
cat <<\! > replacements.sed
/XX/{
s//'"$(date +%F)"'/
s/.*/echo '&'/
}
!
echo "Today is XX" | sed -f replacements.sed | sh
AFAIK, it's not possible. Your best bet will be :
INPUT FILE
aaa
bbb
ccc
SH SCRIPT
#!/bin/sh
STRING="${1//\//\\/}" # using parameter expansion to prevent / collisions
shift
sed "
s/aaa/$STRING/
" "$#"
COMMAND LINE
./sed.sh "fo/obar" <file path>
OUTPUT
fo/obar
bbb
ccc
As others have said, you can't use variables in a sed script, but you might be able to "fake" it using extra leading input that gets added to your hold buffer. For example:
[ghoti#pc ~/tmp]$ cat scr.sed
1{;h;d;};/^--$/g
[ghoti#pc ~/tmp]$ sed -f scr.sed <(date '+%Y-%m-%d'; printf 'foo\n--\nbar\n')
foo
2012-10-10
bar
[ghoti#pc ~/tmp]$
In this example, I'm using process redirection to get input into sed. The "important" data is generated by printf. You could cat a file instead, or run some other program. The "variable" is produced by the date command, and becomes the first line of input to the script.
The sed script takes the first line, puts it in sed's hold buffer, then deletes the line. Then for any subsequent line, if it matches a double dash (our "macro replacement"), it substitutes the contents of the hold buffer. And prints, because that's sed's default action.
Hold buffers (g, G, h, H and x commands) represent "advanced" sed programming. But once you understand how they work, they open up new dimensions of sed fu.
Note: This solution only helps you replace entire lines. Replacing substrings within lines may be possible using the hold buffer, but I can't imagine a way to do it.
(Another note: I'm doing this in FreeBSD, which uses a different sed from what you'll find in Linux. This may work in GNU sed, or it may not; I haven't tested.)
I am in agreement with sputnick. I don't believe that sed would be able to complete that task.
However, you could generate that file on the fly.
You could change the date to a fixed string, like
__DAYOFWEEK__.
Create a temp file, use sed to replace __DAYOFWEEK__ with $(date +%Y).
Then parse your file with sed -f $TEMPFILE.
sed is great, but it might be time to use something like perl that can generate the date on the fly.
To add a newline in the replacement expression using a sed file, what finally worked for me is escaping a literal newline. Example: to append a newline after the string NewLineHere, then this worked for me:
#! /usr/bin/sed -f
s/NewLineHere/NewLineHere\
/g
Not sure it matters but I am on Solaris unix, so not GNU sed for sure.
I'm using CMD on Windows Xp to replace special text with Sed. I'm using this command for replace special characters like $ or * :
sed -i "s/\*/123/g;" 1.txt
the previous command does not work because i have \, " and other special strings that sed use to make regex. The escape character ^ doesn't work well because sed no give me error but nothing change inside files.
To change this text "{\*)(//123/$$ i try use this command:
sed -i "s£"^"{^\^*)(//123/^$^$"£xx£g;" 1.txt £ is the delimiter, xx is new text..but nothing change
How i want to turn text like this? sed -i^/\\*$/$[{" ;" 1.txt into xx
This might work for you:
echo '"{\*)(//123/$$' | sed "s/[\"][{][\\][*][)][(][/][/]123[/][$][$]/xx/"
xx
/usr/bin/sed 's/,/\\n/g' comma-delimited.txt > newline-separated.txt
This doesn't work for me. I just get the ',' removed but the tokens are now just not delimited.
You must have an older version of sed, so you need to put a literal LF char in your substitution, i.e.
/usr/bin/sed 's/,/
/g' comma-delimited.txt > newline-separated.txt
You may even need to escape the LF, so make sure there are no white space chars after the last char '\'
/usr/bin/sed 's/,/\
/g' comma-delimited.txt > newline-separated.txt
This might work for you:
echo a,b,c,d,e | sed 'G;:a;s/,\(.*\(.\)\)/\2\1/;ta;s/.$//'
a
b
c
d
e
Explanation:
Appends a newline to the pattern space. G
Substitute ,'s with the last character in the pattern space i.e. the \n :a;s/,\(.*\(.\)\)/\2\1/;ta
Remove the newline. s/.$//
I tried the following, looks clumsy but does the work. Easy to understand. I use tr to do the replacement of the placeholder §. Only caveat is the placeholder, must be something NOT in the string(s).
ps -fu $USER | grep java | grep DML| sed -e "s/ -/§ -/g" | tr "§" "\n"
will give you an indented output of the commandline. DML is just some servername.
on AIX7 answer #3 worked well:
I need to insert a newline at the beginning of a paragraph so I can do grep -p to filter for 'mksysb' in the resulting 'stanza'
lsnim -l | /usr/bin/sed 's/^[a-zA-Z/\^J&/'
(actually the initial line had an escaped newline:
lsnim -l | /usr/bin/sed 's/^[a-zA-Z/\
&/')
recalling the command showed the ^J syntax ...