I'm trying to find a way to run multiple sed commands that adds lines to the start of different files (on Mac OS).
This works when run from terminal.
sed -i '' '1i\
\\version \"2.19.65\"\
\\language \"english\"\
\\include \"dynamics-defs.ily\"\
altosaxINotes = \\transpose c ef {\
\\relative c {\
' altosaxI.ily
But I want to add slightly different text on a different file:
sed -i '' '1i\
\\version \"2.19.65\"\
\\language \"english\"\
\\include \"dynamics-defs.ily\"\
altosaxIINotes = \\transpose c ef {\
\\relative c {\
' altosaxII.ily
I have about 30 or 40 of these to run, all slightly different. Is it possible to combine them all into one terminal command, or perhaps use Mac's automator, or maybe a better solution?
This might work for you (GNU sed):
# create a function f with one parameter
f(){ cat <<! >tempFile && sed -i '1e cat tempFile' ${1}.ily; }
\\version "2.19.65"
\\language "english"
\\include "dynamics-defs.ily"
${1}Notes = \\transpose c ef {
\\relative c {
!
# call the function
f altosaxI
The function f can then be included in a for-loop or a script.
Related
I'm trying to use sed to replace a specific line within a configuration file:
The pattern for the line I want to replace is:
ALLOWED_HOSTS.*
The text I want to insert is:
'$PublicIP' (Including the single ticks)
But when I run the command:
sed 's/ALLOWED_HOSTS.*/ALLOWED_HOSTS = ['$PublicIP']/g' /root/project/django/mysite/mysite/settings.py
The line is changed to:
ALLOWED_HOSTS = [1.1.1.1]
instead of:
ALLOWED_HOSTS = ['1.1.1.1']
How shall I edit the command to include the single ticks as well?
You could try to escape the single ticks , or better you can reassign the variable including the simple ticks:
PublicIP="'$PublicIP'".
By the way even this sed without redifining var, works ok in my case:
$ a="3.3.3.3"
$ echo "ALLOWED_HOSTS = [2.2.2.2]" |sed 's/2.2.2.2/'"'$a'"'/g'
ALLOWED_HOSTS = ['3.3.3.3']
Even this works ok:
$ echo "ALLOWED_HOSTS = [2.2.2.2]" |sed "s/2.2.2.2/'$a'/g"
ALLOWED_HOSTS = ['3.3.3.3']
I have a html file which includes a section as follows:
<div id='webnews'>
... variable stuff ...
</div>
which I want to comment out as follows:
<!--
<div id='webnews'>
... variable stuff ...
</div>
-->
I can find & print the multiline text as follows:
sed '/<div id="webnews"/, /<\/div>/ { p }' filename.html
Experimenting with h, d, x and G, I have been unable work out how to either wrap the hold buffer or the pattern buffer with '<!--' and '-->'.
Would appreciate help with this challenge.
quick and dirty with sed (not the best idea on html unless you are sure of html content/structure)
sed "/<div id='webnews'/, /<\/div>/ {
/<div id='webnews'/ {
h
d
}
H
/<\/div>/ !d
x
s/^/<!--\\
/
s/$/\\
-->/
}" filename.html
This might work for you (GNU sed):
sed -e '/<div id='\''webnews'\''>/,/<\/dev>/!b;/<div id='\''webnews'\''>/i\<!--' -e '/<\/div>/a\-->' file
Or perhaps:
sed $'/<div id=.webnews.>/,/<\/dev>/{/<div id=.webnews.>/i\<!--\n;/<\/div>/a\-->\n}' file
Sed is not the right tool for the job.
Use sift:
sift -m '(.+)(<div id=.webnews.>.*</div>)(.+)' --replace '$1<!-- $2 -->$3'
I have a report looks like this:
par_a
.xx
.yy
par_b
.zz
.tt
I wish to convert this format into csv format as below using sed 1 liner:
par_a,.xx
par_a,.yy
par_b,.zz
par_b,.tt
please help.
With awk:
awk '/^par_/{v=$0;next}/^ /{$0=v","$1;print}' File
Or to make it more generic:
awk '/^[^[:blank:]]/{v=$0;next} /^[[:blank:]]/{$0=v","$1;print}' File
When a line starts with par_, save the content to variable v. Now, when a line starts with space, change the line to content of v followed by , followed by the first field.
Output:
AMD$ awk '/^par_/{v=$0}/^ /{$0=v","$1;print}' File
par_a,.xx
par_a,.yy
par_b,.zz
par_b,.tt
With sed:
sed '/^par_/ { h; d; }; G; s/^[[:space:]]*//; s/\(.*\)\n\(.*\)/\2,\1/' filename
This works as follows:
/^par_/ { # if a new paragraph begins
h # remember it
d # but don't print anything yet
}
# otherwise:
G # fetch the remembered paragraph line to the pattern space
s/^[[:space:]]*// # remove leading whitespace
s/\(.*\)\n\(.*\)/\2,\1/ # rearrange to desired CSV format
Depending on your actual input data, you may want to replace the /^par_/ with, say, /^[^[:space:]]/. It just has to be a pattern that recognizes the beginning line of a paragraph.
Addendum: Shorter version that avoids regex repetition when using the space pattern to recognize paragraphs:
sed -r '/^\s+/! { h; d; }; s///; G; s/(.*)\n(.*)/\2,\1/' filename
Or, if you have to use BSD sed (as comes with Mac OS X):
sed '/^[[:space:]]\{1,\}/! { h; d; }; s///; G; s/\(.*\)\n\(.*\)/\2,\1/' filename
The latter should be portable to all seds, but as you can see, writing portable sed involves some pain.
ive been tring sooooo many different variations to get this right.
i am simply looking to use sed to remove all words beginning with or containing a backslash.
so string
another test \/ \u7896 \n test ha\ppy
would become
another test test
i've tried soo many different options, but it doesnt seem to want to work. Does anybody have an idea how to do this?
and before everyone starts giving me minus 1 for this question, believe me, i have tried to find the answer.
You could use str.split and a list comprehension:
>>> strs = "another test \/ \u7896 \n test ha\ppy"
>>> [x for x in strs.split() if '\\' not in x]
['another', 'test', 'test']
# use str.join to join the list
>>> ' ' .join([x for x in strs.split() if '\\' not in x])
'another test test'
$ echo "another test \/ \u7896 \n test ha\ppy" | sed -r 's/\S*\\\S*//g' | tr -s '[:blank:]'
another test test
This might work for you (GNU sed):
sed 's/\s*\S*\\\S*//g' file
string = "another test \/ \u7896 \n test ha\ppy"
string_no_slashes = " ".join([x for x in string.split() if "\\" not in x])
For example, given a line a11b12c22d322 e... the fields of break are the numbers or spaces, we want to transform it into
a
b
c
d
e
...
sed need to read the whole line into memory, for gigabytes a line, it would not be efficient, and the job could not be done if we don't have sufficient memory.
EDIT:
Could anyone please explain how do grep, tr, Awk, perl, and python manipulate the memory in reading a large file? What and how much content do they read into memory once a time?
If you use gawk (which is the default awk on Linux, I believe), you can use the RS parameter to specify that multi-digit numbers or spaces are recognized as line terminators instead of a new-line.
awk '{print}' RS="[[:digit:]]+| +" file.txt
As to your second question, all of these programs will need to read some fixed number of bytes and search for its idea of a line separator in an internal buffer to simulate the appearance of reading a single line at a time. To prevent it from reading too much data while searching for the end of the line, you need to change the programs idea of what terminates a line.
Most languages allow you to do this, but only allow you to specify a single character. gawk makes it easy by allowing you to specify a regular expression to recognize an end-of-line character. This saves you from having to implement the fixed-size buffer and end-of-line search yourself.
Fastest... You can do it with help of gcc, here's a version which reads data from given file name if given, otherwise from stdin. If this is still too slow, you can see if you can make it faster by replacing getchar() and putchar() (which may be macros and should optimize very well) with your own buffering code. If we want to get ridiculous, for even faster, you should have three threads, so kernel can copy next block of data with one core, while another core does processing, and third core copies processed output back to kernel.
#!/bin/bash
set -e
BINNAME=$(mktemp)
gcc -xc -O3 -o $BINNAME - <<"EOF"
#include <stdio.h>
#include <stdlib.h>
int main(void)
{
int sep = 0;
/* speed is a requirement, so let's reduce io overhead */
const int bufsize = 1024*1024;
setvbuf(stdin, malloc(bufsize), _IOFBF, bufsize);
setvbuf(stdout, malloc(bufsize), _IOFBF, bufsize);
/* above buffers intentionally not freed, it doesn't really matter here */
int ch;
while((ch = getc(stdin)) >= 0) {
if (isdigit(ch) || isspace(ch)) {
if (!sep) {
if (putc('\n', stdout) == EOF) break;
sep = 1;
}
} else {
sep = 0;
if (putc(ch, stdout) == EOF) break;
}
}
/* flush should happen by on-exit handler, as buffer is not freed,
but this will detect write errors, for program exit code */
fflush(stdout);
return ferror(stdin) || ferror(stdout);
}
EOF
if [ -z "$1" ] ; then
$BINNAME <&0
else
$BINNAME <"$1"
fi
Edit: I happened too look at GNU/Linux stdio.h, some notes: putchar/getchar are not macros, but putc/getc are, so using those instead might be a slight optimization, probably avoiding one function call, changed code to reflect this. Also added checking return code of putc, while at it.
With grep:
$ grep -o '[^0-9 ]' <<< "a11b12c22d322 e"
a
b
c
d
e
With sed:
$ sed 's/[0-9 ]\+/\n/g' <<< "a11b12c22d322 e"
a
b
c
d
e
With awk:
$ awk 'gsub(/[0-9 ]+/,"\n")' <<< "a11b12c22d322 e"
a
b
c
d
e
I'll let you benchmark.
Try with tr:
tr -s '[:digit:][:space:]' '\n' <<< "a11b12c22d322e"
That yields:
a
b
c
d
e