Insert a line at specific line number with sed or awk - sed

I have a script file which I need to modify with another script to insert a text at the 8th line.
String to insert: Project_Name=sowstest, into a file called start.
I tried to use awk and sed, but my command is getting garbled.

sed -i '8i This is Line 8' FILE
inserts at line 8
This is Line 8
into file FILE
-i does the modification directly to file FILE, no output to stdout, as mentioned in the comments by glenn jackman.

An ed answer
ed file << END
8i
Project_Name=sowstest
.
w
q
END
. on its own line ends input mode; w writes; q quits. GNU ed has a wq command to save and quit, but old ed's don't.
Further reading: https://gnu.org/software/ed/manual/ed_manual.html

OS X / macOS / FreeBSD sed
The -i flag works differently on macOS sed than in GNU sed.
Here's the way to use it on macOS / OS X:
sed -i '' '8i\
8 This is Line 8' FILE
See man 1 sed for more info.

the awk answer
awk -v n=8 -v s="Project_Name=sowstest" 'NR == n {print s} {print}' file > file.new

POSIX sed (and for example OS X's sed, the sed below) require i to be followed by a backslash and a newline. Also at least OS X's sed does not include a newline after the inserted text:
$ seq 3|gsed '2i1.5'
1
1.5
2
3
$ seq 3|sed '2i1.5'
sed: 1: "2i1.5": command i expects \ followed by text
$ seq 3|sed $'2i\\\n1.5'
1
1.52
3
$ seq 3|sed $'2i\\\n1.5\n'
1
1.5
2
3
To replace a line, you can use the c (change) or s (substitute) commands with a numeric address:
$ seq 3|sed $'2c\\\n1.5\n'
1
1.5
3
$ seq 3|gsed '2c1.5'
1
1.5
3
$ seq 3|sed '2s/.*/1.5/'
1
1.5
3
Alternatives using awk:
$ seq 3|awk 'NR==2{print 1.5}1'
1
1.5
2
3
$ seq 3|awk '{print NR==2?1.5:$0}'
1
1.5
3
awk interprets backslashes in variables passed with -v but not in variables passed using ENVIRON:
$ seq 3|awk -v v='a\ba' '{print NR==2?v:$0}'
1
a
3
$ seq 3|v='a\ba' awk '{print NR==2?ENVIRON["v"]:$0}'
1
a\ba
3
Both ENVIRON and -v are defined by POSIX.

sed -e '8iProject_Name=sowstest' -i start using GNU sed
Sample run:
[root#node23 ~]# for ((i=1; i<=10; i++)); do echo "Line #$i"; done > a_file
[root#node23 ~]# cat a_file
Line #1
Line #2
Line #3
Line #4
Line #5
Line #6
Line #7
Line #8
Line #9
Line #10
[root#node23 ~]# sed -e '3ixxx inserted line xxx' -i a_file
[root#node23 ~]# cat -An a_file
1 Line #1$
2 Line #2$
3 xxx inserted line xxx$
4 Line #3$
5 Line #4$
6 Line #5$
7 Line #6$
8 Line #7$
9 Line #8$
10 Line #9$
11 Line #10$
[root#node23 ~]#
[root#node23 ~]# sed -e '5ixxx (inserted) "line" xxx' -i a_file
[root#node23 ~]# cat -n a_file
1 Line #1
2 Line #2
3 xxx inserted line xxx
4 Line #3
5 xxx (inserted) "line" xxx
6 Line #4
7 Line #5
8 Line #6
9 Line #7
10 Line #8
11 Line #9
12 Line #10
[root#node23 ~]#

Perl solutions:
quick and dirty:
perl -lpe 'print "Project_Name=sowstest" if $. == 8' file
-l strips newlines and adds them back in, eliminating the need for "\n"
-p loops over the input file, printing every line
-e executes the code in single quotes
$. is the line number
equivalent to #glenn's awk solution, using named arguments:
perl -slpe 'print $s if $. == $n' -- -n=8 -s="Project_Name=sowstest" file
-s enables a rudimentary argument parser
-- prevents -n and -s from being parsed by the standard perl argument parser
positional command arguments:
perl -lpe 'BEGIN{$n=shift; $s=shift}; print $s if $. == $n' 8 "Project_Name=sowstest" file
environment variables:
setenv n 8 ; setenv s "Project_Name=sowstest"
echo $n ; echo $s
perl -slpe 'print $ENV{s} if $. == $ENV{n}' file
ENV is the hash which contains all environment variables
Getopt to parse arguments into hash %o:
perl -MGetopt::Std -lpe 'BEGIN{getopt("ns",\%o)}; print $o{s} if $. == $o{n}' -- -n 8 -s "Project_Name=sowstest" file
Getopt::Long and longer option names
perl -MGetopt::Long -lpe 'BEGIN{GetOptions(\%o,"line=i","string=s")}; print $o{string} if $. == $o{line}' -- --line 8 --string "Project_Name=sowstest" file
Getopt is the recommended standard-library solution.
This may be overkill for one-line perl scripts, but it can be done

For those who are on SunOS which is non-GNU, the following code will help:
sed '1i\^J
line to add' test.dat > tmp.dat
^J is inserted with ^V+^J
Add the newline after '1i.
\ MUST be the last character of the line.
The second part of the command must be in a second line.

sed -i "" -e $'4 a\\\n''Project_Name=sowstest' start
This line works fine in macOS

macOS sed solutions
for example: inserts at line 1
ns
# recommended 👍
# This command only needs to write one line
$ sed -i '' '1s/^/The new First Line\n/' ./your-source-file-name
ni
# not recommended 👎
# This way the command needs to be written on multiple lines
$ sed -i '' '1i\
The new First Line\
' ./your-source-file-name
test demo
$ sed -i '' '1s/^/Perl 🐪 camel\n/' ./multi-line-text.txt

it is working fine in linux to add in 2 lines.
sed '2s/$/ myalias/' file

Thank you umläute
sed -i "" -e $'4 a\\\n''Project_Name=sowstest' filename
the following was usefull on macOS to be able to add a new line after the 4
In order to loop i created an array of folders, ti iterate on them in mac zsh
for foldercc in $foldernames;
sed -i "" -e $'4 a\\\n''Project_Name=sowstest' $foldercc/filenames;

Related

Parse file and insert new line after each occurrence

On a Unix system I am trying to add a new line in a file using sed or perl but it seems I am missing something.
Supposing my file has multiple lines of texts, always ending like this {TNG:}}${1:F01.
I am trying to find a to way to add a new line after the }$, in this way {1 should always start on a new line.
I tried it by escaping $ sign using this:
perl -e '$/ = "\${"; while (<>) { s/\$}\{$/}\n{/; print; }' but it does not work.
Any ideas will be appreciated.
give this a try:
sed 's/{TNG:}}\$/&\n/' file > newfile
The sed will by default use BRE, that is, the {}s are literal characters. But we must escape the $.
kent$ cat f
{TNG:}}${1:F01.
kent$ sed 's/{TNG:}}\$/&\n/' f
{TNG:}}$
{1:F01.
With perl:
$ cat input.txt
line 1 {TNG:}}${1:F01
line 2 {TNG:}}${1:F01
$ perl -pe 's/TNG:\}\}\$\K/\n/' input.txt
line 1 {TNG:}}$
{1:F01
line 2 {TNG:}}$
{1:F01
(Read up on the -p and -n options in perlrun and use them instead of trying to do what they do in a one-liner yourself)

Extract every nth number from a txt file

So I have a txt file where I need to extract every third number and print it to separate file using Terminal. The txt file is just a long list of numbers, tab delimited:
18 25 0 18 24 5 18 23 5 18 22 8.2 ...
I know there is a way to do this using sed or awk, but so far I've only been able to extract every third line by using:
awk 'NR%3==1' testRain.txt > rainOnly.txt
So here's the answer (or rather, the answer I utilized!):
xargs -n1 < input.txt | awk '!(NR%3)' > output.txt
This gives you an output.txt that has every third number of the original file as a separate line.
A quick pipe line to extract every 3rd number:
$ xargs -n1 < file | sed '3~3!d'
0
5
5
8.2
If you don't want each number on a newline throw the result back through xargs:
$ xargs -n1 < file | sed '3~3!d' | xargs
0 5 5 8.2
Use redirection to store the output in a new file:
$ xargs -n1 < file | sed '3~3!d' | xargs > new_file
With awk using a simple for loop you could do:
$ awk '{for(i=3;i<=NF;i+=3)print $i}' file
0
5
5
8.2
or (adds a trailing tab):
$ awk '{for(i=3;i<=NF;i+=3)printf "%s\t",$i;print ""}' file
0 5 5 8.2
Or by setting the value of RS (adds trailing newline):
$ awk '!(NR%3)' RS='\t' file
0
5
5
8.2
$ awk '!(NR%3)' RS='\t' ORS='\t' file
0 5 5 8.2
You can print every third character by substituting the next two with nothing, globally. When the count straddles a newline, using Perl might be the simplest solution:
perl -p000 -e 's/(.)../$1/gs'
If you want the first, fourth etc character from every line, a line-oriented tool like sed suffices:
sed 's/\(.\)../\1/g'
Using grep -P
grep -oP '([^\t]+\t){2}\K[^\t\n]+' file
0
5
5
8.2
This might work for you (GNU sed):
sed -r 's/(\S+\s){3}/\1/g;s/\s$//' file
#user2718946
Your solution was close, but here you are without xarg.
awk 'NR%3==1' RS=" " file
18
18
18
18
Different start:
awk 'NR%3==0' RS=" " file
0
5
5
8.2

Sed or awk: how to call line addresses from separate file?

I have 'file1' with (say) 100 lines. I want to use sed or awk to print lines 23, 71 and 84 (for example) to 'file2'. Those 3 line numbers are in a separate file, 'list', with each number on a separate line.
When I use either of these commands, only line 84 gets printed:
for i in $(cat list); do sed -n "${i}p" file1 > file2; done
for i in $(cat list); do awk 'NR==x {print}' x=$i file1 > file2; done
Can a for loop be used in this way to supply line addresses to sed or awk?
This might work for you (GNU sed):
sed 's/.*/&p/' list | sed -nf - file1 >file2
Use list to build a sed script.
You need to do > after the loop in order to capture everything. Since you are using it inside the loop, the file gets overwritten. Inside the loop you need to do >>.
Good practice is to or use > outside the loop so the file is not open for writing during every loop iteration.
However, you can do everything in awk without for loop.
awk 'NR==FNR{a[$1]++;next}FNR in a' list file1 > file2
You have to >>(append to the file) . But you are overwriting the file. That is why, You are always getting 84 line only in the file2.
Try use,
for i in $(cat list); do sed -n "${i}p" file1 >> file2; done
With sed:
sed -n $(sed -e 's/^/-e /' -e 's/$/p/' list) input
given the example input, the inner command create a string like this: `
-e 23p
-e 71p
-e 84p
so the outer sed then prints out given lines
You can avoid running sed/awk in a for/while loop altgether:
# store all lines numbers in a variable using pipe
lines=$(echo $(<list) | sed 's/ /|/g')
# print lines of specified line numbers and store output
awk -v lineS="^($lines)$" 'NR ~ lineS' file1 > out

Sed replace pattern with line number

I need to replace the pattern ### with the current line number.
I managed to Print in the next line with both AWK and SED.
sed -n "/###/{p;=;}" file prints to the next line, without the p;, it replaces the whole line.
sed -e "s/###/{=;}/g" file used to make sense in my head, since the =; returns the line number of the matched pattern, but it will return me the the text {=;}
What am i Missing? I know this is a silly question. I couldn't find the answer to this question in the sed manual, it's not quite clear.
If possible, point me what was i missing, and what to make it work. Thank you
Simple awk oneliner:
awk '{gsub("###",NR,$0);print}'
Given the limitations of the = command, I think it's easier to divide the job in two (actually, three) parts. With GNU sed you can do:
$ sed -n '/###/=' test > lineno
and then something like
$ sed -e '/###/R lineno' test | sed '/###/{:r;N;s/###\([^\n]*\n\)\([^\n]*\)/\2\1/;tr;:c;s/\n\n/\n/;tc}'
I'm afraid there's no simple way with sed because, as well as the = command, the r and GNU extension R commands don't read files into the pattern space, but rather directly append the lines to the output, so the contents of the file cannot be modified in any way. Hence piping to another sed command.
If the contents of test are
fooo
bar ### aa
test
zz ### bar
the above will produce
fooo
bar 2 aa
test
zz 4 bar
This might work for you (GNU sed):
sed = file | sed 'N;:a;s/\(\(.*\)\n.*\)###/\1\2/;ta;s/.*\n//'
An alternative using cat:
cat -n file | sed -E ':a;s/^(\s*(\S*)\t.*)###/\1\2/;ta;s/.*\t//'
As noted by Lev Levitsky this isn't possible with one invocation of sed, because the line number is sent directly to standard out.
You could have sed write a sed-script for you, and do the replacement in two passes:
infile
a
b
c
d
e
###
###
###
a
b
###
c
d
e
###
Find the lines that contain the pattern:
sed -n '/###/=' infile
Output:
6
7
8
11
15
Pipe that into a sed-script writing a new sed-script:
sed 's:.*:&s/###/&/:'
Output:
6s/###/6/
7s/###/7/
8s/###/8/
11s/###/11/
15s/###/15/
Execute:
sed -n '/###/=' infile | sed 's:.*:&s/^/& \&/:' | sed -f - infile
Output:
a
b
c
d
e
6
7
8
a
b
11
c
d
e
15
is this ok ?
kent$ echo "a
b
c
d
e"|awk '/d/{$0=$0" "NR}1'
a
b
c
d 4
e
if match pattern "d", append line number at the end of the line.
edit
oh, you want to replace the pattern not append the line number... take a look the new cmd:
kent$ echo "a
b
c
d
e"|awk '/d/{gsub(/d/,NR)}1'
a
b
c
4
e
and the line could be written like this as well: awk '1+gsub(/d/,NR)' file
one-liner to modify the FILE in place, replacing LINE with the corresponding line number:
seq 1 `wc -l FILE | awk '{print $1}'` | xargs -IX sed -i 'X s/LINE/X/' FILE
Following on from https://stackoverflow.com/a/53519367/29924
If you try this on osx the version of sed is different and you need to do:
seq 1 `wc -l FILE | awk '{print $1}'` | xargs --verbose -IX sed -i bak "X s/__line__/X/" FILE
see https://markhneedham.com/blog/2011/01/14/sed-sed-1-invalid-command-code-r-on-mac-os-x/

How can I apply Unix's / Sed's / Perl's transliterate (tr) to only a specific column?

I have program output that looks like this (tab delim):
$ ./mycode somefile
0000000000000000000000000000000000 238671
0000000000000000000000000000000001 0
0000000000000000000000000000000002 0
0000000000000000000000000000000003 0
0000000000000000000000000000000010 0
0000000000000000000000000000000011 1548.81
0000000000000000000000000000000012 0
0000000000000000000000000000000013 937.306
What I want to do is on FIRST column only: replace 0 with A, 1 with C, 2 with G, and 3 with T.
Is there a way I can transliterate that output piped directly from "mycode".
Yielding this:
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA 238671
...
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACT 937.306
Using Perl:
C:\> ./mycode file | perl -lpe "($x,$y)=split; $x=~tr/0123/ACGT/; $_=qq{$x\t$y}"
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA 238671
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAC 0
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAG 0
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAT 0
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACA 0
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACC 1548.81
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACG 0
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACT 937.306
You can use single quotes in Bash:
$ ./mycode file | perl -lpe '($x,$y)=split; $x=~tr/0123/ACGT/; $_="$x\t$y"'
As #ysth notes in the comments, perl actually provides the command line options -a and -F:
-a autosplit mode with -n or -p (splits $_ into #F)
...
-F/pattern/ split() pattern for -a switch (//'s are optional)
Using those:
perl -lawnF'\t' -e '$,="\t"; $F[0] =~ y/0123/ACGT/; print #F'
It should be possible to do it with sed, put this in a file (you can do it command-line to, with -e, just don't forget those semicolons, or use separate -e for each line). (EDIT: Keep in mind, since your data is tab delimited, it should in fact be a tab character, not a space, in the first s//, make sure your editor doesn't turn it into spaces)
#!/usr/bin/sed -f
h
s/ .*$//
y/0123/ACGT/
G
s/\n[0-3]*//
and use
./mycode somefile | sed -f sedfile
or chmod 755 sedfile and do
./mycode somefile | sedfile
The steps performed are:
copy buffer to hold space (replacing held content from previous line, if any)
remove trailing stuff (from first space to end of line)
transliterate
append contents from hold space
remove the newline (from the append step) and all digits following it (up to the space)
Worked for me on your data at least.
EDIT:
Ah, you wanted a one-liner...
GNU sed
sed -e "h;s/ .*$//;y/0123/ACGT/;G;s/\n[0-3]*//"
or old-school sed (no semicolons)
sed -e h -e "s/ .*$//" -e "y/0123/ACGT/" -e G -e "s/\n[0-3]*//"
#sarathi
\AWK solution for this
awk '{gsub("0","A",$1);gsub("1","C",$1);gsub("2","G",$1);gsub("3","T",$1); print $1"\t"$2}' temp.txt