I'm trying to add array brackets at the beginning and end of a file using sed (after first removing the trailing comma at the end of the file) to put all the content of the file in an array. I'm first using this sed command to remove the last comma from a file
sed '$ s/,$//' "$path"
After that, I'm using the middle command below to add array brackets at beginning and ending of file
sed '$ s/,$//' "$path" | sed 's/^.*$/[&]/' | tee $filename
This sed 's/^.*$/[&]/' was supposed to match everything (from beginning to end ^$) and then put brackets around the whole match [&] (i.e. as if to make it into an array), but it instead put array brackets around the beginning and end of each line.
Question, how to edit the beginning and ending of a file with sed?
whole script
for path in dirname/* do
name="${path##*/}"
sed '$ s/,$//' "$path" | sed 's/^.*$/[&]/' | tee "newdir/$name"
done
sed is an editor that works line by line, so the command sed 's/^.*$/[&]/' would add brackets to every line. If you want to edit just the beginning and end of a file you need to put line numbers in front of the substitutions ($ stands for the last line):
sed -e '1 s/^/[/' -e '$ s/$/]/'
Since you already have a command that removes trailing ,'s you could combine it with the aforementioned substitutions. Your command line would then look like this:
sed -e '1 s/^/[/' -e '$ s/,*$/]/' "$path" | tee $filename
Related
How to search a pattern and remove the line using sed which contains special characters like "ranasnfs2:/SA_kits/prod"
I tried using a variable to hold the complete string and then recall the variable in sed command but it is not working.
echo $a
ranasnfs2:/SA_kits/prod
sed -i '/"$a"/d' test.txt
cat test.txt | grep -i SA
/SA_kits -rw,suid,soft,retry=4 ranasnfs2:/SA_kits/prod
You need to escape the slash character.
Use this for deleting lines which contain a /:
sed '/\//d' file
Have a file that has been created incorrectly. There are several space delimited fields in the file but one text field has some unwanted newlines. This is causing a big problem.
How can I remove these characters but not the wanted line ends?
file is:
'Number field' 'Text field' 'Number field'
1 Some text 999999
2 more
text 111111111
3 Even more text 8888888888
EOF
So there is a NL after the word "more".
I've tried sed:
sed 's/.$//g' test.txt > test.out
and
sed 's/\n//g' test.txt > test.out
But none of these work. The newlines do not get removed.
tr -d '\n' does too much - I need to remove ONLY the newlines that are preceded by a space.
How can I delete newlines that follow a space?
SunOS 5.10 Generic_144488-09 sun4u sparc SUNW,Sun-Fire-V440
A sed solution is
sed '/ $/{N;s/\n//}'
Explanation:
/ $/: whenever the line ends in space, then
N: append a newline and the next line of input, and
s/\n//: delete the newline.
It might be simplest with Perl:
perl -p0 -e 's/ \n/ /g'
The -0 flag makes Perl read the entire file as one line. Then we can substitute using s in the usual way. You can, of course, also add the -i option to edit the file in-place.
How can I delete newlines that follow a space?
If you want every occurrence of $' \n' in the original file to be replaced by a space ($' '), and if you know of a character (e.g. a control character) that does not appear in the file, then the task can be accomplished quite simply using sed and tr (as you requested). Let's suppose, for example, that control-A is a character that is not in the file. For the sake of simplicity, let's also assume we can use bash. Then the following script should do the job:
#!/bin/bash
A=$'\01'
tr '\n' "$A" | sed "s/ $A/ /g" | tr "$A" '\n'
I have a file in which some lines start by a >
For these lines, and only these ones, I want to keep the first eleven characters.
How can I do that using sed ?
Or maybe something else is better ?
Thanks !
Muriel
Let's start with this test file:
$ cat file
line one with something or other
>1234567890abc
other line in file
To keep only the first 11 characters of lines starting with > while keeping all other lines:
$ sed -r '/^>/ s/(.{11}).*/\1/' file
line one with something or other
>1234567890
other line in file
To keep only the first eleven characters of lines starting with > and deleting all other lines:
$ sed -rn '/^>/ s/(.{11}).*/\1/p' file
>1234567890
The above was tested with GNU sed. For BSD sed, replace the -r option with -E.
Explanation:
/^>/ is a condition. It means that the command which follows only applies to lines that start with >
s/(.{11}).*/\1/ is a substitution command. It replaces the whole line with just the first eleven characters.
-r turns on extended regular expression format, eliminating the need for some escape characters.
-n turns off automatic printing. With -n in effect, lines are only printed if we explicitly ask them to be printed. In the second case above, that is done by adding a p after the substitute command.
Other forms:
$ sed -r 's/(>.{10}).*/\1/' file
line one with something or other
>1234567890
other line in file
And:
$ sed -rn 's/(>.{10}).*/\1/p' file
>1234567890
This code removes newlines, then quotes and separates the characters.
Semicolon isn't working. Piping to a second sed does work, but the semicolon doesn't.
Script
# Piping works
echo "$1" | sed -r ':a;N;s/\n/\\n/;$!ba' | sed -r 's/(\\?.)/'"'\1',/g"
# Semicolon doesn't work on single lines
echo "$1" | sed -r ':a;N;s/\n/\\n/;$!ba;s/(\\?.)/'"'\1',/g"
# Skipping N command on single line works
echo "$1" | sed -r ':a;$bb;N;s/\n/\\n/;$!ba;:b;s/(\\?.)/'"'\1',/g"
Output:
$ wchar "test\n"
't','e','s','t','\n',
test\n
't','e','s','t','\n',
$ wchar "test
test"
't','e','s','t','\n','t','e','s','t',
't','e','s','t','\n','t','e','s','t',
't','e','s','t','\n','t','e','s','t',
The problem you run into is that the first line is also the last one (note that "test\n" does not contain a newline but a backslash), so the first N command is executed before any test whether the current line is the last one and ends up trying to fetch past the end.
Since all you're trying to do with the label and N loop is to assemble the file into the hold buffer (the replacement of the newlines can wait until afterwards), I suggest the following replacement:
echo -e "test\n" | sed -rn '1h;1!H;${x;s/\n/\\n/g; s/(\\?.)/'\''\1'\'', /g;p}'
This follows the basic pattern
sed -n '1 h; 1! H; $ { x; do stuff; p }'
...in other words, it reads the file into the hold buffer, swaps it back into the pattern space when the last line is being handled (i.e., when the whole file is in the hold buffer), does stuff with it and then prints the result. The s commands in place of do stuff are lifted from your code.
how to remove comment lines (as # bal bla ) and empty lines (lines without charecters) from file with one sed command?
THX
lidia
If you're worried about starting two sed processes in a pipeline for performance reasons, you probably shouldn't be, it's still very efficient. But based on your comment that you want to do in-place editing, you can still do that with distinct commands (sed commands rather than invocations of sed itself).
You can either use multiple -e arguments or separate commands with a semicolon, something like (just one of these, not both):
sed -i 's/#.*$//' -e '/^$/d' fileName
sed -i 's/#.*$//;/^$/d' fileName
The following transcript shows this in action:
pax> printf 'Line # with a comment\n\n# Line with only a comment\n' >file
pax> cat file
Line # with a comment
# Line with only a comment
pax> cp file filex ; sed -i 's/#.*$//;/^$/d' filex ; cat filex
Line
pax> cp file filex ; sed -i -e 's/#.*$//' -e '/^$/d' filex ; cat filex
Line
Note how the file is modified in-place even with two -e options. You can see that both commands are executed on each line. The line with a comment first has the comment removed then all is removed because it's empty.
In addition, the original empty line is also removed.
#paxdiablo has a good answer but it can be improved.
(1) The '/^$/d' clause only matches 100% blank lines.
If you want to also match lines that are entirely whitespace (spaces, tabs etc.) use this instead:
'/^\s*$/d'
(2) The 's/#.*$//' clause only matches lines that start with the # character in column 0.
If you want to also match lines that have only whitespace before the first # use this instead:
'/^\s*#.*$/d'
The above criteria may not be universal (e.g. within a HEREDOC block, or in a Python multi-line string the different approaches could be significant), but in many cases the conventional definition of "blank" lines include whitespace-only, and "comment" lines include whitespace-then-#.
(3) Lastly, on OSX at least, the #paxdiablo solution in which the first clause turns comment lines into blank lines, and the second clause strips blank lines (including what were originally comments) doesn't work. It seems to be more portable to make both clauses /d delete actions as I've done.
The revised command incorporating the above is:
sed -e '/^\s*#.*$/d' -e '/^\s*$/d' inputFile
This tiny jewel removes all # comments, no matter where they begin in a line (see caution below):
sed -e 's/\s*#.*$//'
Example:
text="
this is a # test
#this is a test
#this is a #test
this is # another #test
"
$echo "$text" | sed -e 's/\s*#.*$//'
this is a
this is
Next this removes any resulting blank lines:
$echo "$text" | sed -e 's/\s*#.*$//' | sed -e '/^\s*$/d'
Caution: Depending on the syntax and/or interpretation of the lines your processing, this might not be an appropriate solution, as it just stupidly removes end of lines, even if the '#' is part of your data or code. However, for use cases where you'll never use a hash except for as an end of line comment then it works fine. So just as with all coding, context must be taken into consideration.
Alternative variant, using grep:
cat file.txt | grep -Ev '(#.*$)|(^$)'
you can use awk
awk 'NF{gsub(/^[ \t]*#/,"");print}' file
First example(paxdiablo) is very good except its not change file, just output result. If you want to change it inline:
sudo sed -i 's/#.*$//;/^$/d' inputFile
On (one of) my linux boxes, sed understands extended regular expressions with the -r option, so:
sed -r '/(^\s*#)|(^\s*$)/d' squid.conf.installed
is very useful for showing all non-blank, non comment lines.
The regex matches either start of line followed by zero or more spaces or tabs followed by either a hash or end of line, and deletes those matching lines from the input.