I'm trying to change the text in a file from this:
file.txt,c:\path\to\file
file.txt,c:\path with spaces\to\file
file.txt,c:\path\to\file with spaces
file.txt,c:\path\to\file with spaces
file.txt,c:\path\to\file with spaces
To this kind of output (one path to the file):
c:\path\to\file\file.txt
c:\path with spaces\to\file\file.txt
c:\path\to\file with spaces\file.txt
c:\path\to\file with spaces\file.txt
c:\path\to\file with spaces\file.txt
This ALMOST works but requires a ',' on the end of the line :
sed 's#\(.*\),\(.*\),\(.*\)#\2,\1,\3#g' file
Any help would be appreciated, I don't know sed all that well...
EDIT
This worked for me but I would still like to add a "\" in there:
sed 's#\(.*\),\(.*\)#\2,\1,\3#g' file
Escape the backslash, \\.
sed 's/^\(.*\),\(.*\)$/\2\\\1/g' file
--^^--
Also, you only need two capturing groups.
But since I'm more an awk guy.
awk -F, '{ print $2 "\\" $1 }' file
Related
I am cleaning up a dataset (csv dataset). I only want to consider registers in which all fields are complete and have the right type of values. This is what I tried:
sed -r '{
/regex_pattern/!d
more commands follow...
}' $1
The program works just fine and does what it is supposed to do. The problem is that it also removes the very first line (header line) since it does not match the specific regex_pattern. I know there is a way to specify the range in which the command should apply so for example:
sed '2,$ s/A/a/'
will do substitutions on data skipping the header line. Based on this logic I tried:
sed -r '{
2,$/regex_pattern/!d
more commands follow...
}' $1
so that the header line will be untouched however this code does not run at all.So what (and why) would be the right command to do what I am intending?
As an example, imagine my csv file is fruits.csv and that my regex_pattern is [0-9]+,[0-9]+
apples,oranges
20,5
7,3
,4
a,b
12,22
When I call the .sh script that contains the sed commands in should output:
apples,oranges
20,5
7,3
12,22
So, note that:
Header line was not deleted even though it does not match the regex_pattern.
Line number 4, i.e. ",4" was deleted as it does not match the regex_pattern.
Line number 5, i.e. "a,b" was deleted as it does not match the regex_pattern.
Any help is very much appreciated and I wish to thank you all in advance.
Kind regards.
You could write it like this, matching the whole line, starting at the second line:
sed -r '
2,${/^[0-9]+,[0-9]+$/!d}
' file
Output
apples,oranges
20,5
7,3
12,22
If you also want to allow single numbers or more than just 2 comma separated numbers:
sed -r '
2,${/^[0-9]+(,[0-9]+)*$/!d}
' file
Using sed
$ sed '2,${/[0-9]\+,[0-9]\+/!d}' input_file
apples,oranges
20,5
7,3
12,22
any one of these should work in gawk, mawk1/2, or macos nawk
mawk 'NF-_^(NF==NR)' FS='^[0-9]+,[0-9]+$'
nawk '(NF!=NR)!=NF' FS='^[0-9]+,[0-9]+$'
gawk 'NF-(NF!~NR)' FS='^[0-9]+,[0-9]+$'
'
apples,oranges
20,5
7,3
12,22
more concisely would be
mawk -F'[0-9]+,[0-9]+' '(NF<NR)-NF' # using FS
gawk '/[0-9]+,[0-9]+/^+(NF<NR)' # not using FS
nawk '(NF<NR)<=/([0-9]+,?){2}/' # same approach, rev. order
mawk '(NF~NR)-/[0-9]+,[0-9]+/' # truly fringe but
# concise syntax
nawk '(NF~NR)!=/([0-9]+,?){2}/' # same approach, to
# circumvent nawk peculiarities
sed is a bad choice for working with CSVs since it doesn't have any inbuilt functionality for working with fields, nor literal strings, nor variables, doesn't use EREs by default (all of the answers you have so far will only work with GNU sed), etc. To do what you specifically want with any awk in any shell on every Unix box is simply:
$ awk 'NR==1 || /[0-9]+,[0-9]+/' file
apples,oranges
20,5
7,3
12,22
which says "if the current line number (stored in NR) is 1 or the regexp matches the current line contents then print the line". Anything else you want to do with your CSV will also be easier with awk than with sed.
Meh, I would just preserve first line.
sed -r '
1{p;d}
/regex_pattern/!d
more commands follow...
' "$1"
or run it not for first line:
1!{
/regex_pattern/!d
more commands follow...
}
This might work for you (GNU sed):
sed -E '1!{/^[0-9]+,[0-9]+$/!d}' file
If it is not the first line, delete any line that does not match one set of comma separated natural numbers.
Alternative:
sed -E '1b;/^[0-9]+,[0-9]+$/!d' file
Or:
sed -nE '1p;1b;/^[0-9]+,[0-9]+$/p' file
I have some data in the format:
-e, 's/,Chalk/,Cheese/g'
-e, 's/,Black/,White/g'
-e, 's/,Leave/,Remain/g'
in a file data.csv.
Using Gitbash, I use the file command to discover that this is ASCII text with CRLF terminators. If I also use the command cat -v , I see in Gitbash that each line ends ^M .
I want to remove those terminators, to leave a single line.
I've tried the following:
sed -e 's/'\r\n'//g' < data.csv > output.csv
taking care to put the \r\n in single quotes in order that the backslash is treated literally, but it does not work. No error, just no effect.
I'm using Gitbash for Windows.
Quotes within quotes cancel each other out, so you actually undo the quotes around the sed command for the newline characters. You could escape the quotes like 's|'\''\r\n'\''||g', but that would just include them in the string, which would not match anything in your case.
But that is not the only problem; sed by default only processes strings between newlines.
If you have the GNU version of sed, RAM to spare if the file is huge, and are sure the file does not contain data with null characters, try adding the -z argument, like:
sed -z -e 's|\r\n||g' < data.csv > output.csv
Though I guess you probably also want to replace it with a comma:
sed -z -e 's|\r\n|,|g' < data.csv > output.csv
For non-GNU versions of sed, you may have an easier time using tr instead, like:
tr '\r\n' ',' data.csv > output.csv
Have a file that has been created incorrectly. There are several space delimited fields in the file but one text field has some unwanted newlines. This is causing a big problem.
How can I remove these characters but not the wanted line ends?
file is:
'Number field' 'Text field' 'Number field'
1 Some text 999999
2 more
text 111111111
3 Even more text 8888888888
EOF
So there is a NL after the word "more".
I've tried sed:
sed 's/.$//g' test.txt > test.out
and
sed 's/\n//g' test.txt > test.out
But none of these work. The newlines do not get removed.
tr -d '\n' does too much - I need to remove ONLY the newlines that are preceded by a space.
How can I delete newlines that follow a space?
SunOS 5.10 Generic_144488-09 sun4u sparc SUNW,Sun-Fire-V440
A sed solution is
sed '/ $/{N;s/\n//}'
Explanation:
/ $/: whenever the line ends in space, then
N: append a newline and the next line of input, and
s/\n//: delete the newline.
It might be simplest with Perl:
perl -p0 -e 's/ \n/ /g'
The -0 flag makes Perl read the entire file as one line. Then we can substitute using s in the usual way. You can, of course, also add the -i option to edit the file in-place.
How can I delete newlines that follow a space?
If you want every occurrence of $' \n' in the original file to be replaced by a space ($' '), and if you know of a character (e.g. a control character) that does not appear in the file, then the task can be accomplished quite simply using sed and tr (as you requested). Let's suppose, for example, that control-A is a character that is not in the file. For the sake of simplicity, let's also assume we can use bash. Then the following script should do the job:
#!/bin/bash
A=$'\01'
tr '\n' "$A" | sed "s/ $A/ /g" | tr "$A" '\n'
i have several files in which i want to replace a certain word with the name of the file itself..
for example i have 2 files named test1.txt and test2.txt
both files are equal and look like
bla1,bla2,temp
bla2,bla3,temp
with the sed i want to replace the word temp with the name of the file itself
so after the sed operation i have 2 different files
test1.txt , which looks like :
bla1,bla2,test1
bla2,bla3,test1
test2.txt, which looks like :
bla1,bla2,test2
bla2,bla3,test2
so my question ... how do i use the actual name of the input file itself as part of the replace command?
sed "s/temp/ ??filename??/ ??? " *.txt
thanks for your suggestions
I'm not sure you can reference the filename using sed although I could be wrong. You would probably use a shell hack. A better aproach to substitute all occurrences of temp with the filename would be the following awk script:
$ awk '{gsub(/temp/,FILENAME)}1' file
use awk, awk has FILENAME variable:
awk '{sub(/temp/,FILENAME)}7' yourfile
awk 'BEGIN{FS=OFS=","} {$NF=FILENAME}1' file
The difference between this and the sub() solutions is that this will work even if the word "temp" exists elsewhere in your file, e.g. if "bla1" contains the word "temperature".
If you need to strip ".txt" from the file name as it appears from your posted desired output, tweak it to:
awk 'BEGIN{FS=OFS=","} {t=FILENAME; sub(/\.txt$/,"",t); $NF=t}1' file
You can probably edit FILENAME itself but I find it best not to mess with the builtin variables if you don't have to.
You could do it with a little bit of bash to help you out, if that's available.
find . -name "test*.txt" -type f | awk -F '/' '{print $2;}' | while read file; do sed -i "s|temp|$file|" ./$file; done
That's a kind of hacky adaptation of a script I have to do something similar. It can undoubtedly be shortened.
no sed internal variable for the file name so you need some previous batch command for a generic process
for FileName in MyFileShellFilter
do
cat <> ${FileName} | sed "s|,temp$|,${FileName}|"
done
just be carrefull with file name used, they normaly don't have \ but could have & that are s// special meaning. I use | as separator to allow / in file name but for this reason, no unescaped | are allowed in file name (normaly not)
with xargs:
printf "%s\n" *.txt | xargs -I FILE -L 1 sed 's/temp/FILE/' FILE
The filename cannot have: newlines, slashes, ampersand, single quote.
How do I split a file to N files using as a filename the first 2 chars on the line.
Ex input file:
AA23409234TEXT
BA23201202Other Text
AA23509234YADA
BA23202202More Text.
C1000000000000000000
Should generate 3 files:
AA.txt
AA23409234TEXT
AA23509234YADA
BA.txt
BA23201202Other Text
BA23202202More Text.
C1.txt
C1000000000000000000
I'm thinking of using a sed script similar to this
/^(..)/w \1
But what that really does is create a file named '\1' instead of the capture group.
Any ideas?
$ awk '{fname=substr($0, 0, 2); print >>fname}' input.txt
Or
$ while read line; do echo "$line" >>"${line:0:2}"; done <input.txt
The first thing you need to do is determine all of your file names:
filenames=$(sed 's/\(..\).*/\1/' listOfStrings.txt | sort | uniq)
Then, loop through those filenames
for filename in $filenames
do
sed -n '/^$filename/ p' listOfStrings.txt > $filename.txt
done
I have not tested this, but I think it should work.
This might work for you:
sed 's/\(..\).*/echo "&" >>\1.txt/' file | sh
or if you have GNU sed:
sed 's/\(..\).*/echo "&" >>\1.txt/e' file