I need to copy .csv files from one directory to another, and while doing so need to reformat the first column, which is a date column, from 13/04/2012 to 2012-04-13. What is the sed syntax to perform this simple conversion? I am getting awfully confused by the sed stuff I'm reading on the web.
What have you tried so far? You could start with something small like this as a test case:
echo "13/04/2012;Col2;Col3" | sed -E 's#^([^/]+)/([^/]+)/([0-9]+)(.*)#\3-\2-\1\4#'
s = substitution command
# = start of pattern
^ = start of line
([^/]+)/ = group of all non-/-characters followed by a / (day)
([^/]+)/ = group of all non-/-characters followed by a / (month)
([0-9]+) = group of at least one digit (year)
(.*) = rest of line
# = start of replacement
\3 = backward reference to capture of third group (year)
\2 = backward reference to capture of second group (month)
\1 = backward reference to capture of first group (day)
\4 = backward reference to capture of fourth group (rest of line)
# end of command
sed 's|^\([0-9]\+\)/\([0-9]\+\)/\([0-9]\+\)|\3-\2-\1|'
This starts at the beginning of the line (^), records (\(...\)) one or more (\+) numbers ([0-9]), followed by a slash (/), a second set followed by another slash, and a third set, and rearranges the recorded sets (\1,\2,\3) separated by dashes.
This might work for you:
sed 's/^\(..\).\(..\).\(....\)/\3-\2-\1/' file
Related
I have a filename e.g. 15736--1_brand-new-image.jpg
My goal is to get the first letter after the _ in this case the b.
With s/\(.*\)\_\(.*\)$/\2/ I am able to extract brand-new-image.jpg
which is partly based on the info found on https://www.oncrashreboot.com/use-sed-to-split-path-into-filename-extension-and-directory
I've already found get first letter of words using sed but fail to combine the two.
To validate my sed statement I've used https://sed.js.org/
How can I combina a new sed statement on the part I've filtered to get the first letter?
With your shown samples could you please try following.
echo "15736--1_brand-new-image.jpg" | sed 's/[^_]*_\(.\).*/\1/'
Explanation: Simply using substitution operation of sed, then looking till 1st occurrence of _ then saving next 1 char into back reference and mentioning .* will cover everything after it, while substituting simply substituting everything with 1st back reference value which will be after 1st _ in this case its b.
Explanation: Following is only for explanation purposes.
sed ' ##Starting sed program from here.
s/ ##using s to tell sed to perform substitution operation.
[^_]*_\(.\).* ##using regex to match till 1st occurrence of _ then using back reference \(.\) to catch value in temp buffer memory here.
/\1/ ##Substituting whole line with 1st back reference value here which is b in this case.
'
Using a . or \w could also match _ in case there are 2 consecutive __
If you want to match the first word character without matching the _ you could also use
echo "15736--1_brand-new-image.jpg" | sed 's/[^_]*_\([[:alnum:]]\).*/\1/'
Output
b
This might work for you (GNU sed):
sed -nE 's/^[^_]*_[^[:alpha:]]*([[:alpha:]]).*/\1/p' file
Since this a filtering type operation use the -n option to print only when there is a positive match.
Match the first _ from the start of the line and then discard any non-alpha characters until an alpha character and finally discard any other characters.
Print the result if there is a match.
N.B. Anchoring the match to the start of the line, prevents the result containing more than one character i.e. consider the string 123_456_abc might otherwise result in 4 or 123_a.
I am trying to copy the beginning of every line in a text file before a certain character to the end of the same line.
I've tried duplicating each line to the end of itself, and then deleting everything after the character, but the trouble is I haven't been able to figure out how to skip the first instance of the character so the result is that the duplicated text gets deleted as well as everything beyond the first instance of the character.
I've tried things like
sed '/S/ s/$/ append text/' sample.txt > cleaned.txt
but this only adds a fixed text. I've also tried using:
s/\(.*\)/\1 \1/
to duplicate the line, and then deleting everything after the S, but I can't figure out how to get it to go to the 3rd S not the 1st to start deleting.
What I have to start with:
dog 50_50_S5_Scale
cat 10_RV_S76_Scale
mouse 15_SQ_S81_Scale
What I'm trying to get:
dog 50_50_S5_Scale dog 50_50_
cat 10_RV_17_S76_Scale cat 10_RV_17_
mouse 15_EQ_S81_Scale mouse 15_EQ_
Where everything before the first S gets copied to the end of the line.
You may use
sed 's/\([^S]*\)S.*/& \1/' file
See the online demo
Details
\([^S]*\) - Capturing group 1 (\1): any 0+ chars other than S
S.* - S and the rest of the string (actually, line, since sed processes line by line by default).
The replacement is the concatenation of the whole match (&), space and Group 1 value.
You could try:
awk '{print $0 " " substr($0, 0, index($0,"S") - 1)}' file
We take the substring from the first character up to but not including the first occurance of "S".
I need to append a few lines to a configuration file. The format is something like follows:
[Topic1]
param=foo
param=bar
param=foobar
[Topic2]
param=one
param=two
etc...
I am trying to write a script using sed to append parameters to a specific topic. Since all the topics have param=, I can't just insert a line after the last occurrence of that string. Also, I can't count on the value of the last parameter being consistent so for example I can't just insert a line after the string param=two
Any help would be appreciated. I'm not too familiar with mutliline sed-fu.
Thanks!
sed -i -r ':a; N; $!ba; s/\[Topic1\]\n(param=[a-zA-Z]*\n)*/¶m=VALUE\n/g' FILE_NAME
Basically what :a; N; $!ba; doing is append all line when not the last line (N) to the tag created by :a so that we can use \n in our expression.
Then match [Topic1] followed by arbitrary number of param=xxx, and append param=VALUE to the end of the matching result (&).
I'm trying to extract the name of the file name that has been generated by a Java program. This Java program spits out multiple lines and I know exactly what the format of the file name is going to be. The information text that the Java program is spitting out is as follows:
ABCASJASLEKJASDFALDSF
Generated file YANNANI-0008876_17.xml.
TDSFALSFJLSDJF;
I'm capturing the output in a variable and then applying a sed operator in the following format:
sed -n 's/.*\(YANNANI.\([[:digit:]]\).\([xml]\)*\)/\1/p'
The result set is:
YANNANI-0008876_17.xml.
However, my problem is that want the extraction of the filename to stop at .xml. The last dot should never be extracted.
Is there a way to do this using sed?
Let's look at what your capture group actually captures:
$ grep 'YANNANI.\([[:digit:]]\).\([xml]\)*' infile
Generated file YANNANI-0008876_17.xml.
That's probably not what you intended:
\([[:digit:]]\) captures just a single digit (and the capture group around it doesn't do anything)
\([xml]\)* is "any of x, m or l, 0 or more times", so it matches the empty string (as above – or the line wouldn't match at all!), x, xx, lll, mxxxxxmmmmlxlxmxlmxlm, xml, ...
There is no way the final period is removed because you don't match anything after the capture groups
What would make sense instead:
Match "digits or underscores, 0 or more": [[:digit:]_]*
Match .xml, literally (escape the period): \.xml
Make sure the rest of the line (just the period, in this case) is matched by adding .* after the capture group
So the regex for the string you'd like to extract becomes
$ grep 'YANNANI.[[:digit:]_]*\.xml' infile
Generated file YANNANI-0008876_17.xml.
and to remove everything else on the line using sed, we surround regex with .*\( ... \).*:
$ sed -n 's/.*\(YANNANI.[[:digit:]_]*\.xml\).*/\1/p' infile
YANNANI-0008876_17.xml
This assumes you really meant . after YANNANI (any character).
You can call sed twice: first in printing and then in replacement mode:
sed -n 's/.*\(YANNANI.\([[:digit:]]\).\([xml]\)*\)/\1/p' | sed 's/\.$//g'
the last sed will remove all the last . at the end of all the lines fetched by your first sed
or you can go for a awk solution as you prefer:
awk '/.*YANNANI.[0-9]+.[0-9]+.xml/{print substr($NF,1,length($NF)-1)}'
this will print the last field (and truncate the last char of it using substr) of all the lines that do match your regex.
I want to remove duplicate lines from a file, without sorting the file.
Example of why this is useful to me: removing duplicates from Bash's $HISTFILE without changing the chronological order.
This page has a one-liner to do that:
http://sed.sourceforge.net/sed1line.txt
Here's the one-liner:
sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P'
I asked a sysadmin and he told me "you just copy the script and it works, don't go philosophising about this", which is fine, so I am asking here as it's a developer forum and I trust people might be like me, suspicious about using things they don't understand:
Could you kindly provide a pseudo-code explanation of what that "black magic" script is doing, please? I tried parsing the incantation in my head but especially the central part is quite hard.
I'll note that this script does not appear to work with my copy of sed (GNU sed 4.1.5) in my current locale. If I run it with LC_ALL=C it works fine.
Here's an annotated version of the script. sed basically has two registers, one is called "pattern space" and is used for (basically) the current input line, and the other, the "hold space", can be used by scripts for temporary storage etc.
sed -n ' # -n: by default, do not print
G # Append hold space to current input line
s/\n/&&/ # Add empty line after current input line
/^\([ -~]*\n\).*\n\1/d # If the current input line is repeated in the hold space, skip this line
# Otherwise, clean up for storing all input in hold space:
s/\n// # Remove empty line after current input line
h # Copy entire pattern space back to hold space
P # Print current input line'
I guess the adding and removal of an empty line is there so that the central pattern can be kept relatively simple (you can count on there being a newline after the current line and before the beginning of the matching line).
So basically, the entire input file (sans duplicates) is kept (in reverse order) in the hold space, and if the first line of the pattern space (the current input line) is found anywhere in the rest of the pattern space (which was copied from the hold space when the script started processing this line), we skip it and start over.
The regex in the conditional can be further decomposed;
^ # Look at beginning of line (i.e. beginning of pattern space)
\( # This starts group \1
[ -~] # Any printable character (in the C locale)
* # Any number of times
\n # Followed by a newline
\) # End of group \1 -- it contains the current input line
.*\n # Skip any amount of lines as necessary
\1 # Another occurrence of the current input line, with newline and all
If this pattern matches, the script discards the pattern space and starts over with the next input line (d).
You can get it to work independently of locale by changing [ -~] to [[:print:]]
The code doesn't work for me, perhaps due to some locale setting, but this does:
vvv
sed -n 'G; s/\n/&&/; /^\([^\n]*\n\).*\n\1/d; s/\n//; h; P'
^^^
Let's first translate this by the book (i.e. sed info page), into something perlish.
# The standard sed loop
my $hold = "";
while ($my pattern = <>) {
chomp $pattern;
$pattern = "$pattern\n$hold"; # G
$pattern =~ s/(\n)/$1$1/; # s/\n/&&/
if ($pattern =~ /^([^\n]*\n).*\n\1/) { # /…/
next; # d
}
$pattern =~ s/\n//; # s/\n//
$hold = $pattern; # h
$pattern =~ /^([^\n]*\n?)/; print $1; # P
}
OK, the basic idea is that the hold space contains all the lines seen so far.
G: At the beginning of each cycle, append that hold space to the current line. Now we have a single string consisting of the current line and all unique lines which preceeded it.
s/\n/&&/: Turn the newline which separates them into a double newline, so that we can match subsequent and non-subsequent duplicates the same, see the next step.
^\([^\n]*\n\).*\n\1/: Look through the current text for the following: at the beginning of all the lines (^) look for a first line including trailing newline (\([^\n]*\n\)), then anything (.*), then a newline (\n), and then that same first line including newline repeated again (\1). If two subsequent lines are the same, then the .* in the regular expression will match the empty string, but the two \n will still match due to the newline duplication in the preceding step. So basically this asks whether the first line appears again among the other lines.
d: If there is a match, this is a duplicate line. We discard this input, keep the hold space as it is as a buffer of all unique lines seen so far, and continue with the next line of input.
s/\n//: Otherwise, we continue and next turn the double newline back into a single newline.
h: We include the current line in our list of all unique lines.
P: And finally print this new unique line, up to the newline character.
For the actual problem to resolve, here is a simpler solution (at least it looks so) with awk:
awk '!_[$0]++' FILE
In short _[$0] is a counter (of appearance) for each unique line, for any line ($0) appearing for the second time _[$0] >= 1, thus !_[$0] evaluates to false, causing it not to be printed except its first time appearance.
See https://gist.github.com/ryenus/5866268 (credit goes to a recent forum I visited.)