I use torque to submit the test script shown blow
#!/bin/bash
#PBS -N test
#PBS -l nodes=1:ppn=1
#PBS -q ser
#PBS -V
#PBS -S /bin/bash
sed 's/a//' <<< aaabbbaaa
sed 's/\(a\)//' <<< aaabbbaaa
sed 's/a\+//' <<< aaabbbaaa
The expect output should be
aabbbaaa
aabbbaaa
bbbaaa
but actually when i use qsub to submit this file, the output becomes
aabbbaaa
aaabbbaaa
aaabbbaaa
The last two commands do not work. And it seems that the character '\' leads to the fault. But why does this happen.
The sed implementation on the server probably has a different syntax than what you have locally.
Without details about the remote system, this is mildly speculative; but look for the manual page and search for an option to enable Extended Regular Expression syntax (usually -E or -r; if available, probably lose the backslash before the regex specials then).
As a partial and obvious workaround, aa* is equivalent to a\+ but no similar workaround exists for grouping.
Maybe also read up on the differences between POSIX Basic Regular Expression syntax (BRE) and ERE; though chances are your PBS system is using something which even predates POSIX if it's legacy big iron.
If you want to write code which is portable between Linux, MacOS, and whatever behemoth dinosaur OS you have on the PBS system, maybe try Perl instead of sed. Even if you can only rely on Perl 4.x constructs, that's a platform which is a lot more uniform (and also a lot more versatile) than trying to figure out how to write portable sed scripts.
perl -pe 's/(a+)//'
Related
I'm running GnuWin32 under Windows 10
I'm trying to run the following sed one-liner using the Gnu Bash shell:
sed -f <(sed -E 's_(.+)\t(.+)_s/\1/\2/g_' C:/dictionary.txt) C:/content.txt
The file substitute sed statement converts dictionary entries into sed expressions. The main sed uses them for the content replacements.
It is described in How to awk to read a dictionary and replace words in a file?
dictionary.txt looks like this:
aluminium<tab>aluminum
analyse<tab>analyze
white spirit<tab>mineral spirits
stag night<tab>bachelor party
savoury<tab>savory
potato crisp<tab>potato chip
mashed potato<tab>mashed potatoes
content.txt looks like this:
The container of white spirit was made of aluminium.
We will use an aromatic method to analyse properties of white spirit.
No one drank white spirit at stag night.
Many people think that a potato crisp is savoury, but some would rather eat mashed potato.
...
more sentences
When running GnuWin32/sed in GnuBash-shell under windows 10, I receive the following error message:
syntax error near unexpected token <(s
How to re-formulate the script to run under GnuWin32/sed under windows 10?
with thanks to https://stackoverflow.com/users/2836621/mark-setchell and https://stackoverflow.com/users/5403468/tiw the solution works when using cygwin64
One way is to write the inner sed output to a temporary file first, use it, and then delete it:
sed -r "s_(.+)\t(.+)_s/\1/\2/g_" C:/dictionary.txt>tmp_script.sed
sed -f tmp_script.sed C:/content.txt
del tmp_script.sed
Another way, based on Mr. Mark Setchell's comment, plus little tweak, with cygwin installed,
this work on both bash and batch:
sed -r "s_(.+)\t(.+)_s/\1/\2/g_" C:/dictionary.txt | sed -f /dev/stdin C:/content.txt
Running Fedora 25 server edition. sed --version gives me sed (GNU sed) 4.2.2 along with the usual copyright and contact info. I've create a text file sudo vi ./potential_sed_bug. Vi shows the contents of this file (with :set list enabled) as:
don't$
delete$
me$
please$
I then run the following command:
sudo sed -n -i.bak /please/a\testing ./potential_sed_bug
Before we discuss the results; here is what the sed man page says:
-n, --quiet, --silent
suppress automatic printing of pattern space
and
-i[SUFFIX], --in-place[=SUFFIX]
edit files in place (makes backup if extension supplied). The default operation mode is to break symbolic and hard links. This can be changed with --follow-symlinks and --copy.
I've also looked other sed command references to learn how to append with sed. Based on my understanding from the research I've done; the resulting file content should be:
don't
delete
me
please
testing
However, running sudo cat ./potential_sed_bug gives me the following output:
testing
In light of this discrepancy, is my understanding of the command I ran incorrect or is there a bug with sed/the environment?
tl;dr
Don't use -n with -i: unless you use explicit output commands in your sed script, nothing will be written to your file.
Using -i produces no stdout (terminal) output, so there's nothing extra you need to do to make your command quiet.
By default, sed automatically prints the (possibly modified) input lines to whatever its output target is, whether implied or explicitly specified: by default, to stdout (the terminal, unless redirected); with -i, to a temporary file that ultimately replaces the input file.
In both cases, -n suppresses this automatic printing, so that - unless you use explicit output functions such as p or, in your case, a - nothing gets printed to stdout / written to the temporary file.
Note that the automatic printing applies to the so-called pattern space, which is where the (possibly modified) input is held; explicit output functions such as p, a, i and c do not print to the pattern space (for potential subsequent modification), they print directly to the target stream / file, which is why a\testing was able to produce output, despite the use of -n.
Note that with -i, sed's implicit printing / explicit output commands only print to the temporary file, and not also to stdout, so a command using -i is invariably quiet with respect to stdout (terminal) output - there's nothing extra you need to do.
To give a concrete example (GNU sed syntax).
Since the use of -i is incidental to the question, I've omitted it for simplicity. Note that -i prints to a temporary file first, which, on completion, replaces the original. This comes with pitfalls, notably the potential destruction of symlinks; see the lower half of this answer of mine.
# Print input (by default), and append literal 'testing' after
# lines that contain 'please'.
$ sed '/please/ a testing' <<<$'yes\nplease\nmore'
yes
please
testing
more
# Adding `-n` suppresses the default printing, so only `testing` is printed.
# Note that the sequence of processing is exactly the same as without `-n`:
# If and when a line with 'please' is found, 'testing' is appended *at that time*.
$ sed -n '/please/ a testing' <<<$'yes\nplease\nmore'
testing
# Adding an unconditional `p` (print) call undoes the effect of `-n`.
$ sed -n 'p; /please/ a testing' <<<$'yes\nplease\nmore'
yes
please
testing
more
I've got a bash script that wraps mvn (Apache Maven) to add colour to its output. A cut-down version of what it does is:
mvn "$#" | sed -e "s/^\[INFO\] \-.*/$bldblu&$rst/g"
where $bldblu is the ANSI color escape characters for bold blue, and $rst resets the colours.
The issue I'm having is that sometimes mvn writes a line that doesn't end in a newline, thus (as far as I can tell) sed keeps waiting for input and never prints the prompt (which makes it seem like Maven is hanging). I've tried adding -u to sed but that just forces sed to do line-by-line buffering instead of buffering more than one line - not helpful for me.
So far this is what I've come up with:
mvn "$#" | perl -pe "$| = 1; s/^(\[INFO\] \-.*)/$bldblu\$1$rst/g"
but I think the use of -p is not correct here. Any help?
A substitution may be overkill, especially when the replacement pattern has special characters in it. How about this?
export bldblu
export rst
mvn "$#" | perl -pe 'if(/^.INFO. -/){ $_=$ENV{bldblu}.$_.$ENV{rst} }'
or rather than reinventing the wheel
mvn "$#" | perl -MTerm::ANSIColor -pe
'$_=color("bold blue").$_.color("reset") if /^.INFO. -/'
(workaround) Use sed --unbuffered
I couldn't figure out the solution but thankfully this is good enough for my particular usage:
cat - | sed --unbuffered 's/.*?from//g'
But I too would like to know the answer. Perl one line substitution is a key idiom in my toolbelt.
BSD
Looks like there is no common flag for GNU and BSD. For the latter, you'd need:
-l Make output line buffered.
Does sed -i work on AIX?
If not, how can I edit a file "in place" on AIX?
The -i option is a GNU (non-standard) extension to the sed command. It was not part of the classic interface to sed.
You can't edit in situ directly on AIX. You have to do the equivalent of:
sed 's/this/that/' infile > tmp.$$
mv tmp.$$ infile
You can only process one file at a time like this, whereas the -i option permits you to achieve the result for each of many files in its argument list. The -i option simply packages this sequence of events. It is undoubtedly useful, but it is not standard.
If you script this, you need to consider what happens if the command is interrupted; in particular, you do not want to leave temporary files around. This leads to something like:
tmp=tmp.$$ # Or an alternative mechanism for generating a temporary file name
for file in "$#"
do
trap "rm -f $tmp; exit 1" 0 1 2 3 13 15
sed 's/this/that/' $file > $tmp
trap "" 0 1 2 3 13 15
mv $tmp $file
done
This removes the temporary file if a signal (HUP, INT, QUIT, PIPE or TERM) occurs while sed is running. Once the sed is complete, it ignores the signals while the mv occurs.
You can still enhance this by doing things such as creating the temporary file in the same directory as the source file, instead of potentially making the file in a wholly different file system.
The other enhancement is to allow the command (sed 's/this/that' in the example) to be specified on the command line. That gets trickier!
You could look up the overwrite (shell) command that Kernighan and Pike describe in their classic book 'The UNIX Programming Environment'.
#!/bin/ksh
host_name=$1
perl -pi -e "s/#workerid#/$host_name/g" test.conf
Above will replace #workerid# to $host_name inside test.conf
You can simply install GNU version of Unix commands on AIX :
http://www-03.ibm.com/systems/power/software/aix/linux/toolbox/alpha.html
You can use a here construction with vi:
vi file >/dev/null 2>&1 <<#
:1,$ s/old/new/g
:wq
#
When you want to do things in the vi-edit mode, you will need an ESC.
For an ESC press CTRL-V ESC.
When you use this in a non-interactive mode, vi can complain about the TERM not set. The solution is adding export TERM=vt100 before calling vi.
Another option is to use good old ed, like this:
ed fileToModify <<EOF
,s/^ff/gg/
w
q
EOF
you can use perl to do it :
perl -p -i.bak -e 's/old/new/g' test.txt
is going to create a .bak file.
I have the following line in my proftpd log (line 78 to be precise)
Deny from 1.2.3.4
I also have a script which rolls through my logs for people using brute force attacks and then stores their IP (ready for a black listing). What i'm struggling with is inserting (presume with sed) at the end of that specific line - this is what I've got so far:
sed "77i3.4.5.6" /opt/etc/proftpd.conf >> /opt/etc/proftpd.conf
Now one would presume this would work perfectly, however it actually does the following (lines 77 through 78):
3.4.5.6
Deny from 1.2.3.4
I suspect this is due to my dated version of sed, are there any other ways of acheiving the same thing? Also the >> causes the config to be duplicated at the end of the fole (again i'm sure this is a limitation of my version of sed). This is running a homebrew linux kernel on my nas. Sed options below:
root#NAS:~# sed BusyBox v1.7.0
(2009-04-29 19:12:57 JST) multi-call
binary
Usage: sed [-efinr] pattern [files...]
Options:
-e script Add the script to the commands to be executed
-f scriptfile Add script-file contents to the
commands to be executed
-i Edit files in-place
-n Suppress automatic printing of pattern space
-r Use extended regular expression syntax
If no -e or -f is given, the first
non-option argument is taken as the
sed script to interpret. All remaining
arguments are names of input files; if
no input files are specified, then the
standard input is read. Source files
will not be modified unless -i option
is given.
Cheers for your help guys.
This has nothing to do with the version of sed; this is just plain old Doing It Wrong.
sed -i '77s/$/,3.4.5.6/' /opt/etc/proftpd.conf