sed script not executing in a Vagrantfile - sed

I can't for the life of me get sed working with vagrant provisioning. I want to make a inline change to /etc/hosts.
I've verified that the sed command works when run in the shell.
Here is my Vagrantfile:
# vi: set ft=ruby :
########### Global Config ###########
machines = ["admin2"]
num_hdd_per_osd = 3
vagrant_box = %q{bento/ubuntu-18.04}
#####################################
machines.each do |machine|
Vagrant.configure("2") do |config|
config.vm.define machine do |node| #name vagrant uses to reference this VM
node.vm.box = vagrant_box
node.vm.hostname = machine
node.vm.network "private_network", ip: "192.168.0.#{ machines.index(machine) + 10}"
node.vm.provider "virtualbox" do |vb|
# Display the VirtualBox GUI when booting the machine
vb.gui = false
vb.name = machine # name virtualbox uses to refer to this vm
# Customize the amount of memory on the VM:
vb.memory = "1048"
# Core Count
vb.cpus = "2"
end
if node.vm.hostname.include? "admin"
node.vm.provision "shell", inline: <<-SHELL
sed -i.bak -e 's,\\(127\\.0\\.0\\.1[[:space:]]*localhost\\),\\1aa,' /etc/hosts
SHELL
end
end
end
end
I should see /etc/hosts changed to 127.0.0.1 localhostaa but it is unchanged.
What is wrong?
EDIT: I updated the code with the suggestion from Alex below. It now uses inline: <<-SHELL and escaped ALL escapes (so double escape). It Works!

The problem there is your Vagrantfile is Ruby code, and your sed script is inside a Ruby here string.
If you try this simplified Ruby script:
# test.rb
puts <<-SHELL
sudo sed -i.bak -e 's,\(127\.0\.0\.1[[:space:]]*localhost\),\1aa,' /etc/host
SHELL
You may see the problem:
▶ ruby test.rb
sudo sed -i.bak -e 's,(127.0.0.1[[:space:]]*localhost),aa,' /etc/host
That is, the \1 and other \ have been interpreted by Ruby prior to interpolation in the here string.
The best option for you is to use the <<'SHELL' notation, similar to what you would do in Bash:
node.vm.provision "shell", inline: <<-'SHELL'
sed -i.bak -e 's,\(127\.0\.0\.1[[:space:]]*localhost\),\1aa,' /etc/hosts
SHELL
The other option would be to escape the backslash in \1. Also, note that, as far as I can tell, the call to sudo is not required there either.
If, however, you need to interpolate a string in this script, you could do something like this:
# test.rb
mystring = 'aa'
$script = "sed -i.bak -e '" +
's,\(127\.0\.0\.1[[:space:]]*localhost\),\1' + "#{mystring},' /etc/hosts"
And then in your provisioner:
node.vm.provision "shell", inline: $script
See also this related answer.

Related

Command substitution in fish

I'm trying to migrate this working command
docker-compose $(find docker-compose* | sed -e "s/^/-f /") up -d --remove-orphans
from bash to fish. The intention of this command is to get this
docker-compose -f docker-compose.backups.yml ... -f docker-compose.wiki.yml up -d --remove-orphans
My naive try
docker-compose (find docker-compose* | sed -e "s/^/-f /") up -d --remove-orphans
is not working, though. The error is:
ERROR: .FileNotFoundError: [Errno 2] No such file or directory: './ docker-compose.backups.yml'
What is the correct translation?
The difference in behavior is due to the fact fish, sanely, only splits the output of a command capture on line boundaries. Whereas POSIX shells like bash split it on whitespace by default. That is, POSIX shells split the output of $(...) on the value of $IFS which is space, tab, and newline by default.
There are several ways to rewrite that command so it works in fish. The one that requires the smallest change is to change the sed to insert a newline between the -f and the filename:
docker-compose (find docker-compose* | sed -e "s/^/-f\n/") up -d --remove-orphans

sed unterminated s command in vagrant only [duplicate]

I'm currently using Vagrant to set up a development machine running Ubuntu. I want to add a few lines to my .profile that add directories to my $PATH variable, using sed. To this end, I added these lines to my machine's Vagrantfile:
config.vm.provision "shell", inline:
"sudo sed -i \'$ a if [ -d \\\"/usr/local/lib\\\" ]; then\n PATH=\\\"/usr/local/lib:$PATH\\\"\nfi\' /home/vagrant/.profile"
To the best of my knowledge and testing, I've escaped all the characters needed to have Vagrant run this valid shell command:
sudo sed -i '$ a if [ -d \"/usr/local/lib\" ]; then\n PATH=\"/usr/local/lib:$PATH\"\nfi' /home/vagrant/.profile
Which adds these lines to my .profile:
if [ -d "usr/local/lib" ]; then
PATH="usr/local/lib:$PATH"
fi
However, when I do a vagrant up I get the following error when it tries to run the command:
==> default: Running provisioner: shell...
default: Running: inline script
==> default: stdin: is not a tty
==> default: sed: -e expression #1, char 44: extra characters after command
The SSH command responded with a non-zero exit status.
Have I made a mistake somewhere in my Vagrantfile code, or is there something else going wrong here?
To fix "sed: -e expression #1, char 44: extra characters after command", escape \n, that is add another backslash to all \n:
config.vm.provision "shell", inline:
"sudo sed -i \'$ a if [ -d \\\"/usr/local/lib\\\" ] then\\n PATH=\\\"/usr/local/lib:$PATH\\\"\\nfi\' /home/vagrant/.profile"

Replace string literal of unicode character with sed [duplicate]

I've successfully used the following sed command to search/replace text in Linux:
sed -i 's/old_link/new_link/g' *
However, when I try it on my Mac OS X, I get:
"command c expects \ followed by text"
I thought my Mac runs a normal BASH shell. What's up?
EDIT:
According to #High Performance, this is due to Mac sed being of a different (BSD) flavor, so my question would therefore be how do I replicate this command in BSD sed?
EDIT:
Here is an actual example that causes this:
sed -i 's/hello/gbye/g' *
If you use the -i option you need to provide an extension for your backups.
If you have:
File1.txt
File2.cfg
The command (note the lack of space between -i and '' and the -e to make it work on new versions of Mac and on GNU):
sed -i'.original' -e 's/old_link/new_link/g' *
Create 2 backup files like:
File1.txt.original
File2.cfg.original
There is no portable way to avoid making backup files because it is impossible to find a mix of sed commands that works on all cases:
sed -i -e ... - does not work on OS X as it creates -e backups
sed -i'' -e ... - does not work on OS X 10.6 but works on 10.9+
sed -i '' -e ... - not working on GNU
Note Given that there isn't a sed command working on all platforms, you can try to use another command to achieve the same result.
E.g., perl -i -pe's/old_link/new_link/g' *
I believe on OS X when you use -i an extension for the backup files is required. Try:
sed -i .bak 's/hello/gbye/g' *
Using GNU sed the extension is optional.
This works with both GNU and BSD versions of sed:
sed -i'' -e 's/old_link/new_link/g' *
or with backup:
sed -i'.bak' -e 's/old_link/new_link/g' *
Note missing space after -i option! (Necessary for GNU sed)
Had the same problem in Mac and solved it with brew:
brew install gnu-sed
and use as
gsed SED_COMMAND
you can set as well set sed as alias to gsed (if you want):
alias sed=gsed
Or, you can install the GNU version of sed in your Mac, called gsed, and use it using the standard Linux syntax.
For that, install gsed using ports (if you don't have it, get it at http://www.macports.org/) by running sudo port install gsed. Then, you can run sed -i 's/old_link/new_link/g' *
Your Mac does indeed run a BASH shell, but this is more a question of which implementation of sed you are dealing with. On a Mac sed comes from BSD and is subtly different from the sed you might find on a typical Linux box. I suggest you man sed.
Insead of calling sed with sed, I do ./bin/sed
And this is the wrapper script in my ~/project/bin/sed
#!/bin/bash
if [[ "$OSTYPE" == "darwin"* ]]; then
exec "gsed" "$#"
else
exec "sed" "$#"
fi
Don't forget to chmod 755 the wrapper script.
Sinetris' answer is right, but I use this with find command to be more specific about what files I want to change. In general this should work (tested on osx /bin/bash):
find . -name "*.smth" -exec sed -i '' 's/text1/text2/g' {} \;
In general when using sed without find in complex projects is less efficient.
I've created a function to handle sed difference between MacOS (tested on MacOS 10.12) and other OS:
OS=`uname`
# $(replace_in_file pattern file)
function replace_in_file() {
if [ "$OS" = 'Darwin' ]; then
# for MacOS
sed -i '' -e "$1" "$2"
else
# for Linux and Windows
sed -i'' -e "$1" "$2"
fi
}
Usage:
$(replace_in_file 's,MASTER_HOST.*,MASTER_HOST='"$MASTER_IP"',' "./mysql/.env")
Where:
, is a delimeter
's,MASTER_HOST.*,MASTER_HOST='"$MASTER_IP"',' is pattern
"./mysql/.env" is path to file
As the other answers indicate, there is not a way to use sed portably across OS X and Linux without making backup files. So, I instead used this Ruby one-liner to do so:
ruby -pi -e "sub(/ $/, '')" ./config/locales/*.yml
In my case, I needed to call it from a rake task (i.e., inside a Ruby script), so I used this additional level of quoting:
sh %q{ruby -pi -e "sub(/ $/, '')" ./config/locales/*.yml}
Here's how to apply environment variables to template file (no backup need).
1. Create template with {{FOO}} for later replace.
echo "Hello {{FOO}}" > foo.conf.tmpl
2. Replace {{FOO}} with FOO variable and output to new foo.conf file
FOO="world" && sed -e "s/{{FOO}}/$FOO/g" foo.conf.tmpl > foo.conf
Working both macOS 10.12.4 and Ubuntu 14.04.5
Here is an option in bash scripts:
#!/bin/bash
GO_OS=${GO_OS:-"linux"}
function detect_os {
# Detect the OS name
case "$(uname -s)" in
Darwin)
host_os=darwin
;;
Linux)
host_os=linux
;;
*)
echo "Unsupported host OS. Must be Linux or Mac OS X." >&2
exit 1
;;
esac
GO_OS="${host_os}"
}
detect_os
if [ "${GO_OS}" == "darwin" ]; then
sed -i '' -e ...
else
sed -i -e ...
fi
sed -ie 's/old_link/new_link/g' *
Works on both BSD & Linux with gnu sed

Use a Chef recipe to modify a single line in a config file

I'm trying to automate disabling the Transparent Huge Pages (THP) Settings for MongoDB using a Chef Recipe.
The THP setting is explained here: MongoDocs THP Settings
I'm trying to follow the first option "In Boot-Time Configuration (Preferred)" by editing the grub configuration file at "/etc/grub.conf"
All I need to do is append "transparent_hugepage=never" to the end of the existing line that starts with "kernel "
I know I can replace a line with Chef::Util::FileEdit, using something like this:
ruby_block "replace_line" do
block do
file = Chef::Util::FileEdit.new("/etc/grub.conf")
file.search_file_replace_line("/kernel/", "kernel <kernel path> <kernel options> transparent_hugepage=never")
file.write_file
end
end
but I need to keep the existing kernel path and kernel options.
I've tried playing around with Chef::Util::Editor, but haven't been successful initializing the constructor. Chef::Util::FileEdit is initialized with a file path (per above), but the ruby docs say that Chef::Util::Editor is initialized with "lines". I've tried
lines = Chef::Util::Editor.new(<lines>)
where <lines> = file path, = Chef::Util::FileEdit.new(), and = 'test string', but nothing seems to work.
Does anyone have any experience with the Chef::Util::Editor? Or a better solution?
Thanks
I never figured out how to modify a single line in a config file using Chef, but here's the recipe I ended up using to disable THP settings for MongoDB.
Recipe: Install MongoDB
# Install MongoDB on Amazon Linux
# http://docs.mongodb.org/manual/tutorial/install-mongodb-on-amazon/
# 1: configure the package management system (yum)
# 2: install mongodb
# 3: configure mongodb settings
# 3.A: give mongod permission to files
# data & log directories (everything in /srv/mongodb)
# http://stackoverflow.com/questions/7948789/mongodb-mongod-complains-that-there-is-no-data-db-folder
execute "mongod_permission" do
command "sudo chown -R mongod:mongod /srv/mongodb"
#command "sudo chown mongod:mongod /var/run/mongodb/mongod.pid"
#command "sudo chown -R $USER /srv/mongodb"
end
# 3.B: edit Transparent Huge Pages (THP) Settings
# get rid of mongod startup warning
# http://docs.mongodb.org/manual/reference/transparent-huge-pages/#transparent-huge-pages-thp-settings
# 3.B.1: disable
execute "disable_thp_khugepaged_defrag" do
command "echo 0 | sudo tee /sys/kernel/mm/transparent_hugepage/khugepaged/defrag" # different b/c file doesn't have options list
end
execute "disable_thp_hugepage_defrag" do
command "echo 'never > /sys/kernel/mm/transparent_hugepage/defrag' | sudo tee --append /sys/kernel/mm/transparent_hugepage/defrag"
end
execute "disable_thp_hugepage_enables" do
command "echo 'never > /sys/kernel/mm/transparent_hugepage/enabled' | sudo tee --append /sys/kernel/mm/transparent_hugepage/enabled"
end
# 3.B.2: verify disabled on reboot
template "/etc/rc.local" do
source "init-rc.local.erb"
owner 'root'
group 'root'
mode '0775'
end
# 4: use upstart & monit to keep mongod alive
Template: init-rc.local.erb
touch /var/lock/subsys/local
if test -f /sys/kernel/mm/transparent_hugepage/khugepaged/defrag; then
echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/defrag
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
The problem with your own solution is the template can be overwritten by another recipe with it's own rc.local template.
To change that, I add the lines to the existing rc.local
execute "disable_thp_hugepage_defrag" do
command "sudo sed -i -e '$i \\echo never > /sys/kernel/mm/transparent_hugepage/defrag\\n' /etc/rc.local"
not_if 'grep -c "transparent_hugepage/defrag" /etc/rc.local'
end
execute "disable_thp_hugepage_enables" do
command "sudo sed -i -e '$i \\echo never > /sys/kernel/mm/transparent_hugepage/enabled\\n' /etc/rc.local"
not_if 'grep -c "transparent_hugepage/enabled" /etc/rc.local'
end
The grep makes sure that the line is not already in it.
Maybe chef has something better to manage that?
We can efficietly replace contents of file by grouping the elements
e.g.
appending "transparent_hugepage=never" to the end of the existing line that starts with "kernel "
ruby_block "replace_line" do
block do
file = Chef::Util::FileEdit.new("/etc/grub.conf")
file.search_file_replace_line(/kernel.*/, '\0 tansparent_hugepage=never')
file.write_file
end
end
\0 adds whole mached string
note: ' '(single quote)
I disabled hugepages by replicating the following in chef (looks the same as above but with the addition of a not_if statement):
echo never | sudo tee /sys/kernel/mm/transparent_hugepage/defrag
E.G
execute "disable_hugepage_defrag" do
not_if "grep -F '[never]' /sys/kernel/mm/transparent_hugepage/defrag"
command "echo 'never' > /sys/kernel/mm/transparent_hugepage/defrag"
end
I have also had success inserting lines with file.insert_line_if_no_match the ruby line replace feature will probably work for you.
search_file_replace_line(regex, newline) ⇒ Object
ruby_block 'replace_line' do
block do
file = Chef::Util::FileEdit.new('/path/to/file')
file.search_file_replace_line('/Line to find/', 'Line to replace with')
file.write_file
end
end

Escaping whitespace within nested shell/perl scripts

I'm trying to run a perl script from within a bash script (I'll change this design later on, but for now, bear with me). The bash script receives the argument that it will run. The argument to the script is as follows:
test.sh "myscript.pl -g \"Some Example\" -n 1 -p 45"
within the bash script, I simple run the argument that was passed:
#!/bin/sh
$1
However, in my perl script the -g argument only gets "Some (that's with the quotes), instead of the Some Example. Even if I quote it, it cuts off because of the whitespace.
I tried escaping the whitespace, but it doesn't work... any ideas?
To run it as posted test.sh "myscript.pl -g \"Some Example\" -n 1 -p 45" do this:
#!/bin/bash
eval "$1"
This causes the $1 argument to be parsed by the shell so the individual words will be broken up and the quotes removed.
Or if you want you could remove the quotes and run test.sh myscript.pl -g "Some Example" -n 1 -p 45 if you changed your script to:
#!/bin/bash
"$#"
The "$#" gets replaced by all the arguments $1, $2, etc., as many as were passed in on the command line.
Quoting is normally handled by the parser, which isn't seeing them when you substitute the value of $1 in your script.
You may have more luck with:
#!/bin/sh
eval "$1"
which gives:
$ sh test.sh 'perl -le "for (#ARGV) { print; }" "hello world" bye'
hello world
bye
Note that simply forcing the shell to interpret the quoting with "$1" won't work because then it tries to treat the first argument (i.e., the entire command) as the name of the command to be executed. You need the pass through eval to get proper quoting and then re-parsing of the command.
This approach is (obviously?) dangerous and fraught with security risks.
I would suggest you name the perl script in a separate word, then you can quote the parameters when referring to them, and still easily extract the script name without needing the shell to split the words, which is the fundamental problem you have.
test.sh myscript.pl "-g \"Some Example\" -n 1 -p 45"
and then
#!/bin/sh
$1 "$2"
if you really have to do this (for whatever reason) why not just do:
sh test.sh "'Some Example' -n 1 -p 45"
in:
test.sh
RUN=myscript.pl
echo `$RUN $1
(there should be backticks ` before $RUN and after $1)