I use terraform to create an EC2 instance, and I use user_data to place a file in /var/lib/cloud/scripts/per-once. This is not executed - my question now is: is cloud-init run before user_data?
===EDIT===
A longer reply to Dude0001's very helpful answer:
I have tried the following, now - this is my user_data:
#!/bin/bash
cat >/var/lib/cloud/scripts/per-once/install_mysql <<!
#cloud-config
package_update: true
packages:
- mysql-server
!
cat >>/root/.bashrc <<!
set -o vi
unalias -a
alias ll='ls -lp'
!
cat >>/home/admin/.bashrc <<!
set -o vi
unalias -a
alias ll='ls -lp'
!
cat /root/.vimrc <<!
set t_ti= t_te=
set compatible
set expandtab ts=2 sw=2 ai
!
cat >/home/admin/.vimrc <<!
set t_ti= t_te=
set compatible
set expandtab ts=2 sw=2 ai
!
This creates all the files, as expected (I'm really old-fashioned and don't like most of vim's new features). I tried to reboot after the instance was created: no mysqld. I changed the permissions, chmod 755 /var/lib/cloud/scripts/per-once/install_mysql, and rebooted: no result either (the reason I changed permissions is that it appears from the python code that cloud-init looks for executables only).
===EDIT===
Some explanations to my user_data above:
This construction may mystify some, since it isn't too common:
cat >/some/path/to/a/file <<!
...
!
cat is a command that simply read from the standard input and writes to the standard output without change - it is often used with redirection < and >. In the construction above, I direct any output to a file /some/path/to/a/file. The other part, involving <<! and ! is known as a here document, something that has its origin in the JCL language used on mainframes, I suspect, but it is really useful. What is means is read the following lines until the end-marker (here: !, but it could be any string). So, all in all, it says create a file with the following content: ....
The first file, /var/lib/cloud/scripts/per-once/install_mysql, contains:
#cloud-config
package_update: true
packages:
- mysql-server
My hope is that this should tell cloud-init to update the package repository and install mysql-server - this doesn't happen.
The next 4 files are just some setup in the root and admin users' environments; basically, I create a .vimrc and add a few lines to .bashrc to ensure that certain things are set up to my liking.
The files are all created, but the one with #cloud-config doesn't seem to get touched at all. I have done a few experiments yesterday, by placing this file in different directories under /var/lib/cloud/scripts/, but it looks a lot as if these files aren't in place for when cloud-init reads the directories. Reading through cloud-init's source code, it looks as if it runs through 10 stages - user_data is fetched in stage 5, and it should be read in stage 7. I can also see that it seems to require the execute permission bit to be set; however this is what is in the log after a reboot:
2019-10-02 08:06:52,884 - handlers.py[DEBUG]: start: modules-final/config-scripts-per-boot: running config-scripts-per-boot with frequency always
2019-10-02 08:06:52,884 - helpers.py[DEBUG]: Running config-scripts-per-boot using lock (<cloudinit.helpers.DummyLock object at 0x7f677362acc0>)
2019-10-02 08:06:52,885 - util.py[DEBUG]: Running command ['/var/lib/cloud/scripts/per-boot/install_mysql'] with allowed return codes [0] (shell=False, capture=False)
2019-10-02 08:06:52,887 - util.py[WARNING]: Failed running /var/lib/cloud/scripts/per-boot/install_mysql [-]
2019-10-02 08:06:52,887 - util.py[DEBUG]: Failed running /var/lib/cloud/scripts/per-boot/install_mysql [-]
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1992, in subp
env=env, shell=shell)
File "/usr/lib/python3.7/subprocess.py", line 775, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.7/subprocess.py", line 1522, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 8] Exec format error: b'/var/lib/cloud/scripts/per-boot/install_mysql'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 835, in runparts
subp(prefix + [exe_path], capture=False)
File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2000, in subp
stderr="-" if decode else b"-")
cloudinit.util.ProcessExecutionError: Exec format error. Missing #! in script?
Command: ['/var/lib/cloud/scripts/per-boot/install_mysql']
Exit code: -
Reason: [Errno 8] Exec format error: b'/var/lib/cloud/scripts/per-boot/install_mysql'
Stdout: -
Stderr: -
2019-10-02 08:06:52,897 - cc_scripts_per_boot.py[WARNING]: Failed to run module scripts-per-boot (per-boot in /var/lib/cloud/scripts/per-boot)
2019-10-02 08:06:52,898 - handlers.py[DEBUG]: finish: modules-final/config-scripts-per-boot: FAIL: running config-scripts-per-boot with frequency always
2019-10-02 08:06:52,898 - util.py[WARNING]: Running module scripts-per-boot (<module 'cloudinit.config.cc_scripts_per_boot' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_scripts_per_boot.py'>) failed
2019-10-02 08:06:52,898 - util.py[DEBUG]: Running module scripts-per-boot (<module 'cloudinit.config.cc_scripts_per_boot' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_scripts_per_boot.py'>) failed
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 800, in _run_modules
freq=freq)
File "/usr/lib/python3/dist-packages/cloudinit/cloud.py", line 54, in run
return self._runners.run(name, functor, args, freq, clear_on_fail)
File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 187, in run
results = functor(*args)
File "/usr/lib/python3/dist-packages/cloudinit/config/cc_scripts_per_boot.py", line 41, in handle
util.runparts(runparts_path)
File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 842, in runparts
% (len(failed), len(attempted)))
RuntimeError: Runparts: 1 failures in 1 attempted commands
So, it definitely doesn't like the format of the file - it wants to see a #!... or perhaps a binary executable.
I will try out Dude0001's suggestions in more detail now.
===EDIT===
In the end, what does work is using the multipart/mixed format, as suggested by Dude0001:
Content-Type: multipart/mixed; boundary="//"
MIME-Version: 1.0
--//
Content-Type: text/cloud-config; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="cloud-config.txt"
#cloud-config
package_update: yes
package_upgrade: all
packages:
- mariadb-server
- apt-file
--//
Content-Type: text/x-shellscript; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="userdata.txt"
#!/bin/bash
cat >>/root/.bashrc <<!
set -o vi
unalias -a
alias ll='ls -lp'
!
cat >>/home/admin/.bashrc <<!
set -o vi
unalias -a
alias ll='ls -lp'
!
cat /root/.vimrc <<!
set t_ti= t_te=
set compatible
set expandtab ts=2 sw=2 ai
!
cat >/home/admin/.vimrc <<!
set t_ti= t_te=
set compatible
set expandtab ts=2 sw=2 ai
!
--//
Just specifying #cloud-config doesn't seem to work, but this way does. For me, at least. In the present moment.
Short answer:
A user_data value set to a shell script will cause the given shell script to be ran during in the final stage of cloud-init (and I believe after the cloud-init directives in the one-time folder you reference).
If you want to use a custom cloud-init directive and a shell script both in EC2 user_data property you need to use the multipart/mixed mime format https://aws.amazon.com/premiumsupport/knowledge-center/execute-user-data-ec2/
Long answer:
The user_data can hold raw data to be read through the EC2 meta data, a script or a cloud-init directive. Additionally, you can set it up as a multipart/mixed mime type and provide each of these.
If user_data is raw data, it can be fetched with a curl command inside the EC2 instance. It is up to the calling command to interpret the data, it can be whatever the user chooses.
[ec2-user ~]$ curl http://169.254.169.254/latest/user-data
If user_data is a script (e.g. #!/bin/bash in the first line), it is ran as a step in cloud-init in the final stage of cloud-init https://cloudinit.readthedocs.io/en/latest/topics/boot.html#final.
If user_data is a cloud-init directive (e.g. #cloud-config in the first line), it is ran as the specified cloud-init directive.
From https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html#user-data-cloud-init
"To pass cloud-init directives to an instance with user_data... enter your cloud-init directive text in the user_data text."
Like so
#cloud-config
repo_update: true
repo_upgrade: all
packages:
- httpd
- mariadb-server
runcmd:
- [ sh, -c, "amazon-linux-extras install -y lamp-mariadb10.2-php7.2 php7.2" ]
- systemctl start httpd
- sudo systemctl enable httpd
- [ sh, -c, "usermod -a -G apache ec2-user" ]
- [ sh, -c, "chown -R ec2-user:apache /var/www" ]
- chmod 2775 /var/www
- [ find, /var/www, -type, d, -exec, chmod, 2775, {}, \; ]
- [ find, /var/www, -type, f, -exec, chmod, 0664, {}, \; ]
- [ sh, -c, 'echo "<?php phpinfo(); ?>" > /var/www/html/phpinfo.php' ]
The multipart/mixed mime format is described here https://aws.amazon.com/premiumsupport/knowledge-center/execute-user-data-ec2/, with the example
Content-Type: multipart/mixed; boundary="//"
MIME-Version: 1.0
--//
Content-Type: text/cloud-config; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="cloud-config.txt"
#cloud-config
cloud_final_modules:
- [scripts-user, always]
--//
Content-Type: text/x-shellscript; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="userdata.txt"
#!/bin/bash
/bin/echo "Hello World" >> /tmp/testfile.txt
--//
Is there some way in sed to create multiple output lines from a single input line? I have a template file (there are more lines in the file, I'm just simplifying it):
http://hostname:#PORT#
I am currently using sed to replace #PORT# with a real port. However, I'd like to be able to pass in multiple ports, and have sed create a line for each. Is that possible?
I'm assuming you would want to duplicate the whole line for each port number. In that case it's easier to think of it as replacing the port numbers with the URL:
$ cat ports.in
1
2
3
4
5
$ sed 's#^\([0-9]*\$)#http://hostname:\1#' ports.in
http://hostname:1
http://hostname:2
http://hostname:3
http://hostname:4
http://hostname:5
To do it the other way around is easier with awk:
$ cat url.in
http://hostname:#PORT#
$ awk '/^[0-9]/ {ports[++i]=$0} /^http/ {sub(":#PORT#", ":%d\n"); for (p in ports) printf($0, ports[p])}' ports.in url.in
http://hostname:2
http://hostname:3
http://hostname:4
http://hostname:5
http://hostname:1
This reads both ports.in and url.in, and if a line starts with a number it is assumed that it's a port number from ports.in. Otherwise, if the line starts with http it's assumed to be an URL from url.in and will replace the port placeholder with a printf formatting string and then print the URL once for each port number read. It will fail to do the right thing if the files are fed in the wrong order.
A similar solution, but taking the URL from a shell variable:
$ myurl="http://hostname:#PORT#"
$ awk -v url="$myurl" 'BEGIN{sub(":#PORT#", ":%d\n",url)} /^[0-9]/ {ports[++i]=$0} END {for (p in ports) printf(url, ports[p])}' ports.in
http://hostname:2
http://hostname:3
http://hostname:4
http://hostname:5
http://hostname:1
It seems you have multiple templates and multiple ports to apply to them. Here's how to do it in a shell script (tested with bash), but you'll need to do it in two sed executions if you want to keep it simple because you have two multiply valued inputs. It is mathematically a cross product of the templates and the substitution values.
ports='80
8080
8081'
templates='http://domain1.net:%PORT/
http://domain2.org:%PORT/
http://domain3.com:%PORT/'
meta="s/(.*)/g; s|%PORT|\1|p; /p"
sed="`echo \"$ports\" |sed -rn \"$meta\" |tr '\n' ' '`"
echo "$templates" |sed -rn "h; $sed"
The shell variable meta is a meta sed script because it writes another sed script. The h saves the pattern buffer in the sed hold space. The sed commands generated from the meta sed recall, substitute, and print for each port. This is the result.
http://domain1.net:80/
http://domain1.net:8080/
http://domain1.net:8081/
http://domain2.org:80/
http://domain2.org:8080/
http://domain2.org:8081/
http://domain3.com:80/
http://domain3.com:8080/
http://domain3.com:8081/
I have a list of domain names in a file. I have to get 'A Record' for each domain name.
Input:
:~$ cat domainlist.txt
google.com
yahoo.com
facebook.com
Required Output:
google.com
216.58.220.46
yahoo.com
98.139.183.24
206.190.36.45
98.138.253.109
facebook.com
66.220.146.36
So, I have this sed command which helps me to get A Records.
sed 's/^/dig +short A/e' domainlist.txt
But it just outputs the IP. The domain names are not printed with it.
Output:
:~$ sed 's/^/dig +short a/e' domainlist.txt
98.124.199.55
188.125.73.108
74.6.50.150
77.238.184.150
98.137.236.150
199.59.243.120
I could get the required output using bash. But I want to know is there any way to get this ouput using sed as it minimizes the code lines.
Here is what the e flag to the s command does (emphasis mine):
This command allows one to pipe input from a shell command into pattern space. If a substitution was made, the command that is found in pattern space is executed and pattern space is replaced with its output. A trailing newline is suppressed; results are undefined if the command to be executed contains a nul character. This is a GNU sed extension.
So you remove what you'd like to print, but the fix is simple:
$ sed 'p;s/^/dig +short A/e' domainlist.txt
google.com
98.124.199.55
yahoo.com
74.6.50.150
98.137.236.150
77.238.184.150
188.125.73.108
facebook.com
199.59.243.120
Just print the pattern space before replacing it.
I don't have dig on my system but this is the job xargs was created to do:
xargs -n 1 -I % sh -c 'echo %; dig +short A %;' < domainlist.txt
with awk
awk '{cmd="dig "$0" +short";print $0; while((cmd|getline var) > 0){ print var};close(cmd) }' domainlist.txt
Okay, I want to have Nmap scan an IP range for computers with a certain port open (port 80 in this case) and have it output all the IP's it finds into a text file, stored in this format:
192.168.0.1
192.168.0.185
192.168.0.192
192.168.0.195
So to output the file, I tried using this command:
nmap -sT -p 80 -ttl 40 192.168.0.0-255 -oG - | grep "80/open" > output.txt
Where "output.txt" is the output file that contains the results. So a line of output.txt looks
like this:
Host: 192.168.0.1 () Ports: 80/open/tcp//http///
So I basically want it only to output the IP address with port 80 open, and nothing else.
I want it to not output the "Host: " or the "()" and "Ports: 80/open/tcp//http///" lines. So is there anyway I can have Nmap not put that stuff into the output file? Or make it only
output the IP addresses? I tried looking at the map page, it was of little help. And I looked all over the Internet and that wasn't very useful either. So does anyone know how I can do this? Thanks
Awk is your friend!
$ nmap -sT -p 80 192.168.0.0/24 -oG - | awk '/ 80\/open/{print $2}' > output.txt
This will find lines with port 80 open (notice the space before 80, if you plan to scan more than the one port!), and print field 2, splitting on whitespace. Another way to do it would be:
$ nmap -sT -p 80 --open 192.168.0.0/24 -oG - | awk '$4=="Ports:"{print $2}' > output.txt
This one uses the --open argument to Nmap to only produce output for hosts with open ports. The awk command checks that this is a "Ports" line, not a "Status" line (which may only show up when using -v, but I'm not positive) before printing the IP address.
Note that it is usually in your best interests to save the scan results to a file, to avoid needing to repeat the scan if you decide to extract some different information. If you choose to do this, I would recommend using the XML output (-oX), since there are lots of analysis tools that have parsers built for it already.
Having nmap produce exactly what you want would indeed be nice. But as a more general solution:
$ nmap ... | grep ... | tr '/' ' ' | awk '{ print $2,$5; }
192.168.0.1 80
Or maybe:
nmap ... | grep ... | tr '/' ' ' | cut -d' ' -f2,8
I found a script called scanreport.sh very useful. Although its not necessary, you could just use awk as suggested, but thought it might be of interest.
It gives the ability to output the nmap results nicely by service or port (with highlighting). It uses the grep-able output from nmap (-oG) after a quick tidy from grep -v ^# nmapoutput.txt > report.txt
Example
nmap -sS 192.168.1.22 -oG /directory/of/choice/results.txt
grep -v ^# results.txt > report.txt
./scanreport.sh -f report.txt
Host: 192.168.1.22 ()
22 open tcp ssh OpenSSH 5.3p1 Debian 3ubuntu4 (protocol 2.0)
80 open tcp http Apache httpd 2.2.14 ((Ubuntu))
./scanreport.sh -f report.txt -p 80
Host: 192.168.1.22 ()
80 open tcp http Apache httpd 2.2.14 ((Ubuntu))
./scanreport.sh -f report.txt -s ssh
Host: 192.168.1.22 ()
22 open tcp ssh OpenSSH 5.3p1 Debian 3ubuntu4 (protocol 2.0)
Plenty of stuff on google about it but here a link to one ref.
./scanreport.sh
I have the following line in my proftpd log (line 78 to be precise)
Deny from 1.2.3.4
I also have a script which rolls through my logs for people using brute force attacks and then stores their IP (ready for a black listing). What i'm struggling with is inserting (presume with sed) at the end of that specific line - this is what I've got so far:
sed "77i3.4.5.6" /opt/etc/proftpd.conf >> /opt/etc/proftpd.conf
Now one would presume this would work perfectly, however it actually does the following (lines 77 through 78):
3.4.5.6
Deny from 1.2.3.4
I suspect this is due to my dated version of sed, are there any other ways of acheiving the same thing? Also the >> causes the config to be duplicated at the end of the fole (again i'm sure this is a limitation of my version of sed). This is running a homebrew linux kernel on my nas. Sed options below:
root#NAS:~# sed BusyBox v1.7.0
(2009-04-29 19:12:57 JST) multi-call
binary
Usage: sed [-efinr] pattern [files...]
Options:
-e script Add the script to the commands to be executed
-f scriptfile Add script-file contents to the
commands to be executed
-i Edit files in-place
-n Suppress automatic printing of pattern space
-r Use extended regular expression syntax
If no -e or -f is given, the first
non-option argument is taken as the
sed script to interpret. All remaining
arguments are names of input files; if
no input files are specified, then the
standard input is read. Source files
will not be modified unless -i option
is given.
Cheers for your help guys.
This has nothing to do with the version of sed; this is just plain old Doing It Wrong.
sed -i '77s/$/,3.4.5.6/' /opt/etc/proftpd.conf