I have a configuration file that looks like the example below. There are a series of definitions grouped by hostname. I just added the "cpu-service" definition to one host "mothership". Now I need to do this to 100+ more in the same file. What I have already done is scraped from config file all pre-existing host names (100+). So now I have a file with the list of servers that now need to have the cpu-service define comment. They already have ping-service so I just want to add the cpu-service to each one. Obviously manually doing this by hand would be tedious.
Is there a sed/awk script I could use to do this type of work. Basically I need to maybe write a skel file with the define part and leave host_name blank. Then feed the host.txt file into that. I could maybe hack this with some VI trickery as well. Not sure?
Thanks in advance!
define{
use cpu-service
host_name mothership
contact_groups systems manager
}
define{
use ping-service
host_name mothership
contact_groups systems manager
}
Although I got the slight feeling to do your work, try the script below:
awk '
BEGIN {
RS = ORS = "}\n"
FS = "\n"
}
NF > 0 {
print
if (sub(/ping-service/, "cpu-service")) print
}
' file
One tradeoff: Somehow I get a trailing "}" but it is not worth worrying about, unless you got to make that every day - just remove it with an editor.
As always with awk: If your vendor ships an historic version of awk you may want to use nawk.
Three steps mister:
1: Host name file (hostnames.txt)
mothership
motherload
motherofpeal
mothersbaugh
2: script (hostup.sh)
#!/bin/bash
HOSTNAME=$1
TEMPLATE="
define{
use cpu-service
host_name ${HOSTNAME}
contact_groups systems manager
}
define{
use ping-service
host_name ${HOSTNAME}
contact_groups systems manager
}"
echo "${TEMPLATE}"
3: command line
chmod +x hostup.sh
while read name; do hostup.sh $name; done < hostnames.txt
while read name; do hostup.sh $name; done < hostnames.txt >> hosts.conf
Sed can insert newlines, just backslash escape them - e.g. the following will go through each line in your 'hosts' file, and replace it with a full definition for the cpu-service. I'm not sure if this is exactly what you want.
sed -e 's/^(.*)$/define{\
use cpu-service\
host_name \1\
contact_groups systems manager\
}/g' hosts.txt > new_directives
if you're happy with new_directives then you can just
cat new_directives >> config_file
NOTE you may get issues with blank/trailing newlines.
Related
I would like to inplace edit ssd_config file where i need to replace the #Port to a custom port.
Before:
#Port <portnum>
ex: #Port 22
After:
Port <customport>
ex: Port 2022
Here the custom port is coming in a variable $port.
I tried the below script but does nothing.
my $prt = "Port 2022";
my $cmd = "sed -i \'s/#Port [0-9]\+/$prt/g\' sshd_config";
system($cmd);
Tried even with tick operator.
`sed -i \"s/#Port [0-9]\+/\$prt/g\" sshd_config`;
I'd suggest to do all that in Perl, once you are running a Perl program. That way one doesn't have to run external programs, not to mention the benefits of not having to mess with all the quoting and escaping if you actually need a shell (not in the shown example though).
Then we need a few more lines of code, to read the file and and edit its content and write it back out. That shouldn't matter, given all the benefits -- but there are libraries that cut out even that as well, for example the very handy Path::Tiny
use Path::Tiny;
...
path($filename)->edit_lines( sub { s/#Port [0-9]+/Port $prt/ } );
I take it that $filename and $prt have been introduced earlier in the program.
There is also a edit method, which slurps the whole file.
Anything sed can do, Perl can do.
If this is your entire Perl program:
my $prt = "Port 2022";
my $cmd = "sed -i \'s/#Port [0-9]\+/$prt/g\' sshd_config";
system($cmd);
Then you can do it all in Perl itself from the command line.
perl -i -p -e's/#Port [0-9]+/Port 2022/g' sshd_config
system("sed ....") is invoking a shell to parse the command line, which means that everything needs to be properly escaped according to the rules of the shell. This together with string escaping inside Perl makes it hard to get right. A much simpler way is to skip the shell and call sed directly and also use string concatenation to add the $prt at the right place (and don't forget to also include the "Port" string itself since you want to have it in the output):
system('sed','-i','s/#Port [0-9]+/Port ' . $prt . '/', 'sshd_config');
Alternatively one could do it in Perl directly, i.e. something like this:
open(my $fh,'<','sshd_config') or die $!;
my #lines = map { s/#Port \d+/Port $prt/ } <$fh>;
open($fh,'>','sshd_config') or die $!;
print $fh #lines;
close($fh);
This is a bit longer but does not rely on starting an external program and is thus faster. And if there is more to do than a simple replacement it is also more flexible.
I have been searching for this for a while, and can't find a satisfactory answer.
I have a perl script that needs to copy a file from one host to another, essentially
sub copy_file{
my($from_server, $from_path, $to_server, $to_path, $filename) = #_;
my $from_location = "$from_server:\"\\\"${from_path}${filename}\\\"\"";
my $to_location = $to_path . $filename;
$to_location =~ s/\s/\\\\ /g;
$to_location = "${to_server}:\"\\\"${to_location}\\\"\"";
return system("scp -p $from_location $to_location >/dev/null 2>&1"");
}
The problem is, some of my filenames look like this:
BLAH;BLAH;BLAH.TXT
Some really nicely named file( With spaces, prentices, &, etc...).xlx
I am already handling whitespaces, and the code for that is quite ugly since on each side, the files could be local or remote, and the escaping is different for the from and to part of the scp call.
what I am really looking for is either to somehow to escape all possible special characters or somehow bypass the shell expansion entirely by using POSIX system calls. I am ok with writing a XS Module if need be.
I have the correct keys set up in the .ssh directory
Also I am not honestly sure which special characters do and don't cause problems. I would like to support all legal filename characters.
Say you want to copy file foo(s) using scp.
As shown below, scp treats the source and target as shell literals, so you pass the following arguments to scp:
scp
-p
--
host1.com:foo\(s\) or host1.com:'foo(s)'
host2.com:foo\(s\) or host2.com:'foo(s)'
You can do that using the multi-argument syntax of system plus an escaping function.
use String::ShellQuote qw( shell_quote );
my $source = $from_server . ":" . shell_quote("$from_path/$filename");
my $target = $to_server . ":" . shell_quote("$to_path/$filename");
system('scp', '-p', '--', $source, $target);
If you really wanted to build a shell command, use shell_quote as usual.
my $cmd = shell_quote('scp', '-p', '--', $source, $target);
$ ssh ikegami#host.com 'mkdir foo ; touch foo/a foo/b foo/"*" ; ls -1 foo'
*
a
b
$ mkdir foo ; ls -1 foo
$ scp 'ikegami#host.com:foo/*' foo
* 100% 0 0.0KB/s 00:00
a 100% 0 0.0KB/s 00:00
b 100% 0 0.0KB/s 00:00
$ ls -1 foo
*
a
b
$ rm foo/* ; ls -1 foo
$ scp 'ikegami#host.com:foo/\*' foo
* 100% 0 0.0KB/s 00:00
$ ls -1 foo
*
There are three ways to handle this:
Use the multi-argument form of system which will completely avoid the shell:
system 'scp', '-p', $from_location, $to_location;
Disadvantage: you can't use shell features like redirection.
Use String::ShellQuote to escape the characters.
$from_location = shell_quote $from_location;
$to_location = shell_quote $to_location;
Disadvantage: certain strings can exist which can't be quoted safely. Furthermore, this solution is not portable as it assumes Bourne shell syntax.
Use IPC::Run which essentially is a supercharged system command that allows redirections.
run ['scp', '-p', $from_location, $to_location],
'>', '/dev/null', # yes, I know /dev/null isn't portable
'2>', '/dev/null'; # we could probably use IO::Null instead
Disadvantage: a complex module like this has certain limitations (e.g. Windows support is experimental), but I doubt you'll run into any issues here.
I strongly suggest you use IPC::Run.
A few suggestions:
It's not part of the standard Perl distribution, but Net::SSH2 and Net::SSH2::SFTP are two highly ranked CPAN modules that will do what you want. ssh, scp, and sftp all use the same protocol and the sshd daemon. If you can use scp, you can use sftp. This gives you a very nice Perlish way to copy files from one system to another.
The quotemeta command will take a string and quote all non-text ASSCII characters for you. It's better than attempting to do the situation yourself with the s/../../ substitution.
You can use qq(..) and q(...) instead of quotation marks. This will allow you to use quotation marks in your string without having to quote them over and over again. This is especially useful in the system command.
For example:
my $error = system qq(scp $user\#host:"$from_location" "$to_location");
One more little trick: If the system command is passed a single parameter, and that parameter has shell metacharacters in it, the system command will be passed to the default system shell. However, if you pass the system command a list of items, those items are passed to execvp directly without being passed to the shell.
Passing a_list_ of arguments to system via an array is a great way to avoid problems with file names. Spaces, shell metacharacters, and other shell issues are avoided.
my #command;
push #command, 'scp';
push #command, "$from_user\#$from_host:$from_location",
push #command, "$to_user\#$to_host:$to_location"
my $error = system #command;
use Net::OpenSSH:
my $ssh = Net::OpenSSH->new($host);
$ssh->scp_get($remote_path, $local_path);
The way arguments have to be quoted varies depending on the shell running on the remote side. The stable version of the module has support for Bourne compatible shells. The development version available from GitHub has also support for csh and several Windows flavors (Windows quoteing is, err, interesting).
For instance:
my $ssh = Net::OpenSSH->new($host, remote_shell => 'MSWin', ...);
Note that on Windows, there are string that just can not be properly quoted!
I need to find the age of a file in seconds but when I give my Perl line:
perl -e ' my #st=stat("$name"); print time - $st[9];'
a variable from Bash with "." in the filename, it won't find the file and prints the systime instead. Otherwise, if I create a file with a name such "something", it works perfectly well.
I tried escaping the character with \ but that does not work. I really do not know Perl and I don't know how to figure this out.
Thanks for the help!
(Please, do not suggest any BASH only workarounds, lot of stuff here is old, truncated...)
EDIT: I found a workaround but I can't post an answer to my own question since I am a newbie, so here it is:
So I finally found a workaround.
You export the variable in Bash:
export name=".file.txt"
and then call it from the Perl like this: $ENV{name}
and it works just fine.
EDIT 2:
The export idea was just a temporary solution. Better one is using single quotes as perreal suggested.
You need to put the shell variable out of the single quotes:
perl -e ' my #st=stat("'"$name"'"); print time - $st[9];'
or pass it through arguments:
perl -se 'my #st=stat("$name"); print time - $st[9];' -- -name="$name"
I have a file of environment variables that I source in shell scripts, for example:
# This is a comment
ONE=1
TWO=2
THREE=THREE
# End
In my scripts, I source this file (assume it's called './vars') into the current environment, and change (some of) the variables based on user input. For example:
#!/bin/sh
# Read variables
source ./vars
# Change a variable
THREE=3
# Write variables back to the file??
awk 'BEGIN{FS="="}{print $1=$$1}' <./vars >./vars
As you can see, I've been experimenting with awk for writing the variables back, sed too. Without success. The last line of the script fails. Is there a way to do this with awk or sed (preferably preserving comments, even comments with the '=' character)? Or should I combine 'read' with string cutting in a while loop or some other magic? If possible, I'd like to avoid perl/python and just use the tools available in Busybox. Many thanks.
Edit: perhaps a use case might make clear what my problem is. I keep a configuration file consisting of shell environment variable declarations:
# File: network.config
NETWORK_TYPE=wired
NETWORK_ADDRESS_RESOLUTION=dhcp
NETWORK_ADDRESS=
NETWORK_ADDRESS_MASK=
I also have a script called 'setup-network.sh':
#!/bin/sh
# File: setup-network.sh
# Read configuration
source network.config
# Setup network
NETWORK_DEVICE=none
if [ "$NETWORK_TYPE" == "wired" ]; then
NETWORK_DEVICE=eth0
fi
if [ "$NETWORK_TYPE" == "wireless" ]; then
NETWORK_DEVICE=wlan0
fi
ifconfig -i $NETWORK_DEVICE ...etc
I also have a script called 'configure-network.sh':
#!/bin/sh
# File: configure-network.sh
# Read configuration
source network.config
echo "Enter the network connection type:"
echo " 1. Wired network"
echo " 2. Wireless network"
read -p "Type:" -n1 TYPE
if [ "$TYPE" == "1" ]; then
# Update environment variable
NETWORK_TYPE=wired
elif [ "$TYPE" == "2" ]; then
# Update environment variable
NETWORK_TYPE=wireless
fi
# Rewrite configuration file, substituting the updated value
# of NETWORK_TYPE (and any other updated variables already existing
# in the network.config file), so that later invocations of
# 'setup-network.sh' read the updated configuration.
# TODO
How do I rewrite the configuration file, updating only the variables already existing in the configuration file, preferably leaving comments and empty lines intact? Hope this clears things up a little. Thanks again.
You can't use awk and read and write from the same file (is part of your problem).
I prefer to rename the file before I rewrite (but you can save to a tmp and then rename too).
/bin/mv file file.tmp
awk '.... code ...' file.tmp > file
If your env file gets bigger, you'll see that is is getting truncated at the buffer size of your OS.
Also, don't forget that gawk (the std on most Linux installations) has a built in array ENVIRON. You can create what you want from that
awk 'END {
for (key in ENVIRON) {
print key "=" ENVIRON[key]
}
}' /dev/null
Of course you get everything in your environment, so maybe more than you want. But probably a better place to start with what you are trying to accomplish.
Edit
Most specifically
awk -F"=" '{
if ($1 in ENVIRON) {
printf("%s=%s\n", $1, ENVIRON[$1])
}
# else line not printed or add code to meet your situation
}' file > file.tmp
/bin/mv file.tmp file
Edit 2
I think your var=values might need to be export -ed so they are visible to the awk ENVIRON array.
AND
echo PATH=xxx| awk -F= '{print ENVIRON[$1]}'
prints the existing value of PATH.
I hope this helps.
P.S. as you appear to be a new user, if you get an answer that helps you please remember to mark it as accepted, and/or give it a + (or -) as a useful answer.
I don't exactly know what you are trying to do, but if you are trying to change the value of variable THREE ,
awk -F"=" -vt="$THREE" '$1=="THREE" {$2=t}{print $0>FILENAME}' OFS="=" vars
You can do this in just with bash:
rewrite_config() {
local filename="$1"
local tmp=$(mktemp)
# if you want the header
echo "# File: $filename" >> "$tmp"
while IFS='=' read var value; do
declare -p $var | cut -d ' ' -f 3-
done < "$filename" >> "$tmp"
mv "$tmp" "$filename"
}
Use it like
source network.config
# manipulate the variables
rewrite_config network.config
I use a temp file to maintain the existance of the config file for as long as possible.
I'm trying to create a process that renames all my filenames to Camel/Capital Case. The closest I have to getting there is this:
perl -i.bak -ple 's/\b([a-z])/\u$1/g;' *.txt # or similar .extension.
Which seems to create a backup file (which I'll remove when it's verified this does what I want); but instead of renaming the file, it renames the text inside of the file. Is there an easier way to do this? The theory is that I have several office documents in various formats, as I'm a bit anal-retentive, and would like them to look like this:
New Document.odt
Roffle.ogg
Etc.Etc
Bob Cat.flac
Cat Dog.avi
Is this possible with perl, or do I need to change to another language/combination of them?
Also, is there anyway to make this recursive, such that /foo/foo/documents has all files renamed, as does /foo/foo/documents/foo?
You need to use rename .
Here is it's signature:
rename OLDNAME,NEWNAME
To make it recursive, use it along with File::Find
use strict;
use warnings;
use File::Basename;
use File::Find;
#default searches just in current directory
my #directories = (".");
find(\&wanted, #directories);
sub wanted {
#renaming goes here
}
The following snippet, will perform the code inside wanted against all the files that are found. You have to complete some of the code inside the wanted to do what you want to do.
EDIT: I tried to accomplish this task using File::Find, and I don't think you can easily achieve it. You can succeed by following these steps :
if the parameter is a dir, capitalize it and obtain all the files
for each file, if it's a dir, go back at the beginning with this file as argument
if the file is a regular file, capitalize it
Perl just got in my way while writing this script. I wrote this script in ruby :
require "rubygems"
require "ruby-debug"
# camelcase files
class File
class << self
alias :old_rename :rename
end
def self.rename(arg1,arg2)
puts "called with #{arg1} and #{arg2}"
self.old_rename(arg1,arg2)
end
end
def capitalize_dir_and_get_files(dir)
if File.directory?(dir)
path_c = dir.split(/\//)
#base = path_c[0,path_c.size-1].join("/")
path_c[-1].capitalize!
new_dir_name = path_c.join("/")
File.rename(dir,new_dir_name)
files = Dir.entries(new_dir_name) - [".",".."]
files.map! {|file| File.join(new_dir_name,file)}
return files
end
return []
end
def camelize(dir)
files = capitalize_dir_and_get_files(dir)
files.each do |file|
if File.directory?(file)
camelize(file.clone)
else
dir_name = File.dirname(file)
file_name = File.basename(file)
extname = File.extname(file)
file_components = file_name.split(/\s+/)
file_components.map! {|file_component| file_component.capitalize}
new_file_name = File.join(dir_name,file_components.join(" "))
#if extname != ""
# new_file_name += extname
#end
File.rename(file,new_file_name)
end
end
end
camelize(ARGV[0])
I tried the script on my PC and it capitalizes all dirs,subdirs and files by the rule you mentioned. I think this is the behaviour you want. Sorry for not providing a perl version.
Most systems have the rename command ....
NAME
rename - renames multiple files
SYNOPSIS
rename [ -v ] [ -n ] [ -f ] perlexpr [ files ]
DESCRIPTION
"rename" renames the filenames supplied according to the rule specified as the first argument. The perlexpr argument is a Perl expression which
is expected to modify the $_ string in Perl for at least some of the filenames specified. If a given filename is not modified by the expression,
it will not be renamed. If no filenames are given on the command line, filenames will be read via standard input.
For example, to rename all files matching "*.bak" to strip the extension, you might say
rename 's/\.bak$//' *.bak
To translate uppercase names to lower, you’d use
rename 'y/A-Z/a-z/' *
OPTIONS
-v, --verbose
Verbose: print names of files successfully renamed.
-n, --no-act
No Action: show what files would have been renamed.
-f, --force
Force: overwrite existing files.
AUTHOR
Larry Wall
DIAGNOSTICS
If you give an invalid Perl expression you’ll get a syntax error.
Since Perl runs just fine on multiple platforms, let me warn you that FAT (and FAT32, etc) filesystems will ignore renames that only change the case of the file name. This is true under Windows and Linux and is probably true for other platforms that support the FAT filesystem.
Thus, in addition to Geo's answer, note that you may have to actually change the file name (by adding a character to the end, for example) and then change it back to the name you want with the correct case.
If you will only rename files on NTFS filesystems or only on ext2/3/4 filesystems (or other UNIX/Linux filesystems) then you probably don't need to worry about this. I don't know how the Mac OSX filesystem works, but since it is based on BSDs, I assume it will allow you to rename files by only changing the case of the name.
I'd just use the find command to recur the subdirectories and mv to do the renaming, but still leverage Perl to get the renaming right.
find /foo/foo/documents -type f \
-execdir bash -c 'mv "$0" \
"$(echo "$0" \
| perl -pe "s/\b([[:lower:]])/\u\$1/g; \
s/\.(\w+)$/.\l\$1/;")"' \
{} \;
Cryptic, but it works.
Another one:
find . -type f -exec perl -e'
map {
( $p, $n, $s ) = m|(.*/)([^/]*)(\.[^.]*)$|;
$n =~ s/(\w+)/ucfirst($1)/ge;
rename $_, $p . $n . $s;
} #ARGV
' {} +
Keep in mind that on case-remembering filesystems (FAT/NTFS), you'll need to rename the file to something else first, then to the case change. A direct rename from "etc.etc" to "Etc.Etc" will fail or be ignored, so you'll need to do two renames: "etc.etc" to "etc.etc~" then "etc.etc~" to "Etc.Etc", for example.