Map sd?/sdd? names to Solaris disk names? - solaris

Some commands in Solaris (such as iostat) report disk related information using disk names such as sd0 or sdd2. Is there a consistent way to map these names back to the standard /dev/dsk/c?t?d?s? disk names in Solaris?
Edit: As Amit points out, iostat -n produces device names such as eg c0t0d0s0 instead of sd0. But how do I found out that sd0 actually is c0t0d0s0? I'm looking for something that produces a list like this:
sd0=/dev/dsk/c0t0d0s0
...
sdd2=/dev/dsk/c1t0d0s4
...
Maybe I could run iostat twice (with and without -n) and then join up the results and hope that the number of lines and device sorting produced by iostat is identical between the two runs?

Following Amit's idea to answer my own question, this is what I have come up with:
iostat -x|tail -n +3|awk '{print $1}'>/tmp/f0.txt.$$
iostat -nx|tail -n +3|awk '{print "/dev/dsk/"$11}'>/tmp/f1.txt.$$
paste -d= /tmp/f[01].txt.$$
rm /tmp/f[01].txt.$$
Running this on a Solaris 10 server gives the following output:
sd0=/dev/dsk/c0t0d0
sd1=/dev/dsk/c0t1d0
sd4=/dev/dsk/c0t4d0
sd6=/dev/dsk/c0t6d0
sd15=/dev/dsk/c1t0d0
sd16=/dev/dsk/c1t1d0
sd21=/dev/dsk/c1t6d0
ssd0=/dev/dsk/c2t1d0
ssd1=/dev/dsk/c3t5d0
ssd3=/dev/dsk/c3t6d0
ssd4=/dev/dsk/c3t22d0
ssd5=/dev/dsk/c3t20d0
ssd7=/dev/dsk/c3t21d0
ssd8=/dev/dsk/c3t2d0
ssd18=/dev/dsk/c3t3d0
ssd19=/dev/dsk/c3t4d0
ssd28=/dev/dsk/c3t0d0
ssd29=/dev/dsk/c3t18d0
ssd30=/dev/dsk/c3t17d0
ssd32=/dev/dsk/c3t16d0
ssd33=/dev/dsk/c3t19d0
ssd34=/dev/dsk/c3t1d0
The solution is not very elegant (it's not a one-liner), but it seems to work.

One liner version of the accepted answer (I only have 1 reputation so I can't post a comment):
paste -d= <(iostat -x | awk '{print $1}') <(iostat -xn | awk '{print $NF}') | tail -n +3

Try using the '-n' switch. For eg. 'iostat -n'

As pointed out in other answers, you can map the device name back to the instance name via the device path and information contained in /etc/path_to_inst. Here is a Perl script that will accomplish the task:
#!/usr/bin/env perl
use strict;
my #path_to_inst = qx#cat /etc/path_to_inst#;
map {s/"//g} #path_to_inst;
my ($device, $path, #instances);
for my $line (qx#ls -l /dev/dsk/*s2#) {
($device, $path) = (split(/\s+/, $line))[-3, -1];
$path =~ s#.*/devices(.*):c#$1#;
#instances =
map {join("", (split /\s+/)[-1, -2])}
grep {/$path/} #path_to_inst;
*emphasized text*
for my $instance (#instances) {
print "$device $instance\n";
}
}

I found the following in the Solaris Transistion Guide:
"Instance Names
Instance names refer to the nth device in the system (for example, sd20).
Instance names are occasionally reported in driver error messages. You can determine the binding of an instance name to a physical name by looking at dmesg(1M) output, as in the following example.
sd9 at esp2: target 1 lun 1
sd9 is /sbus#1,f8000000/esp#0,800000/sd#1,0
<SUN0424 cyl 1151 alt 2 hd 9 sec 80>
Once the instance name has been assigned to a device, it remains bound to that device.
Instance numbers are encoded in a device's minor number. To keep instance numbers consistent across reboots, the system records them in the /etc/path_to_inst file. This file is read only at boot time, and is currently updated by the add_drv(1M) and drvconf"
So based upon that, I wrote the following script:
for device in /dev/dsk/*s2
do
dpath="$(ls -l $device | nawk '{print $11}')"
dpath="${dpath#*devices/}"
dpath="${dpath%:*}"
iname="$(nawk -v dpath=$dpath '{
if ($0 ~ dpath) {
gsub("\"", "", $3)
print $3 $2
}
}' /etc/path_to_inst)"
echo "$(basename ${device}) = ${iname}"
done
By reading the information directly out of the path_to_inst file, we are allowing for adding and deleting devices, which will skew the instance numbers if you simply count the instances in the /devices directory tree.

I think simplest way to find descriptive name having instance name is:
# iostat -xn sd0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
4.9 0.2 312.1 1.9 0.0 0.0 3.3 3.5 0 1 c1t1d0
#
The last column shows descriptive name for provided instance name.

sd0 sdd0 are instance names of devices.. you can check /etc/path_to_inst to get instance name mapping to physical device name, then check link in /dev/dsk (to which physical device it is pointing) it is 100% sure method, though i dont know how to code it ;)

I found this snippet on the internet some time ago, and it does the trick. This was on Solaris 8:
#!/bin/sh
cd /dev/rdsk
/usr/bin/ls -l *s0 | tee /tmp/d1c |awk '{print "/usr/bin/ls -l "$11}' | \
sh | awk '{print "sd" substr($0,38,4)/8}' >/tmp/d1d
awk '{print substr($9,1,6)}' /tmp/d1c |paste - /tmp/d1d
rm /tmp/d1[cd]

A slight variation to allow for disk names that are longer than 8 characters (encountered when dealing with disk arrays on a SAN)
#!/bin/sh
cd /dev/rdsk
/usr/bin/ls -l *s0 | tee /tmp/d1c | awk '{print "/usr/bin/ls -l "$11}' | \
sh | awk '{print "sd" substr($0,38,4)/8}' >/tmp/d1d
awk '{print substr($9,1,index($9,"s0)-1)}' /tmp/d1c | paste - /tmp/d1d
rm /tmp/d1[cd]

Related

Renaming files/dir's from one date format to another

So I am a coding newbie and have, for some time, wanted to edit the formatting of my fairly extensive live music library. I have looked around on here and various other resources to get to where I am, but I have hit a snag. I have directories named in the following ways:
02.10.90 | 23 East Caberet - Ardmore, PA
02.16.90 | The Paradise - Boston, MA
and I would like to rename these simply to
1990-02-10 | 23 East Caberet - Ardmore, PA
1990-02-16 | The Paradise - Boston, MA
I have been able to rename the date correctly using:
ls -1 | grep 90 | awk '{print $1}' | awk -F. '{printf "%s-%s-%s\n", "19"$3,$1,$2}' > list1.txt
and then pull the rest of the name using
ls -1 | grep 90 | awk '{first = $1; $1 = ""; print $0}'>list2.txt
So, I have a list of directories ranging from years 1990-2004 that I would like to apply this to (they are all in different sub directories so I don't mind manually changing the "grep 90". However, from the two separate lists that I generate, I cant figure out how to make it loop through each row and print "mv original_name list1.txt+list2.txt" so that it would read:
mv 02.10.90 | 23 East Caberet - Ardmore, PA 1990-02-10 | 23 East Caberet - Ardmore, PA
I scanned through many previous posts and couldn't quite figure out the last bit - or better yet, a more elegant solution! Any help is greatly appreciated, thank you in advance!
Don't parse the output from ls, google why, and you don't need grep when you're using awk, nor do you need chains of awk commands but for this task you wouldn't use any of those commands anyway.
The UNIX command to find files is named find so start with that. This will find all directories with names starting with the given globbing pattern:
find . -type d -name '[0-9][0-9].[0-9][0-9].90 *'
Now that you've found the files you need to do something with them. For your needs IMHO the simplest approach is best and that'd be:
find . -type d -name '[0-9][0-9].[0-9][0-9].90 *' -print0 |
while IFS= read -r -d '' old; do
path="$(dirname "$old")"
oldDirName="$(basename "$old")"
if [[ oldDirName =~ ([0-9]+)\.([0-9]+)\.([0-9]+)( .*) ]]; then
newDirName="19${BASH_REMATCH[3]}-${BASH_REMATCH[1]}-${BASH_REMATCH[2]}${BASH_REMATCH[4]}
echo mv -- "${path}/${oldDirName}" "${path}/${newDirName}"
fi
done
The above is using GNU find for -print0 and bash for BASH_REMATCH. Remove the echo when you've debugged it if necessary and are happy with what it's going to do.

Dynamically building a exlude list for both rsync & egrep format

I wonder if anyone out there can assist me in trying to solve a issue with me.
I have written a set of shell scripts with the purpose of auditing remote file systems based on a GOLD build on a audit server.
As part of this, I do the following:
1) Use rsync to work out any new files or directories, any modified or removed files
2) Use find ${source_filesystem} -ls on both local & remote to work out permissions differences
Now as part of this there are certain files or directories that I am excluding, i.e. logs, trace files etc.
So in order to achieve this I use 2 methods:
1) RSYNC - I have an exclude-list that is added using --exclude-from flag
2) find -ls - I use a egrep -v statement to exclude the same as the rsync exclude-list:
e.g. find -L ${source_filesystem} -ls | egrep -v "$SEXCLUDE_supt"
So my issue is that I have to maintain 2 separate lists and this is a bit of a admin nightmare.
I am looking for some assistance or some advice on if it is possible to dynamically build a list of exlusions that can be used for both the rsync or the find -ls?
Here is the format of what the exclude lists look like::
RSYNC:
*.log
*.out
*.csv
logs
shared
tracing
jdk*
8.6_Code
rpsupport
dbarchive
inarchive
comms
PR116PICL
**/lost+found*/
dlxwhsr*
regression
tmp
working
investigation
Investigation
dcsserver_weblogic_*.ear
dcswebrdtEAR_weblogic_*.ear
FIND:
SEXCLUDE_supt="\.log|\.out|\.csv|logs|shared|PR116PICL|tracing|lost\+found|jdk|8\.6\_Code|rpsupport|dbarchive|inarchive|comms|dlxwhsr|regression|tmp|working|investigation|Investigation|dcsserver_weblogic_|dcswebrdtEAR_weblogic_"
You don't need to create a second list for your find command. grep can handle a list of patterns using the -f flag. From the manual:
-f FILE, --file=FILE
Obtain patterns from FILE, one per line. The empty file contains zero
patterns, and therefore matches nothing. (-f is specified by POSIX.)
Here's what I'd do:
find -L ${source_filesystem} -ls | grep -Evf your_rsync_exclude_file_here
This should also work for filenames containing newlines and spaces. Please let me know how it goes.
In the end the grep -Evf was a bit of a nightmare as rsync didnt support regex, it uses regex but not the same.
So I then pursued my other idea of dynamically building the exclude list for egrep by parsing the rsync exclude-list and building variable on the fly to pass into egrep.
This the method I used:
#!/bin/ksh
# Create Signature of current build
AFS=$1
#Create Signature File
crSig()
{
find -L ${SRC} -ls | egrep -v **"$SEXCLUDE"** | awk '{fws = ""; for (i = 11; i <= NF; i++) fws = fws $i " "; print $3, $6, fws}' | sort >${BASE}/${SIFI}.${AFS}
}
#Setup SRC, TRG & SCROOT
LoadAuditReqs()
{
export SRC=`grep ${AFS} ${CONF}/fileSystem.properties | awk {'print $2'}`
export TRG=`grep ${AFS} ${CONF}/fileSystem.properties | awk {'print $3'}`
export SCROOT=`grep ${AFS} ${CONF}/fileSystem.properties | awk {'print $4'}`
**export BEXCLUDE=$(sed -e 's/[*/]//g' -e 's/\([._+-]\)/\\\1/g' ${CONF}/exclude-list.${AFS} | tr "\n" "|")**
**export SEXCLUDE=$(echo ${BEXCLUDE} | sed 's/\(.*\)|/\1/')**
}
#Load Properties File
LoadProperties()
{
. /users/rpapp/rpmonit/audit_tool/conf/environment.properties
}
#Functions
LoadProperties
LoadAuditReqs
crSig
So with these new variables:
**export BEXCLUDE=$(sed -e 's/[*/]//g' -e 's/\([._+-]\)/\\\1/g' ${CONF}/exclude-list.${AFS} | tr "\n" "|")**
**export SEXCLUDE=$(echo ${BEXCLUDE} | sed 's/\(.*\)|/\1/')**
I use them to remove "*" and "/", then match my special characters and prepend with "\" to escape them.
Then it using "tr" replace a newline with "|" and then rerunning that output to remove the trailing "|" to make the variable $SEXCLUDE to use for egrep that is used in the crSig function.
What do you think?

right tool to filter the UUID from the output of blkid program (using grep, cut, or awk, e.t.c)

I want to filter the output of the blkid to get the UUID.
The output of blkid looks like
CASE 1:-
$ blkid
/dev/sda2: LABEL="A" UUID="4CC9-0015"
/dev/sda3: LABEL="B" UUID="70CF-169F"
/dev/sda1: LABEL=" NTFS_partition" UUID="3830C24D30C21234"
In somecases the output of blkid looks like
CASE 2:-
$ blkid
/dev/sda1: UUID="d7ec380e-2521-4fe5-bd8e-b7c02ce41601" TYPE="ext4"
/dev/sda2: UUID="fc54f19a-8ec7-418b-8eca-fbc1af34e57f" TYPE="ext4"
/dev/sda3: UUID="6f218da5-3ba3-4647-a44d-a7be19a64e7a" TYPE="swap"
I want to filter out the UUID.
Using the combination of grep and cut it can be done as
/sbin/blkid | /bin/grep 'sda1' | /bin/grep -o -E 'UUID="[a-zA-Z|0-9|\-]*' | /bin/cut -c 7-
I have tried using awk , grep and cut as below for filtering the UUID
$ /sbin/blkid | /bin/grep 'sda1' | /usr/bin/awk '{print $2}' | /bin/sed 's/\"//g' | cut -c 7-
7ec380e-2521-4fe5-bd8e-b7c02ce41601
The above command(which uses awk) is not reliable since sometimes an extra field such as LABEL may be present in the output of the blkid program as shown in the above output.
What is the best way to create a command using awk which works reliably?
Please post if any other elegant method exits for the job using bin and core utils. I dont want to use perl or python since this has to be run on busybox.
NOTE:-I am using busybox blkid to which /dev/sda1 can not be passed as the args(the version i am using does not support it) hence the grep to filter the line.
UPDATE :- added the CASE 2: -output to show that field position can not be relied upon.
Why are you making it so complex?
Try this:
# blkid -s UUID -o value
d7ec380e-2521-4fe5-bd8e-b7c02ce41601
fc54f19a-8ec7-418b-8eca-fbc1af34e57f
6f218da5-3ba3-4647-a44d-a7be19a64e7a
Or this:
# blkid -s UUID -o value /dev/sda1
d7ec380e-2521-4fe5-bd8e-b7c02ce41601
Install proper blkid package if you don't have it:
sudo apt-get install util-linux
sudo yum install util-linux
For all the UUID's, you can do :
$ blkid | sed -n 's/.*UUID=\"\([^\"]*\)\".*/\1/p'
d7ec380e-2521-4fe5-bd8e-b7c02ce41601
fc54f19a-8ec7-418b-8eca-fbc1af34e57f
6f218da5-3ba3-4647-a44d-a7be19a64e7a
Say, only for a specific sda1:
$ blkid | sed -n '/sda1/s/.*UUID=\"\([^\"]*\)\".*/\1/p'
d7ec380e-2521-4fe5-bd8e-b7c02ce41601
The sed command tries to group the contents present within the double quotes after the UUID keyword, and replaces the entire line with the token.
Here's a short awk solution:
blkid | awk 'BEGIN{FS="[=\"]"} {print $(NF-1)}'
Output:
4CC9-0015
70CF-169F
3830C24D30C21234
Explanation:
BEGIN{FS="[=\"]"} : Use = and " as delimiters
{print $(NF-1)}: NF stands of Number of Fields; here we print the 2nd to last field
This is based on the consistent structure of blkid output: UUID in quotes is at the end of each line.
Alternatively:
blkid | awk 'BEGIN{FS="="} {print $NF}' | sed 's/"//g'
data.txt
/dev/sda2: LABEL="A" UUID="4CC9-0015"
/dev/sda3: LABEL="B" UUID="70CF-169F"
/dev/sda1: LABEL=" NTFS_partition" UUID="3830C24D30C21234"
awk and sed combination
cat data.txt | awk 'BEGIN{FS="UUID";RS="\n"} {print $2}' | sed -e 's/=//' -e 's/"//g'
Explanation:
Set the Field Separator to the string 'UUID', $2 will give the rest output
use sed then to remove the = and " as shown where -e is a switch so that you can give multiple sed commands/expression in one.
All occurrences of " are removed using the ending g option i.e. global.
The question has a "e.t.c" so I'm going to assume python is one of the options ;)
#!/usr/bin/env python3
import subprocess, re, json
# get blkid output
blkid = subprocess.check_output(["blkid"]).decode('utf-8')
devices = []
for line in [x for x in blkid.split('\n') if x]:
parameters = line.split()
for idx, parameter in enumerate(parameters):
if idx is 0:
devices.append({"DEVICE": re.sub(r':$','',parameter)})
continue
key_and_value = parameter.split('=')
devices[-1].update({
key_and_value[0]: re.sub(r'"','',key_and_value[1])
})
uuids = [{dev['DEVICE']: dev['UUID']} for dev in devices if 'UUID' in dev.keys()]
print(json.dumps(uuids, indent=4, sort_keys=True))
Although, this is probably overkill and quite a few error handling/optimization is missing from this script XD
I assume you're using busybox in an initramfs and you are waiting for your e.g. USB drive with the rootfs on it to become available.
You could use the following awk script (busybox awk compliant).
# cat get-ruuid.awk
BEGIN {
ruuid=ENVIRON["RUUID"]
}
/^\/dev\/sd[a-z]/ {
if (index($0, tolower(ruuid)) || index($0, toupper(ruuid))) {
split($1, parts, ":")
printf("%s\n", parts[1])
exit(0) # Return success and stop further scanning.
}
}
END {
exit(1) # If we reach the end, it means RUUID was not found.
}
Call it as follows from e.g. the init script; this is not the most ideal way.
# The UUID of your root partition
export RUUID="<put proper uuid value here>"
for x in 1, 2, 3, 4, 5 ; do
mdev -s
found=$(blkid | awk -f ./get-ruuid.awk)
test -z $found || break; # If no longer zero length, break the loop.
sleep 1
done
But if this is the only reason why you would want to have an initramfs, I would use the 'root=PARTUUID=... waitroot' Linux kernel command line option. Check the kernel docs and sources.
Get the proper PARTUUID (NOT UUID) of your root partition with the blkid command.

Getting the number of unique values in a specific line location in a large text file

All,
I have a large server log with contains the entry:
not found from Module
Immediately after it is a quotation mark followed location of a module. The file is over 4,800 lines long, and each module may appear over 100 times. So, what I'd like to do is something like a wc -l, but then to only show me the number of unique values in the module name portion of the server log. Any ideas?
grep -oP '(?<=not found from Module ")[^"]+(?=")' logfile | sort -u | wc -l
grep 'entry: not found from Module' logfile | sort -u | wc -l
perl -nlwe '$a{$_}++ }{ print "$_ : $a{$_}" for keys %a' log.txt
Create a hash key for each line, then print the keys at the end.
awk '{count[$NF]++} END{ for (mod in count) print mod, count[mod] }' file
is probably close to what you want but it's hard to guess without some sample input and expected output.

How can I tell if a file is on a remote filesystem with Perl?

Is there a quick-and-dirty way to tell programmatically, in shell script or in Perl, whether a path is located on a remote filesystem (nfs or the like) or a local one? Or is the only way to do this to parse /etc/fstab and check the filesystem type?
stat -f -c %T <filename> should do what you want. You might also want -l
You can use "df -T" to get the filesystem type for the directory, or use the -t option to limit reporting to specific types (like nfs) and if it comes back with "no file systems processed", then it's not one of the ones you're looking for.
df -T $dir | tail -1 | awk '{print $2;}'
If you use df on a directory to get info only of the device it resides in, e.g. for the current directory:
df .
Then, you can just parse the output, e.g.
df . | tail -1 | awk '{print $1}'
to get the device name.
I have tested the following on solaris7,8,9 & 10 and it seems to be reliable
/bin/df -g <filename> | tail -2 | head -1 | awk '{print $1}'
Should give you have the fs type rather than trying to match for a "host:path" in your mount point.
On some systems, the device number is negative for NFS files. Thus,
print "remote" if (stat($filename))[0] < 0