Munin plugin in perl doesn't fetch correct data - perl

I'm trying to use a Munin plugin for software raid. Here's the plugin's code: https://github.com/munin-monitoring/contrib/blob/master/plugins/disk/raid
Currently my raid is rebuilding, here's the current output:
# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md2 : active raid1 sda3[0] sdb3[1]
2925544767 blocks super 1.2 [2/2] [UU]
[==>..................] resync = 14.4% (422554560/2925544767) finish=5246.6min speed=7950K/sec
md1 : active raid1 sda2[0] sdb2[1]
524276 blocks super 1.2 [2/2] [UU]
resync=DELAYED
md0 : active raid1 sda1[0] sdb1[1]
4193268 blocks super 1.2 [2/2] [UU]
resync=DELAYED
unused devices: <none>
But when I run the plugin I get the following output (stating that all disks are synced):
# munin-run raid
md2.value 100
md2_rebuild.value 100
md1.value 100
md1_rebuild.value 100
md0.value 100
md0_rebuild.value 100
In the following lines I understand (I'm no programmer) that during the time the code runs, $pct is >= 100, and so $rpct gets set to 100 (which is my output for all raid arrays).
So which values do $nact and $nmem represent in my cat /proc/mdstat output? This would help me find out why $pct is >= 100.
my $pct = 100 * $nact / $nmem;
my $rpct = 100;
if ( $pct < 100 ) {
my #output = `/sbin/mdadm -D /dev/$dev | grep Rebuild`;
if( $output[0] =~ /([0-9]+)% complete/ ) {
$rpct = $1;
} else {
$rpct = 0;
}
I think this regexp holds the answer, but as I said, I'm no programmer :P
while ($text =~ /(md\d+)\s+:\s+active\s+(\(auto-read-only\)\s+|)(\w+)\s+(.*)\n.*\[(\d+)\/(\d+)]\s+\[(\w+)]/ ) {
my($dev,$dummy,$type,$members,$nmem,$nact,$status) = ($1,$2,$3,$4,$5,$6,$7);
Thanks in advance :-)

change this:
if ( $pct < 100 ) {
to this:
if ( $pct <= 100 ) {
and make sure you're running the plugin as root

Related

How to count the numbers of elements in parts of a text file using a loop in Perl?

I´m looking for a way to create a script in Perl to count the elements in my text file and do it in parts. For example, my text file has this form:
ID Position Potential Jury agreement NGlyc result
(PART 1)
NP_073551.1_HCoV229Egp2 23 NTSY 0.5990 (8/9) +
NP_073551.1_HCoV229Egp2 62 NTSS 0.7076 (9/9) ++
NP_073551.1_HCoV229Egp2 171 NTTI 0.5743 (5/9) +
...
(PART 2)
QJY77946.1_NA 20 NGTN 0.7514 (9/9) +++
QJY77946.1_NA 23 NTSH 0.5368 (5/9) +
QJY77946.1_NA 51 NFSF 0.7120 (9/9) ++
QJY77946.1_NA 62 NTSS 0.6947 (9/9) ++
...
(PART 3)
QJY77954.1_NA 20 NGTN 0.7694 (9/9) +++
QJY77954.1_NA 23 NTSH 0.5398 (5/9) +
QJY77954.1_NA 51 NFSF 0.7121 (9/9) ++
...
(PART N°...)
Like you can see the ID is the same in each part (one for PART 1, other to PART 2 and then...). The changes only can see in the columns Position//Potential//Jury agreement//NGlyc result Then, my main goal is to count the line with Potential 0,7 >=.
With this in mind, I´m looking for output like this:
Part 1:
1 (one value 0.7 >=)
Part 2:
2 (two values 0.7 >=)
Part 3:
2 (two values 0.7 >=)
Part N°:
X numbers of values 0.7 >=
This output tells me the number of positive values (0.7 >=) for each ID.
The pseudocode I believe would be something like this:
foreach ID in LIST
foreach LINE in FILE
if (ID is in LINE)
... count the line ...
end foreach LINE
end foreach ID
I´m looking for any suggestion (for a package or script idea) or comment to create a better script.
Thanks! Best!
To count the number of lines, for each part, that match some condition on a certain column, you can just loop over the lines, skip the header, parse the part number, and use an array to count the number of lines matching for each part.
After this you can just loop over the counts recorded in the array and print them out in your specific format.
#!/usr/bin/perl
use strict;
use warnings;
my $part = 0;
my #cnt_part;
while(my $line = <STDIN>) {
if($. == 1) {
next;
}elsif($line =~ m{^\(PART (\d+)\)}) {
$part = $1;
}else {
my #cols = split(m{\s+},$line);
if(#cols == 6) {
my $potential = $cols[3];
if(0.7 <= $potential) {
$cnt_part[$part]++;
};
};
};
};
for(my $i=1;$i<=$#cnt_part;$i++){
print "Part $i:\n";
print "$cnt_part[$i] (values 0.7 <=)\n";
};
To run it, just pipe the entire file through the Perl script:
cat in.txt | perl count.pl
and you get an output like this:
Part 1:
1 (values 0.7 <=)
Part 2:
2 (values 0.7 <=)
Part 3:
2 (values 0.7 <=)
If you want to also display the counts into words, you can use Lingua::EN::Numbers (see this program ) and you get an output very similar to the one in your post:
Part 1:
1 (one values 0.7 <=)
Part 2:
2 (two values 0.7 <=)
Part 3:
2 (two values 0.7 <=)
All the code in this post is also available here.

How to benefit from heap tagging by DLL?

How do I use and benefit from the GFlags setting Enable heap tagging by DLL?
I know how to activate the setting for a process, but I did not find useful information in the output of !heap -t in WinDbg. I was expecting some output like this:
0:000> !heap -t
Index Address Allocated by
1: 005c0000 MyDll.dll
2: 006b0000 AnotherDll.dll
so that I can identify which heap was created by which DLL and then e.g. identify the source of a memory leak.
Is this a misunderstanding of the term "heap tagging by DLL" or do I need some more commands to get to the desired result?
My research so far:
I googled for a tutorial on this topic, but I couldn't find a detailed description
I read WinDbg's .hh !heap but it's not explained there in detail as well. Tag is only used in !heap -b
again a very late answer
to benefit from HeapTagging you need to create a tag first in your code.
as far as i know (that is upto xp-sp3) there were no Documented APIS to Create a tag
(I havent mucked with heap since then so i am not aware of latest apis in os > vista Rewrites were done to heap manager so probably many of the ^^^features^^^ that i post below might have been corrected or bettered or bugs removed )
in xp-sp3 you can use undocumented RtlCreateTagHeap to create a new tag to either Process Heap or Private Heap
and after you create tha tag you need to set the global flag 8000 | 800
htg - Enable heap tagging
htd - Enable heap tagging by DLL
and theoratically all allocs and frees must get tagged .
but practically only allocations > 512 kB gets tagged in xp-sp3 with these basic steps
it either is a bug or a feature that limits tagging to allocations and frees > 512 kB
HeapAlloc goes through ZwAllocateVirtualMemory in case of Allocations > 512 kB in 32 bit process refer HeapCreate / HeapAlloc Documentation in msdn
and as a debuging aid you can patch ntdll.dll on the fly to enable tagging for all Allocations and frees .
below is a sample code that demonstrates the tagging and how to view it all in windbg
compile using cl /Zi /analyze /W4 <src> /link /RELEASE
use windbg to execute the app and watch tagging with !heap * -t command
#include <windows.h>
#include <stdio.h>
//heaptags are kinda broken or they are intentionally
//given only to allocations > 512 kb // allocation > 512 kb
//go through VirtualAlloc Route for Heap created with maxsize
//set to 0 uncomment ALLOCSIZE 0xfdfd2 and recompile to watch
// tagging increase by 100% with ALLOCSIZE 0xfdfd1 only 50 allocs
// and frees that are > 512 kB will be tagged these magic numbers
// are related to comment in HeapCreate Documentation that state
// slightly less than 512 kB will be allocated for 32 bit process
// tagging can be dramatically increased by patching ntdll when
// stopped on system breakpoint patch 7c94b8a4 (xpsp3 ntdll.dll)
// use the below command in windbg for finding the offset of pattern
// command must be in single line no line breaks
// .foreach /pS 4 /ps 4 ( place { !grep -i -e call -c
// "# call*RtlpUpdateTagEntry 7c900000 l?20000" } ) { ub place }
// the instruction we are searching to patch is
//7c94b8a1 81e3ff0fffff and ebx,0FFFF0FFFh
// patch 0f to 00 at system breakpoint with eb 7c94b8a1+3 00
#define BUFFERSIZE 100
#define ALLOCSIZE 0xfdfd1
//#define ALLOCSIZE 0xfdfd2
typedef int ( __stdcall *g_RtlCreateTagHeap) (
HANDLE hHeap ,
void * unknown,
wchar_t * BaseString,
wchar_t * TagString
);
void HeapTagwithHeapAllocPrivate()
{
PCHAR pch[BUFFERSIZE] = {};
HANDLE hHeap = 0;
ULONG tag1 = 0;
ULONG tag2 = 0;
ULONG tag3 = 0;
ULONG tag4 = 0;
ULONG tag5 = 0;
g_RtlCreateTagHeap RtlCreateTagHeap = 0;
HMODULE hMod = LoadLibrary("ntdll.dll");
if(hMod)
{
RtlCreateTagHeap = (g_RtlCreateTagHeap)
GetProcAddress( hMod,"RtlCreateTagHeap");
}
if (hHeap == 0)
{
hHeap = HeapCreate(0,0,0);
if (RtlCreateTagHeap != NULL)
{
tag1 = RtlCreateTagHeap (hHeap,0,L"HeapTag!",L"MyTag1");
tag2 = RtlCreateTagHeap (hHeap,0,L"HeapTag!",L"MyTag2");
tag3 = RtlCreateTagHeap (hHeap,0,L"HeapTag!",L"MyTag3");
tag4 = RtlCreateTagHeap (hHeap,0,L"HeapTag!",L"MyTag4");
}
}
HANDLE DefHeap = GetProcessHeap();
if ( (RtlCreateTagHeap != NULL) && (DefHeap != NULL ))
{
tag5 = RtlCreateTagHeap (DefHeap,0,L"HeapTag!",L"MyTag5");
for ( int i = 0; i < BUFFERSIZE ; i++ )
{
pch[i]= (PCHAR) HeapAlloc( DefHeap,HEAP_ZERO_MEMORY| tag5, 1 );
HeapFree(DefHeap,NULL,pch[i]);
}
}
if(hHeap)
{
for ( int i = 0; i < BUFFERSIZE ; i++ )
{
pch[i]= (PCHAR) HeapAlloc( hHeap,HEAP_ZERO_MEMORY| tag1, 1 );
//lets leak all allocs patch ntdll to see the tagging details
//HeapFree(hHeap,NULL,pch[i]);
}
for ( int i = 0; i < BUFFERSIZE ; i++ )
{
pch[i]= (PCHAR) HeapAlloc( hHeap,HEAP_ZERO_MEMORY| tag2, 100 );
// lets leak 40% allocs patch ntdll to see the tagging details
if(i >= 40)
HeapFree(hHeap,NULL,pch[i]);
}
// slightly less than 512 kb no tagging
for ( int i = 0; i < BUFFERSIZE / 2 ; i++ )
{
pch[i]= (PCHAR) HeapAlloc(
hHeap,HEAP_ZERO_MEMORY| tag3, ALLOCSIZE / 2 );
}
// > 512 kb default tagging
for ( int i = BUFFERSIZE / 2; i < BUFFERSIZE ; i++ )
{
pch[i]= (PCHAR) HeapAlloc(
hHeap,HEAP_ZERO_MEMORY | tag4 ,ALLOCSIZE );
}
for (int i =0 ; i < BUFFERSIZE ; i++)
{
HeapFree(hHeap,NULL,pch[i]);
}
}
}
void _cdecl main()
{
HeapTagwithHeapAllocPrivate();
}
the compiled exe to be run with windbg as below
DEFAULT execution and inspection
**only 50 tags will be visible all of them are > 512 kB Allocations
cdb -c "g;!heap * -t;q" newheaptag.exe | grep Tag**
heaptag:\>cdb -c "g;!heap * -t;q" newheaptag.exe | grep Tag
Tag Name Allocs Frees Diff Allocated
Tag Name Allocs Frees Diff Allocated
Tag Name Allocs Frees Diff Allocated
0004: HeapTag!MyTag4 50 50 0 0
patching ntdll on system breakpoint should make all tags visible
eb = write byte
patch and run the exe on exit inspect heaps with tags
cdb -c "eb 7c94b8a1+3 00;g;!heap * -t;q" newheaptag.exe | grep Tag
heaptag:\>cdb -c "eb 7c94b8a1+3 00;g;!heap * -t;q" newheaptag.exe | grep Tag
Tag Name Allocs Frees Diff Allocated
0012: HeapTag!MyTag5 100 100 0 0 <-our tag in process heap
Tag Name Allocs Frees Diff Allocated
Tag Name Allocs Frees Diff Allocated
0001: HeapTag!MyTag1 100 0 100 3200 <--- leak all
0002: HeapTag!MyTag2 100 60 40 5120 <--- leak 40 %
0003: HeapTag!MyTag3 50 50 0 0 <--- clean < 512 kB
0004: HeapTag!MyTag4 50 50 0 0 <----clean > 512 kB

PowerShell Script Starts Adding Blank Lines After 100+ Lines of Output?

I seem to have encountered a weird bug, or I've missed something in my script.
In my scripts, I'm showing progress with a faux progress bar function. It works by updating the same line of the console repeatedly, mimicking a progress bar. See the function below:
# Update-Progress-Bar : v1.0 : 2013-07-29
# Displays a percentage bar. Meant to be used repeatedly to update the same console line (giving the appearance of incrementing progress).
# - $Percentage : Determines how much progress is shown on the bar.
# - $Message : The message that accompanies the progress bar.
function Update-Progress-Bar ($Percentage, $Message){
# Save the current cursor position so we can come back later.
$CursorPosition = $Host.UI.RawUI.CursorPosition
# Convert the percentage into a proper progress bar.
$ProgressBarMax = "20"
$ProgressBarCount = [Math]::Floor([Decimal]($Percentage / 5))
$ProgressBar = ("#" * $ProgressBarCount) + (" " * ($ProgressBarMax - $ProgressBarCount))
# Change the format of the percentage depending on length.
switch ($Percentage.Length){
1 {$Percentage = " " + $Percentage + "%"}
2 {$Percentage = " " + $Percentage + "%"}
default {$Percentage = $Percentage + "%"}
}
# Trim or pad the message as necessary.
$MessageMaxLength = "50"
if ($Message.Length -gt $MessageMaxLength){ $Message = $Message.Remove($MessageMaxLength) }
else { $Message = $Message + (" " * ($MessageMaxLength - $Message.Length)) }
# Display our progress bar, percentage, and message.
Write-Host -nonewline -ForeGroundColor Blue "[$ProgressBar] $Percentage"
Write-Host " | $Message"
# Revert back to the original cursor position.
$Host.UI.RawUI.CursorPosition = $CursorPosition
}
For whatever reason, after working for about 100+ records (I'm using this as part of a script where I'm regularly performing actions against 1000s of machines), it starts performing double line breaks, which ruins the functionality of the progress bar. So I end up with this...
[ 126 / 2275 ] ComputerName1
[ ] 0% | Verifying network connectivity...
[## ] 10% | Verifying file system access...
[#### ] 20% | Determining installed operating system...
[###### ] 30% | Executing action...
[####################] 100% | Action Completed
[ 127 / 2275 ] ComputerName2
[ ] 0% | Verifying network connectivity...
[## ] 10% | Verifying file system access...
[#### ] 20% | Determining installed operating system...
[###### ] 30% | Executing action...
[####################] 100% | Action Completed
When I should have....
[ 126 / 2275 ] ComputerName1
[####################] 100% | Action Completed
[ 127 / 2275 ] ComputerName2
[####################] 100% | Action Completed
Any thoughts on this issue and a possible workaround?
EDIT #1: Is it possible that this is happening when I hit the buffer height limit of the console (as in, it starts discarding the old lines of output)?
EDIT #2: I have confirmed that this issue goes away if I increase the buffer width and height of my console window. I'm still not sure how to workaround this bug though. Thoughts?
I verified this bug on my own machine. As soon as you start wrapping the buffer, -nonewline fails to live up to its task.
You can do a few things as a workaround:
1) Programmatically increase the BufferSize to a fixed size
$Host.UI.RawUI.BufferSize = New-Object System.Management.Automation.Host.Size(120, 5000)
2) Clear the screen every so often, perhaps every 100 nodes or after each node if you don't need to follow the output
3) Clear the screen only when you've nearly reached the buffer limit
if($Host.UI.RawUI.CursorPosition.Y -gt ($Host.UI.RawUI.BufferSize.Height - 5)) {cls}
4) Clear only old parts of the screen buffer by temporarily reducing the size of the buffer. I used your function, untouched (except renaming to remove the second "-"). With the screen buffer at 100, the odd behavior would start at number 50 (2 output lines per loop)
49 / 2000
[####################] 100% | FINAL
50 / 2000
[#### ] 20% | First
[######## ] 40% | Second
[############ ] 60% | Third
[################ ] 80% | Fourth
[####################] 100% | FINAL
51 / 2000
[#### ] 20% | First
[######## ] 40% | Second
[############ ] 60% | Third
[################ ] 80% | Fourth
[####################] 100% | FINAL
But with the BufferSize switching, I made it all the way to 2000/2000 without a hitch
1998 / 2000
[####################] 100% | FINAL
1999 / 2000
[####################] 100% | FINAL
2000 / 2000
[####################] 100% | FINAL
$range = 1..2000
Test code below:
foreach($i in $range)
{
$Host.UI.RawUI.BufferSize = New-Object System.Management.Automation.Host.Size(120, 3000)
Write-Host
Write-Host $i "/ 2000"
Update-ProgressBar "20" "First"
Update-ProgressBar "40" "Second"
Update-ProgressBar "60" "Third"
Update-ProgressBar "80" "Fourth"
Update-ProgressBar "100" "FINAL"
$Host.UI.RawUI.BufferSize = New-Object System.Management.Automation.Host.Size(120, 2990)
}
Realistically, you could combine 3 and 4 so that you only clear the old parts of the buffer when it is nearly full.

zrangebyscore redis in perl

I'm using Redis.pm in perl scrpit and try to execute next command:
zrevrangebyscore <key> <highscore> 0 WITHSCORES LIMIT 0 1
In appliance with redis documentation i write next and it's working fine
my $data = { $redis->zrevrangebyscore($rkey, $ipl, 0, 'WITHSCORES') };
but when i try to subst 'limit...' in perl command:
my $data = { $redis->zrevrangebyscore($rkey, $ipl, 0, 'WITHSCORES','LIMIT 0 1') };
i got error
[zrevrangebyscore] ERR syntax error, at /usr/local/lib/perl5/site_perl/5.14/Redis.pm line 163
Redis::__ANON__(undef, 'ERR syntax error') called at /usr/local/lib/perl5/site_perl/5.14/Redis.pm line 195
Redis::wait_one_response('Redis=HASH(0x801075300)') called at /usr/local/lib/perl5/site_perl/5.14/Redis.pm line 183
Redis::wait_all_responses('Redis=HASH(0x801075300)') called at /usr/local/lib/perl5/site_perl/5.14/Redis.pm line 172
How i can pass to arg 'LIMIT 0 1' in Redis.pm ?
The answer is:
my $data = { $redis->zrevrangebyscore($rkey, $ipl, 0, 'WITHSCORES', qw{LIMIT 0 1})};
May be it will be usefull for somebody. Thanks!
If you are just looking to iterate through your sorted set, and always fetch the highest entry, just use
zrange
(zrange documentation)
instead of zrevrangebyscore.
my $start = -1; #-1 is last element = the element with the highest score
my $stop = -1;
while (my $data = $redis->zrange($rkey, $start--, $stop--, 'WITHSCORES')) {
#fetch the ultimate element, then the penultimate, etc....
};

Perl: Devel::Gladiator module and memory management

I have a perl script that needs to run in the background constantly. It consists of several .pm module files and a main .pl file. What the program does is to periodically gather some data, do some computation, and finally update the result recorded in a file.
All the critical data structures are declared in the .pl file with our, and there's no package variable declared in any .pm file.
I used the function arena_table() in the Devel::Gladiator module to produce some information about the arena in the main loop, and found that the SVs of type SCALAR and GLOB are increasing slowly, resulting in a gradual increase in the memory usage.
The output of arena_table (I reformat them, omitting the title. after a long enough period, only the first two number is increasing):
2013-05-17#11:24:34 36235 3924 3661 3642 3376 2401 720 201 27 23 18 13 13 10 2 2 1 1 1 1 1 1 1 1 1 1
After running for some time:
2013-05-17#12:05:10 50702 46169 36910 4151 3995 3924 2401 720 274 201 26 23 18 13 13 10 2 2 1 1 1 1 1 1 1 1 1
The main loop is something like:
our %hash1 = ();
our %hash2 = ();
# and some more package variables ...
# all are hashes
do {
my $nowtime = time();
collect_data($nowtime);
if (calculate() == 1) {
update();
}
sleep 1;
get_mem_objects(); # calls arena_table()
} while (1);
Except get_mem_objects, other functions will operate on the global hashes declared by our. In update, the program will do some log rotation, the code is like:
sub rotate_history() {
my $i = $main::HISTORY{'count'};
if ($i == $main::CONFIG{'times'}{'total'}) {
for ($i--; $i >= 1; $i--) {
$main::HISTORY{'data'}{$i} = dclone($main::HISTORY{'data'}{$i-1});
}
} else {
for (; $i >= 1; $i--) {
$main::HISTORY{'data'}{$i} = dclone($main::HISTORY{'data'}{$i-1});
}
}
$main::HISTORY{'data'}{'0'} = dclone(\%main::COLLECT);
if ($main::HISTORY{'count'} < $main::CONFIG{'times'}{'total'}) {
$main::HISTORY{'count'}++;
}
}
If I comment the calls to this function, in the final report given by Devel::Gladiator, only the SVs of type SCALAR is increasing, the number of GLOBs will finally enter a stable state. I doubt the dclone may cause the problem here.
My questions are,
what exactly does the information given by that module mean? The statements in the perldoc is a little vague for a perl newbie like me.
And, what are the common skills to lower the memory usage of long-running perl scripts?
I know that package variables are stored in the arena, but how about the lexical variables? How are the memory consumed by them managed?