I have perl v5.8.4, I cannot install any lib/module, I need to use the vanila version.
I have an perl script that sends HTTP request to an webserver. I'm trying to code an function to print how many requests I'm sending per sec and per minute to the webserver. The idea is to print once per second and then once per minute.
I was thinking on something like the logic below:
# First I get the time I started the script
$time = the time the script started
# Then, for each request I increase $req(for sec) and $reqmin(for minute)
for each request, $req++ and $reqmin++
# When $time hits one sec of load, I will print the number of requests I sent and then I will set back $req to 0, so I can reuse the var for the next second
if $time passed 1 sec, print $req (I think this may give me the TPS)
$req = 0
# Same as above, but for minutes
if $time passed 60sec, print $reqmin
$reqmin = 0
The above is not an perl code, but the explanation of what I'm trying to achieve. I'm not trying to get the runtime, control the traffic or do any benchmarking. I'm just trying to get how many requests I'm sending per sec and per min.
I'm not sure if the logic, explained above, is the correct path that I should follow to calculate the TPS(transactions per second) in my code.
The other problem that I have, is that I'm not sure how to calculate the time using perl. Like, I need to know that 1 second has past since the first run to print the requests per second, same for 1 minute. I believe I should use perl's time but I'm not sure.
I've prepared an example for you. Your algorithm is pretty sound. Here's an implementation that does the seconds only. You should be able to go from there.
It uses Time::HiRes, which is included with your old Perl. We need usleep only to simulate the requests. Thetv_interval function gets the delta between two microsecond-times, and gettimeofday grabs the current microsecond-time.
use strict;
use warnings;
use Time::HiRes qw(tv_interval usleep gettimeofday);
$|++; # disable output buffer
my $req_per_second = 0; # count requests per second
my $sum_towards_second = 0; # sum intervals towards one full second
my $last_timeofday = [gettimeofday]; # start of each interval
foreach my $i ( 1 .. 10000 ) {
do_request();
my $new_timeofday = [gettimeofday]; # end for delta
my $elapsed = tv_interval( $last_timeofday, $new_timeofday ); # get the delta
$last_timeofday = $new_timeofday; # new time is old time for the next round
$sum_towards_second += $elapsed; # add elapsed time to go towards one second
$req_per_second++; # we did one request within the current second
# when we arrive at a full second we reset
if ( $sum_towards_second > 1.0 ) {
printf "approximately %d req/s\n", $req_per_second;
$sum_towards_second = $req_per_second = 0;
}
}
sub do_request {
usleep rand 1000; # emulate the request
}
This algorithm is close to your idea, and also close to what I sketched out in my comment. In every iteration we start with doing the request, then take the current timestamp. We calculate the delta to the last timestamp and add it to a counter. If that counter reaches 1, we print the number of requests we've done in that second. Then we can reset both the time counter and the request counter.
The output looks like this.
approximately 1785 req/s
approximately 1761 req/s
approximately 1759 req/s
approximately 1699 req/s
approximately 1757 req/s
I'll leave counting minutes as an exercise to the reader.
Related
I'm parsing a log file, and I'm required to produce JSON file with results for every 1 second, 5, 5 min, 15, 30, 1 hour ..etc
I'm confused about the concept here, it is possible to write one big infinite while loop that include number of subroutines each will perform its parsing based on different time interval.
Here is I have so far, not sure if this is the right approach
#!/usr/bin/perl
use strict;
use warnings;
$ | = 1; # Ensure output appears
my $interval = 1;
my #command = "echo processing ...";
my $count = 0;
while (1) {
`since access.log | json_convert.pl
sleep $interval - time % $interval;
system #command;
}
Repeated sleep calls will drift rapidly because they don't take into account the time spent creating the JSON data. I would also be very wary of using system calls to do the data manipulation as it is bound to slow things down, and you may well overrun the one-second response time that you are looking for. Do it all within Perl instead
I recommend you look at the excellent EV module. The periodic event type is what you need
Here's a very short example. Your own callback subroutine should examine the time at which it is called to determine what to do, according to whether it is a multiple of 5 seconds, 60 seconds etc. Alternatively you could set up multiple event loops, one for each interval. The module makes it very straighforward
use strict;
use warnings 'all';
use feature 'say';
use EV;
use Time::Piece 'localtime';
STDOUT->autoflush;
my $one_second = EV::periodic 1, 1, 0, \&callback;
sub callback {
say "Periodic event at ", localtime(EV::now)->hms;
};
EV::run;
output
Periodic event at 10:27:36
Periodic event at 10:27:37
Periodic event at 10:27:38
Periodic event at 10:27:39
Periodic event at 10:27:40
etc.
I have the following auto-responder on my bot
on *:TEXT:*sparky*:*: { msg # $read(scripts/name-responses.txt) }
on *:ACTION:*sparky*:*: { msg # $read(scripts/name-responses.txt) }
I wanted to know how can I tell write a code, I'm guessing with an IF statement, that if a user types sparky more than twice that the user gets ignored for 120 seconds. This way, my bot doesn't flood the chat due to the auto-responder feature.
Any help would be appreciated!
I would recommend keeping track of all users that have used the command, and when they have last used it. This can easily be done by saving all data in an INI file.
You can save this information by using the writeini command. To write the data to this file, use something along the lines of the following:
writeini sparky.ini usage $nick $ctime
$ctime will evaluate to the number of seconds elapsed since 1970/01/01. This is generally the way to compare times of events.
Once a user triggers your script again, you can read the value from this INI file and compare it to the current time. If the difference between the times is less than 10 seconds (for example), it can send the command and then ignore them for 120 seconds. You would read the value of their last usage using:
$readini(sparky.ini, n, usage, $nick)
Your final script could look like something along the lines of the following script. I've moved the functionality to a separate alias (/triggerSparky <nick> <channel>) to avoid identical code in the on TEXT and on ACTION event listeners.
on *:TEXT:*sparky*:#: {
triggerSparky
}
on *:ACTION:*sparky*:#: {
triggerSparky
}
alias triggerSparky {
; Send the message
msg $chan $read(scripts/name-responses.txt, n)
if ($calc($ctime - $readini(sparky.ini, n, usage, $nick)) < 10) {
; This user has recently triggered this script (10 seconds ago), ignore him for 120 seconds
ignore -u120 $nick
remini sparky.ini usage $nick
}
else {
writeini sparky.ini usage %nick $ctime
}
}
Of course, a slightly easier way to achieve a similar result is by simply ignoring them for a predefined time without saving their data in an INI file. This would stop you from checking whether they have triggered twice recently, but it would be a good way to only allow them to trigger it once per two minutes, for example.
So I've researched to the ends of the internet (at least I think so) about this issue. I'm trying to set an alarm timeout of 60 seconds for a get() but it does not get caught and will run past 60 seconds, also any time the default timeout (180 sec) is reached for the www::mechanized::timed constructor, I get the error below:
Use of uninitialized value in addition (+) at /usr/lib/perl5/site_perl/5.10.0/WWW/Mechanize/Timed.pm line 52.
code:
use WWW::Mechanize::Timed;
use HTTP::Cookies;
use Try::Tiny;
my $ua = WWW::Mechanize::Timed->new(
autocheck => 0#turning off autocheck becuase any get errors will be fatal need to check ourselves
);
my $cookies = HTTP::Cookies->new(
autosave => 1
);
$ua->cookie_jar($cookies);
$ua->agent_alias("Windows IE 6");
try{
local $SIG{ALRM} = sub { die "alarm\n" };
alarm 60;
$ua->get('https://secure.site.com'); #secure site that timed out
alarm 0;
} catch {
die $_ unless $_ eq "alarm\n";
print "page timed out after 60 seconds!\n";
exit;
};
my $total_time = sprintf '%.3f', ($ua->client_elapsed_time);
unless($ua->success){
print "Error: " . $ua->status;
exit;
}
...
I've gone over these questions to figure out how to get alarm to work without writing my own timeout function.
Perl Mechanize timeout not working with https
and
Ways to do timeouts in Perl?
So far I see recommendations for using LWPx::ParanoidAgent, not sure if I understand the "Use LWPx::ParanoidAgent and mix it into Mech" part
Possible to use timeout in WWW::Mechanize on https?
or patching LWP::UserAgent with
http://search.cpan.org/~sharyanto/LWP-UserAgent-Patch-HTTPSHardTimeout-0.04/lib/LWP/UserAgent/Patch/HTTPSHardTimeout.pm
Any thoughts on how to get the timeout to work with alarm?
Thanks!
The below helped to set an alarm for each get(), Seems much easier than try-catch with sig alarm unless i'm missing something?
use Sys::SigAction qw(timeout_call);
if ( timeout_call( 60 ,sub { $ua->get('https://secured.site.com'); } ))
{
print "ALARM page timed out after 60 seconds!\n" ;
exit;
}
Pretty much the same answer as this question but with actual code Ways to do timeouts in Perl?
text from http://metacpan.org/pod/Sys::SigAction
timeout_call()
$timeout ,$coderef
Given a code reference, and a timeout value (in
seconds), timeout() will (in an eval) setup a signal handler for
SIGALRM (which will die), set an alarm clock, and execute the code
reference. $time (seconds) may be expressed as a floating point
number.
If Time::HiRes is present and useable, timeout_call() can be used with
a timer resolution of 0.000001 seconds. If Time:HiRes is not available
then factional second values less than 1.0 are tranparently converted
to 1.
If the alarm goes off the code will be interrupted. The alarm is
canceled if the code returns before the alarm is fired. The routine
returns true if the code being executed timed out. (was interrupted).
Exceptions thrown by the code executed are propagated out.
The original signal handler is restored, prior to returning to the
caller.
If HiRes is not loadable, Sys::SigAction will do the right thing and
convert
one last thing to consider/keep in mind:
use of Sys::SigAction::timeout_call unsafe?
I have a RFXCOM transceiver for 433 mhz signals. I managed to put together a program that can transmit signals without a problem (and for example turn on a lamp). However I also want to be able to receive signals from my remote control. A bit of googling gave me this working code;
use Device::SerialPort;
my $PortObj=Device::SerialPort->new("/dev/ttyUSB1");
$PortObj->user_msg(ON);
$PortObj->databits(8);
$PortObj->baudrate(38400);
$PortObj->parity("none");
$PortObj->stopbits(1);
$PortObj->handshake("rts");
my $STALL_DEFAULT=10; # how many seconds to wait for new input
my $timeout=$STALL_DEFAULT;
$PortObj->read_char_time(0); # don't wait for each character
$PortObj->read_const_time(1000); # 1 second per unfulfilled "read" call
my $chars=0;
my $buffer="";
while ($timeout>0) {
my ($count,$saw)=$PortObj->read(1); # will read _up to_ 255 chars
if ($count > 0) {
$chars+=$count;
$buffer.=$saw;
print $saw;
# Check here to see if what we want is in the $buffer
# say "last" if we find it
}
else {
$timeout--;
}
}
if ($timeout==0) {
die "Waited $STALL_DEFAULT seconds and never saw what I wanted\n";
}
One thing I can't figure out - this script gives me the output after about 10 seconds, but I want to see the received data instantly. Any idea what I need to change? I don't think it has to do with the timeout part since that just seems to measure the time since the last received signal. Any ideas?
Suffering from buffering? Set
$| = 1;
at the top of your script.
Is it possible to implement some kind of timeout (time limit) for fork using Parallel::ForkManager ?
Basic Parallel::ForkManager script looks like this
use Parallel::ForkManager;
my $pm = Parallel::ForkManager->new( 10 );
for ( 1 .. 1000 ) {
$pm->start and next;
# some job for fork
$pm->finish;
}
$pm->wait_all_children();
I would like to limit time for "# some job for fork". For example, if its not finished in 90 secs. then it (fork) should be killed/terminated.
I thought about using this but I have to say, that I dont know how to use it with Parallel::ForkManager.
EDIT
Thanks hobbs and ikegami. Both your suggestions worked..... but only in this basic example, not in my actual script :(.
These forks will be there forever and - to be honest - I dont know why. I use this script for couple of months. Didnt change anything (although many things depends on outside variables).
Every fork has to download a page from a website, parse it and save results to a file. It should not take more than 30 secs per fork. Timeout is set to 180 secs. Those hanging forks are totally random so its very hard to trace the problem. Thats why I came up with a temporary, simple solution - timeout & kill.
What could possibly disable (interrupt) your methods of timeout in my code ? I dont have any other alarm() anywhere in my code.
EDIT 2
One of the forks, was hanging for 1h38m and returned "timeout PID" - which is what I type in die() for alarm(). So the timeout works... but its late about 1h36,5m ;). Do you have any ideas?
Update
Sorry to update after the close, but I'd be remiss if I didn't point out that Parallel::ForkManager also supports a run_on_start callback. This can be used to install a "child registration" function that takes care of the time()-stamping of PIDs for you.
E.g.,
$pm->run_on_start(sub { my $pid = shift; $workers{$pid} = time(); });
The upshot is that, in conjunction with run_on_wait as described below, the main loop of a P::FM doesn't have to do anything special. That is, it can remain a simple $pm->start and next, and the callbacks will take care of everything else.
Original Answer
Parallel::ForkManager's run_on_wait handler, and a bit of bookkeeping, can force hanging and ALRM-proof children to terminate.
The callback registered by that function can be run, periodically, while the $pm awaits child termination.
use strict; use warnings;
use Parallel::ForkManager;
use constant PATIENCE => 90; # seconds
our %workers;
sub dismiss_hung_workers {
while (my ($pid, $started_at) = each %workers) {
next unless time() - $started_at > PATIENCE;
kill TERM => $pid;
delete $workers{$pid};
}
}
...
sub main {
my $pm = Parallel::ForkManager->new(10);
$pm->run_on_wait(\&dismiss_hung_workers, 1); # 1 second between callback invocations
for (1 .. 1000) {
if (my $pid = $pm->start) {
$workers{$pid} = time();
next;
}
# Here we are child. Do some work.
# (Maybe install a $SIG{TERM} handler for graceful shutdown!)
...
$pm->finish;
}
$pm->wait_all_children;
}
(As others suggest, it's better to have the children regulate themselves via alarm(), but that appears intermittently unworkable for you. You could also resort to wasteful, gross hacks like having each child itself fork() or exec('bash', '-c', 'sleep 90; kill -TERM $PPID').)
All you need is one line:
use Parallel::ForkManager;
my $pm = Parallel::ForkManager->new( 10 );
for ( 1 .. 1000 ) {
$pm->start and next;
alarm 90; # <---
# some job for fork
$pm->finish;
}
$pm->wait_all_children();
You don't need to set up a signal handlers since you do mean for the process to die.
It even works if you exec in the child. It won't work on Windows, but using fork on Windows is questionable in the first place.
Just do what the answer you linked to suggests, inside the child process (i.e. between the $pm->start and next and the end of the loop. There's nothing special you need to do to make it interact with Parallel::ForkManager, other than make sure you don't accidentally kill the parent instead :)