Instagram Real-time API duplicate requests - real-time

I have an issue where when I create a real-time subscription I get duplicate notifications from different Instagram IP addresses. I have it set up so that when I get a notification, I send a request for latest updates using the min_tag_id setting. I store that in my db to use it for the next request. I don't always get duplicates, but when I do, everything about the notification is the same (time, object,changed_aspect), except I can tell they are different from my debugging output which lists two almost identical requests... the only differing info being a different IP address and the REQUEST_TIME_FLOAT is different by about 1/10th of a second. They even have the same HTTP_X_HUB_SIGNATURE value.
My general algorithm is:
process_subscription_update($data){
# get old min_id
$min_tag_id = mysqli_fetch_object(mysqli_query($dbconnecti,sprintf("SELECT instagram_min_id+0 as instaid FROM xxxx WHERE xxxx=%d",$_GET['xxxx'])));
$min_id = $min_tag_id->instaid;
# make api call
$ch = curl_init();
curl_setopt($ch,CURLOPT_URL, 'https://api.instagram.com/v1/tags/'.$_GET['tag'].'/media/recent?client_id=xxxx&min_tag_id='.$min_id.($min_id==0?'&count=1':''));
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
$result = curl_exec($ch);
curl_close($ch);
$i = json_decode($result);
if ($min_id == $i->pagination->min_tag_id) { exit; }
# write new min_id to db
record_min_id($i->pagination->min_tag_id);
$data2 = $i->data;
foreach($data2 as $d) {
process_instagram($d);
}
// debugging output: ****************
$file = file_get_contents($_SERVER['DOCUMENT_ROOT'].'instagram/updates.txt');
$foo = "\n";
foreach($_SERVER as $key_name => $key_value) {
$foo .= $key_name . " = " . $key_value . "\n";
}
$fulldata = $file . "\n\n\n" . $result . "\n min_id = " . $min_id . $foo;
$fulldata .= "\nTIME:".$data[0]->time;
$fulldata .= "\nOBJECT:".$data[0]->object;
$fulldata .= "\nCHANGED_ASPECT:".$data[0]->changed_aspect;
file_put_contents($_SERVER['DOCUMENT_ROOT'].'instagram/updates.txt', $fulldata);
// end debugging output *************
}
I'd like to avoid checking if the instagram message id already exists in my db within the process_instagram function, and with the duplicates only coming 1/10th of a second apart, I don't know if that would work anyway.
Anybody else experience this and/or have a solution?

I solved this. I don't think there is anything I can do about receiving the duplicate notifications. So, when writing the Instagram to my db, I have a field for the Instagram id and put a unique constraint on the field. After doing the mysqli INSERT, I check to see if the errno = 1062, and if it does, I exit.
mysqli_query($dbconnecti,"INSERT INTO xxx (foo, etc, instagram_id ...")
if ($dbconnecti->errno==1062) { exit; }
...
// more script runs here if we don't have a duplicate.

Related

Delayed response to slash command with Mojolicious in Perl

I am trying to create a slack application in Perl with mojolicious and I am having the following use case:
Slack sends a request to my API from a slash command and needs a response in a 3 seconds timeframe. However, Slack also gives me the opportunity to send up to 5 more responses in a 30 minute timeframe but still needs an initial response in 3 seconds (it just sends a "late_response_url" in the initial call back so that I could POST something to that url later on). In my case I would like to send an initial response to slack to inform the user that the operation is "running" and after a while send the actual outcome of my slow function to Slack.
Currently, I can do this by spawning a second process using fork() and using one process to respond imidiately as Slack dictates and the second to do the rest of the work and respond later on.
I am trying to do this with Mojolicious' subprocesses to avoid using fork(). However I can't find a way to get this to work....
a sample code of what I am already doing with fork is like this:
sub withpath
{
my $c = shift;
my $user = $c->param('user_name');
my $response_body = {
response_type => "ephemeral",
text => "Running for $user:",
attachments => [
{ text => 'analyze' },
],
};
my $pid = fork();
if($pid != 0){
$c->render( json => $response_body );
}else{
$output = do_time_consuming_things()
$response_body = {
response_type => "in-channel",
text => "Result for $user:",
attachments => [
{ text => $output },
],
};
my $ua = Mojo::UserAgent->new;
my $tx = $ua->post(
$response_url,
{ Accept => '*/*' },
json => $response_body,
);
if( my $res = $tx->success )
{
print "\n success \n";
}
else
{
my $err = $tx->error;
print "$err->{code} response: $err->{message}\n" if $err->{code};
print "Connection error: $err->{message}\n";
}
}
}
So the problem is that no matter how I tried I couldn't replicate the exact same code with Mojolicious' subproccesses. Any ideas?
Thanks in advance!
Actually I just found a solution to my problem!
So here is my solution:
my $c = shift; #receive request
my $user = $c->param('user_name'); #get parameters
my $response_url = $c->param('response_url');
my $text = $c->param('text');
my $response_body = { #create the imidiate response that Slack is waiting for
response_type => "ephemeral",
text => "Running for $user:",
attachments => [
{ text => 'analyze' },
],
};
my $subprocess = Mojo::IOLoop::Subprocess->new; #create the subprocesses
$subprocess->run(
sub {do_time_consuming_things($user,$response_url,$text)}, #this callback is the
#actuall subprocess that will run in background and contains the POST request
#from my "fork" code (with the output) that should send a late response to Slack
sub {# this is a dummy subprocess doing nothing as this is needed by Mojo.
my ($subprocess, $err, #results) = #_;
say $err if $err;
say "\n\nok\n\n";
}
);
#and here is the actual imidiate response outside of the subprocesses in order
#to avoid making the server wait for the subprocess to finish before responding!
$c->render( json => $response_body );
So I actually simply had to put my code of do_time_consuming_things in the first callback (in order for it to run as a subprocess) and use the second callback (that is actually linked to the parent process) as a dummy one and keep my "imidiate" response in the main body of the whole function instead of putting it inside one of the subprocesses. See code comments for more information!

Perl: How can i test for a URL ( https ) accepting GET requests using "login" parameter

I have a CGI server side script that accepts GET and POST, with login parameters.
I want to test it to make sure it is not vulnerable. So the plan is to use Perl LWP, and send login parameters in GET and POST, and compare the results. the interface has been changed, so that only in POST we can send user-name and password in session cookies ( not sure if that is a great idea ) , so how do i test it ? Here is what i have so far:
#!/usr/bin/perl
use LWP;
print "This is libwww-perl-$LWP::VERSION\n";
# Create a user agent object
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
$ua->agent("MyApp/0.1 ");
# Create a request
#my $req = HTTP::Request->new(POST => 'http://search.cpan.org/search');
#my $req = HTTP::Request->new(GET => 'https://qa.co.net:443/cgi-bin/n-cu.cgi');
my $req = HTTP::Request->new(GET => 'https://qa.co.net:443/cgi-bin/n-cu.cgi?mode=frameset&JScript=1&remote_user&login=foo&password=foo HTTP/1.1');
$req->content_type('application/x-www-form-urlencoded');
$req->content('query=libwww-perl&mode=dist');
# Pass request to the user agent and get a response back
my $res = $ua->request($req);
# Check the outcome of the response
if ($res->is_success) {
print $res->content;
#print $res->code;
#print $res->message;
}
else {
print $res->status_line, "\n";
}
This is not going to do it, since it does not have the session cookie stuff. But might be a good start though. Is this the right way to test the GET and POST ?
Here is what was implemented in the cgi:
#cr_login for POST && login for GET -- leave GET param as it used to be.
if ($m eq 'GET' && defined($req->param('login'))) {
$msg = 'parameter "login" is invalid for this request type.';
+ my $seclog = $event_logging_directory . '/invalid_request.log';
+ open(S, ">>$seclog") or die $!;
+ my $logmsg = sprintf("%4d-%02d-%02d %02d:%02d:%02d",Today_and_Now())
+ . "|mode:" . $req->param('mode')
+ . "|login:" . $req->param('login')
+ . "|remote_addr:" . $ENV{REMOTE_ADDR}
+ . "|$msg\n";
+ print S $logmsg;
and :
POST request to n-cu.cgi should use parameter "cr_login". If the parameter "login" is passed in a post request, it should throw error and return to login screen.
GET request to n-cu.cgi should use the parameter "login". If the parameter "cr_login" is passed in a post request, it should throw error and return to login screen.
so here is how we do it:
Keep the session cookie and context alive :
my $browser = LWP::UserAgent->new(keep_alive => 10);
$browser->cookie_jar( {} );
$browser->agent('Mozilla/8.0');
#$browser->ssl_opts({ verify_hostname => 0 });
$browser->show_progress(1);
and later: print the response
print "Cookies:\n", Dumper($browser->cookie_jar()), "\n\n";
my $content = $response->as_string;
print "$content\n";
Sending password in a cookie? Nope.
Disallow GET for /login.
POST username and password to /login, over SSL.
In CGI, the GET/POST is indicated via the REQUEST_METHOD environment variable.
You cannot stop determined people from issuing a GET request to your server, but you can refuse to process it like so (untested code - you have to fill in details):
if ($ENV{REQUEST_METHOD} ne 'POST') {
# issue a redirect to a suitable error page, then return.
}
my $q = CGI->new();
my $user = $q->params('username');
my $password = $q->params('password');
my $encrypted_password = my_password_encryptor($password);
unless ( can_log_in($user, $encrypted_password) ) {
# issue an error message - redirect&return or fall-through...
}
else {
$session->set_user_logged_in();
}
Most people do not roll their own authentication or session handling. They mostly use one from CPAN, or one included with the larger app framework. If you're doing CGI, you can use CGI::Session.
You might give CGI::Application and/or its offspring a look. Those authors have already solved a bunch of the problems that you're encountering.

Perl - Net::DNS to create PTR record

First, the relevant xkcd comic: http://xkcd.com/979/
Next, the 10-year old thread on PerlMonks: http://www.perlmonks.org/?node_id=210422
Basically, I'm failing in my attempts to use Net::DNS::Update to create a PTR record and I'd like to see how others have manged this.
Below is what I'm trying. $hst is the hostname that I already have an A record for. $rev is the backwards IP address in-addr.arpa thingy.
# Create the update packet:
my $update = Net::DNS::Update->new($OURDOMAIN);
# Add the PTR record:
$update->push(update => rr_add("$rev 3600 PTR $hst"));
# Send the update to the zone's primary master.
my $res = Net::DNS::Resolver->new;
$res->nameservers("$OURNMSERV");
If $OURDOMAIN is your main domain name, you need to know that you can't put PTR records into your own domain, they have to be put in the right .in-addr.arpa zone.
That zone is most likely being run by your ISP, and they're unlikely to support dynamic updates from end users.
Another ten years later, http://xkcd.com/979/ is still relevant. Having read the question and answer and also the linked PerlMonks thread, it was still unclear to me how to properly add PTR records using Net::DNS.
Well, 'when in Rome do as the Romans do' the saying goes. Having noticed that nsupdate can add PTR records to our internal DNS system just fine without me specifying the reverse zone, I was trying to find out what it does to send valid updates. Turned out it actually sends a SOA query first to figure out what the reverse zone is. Copying this behavior seems to be a somewhat robust way to add PTR records via Net::DNS:
use Net::DNS;
my $hostname = 'somehost.example.com.';
my $ip = '10.11.12.13';
my $primary_ns = 'primary.example.com';
# First, construct the reverse name based on the IP address
my #octets = split /\./, $ip;
my #rev_octs = reverse(#octets);
my $rev_name = join(".", #rev_octs) . '.in-addr.arpa.';
# Then, query the SOA record for the constructed reverse name
my $resolver = Net::DNS::Resolver->new();
$resolver->nameservers($primary_ns);
# Note: Need to use 'send' rather than 'query' since we are interested
# in the 'authority section' of the reply only.
my $packet = $resolver->send($rev_name, 'SOA');
my $zone;
if ($packet) {
# We expect only one authority record
my $len = $packet->authority;
if ($len == 0) {
# Server might not be authorative for reverse lookup zone.
warn "No authority section in reply.\n";
exit 1;
}
my $auth = ($packet->authority)[0];
$zone = $auth->name;
} else {
warn 'query failed: ', $resolver->errorstring, "\n";
exit 1;
}
# Armed with this information, we can finally add the PTR records
my $update = Net::DNS::Update->new($zone);
# Make sure that the PTR record does not already exist
$update->push( pre => nxrrset("$rev_name PTR") );
$update->push( update => rr_add("$rev_name 1200 PTR $hostname") );
my $reply = $resolver->send($update);
if ($reply) {
if ($reply->header->rcode eq 'NOERROR') {
print "Successfully added PTR record!\n";
} else {
print 'Update failed: ', $reply->header->rcode, "\n";
}
} else {
print 'Update failed: ', $resolver->errorstring, "\n";
}

Perl - How to get the email address from the FROM part of header?

I am trying to set up this script for my local bands newsletter.
Currently, someone sends an email with a request to be added, we manually add it to newsletter mailer I set up.
(Which works great thanks to help I found here!)
The intent now is to have my script below log into the email account I set up for the list on our server, grab the info to add the email automatically.
I know there are a bunch of apps that do this but, I want to learn myself.
I already have the "add to list" working when there is an email address returned from the header(from) below BUT, sometimes the header(from) is a name and not the email address (eg "persons name" is returned from persons name<email#address> but, not the <email#address>.)
Now, I am not set in stone on the below method but, it works famously... to a point.
I read all the docs on these modules and there was nothing I could find to get the darn email in there all the time.
Can someone help me here? Verbose examples are greatly appreciated since I am struggling learning Perl.
#!/usr/bin/perl -w
##########
use CGI;
use Net::IMAP::Simple;
use Email::Simple;
use IO::Socket::SSL; #optional i think if no ssl is needed
use strict;
use CGI::Carp qw(fatalsToBrowser warningsToBrowser);
######################################################
# fill in your details here
my $username = '#########';
my $password = '#############';
my $mailhost = '##############';
#######################################################
print CGI::header();
# Connect
my $imap = Net::IMAP::Simple->new($mailhost, port=> 143, use_ssl => 0, ) || die "Unable to connect to IMAP: $Net::IMAP::Simple::errstr\n";
# Log in
if ( !$imap->login( $username, $password ) ) {
print STDERR "Login failed: " . $imap->errstr . "\n";
exit(64);
}
# Look in the INBOX
my $nm = $imap->select('INBOX');
# How many messages are there?
my ($unseen, $recent, $num_messages) = $imap->status();
print "unseen: $unseen, <br />recent: $recent, <br />total: $num_messages<br />\n\n";
## Iterate through unseen messages
for ( my $i = 1 ; $i <= $nm ; $i++ ) {
if ( $imap->seen($i) ) {
my $es = Email::Simple->new( join '', #{ $imap->top($i) } );
printf( "[%03d] %s\n\t%s\n", $i, $es->header('From'), $es->header('Subject'));
print "<br />";
next;
}## in the long version these are pushed into different arrays for experimenting purposes
else {
my $es = Email::Simple->new( join '', #{ $imap->top($i) } );
printf( "[%03d] %s\n\t%s\n", $i, $es->header('From'), $es->header('Subject'));
print "<br />";
}
}
# Disconnect
$imap->quit;
exit;
use Email::Address;
my #addresses = Email::Address->parse('persons name <email#address>');
print $addresses[0]->address;
The parse method returns an array, so the above way works for me.
I'm making this a separate answer because even though this information is hidden in the comments of the accepted answer, it took me all day to figure that out.
First you need to get the From header using something like Email::Simple. THEN you need to extract the address portion with Email::Address.
use Email::Simple;
use Email::Address;
my $email = Email::Simple->new($input);
my $from = $email->header('From');
my #addrs = Email::Address->parse($from);
my $from_address = $addrs[0]->address; # finally, the naked From address.
Those 4 steps in that order.
The final step is made confusing by the fact that Email::Address uses some voodoo where if you print the parts that Email::Address->parse returns, they will look like simple strings, but they are actually objects. For example if you print the result of Email::Address->parse like so,
my #addrs = Email::Address->parse($from);
foreach my $addr (#addrs) { say $addr; }
You will get the complete address as output:
"Some Name" <address#example.com>
This was highly confusing when working on this. Granted, I caused the confusion by printing the results in the first place, but I do that out of habit when debugging.

Twitter RSS feed, [domdocument.load]: failed to open stream:

i'm using the following:
<?php
$doc = new DOMDocument();
$doc->load('http://twitter.com/statuses/user_timeline/XXXXXX.rss');
$arrFeeds = array();
foreach ($doc->getElementsByTagName('item') as $node) {
$itemRSS = array (
'title' => $node->getElementsByTagName('title')->item(0)->nodeValue,
'desc' => $node->getElementsByTagName('description')->item(0)->nodeValue,
'link' => $node->getElementsByTagName('link')->item(0)->nodeValue,
'date' => $node->getElementsByTagName('pubDate')->item(0)->nodeValue
);
array_push($arrFeeds, $itemRSS);
}
for($i=0;$i<=3;$i++) {
$tweet=substr($arrFeeds[$i]['title'],17);
$tweetDate=strtotime($arrFeeds[$i]['date']);
$newDate=date('G:ia l F Y ',$tweetDate);
if($i==0) { $b='style="border:none;"'; }
$tweetsBox.='<div class="tweetbox" ' . $b . '>
<div class="tweet"><p>' . $tweet . '</p>
<div class="tweetdate">#' . $newDate .'</div>
</div>
</div>';
}
return $tweetsBox;
?>
to return the 4 most recent tweets from a given timeline (XXXXX is the relevant feed)
It seems to work fine but i've recently been getting the following error sporadically:
PHP error debug
Error: DOMDocument::load(http://twitter.com/statuses/user_timeline/XXXXXX.rss) [domdocument.load]: failed to open stream: HTTP request failed! HTTP/1.1 502 Bad Gateway
I've read that the above code is dependant on Twitter beign available and I know it gets rather busy sometimes. Is there either a better way of receiving twits, or is there any kind of error trapping i could do to just to display "tweets are currently unavailable..." ind of message rather than causing an error. I'm usnig ModX CMS so any parse error kills the site rather than just ouputs a warning.
thanks.
I know this is old, but I was just searching for the same solution for a nearly identical script for grabbing a twitter timeline. I ended up doing this, though I haven't been able to thoroughly test it.
I defined the twitter url as a variable ($feedURL), which I also used in $doc_load. Then, I wrapped everything except for the $feedURL into this conditional statement:
$feedURL = "http://twitter.com/statuses/user_timeline/XXXXXXXX.rss"
$headers = #get_headers($feedURL);
if (preg_match("/200/", $headers[0])){
//the rest of you original code in here
}
else echo "Can't connect user-friendly message (or a fake tweet)";
So, it's just checking the headers of the the feed's page, and if its status is 200 (OK), then the rest of the script will execute. Otherwise, it'll echo a message of your choice.
(reference: http://www.phptalk.com/forum/topic/3940-how-to-check-if-an-external-url-is-valid-andor-get-file-size/ )
ETA: Or even better, save a cached version of the feed (which will also ensure you don't go over your API limit of loads):
<?php
$cache_file = dirname(__FILE__).'/cache/twitter_cache.rss';
// Start with the cache
if(file_exists($cache_file)){
$mtime = (strtotime("now") - filemtime($cache_file));
if($mtime > 600) {
$cache_rss = file_get_contents('http://twitter.com/statuses/user_timeline/75168146.rss');
$cache_static = fopen($cache_file, 'wb');
fwrite($cache_static, $cache_rss);
fclose($cache_static);
}
echo "<!-- twitter cache generated ".date('Y-m-d h:i:s', filemtime($cache_file))." -->";
}
else {
$cache_rss = file_get_contents('http://twitter.com/statuses/user_timeline/75168146.rss');
$cache_static = fopen($cache_file, 'wb');
fwrite($cache_static, $cache_rss);
fclose($cache_static);
}
//End of caching
?>
Then use $cache_file in your $doc->load($cache_file) statement instead of the actual feed url.
(Adapted from here: http://snipplr.com/view/8156/twitter-cache/).