Extracting date parts using DBIC while keeping queries database agnostic - perl

I use a MySQL database in production and a SQLite database for running tests. One part of my application is used to gather monthly statistics for a year. I've successfully done this, however it came at a cost of not being able to automate tests because I'm using MySQL specific functions when querying for the data:
my $criteria = {
status => ['Complete'],
'YEAR(completed_on)' => DateTime->now()->year(),
};
my $attributes = {
select => [ { count => 'title' }, 'completed_on' ],
as => [qw/num_completed datetime/],
group_by => [qw/MONTH(completed_on)/],
};
Notice I'm using YEAR and MONTH MySQL functions.
I know one way I can substitute the where clause to eliminate the use of MySQLs YEAR function, something like this:
my $dtf = $schema->storage->datetime_parser;
my $begin_date = DateTime->from_day_of_year( year => DateTime->now()->year(), day_of_year => 1 ); #inception o_O
my $end_date = DateTime->from_day_of_year( year => DateTime->now()->year(), day_of_year => 36[56] );
my $criteria = {
status => ['Complete'],
completed_on =>
-between => [
$dtf->format_datetime($begin_date),
$dtf->format_datetime($end_date),
]
};
Using the recommended way to query date fields using DBIC
But I'm stumped as to what to do with the group_by clause and how to make the grouping of this fields date value by month database agnostic as well. Wondering if anyone has any ideas?
Thanks!

Sometimes you will have to make engine specific code in DBIx::Class if you're trying to do special things. You can use $schema->storage->sqlt_type to make different SQL.
Note you can also use substr(completed_on,1,4) to get the year in SQLite.
This will solve your problem:
my $type = $schema->storage->sqlt_type;
my $criteria;
my $attributes;
if ($type eq 'MySQL') {
$criteria = {
status => ['Complete'],
'YEAR(completed_on)' => DateTime->now()->year(),
};
$attributes = {
select => [ { count => 'title' }, 'completed_on' ],
as => [qw/num_completed datetime/],
group_by => [qw/MONTH(completed_on)/],
};
}
elsif ($type eq 'SQLite') {
my $dtf = $schema->storage->datetime_parser;
my $begin_date = DateTime->from_day_of_year( year => DateTime->now()->year(), day_of_year => 1 ); #inception o_O
my $end_date = DateTime->from_day_of_year( year => DateTime->now()->year() + 1, day_of_year => 1 )->add( seconds => -1 );
$criteria = {
status => ['Complete'],
completed_on => {
-between => [
$dtf->format_datetime($begin_date),
$dtf->format_datetime($end_date),
]
}
};
$attributes = {
select => [ { count => 'title' }, 'completed_on' ],
as => [qw/num_completed datetime/],
group_by => ['substr(completed_on,6,2)'],
};
}

Related

Can't update description of listing with eBay API

I'm trying to write a Perl script that will update our eBay listings descriptions without having to keep logging in (running across multiple marketplaces if proving tricky to keep stock levels, descriptions etc updated). Here is what I have so far:
my $ebay = new Net::eBay( {
SiteLevel => 'prod',
DeveloperKey => 'x',
ApplicationKey => 'x',
CertificateKey => 'x',
Token => 'x',
} );
$ebay->setDefaults( { API => 2, compatibility => 900 } );
my $new_desc = q|<meta name="viewport" content="width=device-width, initial-scale=1.0">
<p>We are proud to announce our first ever badge! With an easy-to-iron
on backing, fitting couldn't be any easier! We have designed the path to
be a perfect addition to any piece of cosplay costume. Please do send
in the photos of it being used on your costumes, as we would love to
share.</p>
<p>The badge is 7 x 7 cm / 2 x 2 inches in size, and 2mm thi<br></p>|;
my $result = $ebay->submitRequest( "ReviseItem",
{
DetailLevel => "ReturnAll",
ErrorLevel => "1",
SiteId => "1",
Item => {
Description => \$new_desc,
ItemID => 253430606975
},
ItemID => 253430606975
}) || die;
print "Result: " . Dumper( $result ) . "\n";
I get an error when running it though:
'Errors' => [
{
'ShortMessage' => 'Return Policy Attribute Not Valid',
'ErrorClassification' => 'RequestError',
'ErrorCode' => '21920200',
'LongMessage' => 'Return Policy Attribute returnDescription Not Valid On This Site',
'SeverityCode' => 'Warning',
'ErrorParameters' => {
'Value' => 'returnDescription',
'ParamID' => '0'
}
},
{
'ShortMessage' => 'Description is missing.',
'ErrorClassification' => 'RequestError',
'ErrorCode' => '106',
'SeverityCode' => 'Error',
'LongMessage' => 'A description is required.'
}
],
Am I misunderstanding what gets passed in? from what I can understand, you just pass in the params you want to change?
UPDATE: As suggested by Dave, I'm giving Marketplace::Ebay a go. Just testing by trying to select one of my items:
my $ebay = Marketplace::Ebay->new(
production => 1,
site_id => 3,
developer_key => 'xx',
application_key => 'xx',
certificate_key => 'xxx',
token => 'xx',
xsd_file => 'ebaySvc.xsd',
);
my $res = $ebay->api_call('GetItem', { ItemID => 253430606975 });
print Dumper($res);
But I get some weird error:
error: element `{urn:ebay:apis:eBLBaseComponents}GiftIcon' not
processed for {urn:ebay:apis:eBLBaseComponents}GetItemResponse/Item at
//[5]/*[6] $VAR1 = undef;
Any ideas?
Ah ha - got it! The issue seemed to be around the way the HTML was being passed along. If I put it inside a CDATA tag, it works fine:
my $new_desc = q|<![CDATA[
some html etc here
]]>|;
my $result = $ebay->submitRequest( "ReviseItem",
{
DetailLevel => "ReturnAll",
ErrorLevel => "1",
SiteId => "1",
Item => {
Description => $new_desc,
ItemID => 253430606975
},
ItemID => 253430606975
}) || die;
...and updates perfectly

Inserting one hash into another using Perl

I've tried many different versions of using push and splice, but can't seem to combine two hashes as needed. Trying to insert the second hash into the first inside the 'Item' array:
(
ItemData => { Item => { ItemNum => 2, PriceList => "25.00", UOM => " " } },
)
(
Alternate => {
Description => "OIL FILTER",
InFile => "Y",
MfgCode => "FRA",
QtyAvailable => 29,
Stocked => "Y",
},
)
And I need to insert the second 'Alternate' hash into the 'Item' array of the first hash for this result:
(
ItemData => {
Item => {
Alternate => {
Description => "OIL FILTER",
InFile => "Y",
MfgCode => "FRA",
QtyAvailable => 29,
Stocked => "Y",
},
ItemNum => 2,
PriceList => "25.00",
UOM => " ",
},
},
)
Can someone suggest how I can accomplish this?
Assuming you have two hash references, this is straight-forward.
my $item = {
'ItemData' => {
'Item' => {
'PriceList' => '25.00',
'UOM' => ' ',
'ItemNum' => '2'
}
}
};
my $alt = {
'Alternate' => {
'MfgCode' => 'FRA',
'Description' => 'OIL FILTER',
'Stocked' => 'Y',
'InFile' => 'Y',
'QtyAvailable' => '29'
}
};
$item->{ItemData}->{Item}->{Alternate} = $alt->{Alternate};
The trick here is not to actually merge $alt into some part of $item, but to only take the specific part you want and put it where you want it. We take the Alternate key from $alt and put it's content into a new Alternate key inside the guts of $item.
Adam Millerchip pointed out in a hence deleted comment that this is not a copy. If you alter any of the keys inside of $alt->{Alternative} after sticking it into $item, the data will be changed inside of $item as well because we are dealing with references.
$item->{ItemData}->{Item}->{Alternate} = $alt->{Alternate};
$alt->{Alternate}->{InFile} = 'foobar';
This will actually also change the value of $item->{ItemData}->{Item}->{Alternate}->{InFile} to foobar as seen below.
$VAR1 = {
'ItemData' => {
'Item' => {
'ItemNum' => '2',
'Alternate' => {
'Stocked' => 'Y',
'MfgCode' => 'FRA',
'InFile' => 'foobar',
'Description' => 'OIL FILTER',
'QtyAvailable' => '29'
},
'UOM' => ' ',
'PriceList' => '25.00'
}
}
};
References are supposed to do that, because they only reference something. That's what's good about them.
To make a real copy, you need to dereference and create a new anonymous hash reference.
# create a new ref
# deref
$item->{ItemData}->{Item}->{Alternate} = { %{ $alt->{Alternate} } };
This will create a shallow copy. The values directly inside of the Alternate key will be copies, but if they contain references, those will not be copied, but referenced.
If you do want to merge larger data structures where more than the content of one key needs to be merged, take a look at Hash::Merge instead.

Using multiple field query mongo perl

I have my between date queries working great, but am trying to add another and to this and failing. Any ideas?
my $cursor = $collection
->find( { create_date => {'$gte' => $past,'$lt' => $present, status => $eq' => "F" }
#status => {
#'$eq' => "F'
#} } );
You are missing a bracket:
my $cursor = $collection ->find( { create_date => {'$gte' => $past,'$lt' => $present}, status => {'$eq' => "F" }} )

Alias the sum of two columns in a DBIx::Class resultset

SELECT me.id, me.date_created, me.date_updated, me.yes,
me.name, me.description, me.currency, me.locked, me.skip,
me.uri_part, me.user_id,
yes + currency as weight
FROM ideas me having ((weight < 5)) order by weight;
How can I generate that query in DBIx::Class without using literal SQL like this:
my $query = $rs->search({},
{
'+select' => \[
'yes + currency as weight',
],
rows => 1,
order_by => { -desc => [qw/weight name/] },
having => {
weight => { '<' => $self->yes + $self->currency },
},
});
use Data::Dumper;
warn Dumper($query->as_query);
I tried using -as, however, it seems to only be useful for working with columns generated from functions, as this:
'+select' => {
'yes + currency', '-as' => 'weight'
}
generates an error
"Odd number of elements in anonymous hash at
/data/TGC/lib/TGC/DB/Result/Idea.pm line 105, line 1000.
DBIx::Class::SQLMaker::_recurse_fields(): Malformed select argument -
too many keys in hash: SCALAR(0xbf14c40),weight"
Probably the most idiomatic thing I can think of in SQL Abstract expression, without straining too hard:
#!/usr/bin/env perl
use Modern::Perl;
use MySchema;
use Data::Dumper;
my $schema = MySchema->connect('dbi:SQLite:example.db');
my $rs = $schema->resultset('Sample')->search(
{
weight => { '<' => 5 },
},
{
'+select' => [
{ '' => \'`me`.`yes` + `me`.`currency`', -as => 'weight' }
]
}
);
say Dumper( $rs->as_query() );
Which is a contrived wrapping of the column names, but it does the job, sort of. Just don't know of any way to abstract the + here. But stil:
'(SELECT me.name, me.yes, me.currency, ( me.yes + me.currency ) AS weight FROM sample me WHERE ( weight < ? ))',
Unless you are just going for idiomatic perl, in which case:
{ '' => \(join " + ", qw/`me`.`yes` `me`.`currency`/), -as => 'weight' }
But either way seems a little contrived considering both forms are longer than the literal string.
Also note that weight is referenced in WHERE and not HAVING which is because that will blow up on various SQL engines.
The '+as' should be at the same level as '+select'
$idea = $rs->search({-and => [
name => { '<' => $self->name },
]},
{
'+select' => [
'yes + currency'
],
'+as' => [qw/weight/],
rows => 1,
order_by => ['weight', 'name'],
having => {
weight => { '<' => $self->yes + $self->currency },
},
})->single;

How can I do a scrolled search on MetaCPAN?

I'm trying to convert this script to use the new Elasticsearch official client instead of the older (now deprecated) ElasticSearch.pm, but I can't get the scrolled search to work. Here's what I've got:
#! /usr/bin/perl
use strict;
use warnings;
use 5.010;
use Elasticsearch ();
use Elasticsearch::Scroll ();
my $es = Elasticsearch->new(
nodes => 'http://api.metacpan.org:80',
cxn => 'NetCurl',
cxn_pool => 'Static::NoPing',
#log_to => 'Stderr',
#trace_to => 'Stderr',
);
say 'Getting all results at once works:';
my $results = $es->search(
index => 'v0',
type => 'release',
body => {
filter => { range => { date => { gte => '2013-11-28T00:00:00.000Z' } } },
fields => [qw(author archive date)],
},
);
foreach my $hit (#{ $results->{hits}{hits} }) {
my $field = $hit->{fields};
say "#$field{qw(date author archive)}";
}
say "\nUsing a scrolled search does not work:";
my $scroller = Elasticsearch::Scroll->new(
es => $es,
index => 'v0',
search_type => 'scan',
size => 100,
type => 'release',
body => {
filter => { range => { date => { gte => '2013-11-28T00:00:00.000Z' } } },
fields => [qw(author archive date)],
},
);
while (my $hit = $scroller->next) {
my $field = $hit->{fields};
say "#$field{qw(date author archive)}";
} # end while $hit
The first search, where I'm just getting all the results in 1 chunk, works fine. But the second search, where I'm trying to scroll through the results, produces:
Using a scrolled search does not work:
[Request] ** [http://api.metacpan.org:80]-[500]
ActionRequestValidationException[Validation Failed: 1: scrollId is missing;],
called from sub Elasticsearch::Transport::try {...}
at .../Try/Tiny.pm line 83. With vars: {'body' =>
'ActionRequestValidationException[Validation Failed: 1: scrollId is missing;]',
'request' => {'path' => '/_search/scroll','serialize' => 'std',
'body' => 'c2Nhbjs1OzE3MjU0NjM2MjowakFELUU3VFFibTJIZW1ibUo0SUdROzE3MjU0NjM2NDowakFELUU3VFFibTJIZW1ibUo0SUdROzE3MjU0NjM2MTowakFELUU3VFFibTJIZW1ibUo0SUdROzE3MjU0NjM2MDowakFELUU3VFFibTJIZW1ibUo0SUdROzE3MjU0NjM2MzowakFELUU3VFFibTJIZW1ibUo0SUdROzE7dG90YWxfaGl0czoxNDQ7',
'method' => 'GET','qs' => {'scroll' => '1m'},'ignore' => [],
'mime_type' => 'application/json'},'status_code' => 500}
What am I doing wrong? I'm using Elasticsearch 0.75 and Elasticsearch-Cxn-NetCurl 0.02, and Perl 5.18.1.
I finally got it working with the newer Search::Elasticsearch official client. Here's the short version:
#! /usr/bin/perl
use strict;
use warnings;
use 5.010;
use Search::Elasticsearch ();
my $es = Search::Elasticsearch->new(
cxn_pool => 'Static::NoPing',
nodes => 'api.metacpan.org:80',
);
my $scroller = $es->scroll_helper(
index => 'v0',
type => 'release',
search_type => 'scan',
scroll => '2m',
size => 100,
body => {
fields => [qw(author archive date)],
query => { range => { date => { gte => '2015-02-01T00:00:00.000Z' } } },
},
);
while (my $hit = $scroller->next) {
my $field = $hit->{fields};
say "#$field{qw(date author archive)}";
} # end while $hit
Note that the records are not sorted when you do a scrolled search. I wound up dumping the records into a temporary database and sorting them locally. The updated script is on GitHub.
I don't have a direct answer, but I might have an approach to trouble shooting:
I followed your link to the Elasticsearch::Client and found a scroll() method:
https://metacpan.org/pod/Elasticsearch::Client::Direct#scroll
This method takes scroll and scroll_id as parameters. scroll is the number of minutes that you can keep calling the scroll method before the search expires. scroll_id is a marker to the place where the last call to scroll() ended.
$results = $e->scroll(
scroll => '1m',
scroll_id => $id
);
Elasticsearch::Scroll is an object oriented wrapper around scroll() which hides scroll and scroll_id.
I would run perl -d on your script, and step in to $scroller->next and follow that as far down the rabbit hole as you can. Something in there is trying a search which should be populating scroll_id or scrollId and is failing.
My description here is admittedly pretty rough... I ran across an accurate description of what the scroll id is and does during my googling, but I can't seem to find it again.