I am using Hypertable::ThriftClient, and using mutator for mass insertion. Here is the code sample: $master, $port and $namespace are all defined.
Table:
show create table users; # Table schema is below
CREATE TABLE GROUP_COMMIT_INTERVAL "100" users (
'column_name_fake' MAX_VERSIONS 1,
ACCESS GROUP audience ('column_name_fake'),
)
:hypertable$ SELECT * FROM users limit 1; # displays
2342345 sf_following:1000234
2342346 sf_following:1234234
Code:
my $ht = new Hypertable::ThriftClient($master, $port);
my $ns = $ht->namespace_open($namespace);
my $users_mutator = $ht->mutator_open($ns, 'table_name', 2);
Subroutine:
sub batch_insert {
my ($ht, $mutator, $column_family, $row, $val) = #_;
my $keys;
my $cell;
try {
$keys = new Hypertable::ThriftGen::Key({
row => $row,
column_family => $column_family });
$cell = new Hypertable::ThriftGen::Cell({key => $keys, value => $val});
}
catch {
warn Dumper({ 'Error' => $_ });
};
$ht->mutator_set_cell($mutator, $cell);
$ht->mutator_flush($mutator);
return 1;
}
Function called:
for(.....) { # Just for example
batch_insert($ht, $users_mutator, '', $fs, "node:$node_u");
}
Here I get an exception,
Hypertable::ThriftGen::ClientException=HASH(0x54d7380)
Can anyone clarify, why?
Edit: I added table structure for more clarity?
The ClientException class is defined as follows:
exception ClientException {
1: i32 code
2: string message
}
Try catching the exception and printing out the code (in hex) and message. That should help pinpoint what's going on.
FIX: So, this exception is raised is a required parameter 'column_qualifier' was not
passed as per the table design. See below:
sub batch_insert {
my ($ht, $mutator, $cf, $cq, $row, $val) = #_;
my $keys = new Hypertable::ThriftGen::Key({
row => $row,
column_family => $cf,
column_qualifier => $cq });
my $cell = new Hypertable::ThriftGen::Cell({key => $keys, value => $val});
$ht->mutator_set_cell($mutator, $cell);
#$ht->mutator_flush($mutator);
}
Calling the above function:
--------------------------
batch_insert($ht, $users_mutator, 'column_family' , 'column_qualifier, 'row_key', '');
and it worked like a charm. If you are in similar types of issues, Let me know, I can help out. I spent quite a bit of time, reading about the Thrift api.
Related
I tried different possibilities but nothing worked, in the tutorial I couldn't find an example either.
I have a method in my modelclass:
public function getlastImport($filename)
{
//$id = (int) $id;
$rowset = $this->tableGateway->select(['Path' => $filename]);
$row = $rowset->current();
if (! $row) {
throw new RuntimeException(sprintf(
'Could not find row with identifier %d',
$id
));
}
return $row;
}
I want to retrieve the last import of a given filename, so ist must be like in sql:
select max(ID) from table where filename = $filename;
But how would be the right syntax in this case?
The sql query should be
"SELECT * FROM table_name WHERE filename={$filename} ORDER BY id DESC LIMIT 1"
Use as the following in your model
public function getlastImport($filename)
{
$select = $this->tableGateway->getSql()->select();
$select->columns(array('id', 'filename', 'label'));
$select->where(array('filename' => $filename));
$select->order("id DESC");
$select->limit(1);
$result = $this->tableGateway->selectWith($select);
$row = $result->current();
if (! $row) {
throw new RuntimeException(sprintf(
'Could not find row with identifier %d',
$id
));
}
return $row;
}
Hope this would help you!
I want to process a number of files with http://2struc.cryst.bbk.ac.uk/twostruc; to automate this I wrote a perl script using perl's HTML::Form.
This server has a two step submit process: first, upload a file or enter an id; second, select the methods to be used and the output (by chosing one of five submits).
The first step works, but for the second step I seem to be unable to chose any submit button other than the first, even though my script output confirms that I selected the one I want (different from the first).
The two core parts of the code are below, the request function:
sub create_submit_request
{
my $form_arrayref = shift;
my $form_action = shift;
my $value_hashref = shift;
my $submit_name = shift;
my $submit_index = shift;
my $found_form = 0;
my $form;
foreach my $this_form( #$form_arrayref)
{
printf( "# Found form with action=%s\n", $this_form->action);
if( $this_form->action eq $form_action)
{
$found_form = 1;
$form = $this_form;
}
}
die( "# Error: No form with action $form_action") if( $found_form == 0);
my #inputs = $form->inputs;
my $inputs_string;
foreach my $input( #inputs)
{
my $input_name = defined( $input->name) ? $input->name : "<unnamed_input>";
my $input_value = defined( $input->value) ? $input->value : "";
$inputs_string .= $input_name.( length( $input_value) > 0 ? "=".$input_value : "")." (".$input->type."); ";
}
printf( "# Available input names: %s\n", $inputs_string);
printf( "# Filling in form data\n");
while( my( $key, $value) = each( %$value_hashref))
{
$form->value( $key, $value);
}
my #submit_buttons = $form->find_input( $submit_name, "submit", $submit_index); # 1-based counting for the index
die( "# Error: Can only handle a single submit, but found ".scalar( #submit_buttons)) if( scalar( #submit_buttons) != 1);
my %submit_hash = %{ $submit_buttons[ 0]};
# DEBUG
printf( "# Use submit: %s\n", Data::Dumper->Dump( [ \%submit_hash ]));
return $form->click( %submit_hash);
}
and the code using it:
my $request = HTTP::Request->new( GET => $url_server);
my $response = $useragent->request( $request);
# the first page contains the pdb id input and file upload inputs
my #forms = HTML::Form->parse( $response);
my %value_hash = ( "file" => $pdb_file);
# the submit buttons have no name, use undef; chose the first one (w/o javascript)
$request = create_submit_request( \#forms, $form_action1, \%value_hash, undef, 1);
printf( "# Submitting to server\n");
$response = $useragent->request( $request);
# the first page contains the pdb id input and file upload inputs
#forms = HTML::Form->parse( $response);
%value_hash =( "dsspcont" => "on", "stride" => "on");
# this form has 5 submit buttons; select the 5th
$request = create_submit_request( \#forms, $form_action2, \%value_hash, undef, 5);
printf( "# Submitting to server\n");
$response = $useragent->request( $request);
my $response_content = $response->content;
printf( "# Response content: %s\n", $response_content);
Even though the script prints
# Use submit: $VAR1 = {
'name' => 'function_sequenceStructureAlignment',
'onclick' => 'this.form.target=\'_blank\';return true;',
'type' => 'submit',
'value' => 'Sequence Structure Alignments',
'value_name' => ''
};
which is the 5th submit button in the second step, the response is equivalent to pressing the first submit button.
To test the server itself, the file 1UBI.pdb can be downloaded from http://www.rcsb.org/pdb/files/1UBI.pdb and uploaded to the server. The full script is at http://pastebin.com/bSJLvNfc and can be run with
perl 2struc.pl --pdb 1UBI.pdb
Why is the server returning a different output/submit that I seem to select in the script?
(It seems it's not dependend on cookies, because I can clear them after the first step, and still get the correct result for the second step in a web browser.)
You gave a hash as selector for click, which is wrong (see documentation how to specify the selector). But because you have already found the correct submit element you could simply call click directly on it:
--- orig.pl
+++ fixed.pl
## -87,7 +87,7 ##
# DEBUG
printf( "# Use submit: %s\n", Data::Dumper->Dump( [ \%submit_hash ]));
- return $form->click( %submit_hash);
+ return $submit_buttons[0]->click($form);
}
sub predict_pdb
sub loadFileRecon {
my $self = shift;
my $days = shift;
if($days eq '') {
$days = 1;
}
my $insert = $self->{rba}->rbdb->prepare(q{
insert into rba.filerecon (
filename,
records,
start_dtm,
file_type,
managed_file_id,
mf_dtm,
processed_tidemark,
mm_records,
mm_dropped,
mm_erred,
mm_duplicate,
file_source
)
select
i.filename,
i.records,
i.file_dtm start_dtm,
i.file_type,
mf.managed_file_id,
mf.created_dtm mf_dtm,
NULL,
i.orig_records,
i.dropped,
i.erred,
i.duplicate,
i.file_source
from rba.mmfilestats i, managedfile mf, filelog fl
where
i.filename = fl.file_name and
trunc(i.file_dtm) = trunc(sysdate - ?) and
mf.managed_file_id = fl.managed_file_id
}) or die $DBI::errstr;
$insert->execute($days);
$insert->finish;
$self->{rba}->rbdb->commit;
my $update = $self->{rba}->rbdb->prepare(q{
update rba.filerecon fr
set processed_tidemark = (
select processed_tidemark
from jobhasfile j
where j.managed_file_id = fr.managed_file_id
)
where
trunc(start_dtm) = trunc(sysdate - ?) and
processed_tidemark is null
});
$update->execute($days);
$insert->finish;
$self->{rba}->rbdb->commit;
}
If the prepare statement above fails due to table or view not existing, then it should return a value to perl module
you are telling your program to die if the prepare returns false:
}) or die $DBI::errstr;
replace that with what you're looking for:
}) or return $somevalue;
or remove the or entirely and check the value of your statement handle;
my $insert = $self->{rba}->rbdb->prepare(q{
...
});
return $somevalue if ( !$insert );
What you need to do is return the DBI::errstr. Do not die on error.
So do something like this (starting at your prepare and ending with your where but getting rid of the "or die"):
...prepare( ......
where
i.filename = fl.file_name and
trunc(i.file_dtm) = trunc(sysdate - ?) and
mf.managed_file_id = fl.managed_file_id
});
if ($DBI::errstr) {
# oops something is wrong
print $DBI::errstr;
call_error($DBI::errstr);
}
Good luck
Your post of the errorlog shows that your error is raised at point of execute not at the point of prepare.
So this what you do AFTER the execute:
#your execute statement first
$insert->execute($days);
#Now the check on the execute
if ($DBI::errstr) {
# oops something is wrong
return -1;
}
Please let me know if this works
I am looking for a clean and simple example of how to use the "under" functionality in a "Mojolicious" application. All the examples I find are dealing with "Mojolicious::Lite" (which I don't use).
For example I listened to the screencast here http://mojocasts.com/e3 and I think I understand the concept of the under functionality. But I don't use "Mojolicious::Lite", so it seems that I can't follow the example directly. I keep on failing trying to adopt the Lite-example for non-Lite style. (That's probably also because I'm still kind of new to the framework)
The relevant code looks like this:
# Router
my $r = $self->routes;
# Normal route to controller
$r->get('/') ->to('x#a');
$r->get('/y')->to('y#b');
$r->any('/z')->to('z#c');
So all of this routes need to be protected by user/pass. I tried to do something like this:
$r->under = sub { return 1 if ($auth) };
But this does not compile and I just can't find an example matching this code-style...
Can anybody give me the right hint or link here?
And please forgive me if this is somewhere in the docs... they might be complete, but they lack understandable examples for simple minded guys like me :-P
The analogous code to the Lite-examples looks like this:
# Router
my $r = $self->routes;
# This route is public
$r->any('/login')->to('login#form');
# this sub does the auth-stuff
# you can use stuff like: $self->param('password')
# to check user/pw and return true if fine
my $auth = $r->under( sub { return 1 } );
# This routes are protected
$auth->get ('/') ->to('x#a');
$auth->post('/y')->to('y#b');
$auth->any ('/z')->to('z#c');
Hopes this helps anybody!
(Solution found here: http://mojolicio.us/perldoc/Mojolicious/Routes/Route#under)
I am doing it like this - in a full mojo (not lite) app:
in the startup method
$self->_add_routes_authorization();
# only users of type 'cashier' will have access to routes starting with /cashier
my $cashier_routes = $r->route('/cashier')->over( user_type => 'cashier' );
$cashier_routes->route('/bank')->to('cashier#bank');
# only users of type 'client' will have access to routes starting with /user
my $user_routes = $r->route('/user')->over( user_type => 'client' );
$user_routes->get('/orders')->to('user#orders');
below in the main app file:
sub _add_routes_authorization {
my $self = shift;
$self->routes->add_condition(
user_type => sub {
my ( $r, $c, $captures, $user_type ) = #_;
# Keep the weirdos out!
# $self->user is the current logged in user, as a DBIC instance
return
if ( !defined( $self->user )
|| $self->user->user_type()->type() ne $user_type );
# It's ok, we know him
return 1;
}
);
return;
}
I hope this helps
I use this scenario in my application:
my $guest = $r->under->to( "auth#check_level" );
my $user = $r->under->to( "auth#check_level", { required_level => 100 } );
my $admin = $r->under->to( "auth#check_level", { required_level => 200 } );
$guest->get ( '/login' )->to( 'auth#login' );
$user ->get ( '/users/profile' )->to( 'user#show' );
After this all children routes of $r will go over check_level subroutine:
sub check_level {
my( $self ) = #_;
# GRANT If we do not require any access privilege
my $rl = $self->stash->{ required_level };
return 1 if !$rl;
# GRANT If logged in user has required level OR we raise user level one time
my $sl = $self->session->{ user_level };
my $fl = $self->flash( 'user_level' );
return 1 if $sl >= $rl || $fl && $fl >= $rl;
# RESTRICT
$self->render( 'auth/login', status => 403 );
return 0;
}
According to the documentation, you should use the getNumberByUniqueId() method to retrieve the message ID when moving/deleting emails. However, I still seem to get an exception appearing part way through processing a folder: "the single id was not found in response".
The code that is used goes along the lines of:
try {
$emailAddresses = array ();
$invalidAddresses = array ();
$errors = array();
$emailValidator = new Zend_Validate_EmailAddress ();
$trimFilter = new Zend_Filter_StringTrim ();
$mail = new Zend_Mail_Storage_Imap (...);
$mail->selectFolder ( '/Unsubscribe' );
echo 'There are ' . $mail->countMessages () . " messages to process\n";
foreach ( $mail as $messageId => $message ) {
try {
$mail->noop ();
$recipientMatch = $trimFilter->filter ( str_ireplace ( 'REMOVE ', '', $message->subject ) );
if (! substr_count ( $recipientMatch, '#' )) {
// Try the sender if the subject line doesn't have an address in
$matches = array ();
$from = $message->getHeader ( 'from' );
preg_match ( '/<(.+#.+)>/', $from, $matches );
if (sizeof ( $matches ) == 2) {
$recipientMatch = $matches [1];
} else {
$recipientMatch = $from;
}
}
if (! $emailValidator->isValid ( $recipientMatch )) {
$invalidAddresses [] = $recipientMatch;
continue;
}
$emailAddresses [] = $recipientMatch;
$messageUniqueId = $mail->getUniqueId ( $messageId );
$mail->moveMessage ( $mail->getNumberByUniqueId ( $messageUniqueId ), '/Unsubscribe/done' );
} catch ( Exception $e ) {
$errors[] = array($recipientMatch , $e);
}
}
} catch ( Zend_Mail_Storage_Exception $e ) {
echo "There was a problem processing the mail account\n", $e->getMessage ();
} catch ( Exception $e ) {
echo "There was an unmatched exception\n", $e->getMessage ();
}
Any ideas why the exception is being thrown (Zend Framework v1.10.8)?
The stack trace for the exception is:
#0 /usr/share/php/libzend-framework-php/Zend/Mail/Storage/Imap.php(163): Zend_Mail_Protocol_Imap->fetch(Array, 4)
#1 /usr/share/php/libzend-framework-php/Zend/Mail/Storage/Abstract.php(307): Zend_Mail_Storage_Imap->getMessage(4)
#2 /tmp/unsubscribe.php(29): Zend_Mail_Storage_Abstract->current()
#3 /tmp/dummy.php(1): include('/tmp/unsubscribe.php')
#4 {main}
The content of the Array used in Zend_Mail_Protocol_Imap->fetch() is:
Array(
0 => 'FLAGS',
1 => 'RFC822.HEADER'
)
You sholud not use this kind of loop. With this loop $messageId is outdated as soon as you delete a single message. With the next cycle the $messageId would 'point' to the wrong message or even no message (like Index-Out-Of-Bounce). That's why you've got the error message.
Solution: collect all relevant unique ids first into an array, then loop through this array and call moveMessage - with the up-to-date id that you get from getNumberByUniqueId!
I hope I'm not wrong, but I think moveMessage($id, $folder) should be moveMessage($index, $folder). This is more accurate. But correct me if this is wrong.
as it mentioned before... in case if you've got an error: "the single id was not found in response" means that the message sequence changed in expunge process, so when you save all IDs of all messages and try to iterate you have this error, because some messages was deleted in previous, please check: http://framework.zend.com/issues/browse/ZF-5655
You can use "while" for get ACTUAL collection of ids on every iteration, like this:
while($next_id = $gmail->getUniqueId()) {
// move message to label folder "My_archive_folder"
$gmail->moveMessage($next_id, 'My_archive_folder');
}