I use an SQLite database in parallel. Mostly for reading - which means everything works out great. But also for writing and dropping tables. And then suddenly I get this at random times (which indicates a race condition - which is expected running things in parallel):
Error: near line 1: database is locked
Now I know that in 10 ms the database will not be locked, so I would like to just wait 10 ms and try again, but I cannot find a way to catch that error.
How can I catch that error?
Update
Please take note of Georg Mavridis' comment above
It sounds like your child processes are sharing the same database connection and locking out one another
If you want true parallelism then you need to make multiple connections to your database. SQLite will queue requests from separate connections and resolve the conflict for you, unless that behaviour is disabled.
You need to design the error handling of your DBI application. There are three options that may be specified in the connect call
PrintError — on by default — this will cause a warning to be issued if an error arises
RaiseError — off by default — this will cause the process to die if an error arises
HandleError — unset by default — this option must be set to a subroutine reference, and that subroutine will be called if an error arises
If you expect no database errors then it is probably best to use
my $dbh = DBI->connect( ..., { PrintError => 0, RaiseError => 1 } )
Then you can enable error handling for parts of the code where something may go wrong and you want to attempt to fix it
The DBI documentation for the RaiseError option says this
If you want to temporarily turn RaiseError off (inside a library function that is likely to fail, for example), the recommended way is like this:
{
local $h->{RaiseError}; # localize and turn off for this block
...
}
That way, the RaiseError option is implicitly turned back on at the closing brace, and within the block you can check the values returned by execute, which says whether the operation was successful, and errstr, which gives details of the type of error sustained. You can then write retry code in Perl to do whatever you want
The standard sleep call will suspend a process, but has a granularity of one second. If you cannot afford for your program to wait that long between retries, then take a look at the usleep function from the Time::HiRes module, which takes multiples of one microsecond
Related
What happens to the first connection if I assign two DBI connections in a row to a variable?
It doesn't feel like there is an implicit disconnect (no lag).
my $dbh = DBI->connect( $dsn, $user, $pass );
$dbh = DBI->connect( $dsn, $user, $pass );
The destructors of DBDs do close the connection.
I mean, it's possible that there's a DBD that leaks database connections out there, but that's highly unlikely and it would be a bug.
One may connect multiple times with the exact same parameters. So I don't see a reason for connect to be checking whether there is any existing connection, and certainly not to touch (disconnect) it.
However, the object for this new connection then overwrites the Perl variable $dbh. At this point anything could happen since DBI uses tie-ed variables, but I don't see in sources that this would trigger the destructor (DESTROY method) nor can I see that in drivers' sources that I looked up (mySQL and Postgresql).
So it seems to me that in this way the handle on the first connection is just lost, and you effectively got a leak. If that is the case then it is also undefined what happens to possibly open transactions in the first connection.
So I'd add a check on $dbh before connecting, using (defined and) state and/or ping. Then, if previous work with it is indeed done and you want to overwrite it, first close it if needed. (Another option is to structure your code so that a new connection goes into a newly declared variable.)
What is best practice in Perl when data is passed incorrectly to a subroutine? Should the sub die or just return?
Here is what I usually do
my #text = ('line 1', 'line 2');
print_text(\#text)
or die "ERROR: something went wrong in the sub";
sub print_text{
my ($aref_text) = #_;
return unless ref($aref_text) eq "ARRAY";
print "$_\n" for #{$aref_text};
return 1;
}
Here the sub just returns if the passed input is invalid and it expects the caller to check for errors as it does here. I wonder if it is always a better practice to just "die" at the sub level. In big scripts, I'm afraid of doing that because I don't want to kill the entire script just because some simple sub fails.
On the other hand, I'm afraid of just returning because if the caller forgets to check if the sub returns true, then the script will keep going and weird stuff could happen.
Thanks
This falls squarely under the question of how to deal with errors in subroutines in general.
In principle, these are ways to handle errors in subroutines that can't themselves recover
return codes, some of which indicate errors
return "special" values, like undef in Perl
throw exceptions, and a device for that in Perl is die
The caller either checks the return, or tests for undef, or uses eval† to catch and handle the die. What is most suitable depends entirely on the context and on what the code does.
I don't see much reason in modern languages to be restrained to "codes" (like negative values) that indicate errors. For one thing, that either interferes will legitimate returns or it constrains them to go via pointer/reference, which is a big design decision.
Returning undef is often a good middle-of-the-road approach, in particular in code that isn't overly complex. It indicates some "failure" of the sub to perform what it is meant to. However, even in the smallest of subs undef may be suitable to indicate a result that isn't acceptable. Then if it is also used for bad input we have a problem of distinguishing between those failings.
Throwing an exception, based in Perl on the simple die, adds more possibilities. In complex code you may well want to write (or use) an error-handling class that mimics a more elaborate exception handling support from languages that have it, and then throw that
my $error_obj = ErrorHandlingClass->new( params );
... or die $error_obj;
Then the calling code can analyze the object. This would be the most structured way to do it.
A nice and simple example is Path::Tiny, with its own Path::Tiny::Error found in its source.
Again, what is suitable in any one particular case depends on details of that application.
A few comments on direct questions.
The dilemma of what to return is stressed by the information-free message in die (it tells us nothing of what failed). But how do we make the failure informative, in this case?
Note that your or results in a die if the sub returns 0 or an empty string. If we replace it with // (defined-or), so to die on undef, we still can't print a specific message if undef may also indicate a bad result.
So in this case you may want the function to die on bad input, with a suitable message.
That would do it for debugging after there's been a problem. If the code needs to be able to recover then you'd better return more structured information -- throw (or return) an object of an error handling class you'd write. (As an ad hoc stop-gap measure you can parse the message from die.)
As for the age-old question of discipline to check returns, a die is a good tool. There is no "simple sub" that is unworthy – you do not want to proceed with an error so it's OK to die. And in complex projects error handling is more complex, so we need more tools and structure, not less.
Recall that exceptions "bubble up", propagate up the call stack if unhandled, and so does die. This can be used nicely for debugging without having eval on every single call. In the end, most of this is a part of debugging.
There is no "best practice" for this. But a default of die-ing is rather reasonable.
† By now we seem to be getting a try-catch style handling of an exception (die) support in the core. It is introduced as experimental in 5.34.0, but they recommend using Feature::Compat::Try for now. This is
ported from Syntax::Keyword::Try.
Let's say I have a subroutine that connects to a DB. Then I want to make a query and receive the output and do something with it, of course, but what if the query is invalid?
So let's assume I have something like:
$dbh = DBI->connect(<db>, <user>, <pass>);
$query = 'select a, b, c from table';
$sth = $dbh->prepare($query);
$sth->execute();
I realise I could also use a "do", but this will be easier for me to understand. So in a scenario where "table" doesn't have a column "c", the query would be invalid, making the handle $sth invalid and failing to execute. Right?
Now, in case of a failure as such, what happens to $sth? I can't call finish on it, because it is invalid.
So in a scenario where "table" doesn't have a column "c", the query would be invalid, making the handle $sth invalid and failing to execute. Right?
It's not clear what you mean by "invalid".
Some drivers, like DBD::Oracle, may send the SQL statement to the server when you call prepare, in which case the error will be detected by the server and prepare will fail, returning an undef. (See also #Borodin's note about RaiseError.)
If the driver doesn't send the statement to the server when prepare is called then driver creates and returns a statement handle that has the statement string stored within it.
Now, in case of a failure as such, what happens to $sth? I can't call finish on it, because it is invalid.
Again, it's not clear what you mean by "invalid". (I find it helpful to try to be really clear about the concepts and situations I'm thinking about. Forcing myself to do that often means I find the answers to my own questions because the 'problem' was in the words I was using.)
So, as I described above, either prepare failed and returned an undef (or threw an exception due to RaiseError - recommended), or prepare returned a valid $sth.
If prepare returned an undef then perl will throw an exception when you try to call execute.
If prepare returned a valid $sth then execute will be called, the statement will be sent to the server, the server will detect the error, and execute will return false (or throw an exception if RaiseError is set - recommended).
At no point would I call $sth "invalid". Either you don't have an $sth at all because prepare failed, or you have an $sth that's perfectly valid but experienced an error when executed.
In that case calling $sth->finish would work but, as #ThisSuitIsBlackNot correctly noted, it would be pointless. (I should have renamed finish to discard_pending_rows many many years ago.)
This is by far the best way to do things. Stick with prepare (which you need to do only once) and execute and avoid do. You should also use question marks as placeholders for any parameters to the query and pass the values in the call to execute. Amongst other things, that way handles any necessary quoting of the values for you
It is very rare that a call to finish or disconnect is necessary. You should leave them out until you know your way around DBI better
You're correct, $sth will be undef, and the error code and error message will be returned by $dbh->err and $dbh->errstr. Also, by default the PrintError option is set, which causes DBI to generate a warning if there are any errors in execution. If you want your program to stop altogether instead then disable PrintError and enable RaiseError
Apart from that it's really down to what you choose to do in the case of an error. For instance, some errors might be recoverable, in which case you can disable both PrintError and RaiseError and test whether the returned value is undef. Then you would perform some action possibly based on the error code returned by $dbh->err
I hope that helps
Consider the following actions:
sub get_stuff :Chained('/') :PathPart('stuff') :CaptureArgs(1) {
my ($self,$c,$stuff_id) = #_;
die "ARRRRRRGGGG";
}
sub view_stuff :Chained('get_stuff') :PathPart('') :Args(0){
die "DO'H";
}
Now if you request '/stuff/314/' , you'll get
Error: ARRRRG in get_stuff at ...
Error: DO'H in view_stuff at ...
Is there a reason why not just throw the error at the first failing chain link?
Why is catalyst trying to carry on the chain?
I'm not sure of the answer as to 'why' but I presume it was done that way to give flexibility.
You should probably catch the error with eval (or preferably something like Try::Tiny or TryCatch) and call $c->detach if you want to stop processing actions.
Catalyst::Plugin::MortalForward
The chosen answer may be outdated. Catalyst can die early, when the application's config key abort_chain_on_error_fix is set.
__PACKAGE__->config(abort_chain_on_error_fix => 1);
See the documentation about Catalyst configurations. It also states, that this behaviour may be standard in future.
Cucabit is right, detach is the way to go. As to the why, normally in a perl process, 'die' stops the process. In Catalyst you don't want that. If for instance you run your Catalyst app under FastCGI, you spawn one or more standalone processes that handle multiple requests. If the first request would kill the process itself, the web server would have to respawn the FastCGI process in order to be able to handle the next call. I think for that, Catalyst catches 'die' (its used a lot as the default 'do_something() or die $!') and turns it into an Exception.
You could also end the process with 'exit' I guess, but you end up with the same problems as above, killing the process.
What you can of course do is create your own 'die' method that logs the error passed with the default log object and then calls detach or something. it should also be possible to redefine the Catalyst exception handling as anything is possible with Catalyst :)
I've got a couple of methods that should be executed only in the case my DBI driver class is currently into a transaction to ensure data integrity. I'm looking to write something like this:
sub m{
my ($self , $dbh ) = #_ ;
unless( $dbh->isInTransaction()){
die "Use this only within a transaction\n" ;
}
etc ...
}
From the docs for begin_work, I understand that begin_work will set AutoCommit to off during the time of the transaction and will set it back to 'on' on commit or rollback, but I wonder if testing for the AutoCommit attribute value is a safe way to implement isInTransaction.
Thanks for your help.
J.
If you enable AutoCommit and start transactions with $dbh->begin_work, you can test to see if you're in a transaction:
if ($dbh->{BegunWork}) {
If you disable AutoCommit, DBI doesn't help much: you can only check for active statements connected to the database handle:
if ($dbh->{ActiveKids}) {
I've never had to check if there was a transaction active--it surprises me there's no support for it. You should probably track transactions yourself in a wrapper about DBI (or inject methods into DBI). Extending BegunWork to be useful with AutoCommit disabled looks like a core DBI fix.
If you are writing your own wrapper class, you can wrap begin_work and the other transaction methods so you can maintain your own state. Otherwise, you're depending on undocumented features or assumptions that may change, especially if you have to switch to another driver.
Is your code database-independent? If so carefully read the section on AutoCommit because there are some important differences between databases depending on how they handle transactions. But if you already know your database deals with transactions in the way you need, then AutoCommit should be fine.