This has been driving me crazy. We have IIS (6) and windows 2008 and ActiveState Perl 5.10. For some reason whenever we do a warn or a carp it eventually corrupts the app pool. Of course, that's a pretty big deal since it means that our errors actually cause problems.
This happened with the previous version of Perl (5.8) and Windows (2003) and IIS (5.) Anyway, basically I put in a carp or a warn and I get an error message and then some garbage text. Any thoughts?
Check to make sure that IIS and the perl DLL are linked with the same version of the C runtime library. (Use depends.exe or dumpbin /dependents).
To expand: the problem may be that IIS has its FILE* table in one place, and the perl DLL thinks it's going to be in a slightly different place. When perl goes to find the stderr handle, it treats random memory as a file handle, with predictable results.
Try adding the following to the top of your scripts:
BEGIN {
open STDERR, '>> c:/iisError.log'
or die "Can't write to c:/issError.log: $!\n";
binmode STDERR;
}
I'm not sure why you would have this problem. But several "wild" guesses as to sources for such a problem would be addressed by the above code.
(It has been a while since I read the source code for appending to files in Win32, but, as I recall, the >> mode plus binmode means that writes to the file from different processes are unlikely to collide, preventing overlapping text in the log.)
A couple of suggestions:
Make sure that the id of the worker
process has write permission to the
directory/file you are writing. I
probably wouldn't give it full
control of C:, though. Better to
make a sub-directory.
Write to the event log instead of a file using
Win32::EventLog
Update: I discovered that this error only happens when you have a variable in the warn. If the warn is just regular text there are no issues. Also, the variable cannot be empty and it looks like you have to have two warns with nonempty variables to hit the bug.
Related
In a small test file, I can run
#!/usr/bin/perl
use warnings;
use strict;
use open qw{:utf8 :std};
use XML::Simple;
my #cmdline = ("hg", "log", "-v", "--style", "xml");
open my $xml, "#cmdline |";
my $xmllog = XMLin($xml, ForceArray => ['logentry', 'parent', 'copy', 'path']);
foreach my $rev (#{$xmllog->{logentry}}) {
#do stuff
}
and it works fine. When I run the same code in a larger program (with the same XML input), it terminates with
*** glibc detected *** /usr/bin/perl: malloc(): memory corruption: 0x0a40e308 ***
(full crash log # pastebin.com)
However, if I do the exchange
#open my $xml, "#cmdline |";
my $xml = `#cmdline`;
then it works (in both files), so this is more a question of curiosity than a real problem for me.
Does anyone have any pointers on what the difference between my test case and the larger code base might be?
Is there a speed/memory/? difference in the different command calls? Best practices?
Debian Sid: Perl 5.12.4-1.
(This is my first Perl encounter, so don't assume too much about what I "should" know about the language. I just dove into existing code.)
(The larger program is ikiwiki, so the code is not a secret, but I don't know where to look for trouble, and I can't include all the code in this post for practical reasons. This concerns the Mercurial backend.)
As per suggestion from cjm, I added print "$_\n" for sort grep /XML/, keys %INC; which gave output
RPC/XML.pm
RPC/XML/Client.pm
RPC/XML/ParserFactory.pm
XML/NamespaceSupport.pm
XML/Parser.pm
XML/Parser/Expat.pm
XML/SAX.pm
XML/SAX/Base.pm
XML/SAX/Exception.pm
XML/SAX/Expat.pm
XML/SAX/ParserFactory.pm
XML/Simple.pm
in the large project, and
XML/NamespaceSupport.pm
XML/Parser.pm
XML/Parser/Expat.pm
XML/SAX.pm
XML/SAX/Base.pm
XML/SAX/Exception.pm
XML/SAX/Expat.pm
XML/SAX/ParserFactory.pm
XML/Simple.pm
in the test file.
Update: I installed the Debian package libxml-libxml-perl and added $XML::SAX::ParserPackage = "XML::LibXML::SAX"; as suggested. This also crashed, with a different message this time:
*** stack smashing detected ***: /usr/bin/perl terminated
full backtrace # pastebin.com
This time it happened consistently in both the large and the small file, though. Also, only when using open, not when using backticks.
I also installed libxml-libxml-simple-perl, but that was not supposed to be more than in practice a wrapper to always use XML::LibXML as parser. It also behaved differently and complained about the options to XMLin() that was set, so I discarded it.
Trying to explicitly (and blindly) make the program use each of the alternatives given by print "$_\n" for sort grep /XML/, keys %INC; seems to point towards that XML::SAX::Expat is used by default as cjm said (since all other alternatives exit with errors, and XML::SAX:Expat behaves exactly like the original problem in both files. Explicitly demanding XML::Simple goes into a loop that allocates all my memory).
I'm thankful for learning about different XML parsers and that XML::Simple automatically chooses different ones. Both parts of my original question somewhat remain though:
Why do the programs behave differently? Even if I explicitly set $XML::SAX::ParserPackage = "XML::SAX::Expat" in both programs, one crashes (using open) and the other works.
Should I use another method to receive output from the external command? Is it even wrong to expect XMLin() ta work with open (but why does it work in one case, then?)?
Or are they simple the "wrong" questions to ask (i.e. irrelevant)?
UPDATE: More than a week has passed, not a flurry of activity here, and I solve it a bit differently now, without problems. I mark cjm's answer as correct, since it got me further in the error analysis. Thanks!
XML::Simple is pure-Perl, so it's unlikely to cause the memory corruption you report. It depends on a lower-level XML parser, and it's likely the bug you've encountered is in there. But there are multiple parsers it could be using, and we'd need to know which one.
Try adding this line right after the XMLin line in your sample program, and update your question with the results:
print "$_\n" for sort grep /XML/, keys %INC;
This will tell us which XML parser you're actually using on your system.
Update: Since it looks like you're using XML::Parser (through its SAX interface XML::SAX::Expat, I'd suggest trying XML::LibXML::SAX instead. Libxml2 is considered one of the better XML parsers.
If you don't already have XML::LibXML::SAX installed, just installing it should switch your default SAX parser to it. If it is installed, try putting
$XML::SAX::ParserPackage = "XML::LibXML::SAX";
at the beginning of your program. (See XML::SAX::ParserFactory for how the SAX parser is selected.)
In the tchrists broilerplate i found this explicit closing of STDOUT in the END block.
END { close STDOUT }
I know END and close, but i'm missing why it is needed.
When start searching about it, found in the perlfaq8 the following:
For example, you can use this to make
sure your filter program managed to
finish its output without filling up
the disk:
END {
close(STDOUT) || die "stdout close failed: $!";
}
and don't understand it anyway. :(
Can someone explain (maybe with some code-examples):
why and when it is needed
how and in what cases can my perl filter fill up the disk and so on.
when things getting wrong without it...
etc??
A lot of systems implement "optimistic" file operations. By this I mean that a call to for instance print which should add some data to a file can return successfully before the data is actually written to the file, or even before enough space is reserved on disk for the write to succeed.
In these cases, if you disk is nearly full, all your prints can appear successful, but when it is time to close the file, and flush it out to disk, the system realizes that there is no room left. You then get an error when closing the file.
This error means that all the output you thought you saved might actually not have been saved at all (or partially saved). If that was important, your program needs to report an error (or try to correct the situation, or ...).
All this can happen on the STDOUT filehandle if it is connected to a file, e.g. if your script is run as:
perl script.pl > output.txt
If the data you're outputting is important, and you need to know if all of it was indeed written correctly, then you can use the statement you quoted to detect a problem. For example, in your second snippet, the script explicitly calls die if close reports an error; tchrist's boilerplate runs under use autodie, which automatically invokes die if close fails.
(This will not guarantee that the data is stored persistently on disk though, other factors come into play there as well, but it's a good error indication. i.e. if that close fails, you know you have a problem.)
I believe Mat is mistaken.
Both Perl and the system have buffers. close causes Perl's buffers to be flushed to the system. It does not necessarily cause the system's buffers to be written to disk as Mat claimed. That's what fsync does.
Now, this would happen anyway on exit, but calling close gives you a chance to handle any error it encountered flushing the buffers.
The other thing close does is report earlier errors in attempts by the system to flush its buffers to disk.
I have Perl program based on IO::Async, and it sometimes just exits after a few hours/days without printing any error message whatsoever. There's nothing in dmesg or /var/log either. STDOUT/STDERR are both autoflush(1) so data shouldn't be lost in buffers. It doesn't actually exit from IO::Async::Loop->loop_forever - print I put there just to make sure of that never gets triggered.
Now one way would be to keep peppering the program with more and more prints and hope one of them gives me some clue. Is there better way to get information what was going on in a program that made it exit/silently crash?
One trick I've used is to run the program under strace or ltrace (or attach to the process using strace). Naturally that was under Linux. Under other operating systems you'd use ktrace or dtrace or whatever is appropriate.
A trick I've used for programs which only exhibit sparse issues over days or week and then only over handfuls among hundreds of systems is to direct the output from my tracer to a FIFO, and have a custom program keep only 10K lines in a ring buffer (and with a handler on SIGPIPE and SIGHUP to dump the current buffer contents into a file. (It's a simple program, but I don't have a copy handy and I'm not going to re-write it tonight; my copy was written for internal use and is owned by a former employer).
The ring buffer allows the program to run indefinitely with fear of running systems out of disk space ... we usually only need a few hundred, even a couple thousand lines of the trace in such matters.
If you are capturing STDERR, you could start the program as perl -MCarp::Always foo_prog. Carp::Always forces a stack trace on all errors.
A sudden exit without any error message is possibly a SIGPIPE. Traditionally SIGPIPE is used to stop things like the cat command in the following pipeline:
cat file | head -10
It doesn't usually result in anything being printed either by libc or perl to indicate what happened.
Since in an IO::Async-based program you'd not want to silently exit on SIGPIPE, my suggestion would be to put somewhere in the main file of the program a line something like
$SIG{PIPE} = sub { die "Aborting on SIGPIPE\n" };
which will at least alert you to this fact. If instead you use Carp::croak without the \n you might even be lucky enough to get the file/line number of the syswrite, etc... that caused the SIGPIPE.
It is generally advised not to use additional linux tools in a Perl code;
e.g if someone intends to print the last line of a text file he can:
$last_line = `tail -1 $file` ;
or otherwise, open the file and read it line by line
open(INFO,$file);
while(<INFO>) {
$last_line = $_ if eof;
}
What are the pitfalls of using the previous and why should I avoid using shell tools in my code?
thanx,
Efficiency - you don't have to spawn a new process
Portability - you don't have to worry about an executable not existing, accepting different switches, or having different output
Ease of use - you don't have to parse the output, the results are already in a usable form
Error handling - you have finer-grained control over errors and what to do about them in Perl.
It's better to keep all the action in Perl because it's faster and because it's more secure. It's faster because you're not spawning a new process, and it's more secure because you don't have to worry about shell meta character trickery.
For example, in your first case if $file contained "afilename ; rm -rf ~" you would be a very unhappy camper.
P.S. The best all-Perlway to do the tail is to use File::ReadBackwards
One of the primary reasons (besides portability) for not executing shell commands is that it introduces overhead by spawning another process. That's why much of the same functionality is available via CPAN in Perl modules.
One reason is that your Perl code might be running in an environment where there is no shell tool called 'tail'.
It's a personal call depending on the project:
Is it going to be always used in shell environments with tail?
Do you care about only using pure Perl code?
Using tail? Fine. But that's really a special case, since it's so easy to use and since it is so trivial.
The problem in general is not really efficiency or portability, that is largely irrelevant; the issue is ease of use. To run an external utility, you have to find out what arguments it accepts, write code to transform your program's data structures to that format, quote them properly, build the command line, and run the application. Then, you might have to feed it data and read data from it (involving complexity like an event loop, worrying about deadlocking, etc.), and finally interpret the return value. (UNIX processes consider "0" true and anything else false, but Perl assumes the opposite. foo() and die is hard to read.) This is a lot of work to do, and that's why people avoid it. It's much easier to create an instance of a class and call methods on it to get the data you need.
(You can abstract away processes this way; see Crypt::GpgME for example. It handles the complexity associated with invoking gpg, which would normally involve creating multiple filehandles other than STDOUT, STDIN, and STDERR, among other things.)
The main reason I see for doing it all in Perl would be for robustness. Your use of tail will fail if the filename has shell metacharacters or spaces or doesn't exist or isn't accessible. From Perl, characters in the filename aren't an issue, and you can distinguish between errors in accessing the file. Sometimes being robust is more important than speedy coding and sometimes it's not.
I have a Perl script running in an AIX box.
The script tries to open a file from a certain directory and it fails to read the file because file has no read permission, but I get a different error saying inappropriate ioctl for device.
Shouldn't it say something like no read permissions for file or something similar?
What does this inappropriate ioctl for device message mean?
How can I fix it?
EDIT: This is what I found when I did strace.
open("/local/logs/xxx/xxxxServer.log", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE,
0666) = 4 _llseek(4, 0, [77146], SEEK_END) = 0
ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbffc14f8) = -1 ENOTTY
(Inappropriate ioctl for device)
Most likely it means that the open didn't fail.
When Perl opens a file, it checks whether or not the file is a TTY (so that it can answer the -T $fh filetest operator) by issuing the TCGETS ioctl against it. If the file is a regular file and not a tty, the ioctl fails and sets errno to ENOTTY (string value: "Inappropriate ioctl for device"). As ysth says, the most common reason for seeing an unexpected value in $! is checking it when it's not valid -- that is, anywhere other than immediately after a syscall failed, so testing the result codes of your operations is critically important.
If open actually did return false for you, and you found ENOTTY in $! then I would consider this a small bug (giving a useless value of $!) but I would also be very curious as to how it happened. Code and/or truss output would be nifty.
Odd errors like "inappropriate ioctl for device" are usually a result of checking $! at some point other than just after a system call failed. If you'd show your code, I bet someone would rapidly point out your error.
"inappropriate ioctl for device" is the error string for the ENOTTY error. It used to be triggerred primarily by attempts to configure terminal properties (e.g. echo mode) on a file descriptor that was no terminal (but, say, a regular file), hence ENOTTY. More generally, it is triggered when doing an ioctl on a device that does not support that ioctl, hence the error string.
To find out what ioctl is being made that fails, and on what file descriptor, run the script under strace/truss. You'll recognize ENOTTY, followed by the actual printing of the error message. Then find out what file number was used, and what open() call returned that file number.
Since this is a fatal error and also quite difficult to debug, maybe the fix could be put somewhere (in the provided command line?):
export GPG_TTY=$(tty)
From: https://github.com/keybase/keybase-issues/issues/2798
"files" in *nix type systems are very much an abstract concept.
They can be areas on disk organized by a file system, but they could equally well be a network connection, a bit of shared memory, the buffer output from another process, a screen or a keyboard.
In order for perl to be really useful it mirrors this model very closely, and does not treat files by emulating a magnetic tape as many 4gls do.
So it tried an "IOCTL" operation 'open for write' on a file handle which does not allow write operations which is an inappropriate IOCTL operation for that device/file.
The easiest thing to do is stick an " or die 'Cannot open $myfile' statement at the end of you open and you can choose your own meaningful message.
I just fixed this perl bug.
See https://rt.perl.org/Ticket/Display.html?id=124232
When we push the buffer layer to PerlIO and do a failing isatty() check
which obviously fails on all normal files, ignore the wrong errno ENOTTY.
Eureka moment!
I have had this error before.
Did you invoke the perl debugger with something like :-
perl -d yourprog.pl > log.txt
If so whats going on is perl debug tries to query and perhaps reset the terminal width.
When stdout is not a terminal this fails with the IOCTL message.
The alternative would be for your debug session to hang forever because you did not see the prompt for instructions.
Ran into this error today while trying to use code to delete a folder/files that are living on a Windoze 7 box that's mounted as a share on a Centos server. Got the inappropriate icotl for device error and tried everything that came to mind. Read just about every post on the net related to this.
Obviously the problem was isolated to the mounted Windoze share on the Linux server. Looked
at the file permissions on the Windoze box and noted the files had their permissions set to read only.
Changed those, went back to the Linux server and all worked as expected. This may not be the solution for most but hopefully it saves someone some time.
I tried the following code that seemed to work:
if(open(my $FILE, "<File.txt")) {
while(<$FILE>){
print "$_";}
} else {
print "File could not be opened or did not exists\n";
}
I got the error Can't open file for reading. Inappropriate ioctl for device recently when I migrated an old UB2K forum with a DBM file-based database to a new host. Apparently there are multiple, incompatible implementations of DBM. I had a backup of the database, so I was able to load that, but it seems there are other options e.g. moving a perl script/dbm to a new server, and shifting out of dbm?.
I also get this error "inappropriate ioctl for device" when try to fetch file stat.
It was first time when I got a chance to work on perl script.
my $mtime = (stat("/home/ec2-user/sample/test/status.xml"))[9]
Above code snippet was throwing error. Perl script was written in version 5.12 on Windows, and I have to run it on amazon linux having perl 5.15.
In my case error was because of Array index out of bond ( In java language sense).
When I modified code my $var = (stat("/home/ec2-user/sample/test/status.xml"))[0][9]; error gone and I get correct value.
Of course, it is too late to answer, but I am posting my finding so that it can be helpful for developer community.
If some perl expert can verify this, it will be great..