From what I can tell neither Log4Perl or any of its related modules in CPAN supports rotate & compression of log files.
Rotation can be accomplished by using:
Log::Log4perl::Appender::File
Log::Dispatch::FileRotate.
But neither modules supports rotation and compression. (Log::Dispatch::FileRotate has it in its todo list, but it's not currently implemented).
It is possible to do this using the standard Logrotate facility in Linux, by using either Log::Log4perl::Appender::File's recreate_check_interval or recreate_check_signal.
From initial tests, it looks like using Logrotate with the delaycompress option will do the trick - even on a machine with high load, as once the file is moved, log4perl will continue logging to the same filehandle, until the signal is cought.
However, if delaycompress is not used, and there is (even a slight delay) between the compressing of the log file, and the catching the signal by the Perl program, some logging data might be lost.
What do you think? Are there other options we did not think of?
Over the years, I've found that you almost always want to use external methods for log file rotation with Log4perl. You simply avoid a lot of subtle issues (log delays, permission issues) that internal log rotates inevitably run into.
You've mentioned two methods that work with logrotate on Linux, why not stick with them? The Log4perl FAQ describes using newsyslog which is the FreeBSD equivalent of logrotate and provides similar features.
Have you thought about working with the Log::Dispatch::FileRotate's maintainers to add the features its missing and you need? It is open source after all. :)
If you don't want to deal with that yourself, there are various CPAN support consultancies that do that for you.
I contacted the author of Log::Dispatch::FileRotate, as suggested here, and he explained the reason why compression is not yet implemented in Log::Dispatch::FileRotate.
Basically, compressing right after rotation, might block the running process, during the compression which is pretty expensive.
The options suggested were to allow the user of Log::Dispatch::FileRotate to execute an arbitrary application on the file, just after rotation, thus doing it in another non blocking process.
Another suggestion was to have a filesystem trigger (like inotify) trigger the compression when the file is closed to writing by the main process.
Yet another suggestion , is to write the log file compressed through a gzip pipe, or one of the perl gzip modules. This works, but causes some problems (grep/less) won't work. zgrep and zless will work, but zgrep gives an ugly warning when grepping on a gzip file which is still open for writing. Using "tail" on the file will also not work - so this option isn't practical.
Related
I am using Eclipse Kepler Service Release 2 , EPIC 0.5.46 and Strawberry Perl 5 version 18 for perl programming. For debugging I am using Eclipse debugger and PadWalker .
I have an interactive perl program that writes to files based on answers provided by the users to multiple prompts. While debugging , every time i change a single line of code I have to rerun the whole program again and provide inputs to every prompt , which is really time consuming.
Is there a way to make changes to the code in a sub routine , in the middle of debugging session such that the instruction pointer resets itself to the first line of that sub routine. This way i do not have to restart the session to recompile the new code.
Appreciate your inputs and suggestions. Thank You!!!
What you want to do can be done, and I've done it many times in Perl myself. For example, see this.
However although what you describe may work (and is a bit dangerous), the way it is generally done a bit different and safer.
First one has to assume a regular kind of command structure like a command processor, or say a web server.
In a command processor or web server, you read a command (or get a web request), perform an action, then read another command, perform another action and so on. From your description, it sounds like you have such a structure.
In my case, I have each debugger command stored as in Perl file. This is helpful not only for facilitating this task, but also for understanding, testing and changing the code.
Given this kind of program structure, instead of trying to change the program counter, you complete the command and at the level where you are about to read a new command, you make the change and then reload the file which changes the code.
The specific Perl construct to do this is called do. Don't use require or use which will load in a Perl file only if that file or module hasn't been previously loaded. In your situation, you want to reload even if it has been loaded before.
So now how do you get to be able to issue a do command? As you suggest, you could do it through a debugger. Assuming you have this overall program stucture as described above, you put the breakpoint somewhere a common point in the caller which loops over things to process, rather than try to change things in indvidual commands.
And you don't even need a debugger to do this! Many web frameworks like Ruby on Rails, have a "development" mode where they save timestamps on files that implement functionality. If the file has changed they issue the "do" command before running the request.
I am using a tWaitForFile component from a Talend Studio Project and I want to know if there is a way to be sure a file to trig the event when this file is fully written on disk.
I tried to set the advanced property : "Wait the file to be released"
but it seems this is useless, the file trigs the component even it is not finished to be transmitted.
Does anybody have the same behaviour and a solution to fix that?
The version of Tos is: 4.2.3
The advanced setting "Wait for file to be released" only works on Windows. It has no effect on Unix, which probably explains why it did not work for you.
It is generally difficult, or even impossible, for a Unix process to figure out if a file has been written completely or not. Consequently, there is no easy way to do this in Talend, either.
(For example, if you wanted to wait until the file size does not change anymore -- how long do you wait?)
A common solution involves the process writing to the file: Create the file under a different name first, and when it is written completely, rename it to the name that the other process expects. That way, it will show up in its full size immediately.
Is anyone aware of a hybrid version control and synchronising system?
I'm currently a happy mercurial user, but my projects usually contain a mixture of files.
Most of these (code, documentation, ...) I want to be version-controlled. This is why I use mercurial.
However, on the rare occasion I have files that I would like to synchronise between my working copies, but not version control.
For example, I version control the code I write to do image processing. This code can produce a whole bunch of output images which I'd like to have synchronised so I don't have to remember to shuffle them around my various computers, but there's no point having these version controlled.
To clarify - I am aware of extension to mercurial such as bfiles and bigfiles, which are handy for my image example, but I was just wondering if anyone out there knows of alternative ways to handle this. I just want the one system that I can tell "version control all files except those ones, which should be synced but have no history".
cheers!
EDIT: I could do something like adding a hg marksync <filename> that added <filename> to a list of files to be synced, and then adding a hook to hg push/hg pull that would (say) run rsync (or whichever sync tool) in the background, but I wondered if there was a less hacky solution (I think bfiles/bigfiles do something along these lines anyway).
Version Control System (any) doesn't care about synchronization of
not versioned data
besides default pathes
If you want sync any files - use specially designed for this task tools: f.e. rsync
This code can produce a whole bunch of output images which I'd like to have synchronised
Is this DATA or part of your CODE?
If data: Keep out of your versioning system, just don't go there. If it is part of your code (like layout images) check it in. Those are the only ways which are the generally accepted.
A nice solution for the data would be syncing OR generating them. So you might add a step after deployment to a server: GenerateImages().
edit: In addition to the comment made by the thread starter:
If the images are data and you need to process them on a different system don't think about the version control for your code. It is unrelated. The steps which would make sense to me, in order of processing:
Start with updating your image code, check it in versioning. Then deploy (yes this is deployment) the updated code to the cruncher computer. Now code is done.
Then you have tasks which the number cruncher should handle. Like processing the images. So start that processing from either the cruncher itself (probably some queue happens there) or from a central dispatcher.
Then you have the results locally at the cruncher. Now something has to happen with that data, so that's also part of your software. Decide whether you want the cruncher to send them to some central storage, your workstation or another location. Let the software handle that. This is the most hard part as I read through your question. Many solutions are possible from just FTP/network transfers to specific storage solutions. Willing to help but need more info about the real issues, amounts, sizes etc. on these parts.
If the new updated version of the image processor makes the old generated images obsolete implement that also in your code, by for example attaching an attribute to the files generated, a seperate folder or another indication. That way you could request the cruncher after update to re-generate any obsolete files.
I keep typing "t eh" instead of " the" which is, of course, annoying in the amount of time it takes me to correct myself.
The obvious answer is "Learn to type, noob!" or at least to type more slowly and/or more correctly. This error is frighteningly consistent so it appears I've trained my muscle memory for that pattern already.
But I'm wondering if it's possible to write a small, windows portable script or application that, when it detects the incorrect sequence, backspaces and corrects it automatically at a layer where it would apply to any keyboard input.
Does C# have access to that layer of the OS that intercepts keypresses systemwide?
Will I run into UAC issues with Vista?
Am I re-inventing the wheel (ie, are there open source tools I can modify or use out of the box)?
In DOS this sort of thing was quite easy and one could make TSRs (Terminate and Stay Resident) programs that would, for instance, give you a calculator onscreen with a special keypress. Not to mention the many, many practical joke programs based on this concept (dial "M" for monster!)...
I would, of course, never suggest such a utility could be used that way for co-workers...
-Adam
On windows you could use AutoHotKey. That allows you to create little scripts or macros to automate and correct things like mistypes.
One use was posted on lifehacker which took the common mistyped words and corrected them. It is at http://lifehacker.com/192506/download-of-the-day-universal-autocorrect
UPDATE Per Comment: This is Free software and windows only as far as I know.
The above script is just an example of what it can do. There are a slew of scripts available at AutoHotkeys Site
I suggest AutoHotKey. If you've never used it before, have a quick read of the tutorial: http://www.autohotkey.com/docs/Tutorial.htm
The feature you are looking for is called "hotstrings." http://www.autohotkey.com/docs/Hotstrings.htm
In your case, your script would look something like:
::teh::the
That's it! Add other things you want corrected on additional lines. AutoHotkey scripts can be compiled so you don't have to install AutoHotKey on all of your machines.
It's a very cool program. It's primary use (for making custom hotkeys) rocks! These scripts are system wide so you'll also probably want to make a hotkey to be able to turn them off too!
EDIT: In a comment, it was mentioned that he actually types "t eh" (with a space in it) and I wondered if something additional would be needed for it to work. I just tested it and it works fine. Just install autohotkey, and create a file with the .AHK extension. In that file put in the following line
::t eh::the
and save the file. Then double-click on the AHK file to load AutoHotKey with your script (you'll see a green square in your system tray to let you know it is running). It should work fine!
Yes, you can use pinvoke commands from C# to intercept the low-level os commands. I recommend you take a look at http://www.pinvoke.net. The coding isn't easy but it does work.
I suggest learning to type more slowly. I also suffer from "teh" and "ahve" in part due to autocorrect giving me the leniency. If you forced yourself to retrain then you would not be at a disadvantage when using someone else's machine.
Not to mention the unfortunate event when you need to write "t eh" and are being prevented by an overzealous 'corrector'.
Whenever I alter (or even just resave without altering) a Perl file, it completely takes down our backend. I have no idea what the problem could be. Permissions are correct. Encoding is correct. Encoding is UTF-8. Transfer mode was ASCII.
I might not deal with Perl too much but I have no idea what the problem could be. The network admin hosting our website has no idea what the problem could be.
Text editors I tried: Dreamweaver, TextMate, Vim
Operating systems I tried: Mac OS X, Linux (Ubuntu)
FTP clients I tried: Transmit (Mac), Filezilla (Linux (Ubuntu))
It's not that it's bad code, I even tried to open and solely save and my backend still goes down.
The network admin told me that he ran the files through a dos2unix converter and it worked immediately. I of course tried this and it did not, more so it wouldn't make any sense, since I tried this in some of the most respected editors and I don't think it would make such drastic changes to the file type without any user input. (when I say respected editors Dreamweaver is not included in that sentiment).
I personally think it is some sort of server-side issue because I have crossed my t's and dotted my i's in regards to any possible client side issue but I have tried everything. Any opinions as to what the root of this problem is, and any possible solutions? Thanks in advance.
Try setting binary mode in your FTP client. That will allow you to experiment with different line endings (dos2unix) on the client side, without worrying about them being translated during transfer.
I've had this problem in the past and line-feeds were indeed the culprit.
Your editor and/or FTP program may be mangling the linefeeds.
Running dos2unix on the server is a good test as to the problem but not the cause.
Generate an MD5 hash of the file after each step in saving and transport to find where it changes.
You do not say what kind of framework/server you are using.
Maybe the server reloads the file while it is still being written by FTP or whatever? (I.e. that the file is not complete when the server reads it?)
Will a server restart fix the problem once the file is uploaded?
It sounds like you are using dos2unix before the transfer but the network admin is using it after. Perhaps it's doing something different in that case.
How many lines are in the file? What is the file size before and after you save it, after you transfer it, and after transfer and running dos2unix on it?
If this is just a line ending problem, you might point your network admin at http://www.perlmonks.org/?node_id=586942.
Response to rebra: No frameworks are used, and I don't know what kind of server this is on. This is basically a one man project on a shared host which was pretty horribly maintained and I'm trying to clean house.
Yeah that does make sense and I asked the server people about that, one of my first questions actually, but even if that is the case, I can't reboot via Plesk (kind of like cPanel). But thanks for that, you put into technical words/explanation what I was thinking of the whole time.