I'm using ExifTool to add keywords to a file. I do this in batch, spawning ExifTool as a child process from my app.
exiftool -keywords+="courgettes" -keywords+="edible gardening" -keywords+="vegetables" -keywords+="Glasshouse" -keywords+="Greenhouse" -overwrite_original "/pictures/garden/P8151927.JPG"
However, it seems that with -overwrite_original the process isn't 100% transactional and atomic; occasionally it fails for some reason or other, but instead of leaving the original file untouched, it leaves it renamed to P8151927.JPG_exiftool_tmp. This means it looks like the original image has gone missing.
Is there any way to guarantee that if the ExifTool update fails, the original won't be touched? My understanding was that it would create a temp file with the new/changed keywords, and then carry out a transactional delete/rename which would either work or not, but it seems the delete can succeed, but the rename from the temp file can then fail.
Is there any way to work around this - any additional options I can use to make it 100% resilient? If not, my only alternative is to copy the file to a temp location, run exiftool on it, and then handle the rename/replace myself. Alternatively, I could check the return code, and if non-zero, I can rename the .JPG_exiftool_tmp file back to the original, I suppose.
Any better ideas?
Related
I'm looking for solution to delete or (preferably directly) overwrite source of an exe file while it is running.
To explain further before you get it all wrong, I'll give an example:
I have an exe file on drive D:\ which I run (with previously posted question's answer, giving params to "Start in" folder on C:\Program Files\MyProgram\" so it finds its dlls.
Now after the file is running, I'd like to rewrite the file's byte stream (just like opening it in hex editor...), or at least delete it so I can copy over new exe file directly using same name.
So far the solution I'm using is that I trigger format D: command for the whole drive D:\ (which, in my case is ramdisk and thumb-drive, as I only have this exe on it, I copy it there as necessary), since that removes the file and let's me copy new file there.
Trying to use del myProgram.exe even with -force flag triggers error that access to the file is denied. Same goes if I try to overwrite the contents of the file.
Is there any alternative to do that without using the format command, as that requires to have partition drive only for the purpose?
Update: Note: MoveFileEx and similar techniques that require termination of the process or system restart/reboot are not qualified as a solution. This should be done while the process is running without further actions that can compromise the process's run state.
On a side note, when formatting the drive using the Powershell's format command, the file is gone, although if viewing the partition using Hex viewer tool, there is full binary (hex) content of the exe visible there and an be restored using just as simple as copy-paste technique. This is one of the points as to where overwriting the file contents would be preferable than deleting the file directly.
Please note: This is a knowledge and skills based question, and would therefore appreciate sparing the moral and security-concerning comments about such actions and behaviour.
For deleting/replacing/overwriting a file at least two conditions must be met:
The user performing the operation must have the required permissions to do so. This can be verified for instance via Get-Acl or icacls.
Windows must not have an open handle to the file. This can be checked for instance with tools like Process Explorer or handle. These tools can also be used to forcibly close open handles, although that's not recommended as it may cause data loss and/or damage to the files in question. I'm not sure, though, if it's actually possible to close handles to an executable without terminating the process.
Note that antivirus software is likely to interfere with this kind of operation.
The basic problem here is that Windows loads from the .EXE upon demand, it's not all read in at once.
If you destroy the original file what happens when it tries to load in a page that no longer exists?
If I had to write something of this sort I would copy the .exe to a temporary location (beware that running code from the temp directory may be prohibited), run the new .exe, terminate the old one and then do what I want to it.
I am brand new to scripts and would like to be able to write a script to copy a file (example.doc) from the source C:\Test to destination D:\Destination which checks that the copy has been successful and then deletes the original file.
A couple answers:
Research "move" (maybe, Google "batch files move"). The short documentation at the command prompt (move /?) doesn't specifically address whether it checks that the file arrived correctly before erasing the original, but it seems like a kinda obvious thing for the developers to do. There's probably someplace on the web that'll confirm whether they do that.
If that works, try "move C:\Test\example.doc D:\Destination". That'll probably do it.
Or, write a batch file, maybe mymove.bat
copy C:\Test\example.doc D:\Destination
if exist D:\Destination\example.doc del C:\Test\example.doc
You can get more elaborate: using a parameter to specify the file name, checking whether D:\Destination\example.doc and C:\Test\example.doc have the same size before you delete, using some switches to keep from getting prompted about things and similar. Maybe this'll get you started.
For all these batch file commands (move, copy, del, dir), you can get some good documentation for your particular operating system at the command prompt. Type <cmd> /? (like the move /? I mentioned above).
Batch file programming is tedious and frustrating, but it's available to all of us! Good luck.
My scenario: A text file(s) will keep coming into say a folder, I need to detect the new text file, and read particular information from it, say format being (word : info, OR word and under it a column of info, etc.). And, this process needs to keep going on always.
Problem: How should I go about doing this, I guess use perl scipt, but where to go from there ?, I am getting ideas, and also help on the internet, but I thought asking it here might make my thoughts clearer.
Kindly help, please suggest a path to do this.
Regards,
Chirayu
First thing: you want a daemon process, so you may want to have a look at Proc::Daemon.
Second thing, you need to read & parse your file. Parsing it, depends on its format, and your question is not really clear about it.
Finally, you may want to consider moving a newly detected file (or renaming it) while processing it, end then (possibly) deleting it after having processed. This depends on the requirements that you have. Alternatively, you may want to move the newly detected file into an archive directory after having processed them.
One approach might be to have a perl process that regularly (say every 5 seconds, every 5 minutes or every 5 hours, your call really) scans said directory and as soon as any new text file appears, spawn a child process that process it.
The child process might be another perl script which gets the name of the text file as it's argument and which reads the file, detects the word you mention and then extracts the information you are interested in (and then does whatever you consider necessary with that information).
Things to look out for is what to do with the text files once they are processed. Are they supposed to stay around? Then you need to keep track of which of them you have processed, so they do not get processed again in the case your master process (the one that scans the directory and spawn perl children) has to be restarted (due to either a crash or a deliberate restart).
If the text files are supposed to disappear once they are processed, then I assume it could be a good idea to either let the children remove them after completion or to let the master process remove them provided the master process always waits for the children to complete before it continues running. The drawback with a master process waiting for children to complete is that children then cannot be run in parallell but has to be run in strict sequence (not necessary a drawback depending on your situation).
(If you have a master process always waiting for the child process to run, you can actually skip having child processes altogether and create a subroutine in the master program which reads and processes the text file).
High level description but hope it helps.
What is the operating system you are using?
On Windows, you can use Win32::ChangeNotify and on Linux, you can use Linux::Inotify2 to be notified of changes to the contents of a directory.
Your script can simply wait to be notified and take action when notified instead of polling the contents of the directory which will either waste resources or potentially miss some changes.
I have implemented a log file that will be storing the cpu and memory state of a process after every minute.I have limited the maximum size of the file to 3MB (thats enough for my purpose).
The script will be called by a cron job after every minute and the script will log the details for that minute and will rename the file as "Log_.log".
When the size reaches "3MB - 100 bytes" I reset the file pointer to point to the begining and will overwrite the first entry in the log file and will now rename the file as "Log_<0+some offset>.log".
As I am renaming the file after every minute to update the file pointer position, is it a good/efficient way ?
I do not want to maintain more than one log file for this purpose.
Another option for me is to maintain the file pointer position in a file ,but ....another file !! not interested in maintaining one if this option is good :)
Thanks in Advance.
Are you an engineer? This is a nice example of some simple task, solved by a perfectly working but overly complex solution.
Unless the content you put in takes exactly as many bytes as the content you take out, writing "in" a file will actually cause the whole following part after your writing position to be rewritten to disk. Append is much cheaper.
Renaming the file to store the pointer works - but it's not very elegant, and makes stuff more complex (for one, your process needs write rights to the directory in which the file resides - else just write access to two files is sufficient)
Unless disk space is an issue (and really, it rarely is), your approach is less efficient than say, append everything to a file, and rotate the file when it reaches its maximum size. This way you always have the last 3MB of logs available, and maximum 3MB more in your current file. It will make parsing the file a lot easier too, instead of recalculating the entire pointer position thing.
Update to answer your comment:
Renaming a file every minute (or even every second) shouldn't slow down your system significantly, don't worry about that.
Our concerns are mainly with "why you think you need to rename the file". It's not better technically, it's not better from a logical point of view, it makes a lot of other (future) tasks harder. You could store the file pointer in a seperate file, or at the end of your file, and there are better^H^H^H^H^H^H simpler solutions that don't require the file pointer at all.
I'm confused why you would rename your file. What does this accomplish?
Are the log entries fixed size? Or variable size?
If the entries are fixed size, then there is no trouble in re-writing the existing file from the start: you won't ever have incomplete entries in your file, and if you are writing a counter or timestamps to the file, it should be clear where the 'cursor' is located.
If the entries are variable size, then you should probably not begin re-writing the file from the beginning without somehow making it clear where the 'cursor' is located in the file, and write code that is resilient to reading truncated log entries.
Can you re-use existing tools such as RRDtool?
I am trying to find a way of monitoring directories in Perl, in particular the size of a directory, and upon detecting a change in directory size, perform a particular action.
The issue I have is with large files that require a noticeable amount of time to copy into this directory, i.e. > 100MB. What happens (in Windows, not Unix) is the system reserves enough disk space for the entire file, even though the file is still copying in progress. This causes problems for me, because my script will try to perform an action on this file that has not finished copying over. I can easily detect directory size changes in Unix via 'du', but 'du' in Windows does not behave the same way.
Are there any accurate methods of detecting directory size changes in Perl?
Edit: Some points to clarify:
- My Perl script is only monitoring a particular directory, and upon detecting a new file or a new directory, perform an action on this new file or directory. It is not copying any files; users on the network will be copying files into the directory I am monitoring.
- The problem occurs when a new file or directory appears (copied, not moved) that is significantly large (> 100MB, but usually a couple GB) and my program fires before this copy completes
- In Unix I can easily 'du' to see that the file/directory in question is growing in size, and take the appropriate action
- In Windows the size is static, so I cannot detect this change
- opendir/readdir/closedir is not feasible, as some of the directories that appear may contain thousands of files, and I want to avoid the overhead of
Ideally I would like my program to be triggered on change, but I am not sure how to do this. As of right now it busy waits until it detects a change. The change in file/directory size is not in my control.
You seem to be working around the underlying issue rather than addressing it -- your program is not properly sending a notification when it is finished copying a file. Why not do that instead of using OS-specific mechanisms to try to indirectly determine when the operation is complete?
You can use Linux::Inotify2 or Win32::ChangeNotify to detect directory/file changes.
EDIT: File::ChangeNotify seems a better option (cross-platform & used by Catalyst)
As I understand it, you are polling a directory with thousands of files. When you see a new file, there is an action that is taken on the file. This causes problems if the file is in use or still being copied, correct?
There are potentially several solutions:
1) Use flock to detect if the file is still in use by another process (test if it works properly on your OS, file system, and Perl version).
2) Use a LockFile call on Windows. If it fails, the OS or another process is using that file.
3) Change the poll interval to a non busy time on the server and take the directory off line while your process completes.
Evaluating the size of a directory is something all but the most inexperienced Perl programmers should be able to do. You can write your own portable version of du in 15 lines of code if you know about:
Either glob or opendir / readdir / closedir to iterate through the files in a directory
The filetest operators (-f file, -d file, etc.) to distinguish between regular files and directory names
The stat function or file size operator -s file to obtain the size of a file
There is a nice module called File::Monitor, it will detect new files, deleted files, changes in size and any other attribute that can be done with stat. It will then go and out put the files for you.
http://metacpan.org/pod/File::Monitor
You set up a baseline scan, then set up a call back for each item you are looking for, so new changes you can see via
$monitor->watch( {
name => 'somedir',
recurse => 1,
callback => {
files_created => sub {
my ($name, $event, $change) = #_;
# Do stuff
}
}
} );
If you need to go deeper than one level just do it to whatever level you need. After this is done and it finds new files you can trigger you application to do what you want on the files.