Inotify vim modification - inotify

I'm trying to use inotify to watch a file for being modified in an infinite loop. I'm having some problems with it:
1) I have a while(1), and the read doesn't work unless i create a new file descriptor and a new watch descriptor for every iteration of the while (what i wanted to do is to open those descriptors before the infinite loop, but if the other solution is acceptable, then i can use it). This is the version that works:
while(1){
int file_descriptor = inotify_init();
if (file_descriptor < 0) {
perror("inotify_init");
}
int watch_descriptor = inotify_add_watch(file_descriptor, "/home/user/hello.cfg", IN_CLOSE_WRITE);
....
2) I tryed using the mask IN_MODIFY, but I read that it doesn't work well with vim, so instead i use IN_CLOSE_WRITE. The problem is that when i modify the file with vim, the event is read, but the mask of the event is IN_IGNORED (Mask 0x00008000). When I use gedit, sometimes the mask of the event is IN_IGNORED, and sometimes is IN_CLOSE_WRITE (Mask 0x0000008). I would like to know why i get IN_IGNORED if i'm modifying the file, and why the event is not IN_CLOSE_WRITE. Is there another way to watch for modifications of a single file? Is IN_CLOSE_WRITE the correct mask?

I found the same behavior (kernel 3.14.4) while writing a daemon that automatically reloads its configuration file if changed while running. What I found was that some programs (vim among them) cause inotify to generate IN_IGNORED because they delete the original file and replace it with a "swap" file.
From the inotify documentation,
IN_IGNORED
Watch was removed explicitly (inotify_rm_watch(2)) or
automatically (file was deleted, or filesystem was
unmounted). See also BUGS.
You can see this by running
strace -o log vi myfile.txt
and examining the strace result in log, which will show creation and deletion of .myfile.txt.swp.
The solution is to catch the IN_IGNORED event and re-add the watch for your file, i.e.
int watch_descriptor = inotify_add_watch(file_descriptor, "/home/user/hello.cfg", (IN_CLOSE_WRITE | IN_IGNORED));
IN_CLOSE_WRITE is the correct event to handle, but there is a gotcha with it as well. Some programs open a file for reading and writing, but only read from it. When they close the file, IN_CLOSE_WRITE is generated even though the file was not modified.
I found the best approach was to maintain a flag for every file I am watching and toggle it when I catch IN_MODIFY. Then, when IN_CLOSE_WRITE is generated, the flag can be examined to decide if IN_CLOSE_WRITE was meaningful or not.

Related

Make ExifTool overwrite_original is transactional and atomic?

I'm using ExifTool to add keywords to a file. I do this in batch, spawning ExifTool as a child process from my app.
exiftool -keywords+="courgettes" -keywords+="edible gardening" -keywords+="vegetables" -keywords+="Glasshouse" -keywords+="Greenhouse" -overwrite_original "/pictures/garden/P8151927.JPG"
However, it seems that with -overwrite_original the process isn't 100% transactional and atomic; occasionally it fails for some reason or other, but instead of leaving the original file untouched, it leaves it renamed to P8151927.JPG_exiftool_tmp. This means it looks like the original image has gone missing.
Is there any way to guarantee that if the ExifTool update fails, the original won't be touched? My understanding was that it would create a temp file with the new/changed keywords, and then carry out a transactional delete/rename which would either work or not, but it seems the delete can succeed, but the rename from the temp file can then fail.
Is there any way to work around this - any additional options I can use to make it 100% resilient? If not, my only alternative is to copy the file to a temp location, run exiftool on it, and then handle the rename/replace myself. Alternatively, I could check the return code, and if non-zero, I can rename the .JPG_exiftool_tmp file back to the original, I suppose.
Any better ideas?

Force overwrite or delete file in use (executable that currently runs)

I'm looking for solution to delete or (preferably directly) overwrite source of an exe file while it is running.
To explain further before you get it all wrong, I'll give an example:
I have an exe file on drive D:\ which I run (with previously posted question's answer, giving params to "Start in" folder on C:\Program Files\MyProgram\" so it finds its dlls.
Now after the file is running, I'd like to rewrite the file's byte stream (just like opening it in hex editor...), or at least delete it so I can copy over new exe file directly using same name.
So far the solution I'm using is that I trigger format D: command for the whole drive D:\ (which, in my case is ramdisk and thumb-drive, as I only have this exe on it, I copy it there as necessary), since that removes the file and let's me copy new file there.
Trying to use del myProgram.exe even with -force flag triggers error that access to the file is denied. Same goes if I try to overwrite the contents of the file.
Is there any alternative to do that without using the format command, as that requires to have partition drive only for the purpose?
Update: Note: MoveFileEx and similar techniques that require termination of the process or system restart/reboot are not qualified as a solution. This should be done while the process is running without further actions that can compromise the process's run state.
On a side note, when formatting the drive using the Powershell's format command, the file is gone, although if viewing the partition using Hex viewer tool, there is full binary (hex) content of the exe visible there and an be restored using just as simple as copy-paste technique. This is one of the points as to where overwriting the file contents would be preferable than deleting the file directly.
Please note: This is a knowledge and skills based question, and would therefore appreciate sparing the moral and security-concerning comments about such actions and behaviour.
For deleting/replacing/overwriting a file at least two conditions must be met:
The user performing the operation must have the required permissions to do so. This can be verified for instance via Get-Acl or icacls.
Windows must not have an open handle to the file. This can be checked for instance with tools like Process Explorer or handle. These tools can also be used to forcibly close open handles, although that's not recommended as it may cause data loss and/or damage to the files in question. I'm not sure, though, if it's actually possible to close handles to an executable without terminating the process.
Note that antivirus software is likely to interfere with this kind of operation.
The basic problem here is that Windows loads from the .EXE upon demand, it's not all read in at once.
If you destroy the original file what happens when it tries to load in a page that no longer exists?
If I had to write something of this sort I would copy the .exe to a temporary location (beware that running code from the temp directory may be prohibited), run the new .exe, terminate the old one and then do what I want to it.

How to have Autohotkey 'listen' for a change to a file?

I have a text file, lets call it C:\to_run.txt. I'd like my Autohotkey script to 'listen' to this file in such a way that when it detects a change, it performs an action immediately based on the contents of the file, and then make the file blank again.
I can handle the last parts, so really I'm asking for an efficient way to detect file changes? I say efficient because my Autohotkey script is getting rather long and I don't want this listening function to hang up the rest of the script in any way.
Assuming we are really talking of only one file to check on:
Surely not as beautiful as Sidola's answer, but without the need for external libraries:
#persistent
lastFileContent := ""
setTimer, checkFile, 20
return
checkFile:
fileread newFileContent, changingDocument.txt
if(newFileContent != lastFileContent) {
lastFileContent := newFileContent
msgbox, content changed to: %newFileContent%
}
return
In this case, for checking on larger files, it might be better to compare MD5-checksums instead of the whole file content.
Note: I have not tested the performance implications on this. This script opens up the file 50 times per second, could be pretty hard drive consuming.
Check out WatchDirectory()
Just make sure you're running the latest version of AHK.
To get it up and running, first download these three scripts and save them to your /lib folder.
WatchDirectory.ahk
_Struct.ahk
sizeof.ahk
After that, simply point to to wherever you want to look, provide a callback function and if you want, a third param to watch for specific changes. Refer to this forum post for full documentation.
If you drop this script in its own folder, save it and run it, then save it again, it should detect changes to that script file.
#Persistent
WatchDirectory(A_ScriptDir "\|.ahk\", "Callback", 0x10)
return
Callback(param1, param2) {
msgBox % param1 "`n" param2
}
Note however, it will fire twice whenever the file is changed. This seems to be a Windows behaviour from what I can gather.

ExtAudioFileSeek and ExtAudioFileWrite together on the same file

I have a situation where I can save a post-processing pass through the audio by taking some manipulated buffer from the end of the track and writing them to the beginning of my output file.
I originally thought I could do this by resetting the write pointer using ExtAudioFileSeek, and was about to implement it when I saw this line in the docs
Ensure that the file you are seeking in is open for reading only. This function’s behavior with files open for writing is undefined.
Now I know I could close the file for writing then reopen it, but the process is a little more complicated than that. Part of the manipulation I am doing is reading from buffers that are in the file I am writing to. The overall process looks like this:
Read buffers from the end of the read file
Read buffers from the beginning of the write file
Process the buffers
Write the buffers back to the beginning of the write file, overwriting the buffers I read in step 2
Logically, this can be done in 1 pass no problem. Programmatically, how can I achieve the same thing without corrupting my data, becoming less-efficient (opposite of my goal) or potentially imploding the universe?
Yes, using a single audio file for both reading and writing may, as you put it, implode the universe, or at least lead to other nastiness. I think that the key to solving this problem is in step 4, where you should write the output to a new file instead of trying to "recycle" the initial write file. After your processing is complete, you can simply scrap the intermediate write file.
Or have I misunderstood the problem?
Oh, and also, you should use ExtAudioFileWriteAsync instead of ExtAudioFileWrite for your writes if you are doing this in realtime. Otherwise the I/O load will cause audio dropouts.

How to detect changing directory size in Perl

I am trying to find a way of monitoring directories in Perl, in particular the size of a directory, and upon detecting a change in directory size, perform a particular action.
The issue I have is with large files that require a noticeable amount of time to copy into this directory, i.e. > 100MB. What happens (in Windows, not Unix) is the system reserves enough disk space for the entire file, even though the file is still copying in progress. This causes problems for me, because my script will try to perform an action on this file that has not finished copying over. I can easily detect directory size changes in Unix via 'du', but 'du' in Windows does not behave the same way.
Are there any accurate methods of detecting directory size changes in Perl?
Edit: Some points to clarify:
- My Perl script is only monitoring a particular directory, and upon detecting a new file or a new directory, perform an action on this new file or directory. It is not copying any files; users on the network will be copying files into the directory I am monitoring.
- The problem occurs when a new file or directory appears (copied, not moved) that is significantly large (> 100MB, but usually a couple GB) and my program fires before this copy completes
- In Unix I can easily 'du' to see that the file/directory in question is growing in size, and take the appropriate action
- In Windows the size is static, so I cannot detect this change
- opendir/readdir/closedir is not feasible, as some of the directories that appear may contain thousands of files, and I want to avoid the overhead of
Ideally I would like my program to be triggered on change, but I am not sure how to do this. As of right now it busy waits until it detects a change. The change in file/directory size is not in my control.
You seem to be working around the underlying issue rather than addressing it -- your program is not properly sending a notification when it is finished copying a file. Why not do that instead of using OS-specific mechanisms to try to indirectly determine when the operation is complete?
You can use Linux::Inotify2 or Win32::ChangeNotify to detect directory/file changes.
EDIT: File::ChangeNotify seems a better option (cross-platform & used by Catalyst)
As I understand it, you are polling a directory with thousands of files. When you see a new file, there is an action that is taken on the file. This causes problems if the file is in use or still being copied, correct?
There are potentially several solutions:
1) Use flock to detect if the file is still in use by another process (test if it works properly on your OS, file system, and Perl version).
2) Use a LockFile call on Windows. If it fails, the OS or another process is using that file.
3) Change the poll interval to a non busy time on the server and take the directory off line while your process completes.
Evaluating the size of a directory is something all but the most inexperienced Perl programmers should be able to do. You can write your own portable version of du in 15 lines of code if you know about:
Either glob or opendir / readdir / closedir to iterate through the files in a directory
The filetest operators (-f file, -d file, etc.) to distinguish between regular files and directory names
The stat function or file size operator -s file to obtain the size of a file
There is a nice module called File::Monitor, it will detect new files, deleted files, changes in size and any other attribute that can be done with stat. It will then go and out put the files for you.
http://metacpan.org/pod/File::Monitor
You set up a baseline scan, then set up a call back for each item you are looking for, so new changes you can see via
$monitor->watch( {
name => 'somedir',
recurse => 1,
callback => {
files_created => sub {
my ($name, $event, $change) = #_;
# Do stuff
}
}
} );
If you need to go deeper than one level just do it to whatever level you need. After this is done and it finds new files you can trigger you application to do what you want on the files.