I am trying to find a way to monitor the contents of a directory for changes. I have tried two approaches.
Use kqueue to monitor the directory
Use GCD to monitor the directory
The problem I am encountering is that I can't find a way to detect which file has changed. I am attempting to monitor a directory with potentially thousands of files in it and I do not want to call stat on every one of them to find out which ones changed. I also do not want to set up a separate dispatch source for every file in that directory. Is this currently possible?
Note: I have documented my experiments monitoring files with kqueue and GCD
My advice is to just bite the bullet and do a directory scan in another thread, even if you're talking about thousands of files. But if you insist, here's the answer:
There's no way to do this without rolling up your sleeves and going kernel-diving.
Your first option is to use the FSEvents framework, which sends out notifications when a file is created, edited or deleted (as well as things to do with attributes). Overview is here, and someone wrote an Objective C wrapper around the API, although I haven't tried it. But the overview doesn't mention the part about events surrounding file changes, just directories (like with kqueue). I ended up using the code from here along with the header file here to compile my own logger which I could use to get events at the individual file level. You'd have to write some code in your app to run the logger in the background and monitor it.
Alternatively, take a look at the "fs_usage" command, which constantly monitors all filesystem activity (and I do mean all). This comes with Darwin already, so you don't have to compile it yourself. You can use kqueue to listen for directory changes, while at the same time monitoring the output from "fs_usage". If you get a notification from kqueue that a directory has changed, you can look at the output from fs_usage, see which files were written to, and check the filenames against the directory that was modified. fs_usage is a firehose, so be prepared to use some options along with Grep to tame it.
To make things more fun, both your FSEvents logger and fs_usage require root access, so you'll have to get authorization from the user before you can use them in your OS X app (check out the Authorization Services Programming Guide for info on how to do it).
If this all sounds horribly complicated, that's because it is. But at least you didn't have to find out the hard way!
Related
I'd like to get an indication about the context in which my process is running from. I'd like to distinguish between the following cases :
It runs as a persistent scheduled task (launchDaemon/launchAgent)
It was called on-demand and created by launchd using open command-line or double-click.
It was called directly from command-line terminal (i.e. > /bin/myProg from terminal )
Perhaps is there any indication about the process context using Objective-c/swift framework or any other way ? I wish to avoid inventing the wheel here :-)
thanks
There is definetely no simple public API or framework for doing this, and doing this is hard.
Some parts of this info possibly could be retreived by your process itslef with some side-ways which will work on some system versions:
There is a launchctl C-based API, which you can try to use to enumerate all
launch daemon/agent tasks and search for your app path/pid. You may
require a root rights for your process for doing this.
Using open command-line sometimes could be traced with environment
variables it sets for your process.
Running directly from command-line could leave responsible_pid filled correctly (which is private API from libquarantine, unless you are observing it with Endpoint Security starting from 11.smth version)
All this things, except launchctl API, are not public, not reliable, could be broken at any time by Apple, and may be not sufficient for your needs.
But it is worth to take them a try, because there is nothing better :)
You could potentially distinguish all cases you want using system events monitoring from some other (root-permitted) process you control, possibly adopting Endpoint Security Framework (requires an entitlement from Apple, can't be distributed via AppStore), calling a lot of private APIs and a doing bunch of reversing tricks.
The open resource I could suggest on this topic is here
I want to open two command line prompts (I am using CMDer) from the same directory and run different commands at the same time.
Would those two commands interrupt each other?
One is for compiling a web application I am building (takes like 7 minutes to compile), and the other is to see the history of the commands I ran (this one should be done quickly).
Thank you!
Assuming that CMDer does nothing else than to issue the same commands to the operating system as a standard cmd.exe console would do, then the answer is a clear "Yes, they do interfere, but it depends" :D
Break down:
The first part "opening multiple consoles" is certainly possible. You can open up N console windows and in each of them switch to the same directory without any problems (Except maybe RAM restrictions).
The second part "run commands which do or do not interfere" is the tricky part. If your idea is that a console window presents you something like an isolated environment where you can do things as you like and if you close the window everything is back to normal as if you never ever touched anything (think of a virtual machine snapshot which is lost/reverted when closing the VM) - then the answer is: This is not the case. There will be cross console effects observable.
Think about deleting a file in one console window and then opening this file in a second console window: It would not be very intuitive if the file would not have been vanished in the second console window as well.
However, sometimes there are delays until changes to the file system are visible to another console window. It could be, that you delete the file in one console and make a dir where the file is sitting in another console and still see that file in the listing. But if you try to access it, the operating system will certainly quit with an error message of the kind "File not found".
Generally you should consider a console window to be a "View" on your system. If you do something in one window, the effect will be present in the other, because you changed the underlying system which exists only once (the system is the "Model" - as in "Model-View-Controller Design Pattern" you may have heard of).
An exception to this might be changes to the environment variables. These are copied from the current state when a console window is started. And if you change the value of such a variable, the other console windows will stay unaffected.
So, in your scenario, if you let a build/compile operation run and during this process some files on your file system are created, read (locked), altered or deleted then this would be a possible conflicting situation if the other console window tries to access the same files. It will be a so called "race condition", that is, a non-deterministic process, which state of a file will be actual to the second console window (or both, if the second one also changes files which the first one wants to work with).
If there is no interference on a file level (reading the same files is allowed, writing to the same file is not), then there should be no problem of letting both tasks run at the same time.
However, on a very detailed view, both processes would interfere in that they need the same limited but vastly available CPU and RAM resources of your system. This should not pose any problems with the todays PC computing power, considering features like X separate cores, 16GB of RAM, Terabytes of hard drive storage or fast SSDs, and so on.
Unless there is a very demanding, highly parallelizable, high priority task to be considered, which eats up 98% CPU time, for example. Then there might be a considerable slow down impact on other processes.
Normally, the operating system's scheduler does a good job on giving each user-process enough CPU time to finish as quickly as possible, while still presenting a responsive mouse cursor, playing some music in the background, allowing a Chrome running with more than 2 tabs ;) and uploading the newest telemetry data to some servers on the internet, all at the same time.
There are techniques which make it possible that a file is available as certain snapshots to a given timestamp. The key word would be "Shadow Copy" under Windows. Without going into details, this technique allows for example defragmenting a file while it is being edited in some application or a backup could copy a (large) file while a delete operation is run at the same file. The operating system ensures that the access time is considered when a process requests access to a file. So the OS could let the backup finish first, until it schedules the delete operation to run, since this was started after the backup (in this example) or could do even more sophisticated things to present a synchronized file system state, even if it is actually changing at the moment.
I just want to get some sort of notification in my perl script when particular file is modified.
Is there any other way rather than just polling it constantly?
Win32::ChangeNotify, Linux::Inotify2
AnyEvent::Filesys::Notify: Check out the section which says "Why Another Module For File System Notifications".
Excerpt:
AnyEvent::Filesys::Notify exists because I need a way to simply tie
the functionality those modules provide into an event framework.
Neither of the existing modules seem to work with well with an event
loop. Filesys::Notify::Simple does not supply a non-blocking interface
and File::ChangeNotify requires you to poll an method for new events.
I have a perl app which processes text files from the local filesystem (think about it as an overly-complicated grep).
I want to design a webapp which allows remote users to invoke the perl app by setting the required parameters.
Once it's running it would be desirable some sort of communication between the perl app and the webapp about the status of the process (running, % done, finished).
Which would be a recommended way of communication between the two processes? I was thinking in a database table, but I'm not really sure it's a good idea.
any suggestions are appreciated.
Stackers, go ahead and edit this answer to add code examples or links to them.
DrNoone, two approaches come to mind.
callback
Your greppy app needs to offer a callback function that returns the status and which is periodically called by the Web app.
event
This makes sense if you are already using a Web server/app framework which exposes an event loop usable from external applications (rather unlikely in Perl land). The greppy app fires events on status changes and the Web app attaches/listens to them and acts accordingly.
For IPC as you envision it, a plain database is not so suitable. Look into message queues instead. For great interop, pick AMPQ compliant implementation.
If you run the process using open($handle, "cmd |") you can read the results in real time and print them straight to STDOUT while your response is open. That's probably the simplest approach.
Suppose two asynchronous operations try to read or write in the same file on a given folder. How does iOS deals with that? Is the operations queued? The app crashes? the file get corrupted?
In case it is up to the programmer to deal with this, where do I find informations or an example on how to deal with race conditions, queues, etc.?
Can you guys give me a hint?
thanks.
I think that when trying to acquire the file descriptor for your file, it should fail if it is already in use.
As there is apparently no way to know if the file is already opened, there might be a way to lock a file.
Links that may help you:
http://developer.apple.com/library/ios/#documentation/Cocoa/Conceptual/LowLevelFileMgmt/Introduction.html
http://developer.apple.com/library/ios/#documentation/Cocoa/Reference/Foundation/Classes/NSFileManager_Class/Reference/Reference.html
http://developer.apple.com/library/ios/#documentation/Cocoa/Reference/Foundation/Classes/NSFileHandle_Class/Reference/Reference.html
http://developer.apple.com/library/ios/#documentation/Performance/Conceptual/FileSystem/FileSystem.html
http://developer.apple.com/library/ios/#documentation/General/Conceptual/ConcurrencyProgrammingGuide/Glossary/Glossary.html
http://developer.apple.com/library/ios/#documentation/CoreFoundation/Reference/CFFileDescriptorRef/Reference/reference.html
The most likely answer is "the file gets corrupted". The OS will not do anything to prevent multiple threads from writing to a file simultaneously, and if they're writing to overlapping sections of the file, all bets are off.
So, your best bet is to enforce your own queueing on access to the file. There are lots of ways to to that. At a low level, you can use an NSLock to prevent simultaneous access, or you might want to use a Dispatch Queue. The Concurrency Programming Guide mentioned by Julio's answer is a good reference.