Inotify: Odd behavior with directory creations - inotify

Question

I have an inotify/kernel question. I'm using the "inotify" Python project in order to make my observations, but my question is still inherently about the core inotify kernel implementation.
The Python inotify project handles recursive inotify watches. It provides a nice generator that allows you to loop over the events. It implements recursive watches by identifying directory-create events and automatically adding those watches before yielding the event.
I noticed some weird behavior with "mkdir -p" calls. Whereas I can rapidly, incrementally create individual directories and see them from the event-loop, "mkdir -p" never produces events for the subdirectory of a subdirectory or a file created in that subdirectory.
Does anyone have any thoughts?
WORKS: "mkdir aa && mkdir aa/bb && touch aa/bb/filename":
(_INOTIFY_EVENT(wd=1, mask=1073742080, cookie=0, len=16), ['IN_ISDIR', 'IN_CREATE'], '/tmp/tmpt3MlIQ', u'aa')
(_INOTIFY_EVENT(wd=2, mask=1073742080, cookie=0, len=16), ['IN_ISDIR', 'IN_CREATE'], u'/tmp/tmpt3MlIQ/aa', u'bb')
(_INOTIFY_EVENT(wd=3, mask=256, cookie=0, len=16), ['IN_CREATE'], u'/tmp/tmpt3MlIQ/aa/bb', u'filename')
(_INOTIFY_EVENT(wd=3, mask=32, cookie=0, len=16), ['IN_OPEN'], u'/tmp/tmpt3MlIQ/aa/bb', u'filename')
(_INOTIFY_EVENT(wd=3, mask=4, cookie=0, len=16), ['IN_ATTRIB'], u'/tmp/tmpt3MlIQ/aa/bb', u'filename')
(_INOTIFY_EVENT(wd=3, mask=8, cookie=0, len=16), ['IN_CLOSE_WRITE'], u'/tmp/tmpt3MlIQ/aa/bb', u'filename')
DOESN'T WORK: "mkdir -p aa/bb && touch aa/bb/filename":
(_INOTIFY_EVENT(wd=1, mask=1073742080, cookie=0, len=16), ['IN_ISDIR', 'IN_CREATE'], '/tmp/tmpuTSxYl', u'aa')
(_INOTIFY_EVENT(wd=1, mask=1073741856, cookie=0, len=16), ['IN_ISDIR', 'IN_OPEN'], '/tmp/tmpuTSxYl', u'aa')
(_INOTIFY_EVENT(wd=1, mask=1073741840, cookie=0, len=16), ['IN_ISDIR', 'IN_CLOSE_NOWRITE'], '/tmp/tmpuTSxYl', u'aa')
Naturally, I did the next obvious, brainless thing I could think of and added the "-p" flag to the "mkdir aa && mkdir aa/bb", just to make sure there wasn't any "-p"-specific anomalies, but it didn't make a difference.
The GNU implementation of "mkdir -p" just iterates from separator to separator in the path. No magic. The Python implementation of os.makedirs (same functionality) also just splits the path and enumerates the parts. However, the GNU doesn't work but the Python one does. This seems to imply a race condition, except that the results are identical no matter what I manipulate the conditions. I even started using a trivial/miniscule timeout on the epoll that we're doing to read the events (read: if there was any delay with the original timeout value then that's no longer a factory). It's almost as if inotify in the kernel seems to be totally missing the subsequent creations in "mkdir -p".
I'm sure I'm just missing something.
For reference, the calls involved in the GNU implementation:
http://code.metager.de/source/xref/gnu/coreutils/src/mkdir.c
http://code.metager.de/source/xref/gnu/octave/gnulib-hg/lib/mkdir-p.c#85
http://code.metager.de/source/xref/gnu/octave/gnulib-hg/lib/mkancesdirs.c#67
Note that we start in GNU's "coreutils" and apparently proceed into GNU Octave for the implementation of the "mkdir -p". It's the only reference that OpenGrok provided. I can't explain this and I'm in unfamiliar territory.
Python's implementation:
https://github.com/python/cpython/blob/master/Lib/os.py#L196
Am I overlooking some detail of inotify's behavior?

Answer 1

A very interesting catch you've got there!
No, the native kernel inotify library does exactly what the documentation says. GNU mkdir -p is totally fine as well.
I noticed some weird behavior with "mkdir -p" calls. Whereas I can
rapidly, incrementally create individual directories and see them from
the event-loop, "mkdir -p" never produces events for the subdirectory
of a subdirectory or a file created in that subdirectory.
You will have to try it with other inotify implementations to assert the credibility of PyInotify's recursive watch. Simply the fact that it is a popular Python implementation alone doesn't earn my trust!
I make the following statement presuming you haven't messed up the sample Python output that you have produced for reference. It is PyInotify implementation which is slacking, I would say, rather not very good at what it does.
For our speculation here, let's split the flow and start with creation of dirs before considering file creation.
Did you notice that even with your first scenario, IN_CREATE for the dir aa
(_INOTIFY_EVENT(wd=1, mask=1073742080, cookie=0, len=16), ['IN_ISDIR', 'IN_CREATE'], '/tmp/tmpt3MlIQ', u'aa')
does not have correspoding IN_OPEN and IN_CLOSE_NOWRITE events which for some odd reason seems to be present in the second scenario though the action is just mkdir?
(_INOTIFY_EVENT(wd=1, mask=1073741856, cookie=0, len=16), ['IN_ISDIR', 'IN_OPEN'], '/tmp/tmpuTSxYl', u'aa')
(_INOTIFY_EVENT(wd=1, mask=1073741840, cookie=0, len=16), ['IN_ISDIR', 'IN_CLOSE_NOWRITE'], '/tmp/tmpuTSxYl', u'aa')
There's clearly something fishy about this. Definitely inconsistent.
I have not taken a look at PyInotify implementation yet, and I don't intend to waste my time with it. However, I have worked with the native inotify interface rather closely to vouch for its accuracy! It has never missed reporting a single event, that is, if it occurs at all.
Now let's move on to the file creation part, which seems to be your major concern- mkdir -p never produces events for the subdirectory of a subdirectory or a file created in that subdirectory. This is not true always; depends on how poorly the implementation stands.
I have reproduced the same set of actions, with a better inotify implementation, that you've performed. Yes, just to prove my claims.
Notice the event flow which is clearly more accurate than PyInotify's report?
Reproduction:
case1: mkdir aa && mkdir aa/bb && touch aa/bb/filename
root#six-k:/opt/test# ls -la
total 8
drwxr-xr-x 2 root root 4096 Mar 18 13:55 .
drwxr-xr-x 20 root root 4096 Mar 18 13:53 ..
root#six-k:/opt/test# fluffyctl -w ./
root#six-k:/opt/test# mkdir aa && mkdir aa/bb && touch aa/bb/filename
events caught:
root#six-k:/home/lab/fluffy# fluffy
event: CREATE, ISDIR,
path: /opt/test/aa
event: ACCESS, ISDIR,
path: /opt/test/aa
event: ACCESS, ISDIR,
path: /opt/test/aa
event: CLOSE_NOWRITE, ISDIR,
path: /opt/test/aa
event: CREATE, ISDIR,
path: /opt/test/aa/bb
event: ACCESS, ISDIR,
path: /opt/test/aa/bb
event: ACCESS, ISDIR,
path: /opt/test/aa/bb
event: CLOSE_NOWRITE, ISDIR,
path: /opt/test/aa/bb
event: CREATE,
path: /opt/test/aa/bb/filename
event: OPEN,
path: /opt/test/aa/bb/filename
event: ATTRIB,
path: /opt/test/aa/bb/filename
event: CLOSE_WRITE,
path: /opt/test/aa/bb/filename
case 2: mkdir -p aa/bb && touch aa/bb/filename
root#six-k:/opt/test# cd ../
root#six-k:/opt# mkdir test2
root#six-k:/opt# cd test2/
root#six-k:/opt/test2# fluffyctl -w ./
root#six-k:/opt/test2# mkdir -p aa/bb && touch aa/bb/filename
root#six-k:/opt/test2#
events caught:
root#six-k:/home/lab/fluffy# fluffy
event: CREATE, ISDIR,
path: /opt/test2/aa
event: ACCESS, ISDIR,
path: /opt/test2/aa
event: ACCESS, ISDIR,
path: /opt/test2/aa/bb
event: ACCESS, ISDIR,
path: /opt/test2/aa/bb
event: CLOSE_NOWRITE, ISDIR,
path: /opt/test2/aa/bb
event: ACCESS, ISDIR,
path: /opt/test2/aa
event: CLOSE_NOWRITE, ISDIR,
path: /opt/test2/aa
event: CREATE,
path: /opt/test2/aa/bb/filename
event: OPEN,
path: /opt/test2/aa/bb/filename
event: ATTRIB,
path: /opt/test2/aa/bb/filename
event: CLOSE_WRITE,
path: /opt/test2/aa/bb/filename
There you go, events on the sub directory and the file in it.
The answer is getting lengthy!
Neverthless, no recursive implementation built on top of the native inotify library can guarantee all the events. It's not feasible! If it were, it would have been rather simple for the kernel guys who authored inotify to have introduced recursive watches natively.
Gotchas:
Notice that there is indeed a difference in the create event from my reproduction snippet? There's none reported for the sub directory in the second case(mkdir -p). Why? Though everything happens very quickly, recursive setups aren't quick enough. By the time the first create event on dir aa is caught, mkdir -p aa/bb finishes up creating dir bb as well. So, there's not create event for dir bb. Again, reminder, this not the native inotify library's fault; it's because we haven't event set up a watch on dir aa yet, how in the world are we going to receive events on it?
I hope that cleared things up!
Hold on, if the create event of dir bb wasn't even caught, how does fluffy seem to have set a watch on it and there by has reported subsequent events on dir 'bb`?
Good, you are following! You guessed right, but not entirely. fluffy received the first create event of dir aa. By the time it processed this event, mkdir -p finished up it's work, so no dir bb create event. Right. But, while fluffy setups watches on dir aa, dir bb was already present. So, fluffy, pulls bb in to it's watch because it is indeed a descendant of dir aa. The rest you already know. Since it's being watched, it reports subsequent events, which included the file creations.
Feel free to quote this answer or point to fluffy if you(anyone reading this) mean to raise a ticket/issue about this on PyInotify GitHub project page. If you need more info, I'll gladly provide. You can open an issue at fluffy's GH page for general discussions/suggestions/opinions as well. fluffy could use your help to better it.

Answer 2

It seems to me that it is tricky to catch all of the events you are hoping to get notifications for. My experience is with inotify is in C. I am sure the python package has a few things added for convenience. So, I have tailored my answer here to speak to inotify in fairly general terms.
There are no guarantees on the order in which things happen. The first directory created will trigger notification in the parent. Sometime later, the next directory gets created, and the first directory gets registered for event notification. There are two possible orders that these actions can happen.
If inotify is updated first, then you will get notified when the second directory is created.
When the other order happens, notification seems unlikely. However, one might opendir and check for entries. For each directory, update inotify to also watch that.
Inotify is a great tool for getting the initial signal to do work. But, it does have its limits. Especially with new directories being created, you need to check for gaps and races.