Do the methods in Apache's FileUtils perform synchronous (blocking) i/o?
I am making a call to FileUtils.copyDirectoryToDirectory. In my next line, I want to delete the directory that I copied.
Example:
FileUtils.copyDirectoryToDirectory(source, destination);
FileUtils.deleteDirectory(source);
Just want to make sure this is "safe" and asynchronous (non-blocking) i/o isn't happening.
Thanks.
Two things:
FileUtils is not part of the standard JDK, it a class in the Apache Commons IO library.
The operations you mentioned do not use non-blocking IO.
So to answer your question, yes, your overall operation is safe.
Related
Target:
I would like to see Akka's file IO to be as asynchronous and non-blocking as possible.
My status of knowledge so far:
In older project documentation you can read this:
Note
Since the current version of Akka (2.3.x) needs to support JDK6, the
currently provided File IO implementations are not able to utilise
Asynchronous File IO operations, as these were introduced in JDK7 (and
newer). Once Akka is free to require JDK8 (from 2.4.x) these
implementations will be updated to make use of the new NIO APIs (i.e.
AsynchronousFileChannel).
The current akka version is '2.5.4'. The current version of akka-stream is '2.11' or '2.12'. In the current documentation the note from abhove is missing and it is only explicitly mentioned that file IO means blocking operations which are processed by a dispatcher dedicated to IO operations.
In the 'MANIFEST.MF' file inside the akka-streams Jar-File there is a line:
Require-Capability: osgi.ee;filter:="(&(osgi.ee=JavaSE)(version=1.8))"
So I guess it requires Java 8.
There is a question related to scala asynchronous file IO but its from January 2015. One of the answers contains:
Akka IO, while not providing file IO in it's core, has a module
developed by Dario Rexin, which allows to use AsynchronousFileChannel
with Akka IO in a very simple manner. Have a look at this library to
make use of it: https://github.com/drexin/akka-io-file
Questions:
How asynchronous is the current state of akka streams file IO?
Does akka streams file IO use 'AsynchronousFileChannel' from Java's NIO?
Do I have to do something to use 'AsynchronousFileChannel' from Java's NIO?
How asynchronous is the current state of akka streams file IO?
Perusing the source code shows that Akka Stream's FileIO uses java.nio.ByteBuffer and java.nio.channels.FileChannel. And as the documentation states, the file IO operations run in isolation on a dedicated dispatcher.
There is an open pull request that attempts to use AsynchronousFileChannel. Based on the benchmark results reported in that PR, the PR might be closed in favor of trying a newer approach with synchronous NIO as captured in another PR.
Does akka streams file IO use 'AsynchronousFileChannel' from Java's NIO?
No.
Do I have to do something to use 'AsynchronousFileChannel' from Java's NIO?
This question is moot, because Akka Streams does not use AsynchronousFileChannel.
I want to add an RPC service to my unix daemon. The daemon is written in C and has an event driven loop implemented using select(). I've looked at a number of RPC implementations but they all seem to involve calling a library routine, or auto generated code, which blocks indefinitely.
Are there any RPC frameworks out there where the library code/autogenerated code doesn't block or start threads. Ideally, I'd like to create the input/output sockets myself and pass them into my select loop.
Regards,
Alex - first time poster! :-)
I'm assuming that you can use C++ Apache Thrift is good - FAST RPC is also useful.
I evaluated a fair few libraries at the start of 2012 and eventually ended up going with ZeroMQ as it was more adaptable and (I found it) easier and a lot more flexible. I did consider using a Google protobuf implementation but ended up using a simpler structured command text approach.
I probably wouldn't consider doing this in C unless I had to, in which case I'd probably start with the standard rpc(3) stuff, for a good overview see this overview of Remote Procedure Calls (RPC).
I've been writing some code that replaces some existing:
while(runEventLoop){
if(select(openSockets, readFDS, writeFDS, errFDS, timeout) > 0){
// check file descriptors for activity and dispatch events based on same
}
}
socket reading code. I'd like to change this to use a GCD queue, so that I can pop events on to the queue using dispatch_async instead of maintaining a "must be called on next iteration" array. I also am already using a GCD queue to /contain/ this particular action, hence wanting to devolve it to a more natural GCD dispatch form. ( not a while() loop monopolizing a serial queue )
However, when I tried to refactor this into a form that relied on dispatch sources fired from event handlers tied to DISPATCH_SOURCE_TYPE_READ and DISPATCH_SOURCE_TYPE_WRITE on the socket descriptors, the library code that depended on this scheduling stopped working. My first assumption is that I'm misunderstanding the use of DISPATCH_SOURCE_TYPE_READ and DISPATCH_SOURCE_TYPE_WRITE - I had assumed that they would yield roughly the same behavior as calling select() with those socket descriptors.
Do I misunderstand GCD dispatch sources? Or, regarding the refactor, am I using it in a situation where it is not best suited?
The short answer to your question is: none. There are no differences, both GCD dispatch sources and select() do the same thing: they notify the user that a specific kernel event happened or that a particular condition holds true.
Note that, on a mac or iOS device you should not use select(), but rather the more advanced kqueue() and kevent() (or kevent64()).
You may certainly convert the code to use GCD dispatch sources, but you need to be careful not to break other code relying on this. So, this needs a complete inspection of the whole code handling signals, file descriptors, socket and all of the other low level kernel events.
May be a simpler solution could be to maintain the original code, simply adding GCD code in the part that react to events. Here, you dispatch events on different queues depending on the particular type of event.
Is inet_aton Thread-Safe? I know according to UNP that POSIX doesn't require a lot of the Sockets API to be thread safe, and so I have to assume they're not, but in general how do I know if something is thread safe in Perl? To what extent do I need to lock library function that I call? And how do I lock them? When I try something like lock(&inet_aton) it gives me an error: Can't modify non-lvalue subroutine call in lock.
And yes, I've read: Thread-Safety of System Libraries
If you read the inet_aton manpage carefully you will see that this call does not use any shared state (contrary to the inet_ntoa function described in the same manpage), and thus should be thread safe.
That the function writes its result into a caller-provided structure also supports this.
Perl uses a thin wrapper on top of those functions and thus doesn't change the thread safety of the underlying library.
The function inet_aton doesn't have any state it keeps between function calls, so I don't see any reason why it wouldn't be thread safe (provided the arguments you pass it aren't shared between threads).
I'd like to use POSIX semaphores to manage atomic get and put from a file representing a queue. I want the flexibility of having something named in the filesystem, so that completely unrelated processes can share a queue. I think this plan rules out pthreads. The named posix semaphores are great for putting something in the filesystem that any process can see, but I can't find the standard CondWait primitive:
... decide we have to wait ....
CondWait(sem, cond);
When CondWait is called by a process it atomically posts to sem and waits on cond. When some other process posts to cond, the waiting process wakes up only if it can atomically decrement sem as well. The alternative of
... decide we have to wait ....
sem_post(sem);
sem_wait(cond);
sem_wait(sem);
is subject to a race condition in which some other process signals cond just before this process waits on it.
I hardly ever do any concurrent programming, so I thought I would ask SO: if I use a standard POSIX counting semaphore for the condition variable, is it possible that this race is benign?
Just in case anybody wants the larger context, I am building get and put operations for an atomic queue that can be called from shell scripts.
Since there are no other answers I will follow up with what I've learned:
Pthreads will not work with my application because I have processes without a common ancestor which need to share an atomic queue.
Posix semaphores are subject to the wakeup-waiting race, but because unlike classic condition variables they are counting semaphores, the race is benign. I don't have a proof of this claim but I have had a system running for two days now and working well. (Completely meaningless I know, but at least it meant I got the job done.)
Named Posix semaphores are difficult to garbage-collect from the filesystem.
To summarize, named Posix semaphores turned out to be a good basis for implementing an atomic queue abstraction to be shared among unrelated processes.
I would like to have a proof or a validated SPIN model, but as my need for the application is limited, it seems unlikely that I will write one. I hope this helps someone else who may want to use Posix semaphores.
According to the POSIX standard, the set of semaphore routines is:
sem_close()
sem_destroy()
sem_getvalue()
sem_init()
sem_open()
sem_post()
sem_timedwait()
sem_trywait()
sem_unlink()
sem_wait()
The sem_trywait() and sem_timedwait() functions might be what you are looking for.
I know this question is old, but the obvious solution would be to just use process-shared mutexes and condition variables located in a file you can mmap.
You are looking for: pthread_cond_wait, pthread_cond_signal, I think.
That's if you are using posix threads, then the pthread methods would supply the functionality of CondWait and Signal.
Look here for source code on multiprocess pthreads via shared memory.
http://linux.die.net/man/3/pthread_mutexattr_init
That's for Linux, but the documents are posix. They're similar to Solaris, but you'll want to peruse the man pages on your OS.