Should named pipes opened with mkfifo be closed and how? - matlab

I am using a named pipe to capture the output of an external program (wgrib2) within another program (MATLAB). The MATLAB code is below, and system() accesses the command line to make the pipe:
system('mkfifo myfifo'); % Make a named pipe myfifo
% Call the external program wgrib2 and dump its output to the named pipe myfifo
system('wgrib2.exe multi_1.glo_30m.hs.201212.grb2 -ij 1 165 -ij 1 166 > myfifo &');
fid = fopen('myfifo', 'r'); % Open the named pipe
a = fscanf(fid, '%c'); % Read the output as character
fclose(fid); % Close the "file" (myfifo still exists afterward)
Here are my questions:
Do I have to close the named pipe myfifo after I use it? It seems to persist after the code is run.
If myfifo needs to be closed, what is the command to close it?
I will be running the code sample above many times (>1000), so is it OK if I reuse the named pipe and do not close it until the end?

No. Unix treats everything like a file. Named pipes are not different. If you’re done using it, you probably want to close it so you don’t clutter up your machine with named pipes, but you don’t need to close it.
Edited to reflect below comment, which is correct. Deleting != closing.
You close the named pipe the same way you close any file:
fclose(mFifo)
As mentioned in the accepted answer, closing will not delete the fifo. You may need to do that separately.
There’s nothing wrong with re-using a named pipe. It’s up to you, however, to know when you’re done reading/writing to it for each iteration. Once all the data has been read out of the pipe, you’re free to use it again as many times as you want.

You probably confused closing with the opposite of mkfifo command.
An accepted answer is definitely the best solution for MATLAB users, but I'd like to clear things out for those who came here for named pipes.
On Unix-likes named pipe (FIFO) is a special type of file with no content. The mkfifo command creates the pipe on a file system (assigns a name to it), but doesn't open it. You need to open and close it separately like any other file.
Do I have to close the named pipe myfifo after I use it? It seems to persist after the code is run.
It's generally a good idea to close/delete/free things up as soon as you don't need them anymore.
The pipe itself (and its content) gets destroyed when all descriptors to it are closed. What you see left is just a name.
If myfifo needs to be closed, what is the command to close it?
Named pipe can be closed with fclose() function. To make the pipe anonymous and unavailable under the given name (can be done when the pipe is still open) you could use the MATLAB's delete function or the rm console command.
I will be running the code sample above many times (>1000), so is it OK if I reuse the named pipe and do not close it until the end?
It's OK to reuse a named pipe as long as each iteration starts with an empty pipe (according to MATLAB documentation, fscanf() function will do this for you).
A named pipe can be reused in two ways:
By reusing the pipe itself (without closing it):
system('mkfifo myfifo');
tmp = fopen('myfifo', 'r+'); % Open the pipe in both ways (otherwise it will block)
fid = fopen('myfifo', 'r'); % Open the pipe for reading (otherwise `fscanf` will block)
fclose(tmp); % Close the auxiliary descriptor
% Use the pipe multiple times...
system('wgrib2.exe multi_1.glo_30m.hs.201212.grb2 -ij 1 165 -ij 1 166 > myfifo');
a = fscanf(fid, '%c');
...
% Close and delete the pipe
fclose(fid);
delete myfifo;
By reusing the name to open a pipe (the way you were using it).

You probably shouldn't use a named pipe for capturing system output in MATLAB. Per MATLAB's system() documentation, you can capture output directly in the system() call:
[status, cmdout] = system('wgrib2.exe multi_1.glo_30m.hs.201212.grb2 -ij 1 165 -ij 1 166');
a = cmdout
However, if you insist on using a named pipe, then Yes, you should close it. You should always close resources you open. But closing a named pipe does not delete it.

Cannot agree. You close fifo by closing it. The system closes fifo when it is finished (the system() call above). When the fifo is closed the other side knows that there's no more data (EOF condition, there is no explicit EOF check in the example code but it is very probable in the real one) and finishes.

Related

How to write a test to confirm that a perl script waits (or doesn't wait) for interactive user input (based on supplied options)

I have a perl script for which I would like to write a few tests. If the perl script gets supplied a specific option, the user is allowed to paste or type out multi-line input and end their input using control-d.
These are the tests I want to write:
The script (when an interactive flag is supplied) waits for (multiple lines of) input on STDIN until the user hits control-d
[this test is already implemented - but provided for completeness] The script (when a flag is supplied that indicates a redirect/pipe) consumes all input on STDIN and does not wait for control-d
The script (when no input flag is provided [interactive or redirect]) does not wait for interactive user input
Test 1 toy example
I wrote a test for test 1 that confirms input was received (it works by the parent test script printing to the child's input handle and the child modifies and prints that input back out), but that test doesn't wait for an end of input signal (e.g. control-d) (which I don't know how to send anyway). So in other words, I can confirm it receives input, but I don't know how to confirm that it waits for the user to intentionally end the input entry. I.e. How do I know that the input consumption won't stop until the user types control-d?
Here's what I have so far for test 1. I wrote a 3rd little IO::Pipe::Consumer module to be able to send input to the child process that I'm testing and I wrote a toy example of the script that allows input on STDIN from a tty.
Here is a toy version of the script I'm testing:
>perl -e 'while(<STDIN>){print("*$_")}'
test
*test
^d
>
And here is the toy test code (that I want to improve) for the above script:
>perl -e '
use IO::Pipe::Consumer;
$obj = new IO::Pipe::Consumer;
$si = $obj->getSubroutineConsumer(
sub { while(<STDIN>) print("*$_") } }
);
print $si "test\n"
'
*test
>
I thought the parent would have to print an EOF (e.g. like what you get from "control-d") to end the input in the test, but the test ends immediately even though I'm not sending any such end-of-input character. I can see that it's getting, modifying, and printing the input. Is that sufficient to confirm that the script will wait for user input (and that the user will be able to intentionally end the input) or is there something else I should do to confirm it waits for all user input until the user intends to end it?
Test 2 - done
Test 3 toy - don't know how to write it...
Even if modified input spit back out is sufficient proof of "waiting for input" for test 1, I also wish to test that a script doesn't consume input on STDIN when no input option (interactive or redirect) is provided - but since it doesn't seem to wait even when I do send it input without an end-of-input signal, how would I test that the script wouldn't hang waiting for input? Note, the script has other options for consuming redirected or piped input, so my intent is specifically to know if it's waiting on input from the tty. All of the STDIN consumption options (whether from the tty or via redirect/pipe) are optional, which is why I want to write these tests.
My manual testing shows everything works as intended. I would just like some tests to assure that behavior for the future.
IO::Pipe::Consumer
I feel like the thing I'm missing is not relevant to IO::Pipe::Consumer, so WRT that, I'll just describe it instead of paste in 30 or so lines of code... All it does is it sets a pipe to the child's STDIN and gives that handle back to the parent for it to print to. I haven't put it on cpan. I'm just experimenting to see if I can use it to write these tests.
IO::Pipe::Consumer is basically the opposite of IO::Pipe::Producer (a module I published on cpan looong ago, say 2001-ish, when I was new to perl, or programming for that matter). The main difference, aside from swapping STDIN for STDOUT and Reader with Writer (and vice versa), is that the open is open(STDIN,"<",\${$stdin_pipe}).
I thought the parent would have to print an "end-of-input" (e.g. "control-d") character to end the input in the test,
Ctrl-D doesn't produce an "end of input character"; it causes the terminal to return EOF.
I don't know what IO::Pipe::Consumer is —it's not on CPAN— but I presume it creates a pipe. Exiting the program causes the writer end of the pipe to be closed and thus cause the reader end to return EOF.
is there something else I should do to confirm it waits for all user input until the user intends to end it?
<> reads until one of the following things happen:
A line feed is encountered (returning what was read including the line feed)
EOF is encountered (returning what was read up to an including the line feed)
An error is encountered (returning false).
You can confirm it waits by putting sleep statements between what you send. Be aware that buffering may interfere.

Suppress output to stderr in matlab

I'm trying to suppress output from a code section in a script (namely the network initialization from a Caffe network). I've tried wrapping the corresponding bit of code in an evalc command
[suppressed_output, var_output] = evalc('someFunction(input)');
But this doesn't work. I've still got loads of lines of (non-error) output from the network initialization that are clogging my logs (amidst all the wanted output printed via fprintf('') in the script). I think this happens because the corresponding function is writing to STDERR (instead of STDOUT?) - the first line it prints is this warning:
WARNING: Logging before InitGoogleLogging() is written to STDERR
... and then hundreds of lines of what it is doing follow, e.g.:
I0215 15:01:51.840272 28620 upgrade_proto.cpp:66] Attempting to upgrade input file specified using deprecated input fields: tmp-def.prototxt
I0215 15:01:51.840360 28620 upgrade_proto.cpp:69] Successfully upgraded file specified using deprecated input fields.
...
Can I somehow suppress the output to STDERR (without messing with the function content)? Ideally only locally for this specific function, since I'd still like to get potential error messages.
In case it is relevant:
I call myScript via matlab command line and its output written to a log (mlexec.log) with tee:
matlab -nodesktop -nosplash -display :1 -r "try, myScript; catch e, disp(getReport(e)), end, quit force" 2>&1| tee mlexec.log
The problem here is, that in the matlab command line call, the output from STDERR is streamed to STDOUT by this "command": 2>&1. Since the .cpp file seems to stream its output to STDERR (according to the Warning), it will be forwarded to STDOUT and eventually the log.
Streaming STDERR (2) to Nirvana with 2>NUL or a different log file (e.g. 2>mlexec.stderr.log) solves the problem.
I wanted to post this in a comment but it said I had to have 50 reputation (I have 49 now...)
I think this is what you're looking for
EDIT/UPDATE:
One thing you can do is enclose a section of your code with warning on/off statements as follows:
warning('off','all')
%your code here
warning('on','all')
This should stop any warnings being output to stderr from that section. I personally do not recommend this, it's good to know what you're doing that the MATLAB runtime does not like.

opening a batch file that opens a text file in python

I am writing a script that can execute a batch file, which needs to open a file in the same folder first. My current code is:
from subprocess import Popen
p = Popen("Mad8dl.bat <RUNTHISTO.txt>", cwd=r"C:\...\test")
stdout, stderr = p.communicate()
where the ... is just the path to the folder. However, everytime I run it I get the syntax error:
The syntax of the command is incorrect
Any help regarding the syntax would be greatly appreciated.
First, you should probably remove the < and > angle brackets from your code; just pass the filename, without any brackets, to your batch file. (Unless your filename really does contain < and > characters, in which case I really want to know how you managed it since those characters are forbidden in filenames in Windows).
Second, your code should look like:
from subprocess import Popen, PIPE
p = Popen(["Mad8dl.bat", "RUNTHISTOO.txt"], cwd=r"C:\...\test", stdout=PIPE, stderr=PIPE)
stdout, stderr = p.communicate()
Note the list containing the components of the call, rather than a single string. Also note that you need to specify stdout=PIPE and stderr=PIPE in your Popen() call if you want to use communicate() later on.

Can kdb read from a named pipe?

I hope I'm doing something wrong, but it seems like kdb can't read data from named pipes (at least on Solaris). It blocks until they're written to but then returns none of the data that was written.
I can create a text file:
$ echo Mary had a little lamb > lamb.txt
and kdb will happily read it:
q) read0 `:/tmp/lamb.txt
enlist "Mary had a little lamb"
I can create a named pipe:
$ mkfifo lamb.pipe
and trying to read from it:
q) read0 `:/tmp/lamb.pipe
will cause kdb to block. Writing to the pipe:
$ cat lamb.txt > lamb.pipe
will cause kdb to return the empty list:
()
Can kdb read from named pipes? Should I just give up? I don't think it's a permissions thing (I tried setting -m 777 on my mkfifo command but that made no difference).
With release kdb+ v3.4 Q has support for named pipes: Depending on whether you want to implement a streaming algorithm or just read from the pipe use either .Q.fps or read1 on a fifo pipe:
To implement streaming you can do something like:
q).Q.fps[0N!]`:lamb.pipe
Then $ cat lamb.txt > lamb.pipe
will print
,"Mary had a little lamb"
in your q session. More meaningful algorithms can be implemented by replacing 0N! with an appropriate function.
To read the context of your file into a variable do:
q)h:hopen`:fifo://lamb.pipe
q)myText: `char$read1(h)
q)myText
"Mary had a little lamb\n"
See more about named pipes here.
when read0 fails, you can frequently fake it with system"cat ...". (i found this originally when trying to read stuff from /proc that also doesn't cooperate with read0.)
q)system"cat /tmp/lamb.pipe"
<blocks until you cat into the pipe in the other window>
"Mary had a little lamb"
q)
just be aware there's a reasonably high overhead (as such things go in q) for invoking system—it spawns a whole shell process just to run whatever your command is
you might also be able to do it directly with a custom C extension, probably calling read(2) directly…
The algorithm for read0 is not available to see what it is doing under the hood but, as far as I can tell, it expects a finite stream and not a continuous one; so it will will block until it receives an EOF signal.
Streaming from pipe is supported from v3.4
Details steps:
Check duplicated pipe file
rm -f /path/dataPipeFileName
Create named pipe
mkfifo /path/dataPipeFileName
Feed data
q).util.system[$1]; $1=command to fetch data > /path/dataPipeFileName &
Connect pipe using kdb .Q.fps
q).Q.fps[0N!]`$":/path/",dataPipeFileName;
Reference:
.Q.fps (streaming algorithm)
Syntax: .Q.fps[x;y] Where x is a unary function and y is a filepath
.Q.fs for pipes. (Since V3.4) Reads conveniently sized lumps of complete "\n" delimited records from a pipe and applies a function to each record. This enables you to implement a streaming algorithm to convert a large CSV file into an on-disk kdb+ database without holding the data in memory all at once.

Running a matlab program with arguments

I have a matlab file that takes in a file. I would like to run that program in the matlab shell, such as prog. I need to implement it so that it takes a number of arguments, such as "prog filename.txt 1 2 which would mean that i can use filename.txt and 1 2 as variables in my program.
Thank you!
In order to make a script accept arguments from the command line, you must first turn it into a function that will get the arguments you want, i.e if your script is named prog.m, put as the first line
function []=prog(arg1, arg2)
and add an end at the end (assuming that the file has only one function). It's very important that you call the function the same name as the file.
The next thing is that you need to make sure that the script file is located at the same place from where you call the script, or it's located at the Matlab working path, otherwise it'll not be able to recognize your script.
Finally, to execute the script you use
matlab -r "prog arg1 arg2"
which is equivalent to calling
prog(arg1,arg2)
from inside Matlab.
*- tested in Windows and Linux environments
Once your function is written in a separate file, as discussed by the other answer you can call it with a slightly more complicated setup to make it easier to catch errors etc.
There is useful advice in this thread about ensuring that Matlab doesn't launch the graphical interface and quits after finishing the script, and reports the error nicely if there is one.
For example:
matlab -nodisplay -nosplash -r "try, prog(1, 'file.txt'), catch me, fprintf('%s / %s\n',me.identifier,me.message), exit(1), end, exit(0)"
The script given to Matlab would read as follows if line spaces were added:
% Try running the script
try
prog(1, 'file.txt')
catch me
% On error, print error message and exit with failure
fprintf('%s / %s\n',me.identifier,me.message)
exit(1)
end
% Else, exit with success
exit(0)