Thanks to this great answer from #Przemyslaw Szufel, we can easily redirect stdio output into a file providing its path.
I wanted to know if it is possible to both redirect stio output to a file and still continue print it in the terminal. That is probably an easy feature to achieve but I could not find it!
And obviously not running the code twice :)
You need a Tee Stream such as https://github.com/fredrikekre/TeeStreams.jl/.
Unfortunately the interface of redirect_stdio does not accept abstract IO for its sink. It requires it to be an IOStream which is a concrete Julia type.
So it seems like there is no good way with the redirect_stdio via streams.
However redirect_stdio does accept sockets and named pipes and this is a way you could consider (yet too complex for this simple tasks - maybe someone finds a more elegant solution).
First you need to create a pipe server that will be running asynchronously in your REPL session (this is code for Windows for Linux you will probably need to remove \\.\pipe\ from the pipe name):
using Sockets
srv = Sockets.listen(raw"\\.\pipe\TeePipe")
mystdout = stdout
#async begin
open("log.txt", "w") do f
sock = accept(srv)
while !eof(sock)
line = readline(sock)
println(mystdout, line)
flush(mystdout)
println(f, line)
flush(f)
end
end
end
Now at the same REPL session you create the client:
teelogger = Sockets.connect(raw"\\.\pipe\TeePipe")
And here is a sample REPL session showing that this works:
julia> redirect_stdio(stdout=teelogger) do
println("Line1")
println("Line2")
#show 3+4
end
Line1
Line2
3 + 4 = 7
7
shell> more log.txt
Line1
Line2
3 + 4 = 7
Related
I have a perl script for which I would like to write a few tests. If the perl script gets supplied a specific option, the user is allowed to paste or type out multi-line input and end their input using control-d.
These are the tests I want to write:
The script (when an interactive flag is supplied) waits for (multiple lines of) input on STDIN until the user hits control-d
[this test is already implemented - but provided for completeness] The script (when a flag is supplied that indicates a redirect/pipe) consumes all input on STDIN and does not wait for control-d
The script (when no input flag is provided [interactive or redirect]) does not wait for interactive user input
Test 1 toy example
I wrote a test for test 1 that confirms input was received (it works by the parent test script printing to the child's input handle and the child modifies and prints that input back out), but that test doesn't wait for an end of input signal (e.g. control-d) (which I don't know how to send anyway). So in other words, I can confirm it receives input, but I don't know how to confirm that it waits for the user to intentionally end the input entry. I.e. How do I know that the input consumption won't stop until the user types control-d?
Here's what I have so far for test 1. I wrote a 3rd little IO::Pipe::Consumer module to be able to send input to the child process that I'm testing and I wrote a toy example of the script that allows input on STDIN from a tty.
Here is a toy version of the script I'm testing:
>perl -e 'while(<STDIN>){print("*$_")}'
test
*test
^d
>
And here is the toy test code (that I want to improve) for the above script:
>perl -e '
use IO::Pipe::Consumer;
$obj = new IO::Pipe::Consumer;
$si = $obj->getSubroutineConsumer(
sub { while(<STDIN>) print("*$_") } }
);
print $si "test\n"
'
*test
>
I thought the parent would have to print an EOF (e.g. like what you get from "control-d") to end the input in the test, but the test ends immediately even though I'm not sending any such end-of-input character. I can see that it's getting, modifying, and printing the input. Is that sufficient to confirm that the script will wait for user input (and that the user will be able to intentionally end the input) or is there something else I should do to confirm it waits for all user input until the user intends to end it?
Test 2 - done
Test 3 toy - don't know how to write it...
Even if modified input spit back out is sufficient proof of "waiting for input" for test 1, I also wish to test that a script doesn't consume input on STDIN when no input option (interactive or redirect) is provided - but since it doesn't seem to wait even when I do send it input without an end-of-input signal, how would I test that the script wouldn't hang waiting for input? Note, the script has other options for consuming redirected or piped input, so my intent is specifically to know if it's waiting on input from the tty. All of the STDIN consumption options (whether from the tty or via redirect/pipe) are optional, which is why I want to write these tests.
My manual testing shows everything works as intended. I would just like some tests to assure that behavior for the future.
IO::Pipe::Consumer
I feel like the thing I'm missing is not relevant to IO::Pipe::Consumer, so WRT that, I'll just describe it instead of paste in 30 or so lines of code... All it does is it sets a pipe to the child's STDIN and gives that handle back to the parent for it to print to. I haven't put it on cpan. I'm just experimenting to see if I can use it to write these tests.
IO::Pipe::Consumer is basically the opposite of IO::Pipe::Producer (a module I published on cpan looong ago, say 2001-ish, when I was new to perl, or programming for that matter). The main difference, aside from swapping STDIN for STDOUT and Reader with Writer (and vice versa), is that the open is open(STDIN,"<",\${$stdin_pipe}).
I thought the parent would have to print an "end-of-input" (e.g. "control-d") character to end the input in the test,
Ctrl-D doesn't produce an "end of input character"; it causes the terminal to return EOF.
I don't know what IO::Pipe::Consumer is —it's not on CPAN— but I presume it creates a pipe. Exiting the program causes the writer end of the pipe to be closed and thus cause the reader end to return EOF.
is there something else I should do to confirm it waits for all user input until the user intends to end it?
<> reads until one of the following things happen:
A line feed is encountered (returning what was read including the line feed)
EOF is encountered (returning what was read up to an including the line feed)
An error is encountered (returning false).
You can confirm it waits by putting sleep statements between what you send. Be aware that buffering may interfere.
I am using a named pipe to capture the output of an external program (wgrib2) within another program (MATLAB). The MATLAB code is below, and system() accesses the command line to make the pipe:
system('mkfifo myfifo'); % Make a named pipe myfifo
% Call the external program wgrib2 and dump its output to the named pipe myfifo
system('wgrib2.exe multi_1.glo_30m.hs.201212.grb2 -ij 1 165 -ij 1 166 > myfifo &');
fid = fopen('myfifo', 'r'); % Open the named pipe
a = fscanf(fid, '%c'); % Read the output as character
fclose(fid); % Close the "file" (myfifo still exists afterward)
Here are my questions:
Do I have to close the named pipe myfifo after I use it? It seems to persist after the code is run.
If myfifo needs to be closed, what is the command to close it?
I will be running the code sample above many times (>1000), so is it OK if I reuse the named pipe and do not close it until the end?
No. Unix treats everything like a file. Named pipes are not different. If you’re done using it, you probably want to close it so you don’t clutter up your machine with named pipes, but you don’t need to close it.
Edited to reflect below comment, which is correct. Deleting != closing.
You close the named pipe the same way you close any file:
fclose(mFifo)
As mentioned in the accepted answer, closing will not delete the fifo. You may need to do that separately.
There’s nothing wrong with re-using a named pipe. It’s up to you, however, to know when you’re done reading/writing to it for each iteration. Once all the data has been read out of the pipe, you’re free to use it again as many times as you want.
You probably confused closing with the opposite of mkfifo command.
An accepted answer is definitely the best solution for MATLAB users, but I'd like to clear things out for those who came here for named pipes.
On Unix-likes named pipe (FIFO) is a special type of file with no content. The mkfifo command creates the pipe on a file system (assigns a name to it), but doesn't open it. You need to open and close it separately like any other file.
Do I have to close the named pipe myfifo after I use it? It seems to persist after the code is run.
It's generally a good idea to close/delete/free things up as soon as you don't need them anymore.
The pipe itself (and its content) gets destroyed when all descriptors to it are closed. What you see left is just a name.
If myfifo needs to be closed, what is the command to close it?
Named pipe can be closed with fclose() function. To make the pipe anonymous and unavailable under the given name (can be done when the pipe is still open) you could use the MATLAB's delete function or the rm console command.
I will be running the code sample above many times (>1000), so is it OK if I reuse the named pipe and do not close it until the end?
It's OK to reuse a named pipe as long as each iteration starts with an empty pipe (according to MATLAB documentation, fscanf() function will do this for you).
A named pipe can be reused in two ways:
By reusing the pipe itself (without closing it):
system('mkfifo myfifo');
tmp = fopen('myfifo', 'r+'); % Open the pipe in both ways (otherwise it will block)
fid = fopen('myfifo', 'r'); % Open the pipe for reading (otherwise `fscanf` will block)
fclose(tmp); % Close the auxiliary descriptor
% Use the pipe multiple times...
system('wgrib2.exe multi_1.glo_30m.hs.201212.grb2 -ij 1 165 -ij 1 166 > myfifo');
a = fscanf(fid, '%c');
...
% Close and delete the pipe
fclose(fid);
delete myfifo;
By reusing the name to open a pipe (the way you were using it).
You probably shouldn't use a named pipe for capturing system output in MATLAB. Per MATLAB's system() documentation, you can capture output directly in the system() call:
[status, cmdout] = system('wgrib2.exe multi_1.glo_30m.hs.201212.grb2 -ij 1 165 -ij 1 166');
a = cmdout
However, if you insist on using a named pipe, then Yes, you should close it. You should always close resources you open. But closing a named pipe does not delete it.
Cannot agree. You close fifo by closing it. The system closes fifo when it is finished (the system() call above). When the fifo is closed the other side knows that there's no more data (EOF condition, there is no explicit EOF check in the example code but it is very probable in the real one) and finishes.
I hope I'm doing something wrong, but it seems like kdb can't read data from named pipes (at least on Solaris). It blocks until they're written to but then returns none of the data that was written.
I can create a text file:
$ echo Mary had a little lamb > lamb.txt
and kdb will happily read it:
q) read0 `:/tmp/lamb.txt
enlist "Mary had a little lamb"
I can create a named pipe:
$ mkfifo lamb.pipe
and trying to read from it:
q) read0 `:/tmp/lamb.pipe
will cause kdb to block. Writing to the pipe:
$ cat lamb.txt > lamb.pipe
will cause kdb to return the empty list:
()
Can kdb read from named pipes? Should I just give up? I don't think it's a permissions thing (I tried setting -m 777 on my mkfifo command but that made no difference).
With release kdb+ v3.4 Q has support for named pipes: Depending on whether you want to implement a streaming algorithm or just read from the pipe use either .Q.fps or read1 on a fifo pipe:
To implement streaming you can do something like:
q).Q.fps[0N!]`:lamb.pipe
Then $ cat lamb.txt > lamb.pipe
will print
,"Mary had a little lamb"
in your q session. More meaningful algorithms can be implemented by replacing 0N! with an appropriate function.
To read the context of your file into a variable do:
q)h:hopen`:fifo://lamb.pipe
q)myText: `char$read1(h)
q)myText
"Mary had a little lamb\n"
See more about named pipes here.
when read0 fails, you can frequently fake it with system"cat ...". (i found this originally when trying to read stuff from /proc that also doesn't cooperate with read0.)
q)system"cat /tmp/lamb.pipe"
<blocks until you cat into the pipe in the other window>
"Mary had a little lamb"
q)
just be aware there's a reasonably high overhead (as such things go in q) for invoking system—it spawns a whole shell process just to run whatever your command is
you might also be able to do it directly with a custom C extension, probably calling read(2) directly…
The algorithm for read0 is not available to see what it is doing under the hood but, as far as I can tell, it expects a finite stream and not a continuous one; so it will will block until it receives an EOF signal.
Streaming from pipe is supported from v3.4
Details steps:
Check duplicated pipe file
rm -f /path/dataPipeFileName
Create named pipe
mkfifo /path/dataPipeFileName
Feed data
q).util.system[$1]; $1=command to fetch data > /path/dataPipeFileName &
Connect pipe using kdb .Q.fps
q).Q.fps[0N!]`$":/path/",dataPipeFileName;
Reference:
.Q.fps (streaming algorithm)
Syntax: .Q.fps[x;y] Where x is a unary function and y is a filepath
.Q.fs for pipes. (Since V3.4) Reads conveniently sized lumps of complete "\n" delimited records from a pipe and applies a function to each record. This enables you to implement a streaming algorithm to convert a large CSV file into an on-disk kdb+ database without holding the data in memory all at once.
I'm trying to solve the bilateral problem on Spotify's Tech Puzzles. http://www.spotify.com/us/jobs/tech/bilateral-projects/ I have something that is working on my computer that reads input from a file input.txt, and it outputs to ouput.txt. My problem is that I cannot figure out how to make my code work when I submit it where it must read from stdin. I have looked at several other posts and I don't see anything that makes sense to me. I see some people just use raw_input - but this produces a user prompt?? Not sure what to do. Here is the protion of my code that is suposed to read the input, and write the output. Any suggestions on how this might need changed? Also how would I test the code once it is changed to read from stdin? How can I put test data in stdin? The error i get back from spotify says Run Time Error - NameError.
import sys
# Read input
Input = []
for line in sys.stdin.readlines():
if len(line) <9:
teamCount = int(line)
if len(line) > 8:
subList = []
a = line[0:4]
b = line[5:9]
subList.append(a)
subList.append(b)
Input.append(subList)
##### algorithm here
#write output
print listLength
for empWin in win:
print empWin
You are actually doing ok.
for line in sys.stdin.readlines():
will read lines from stdin. It can however be shortened to:
for line in sys.stdin:
I don't use Windows, but to test your solution from a command line, you should run it like this:
python bilateral.py < input.txt > output.txt
If I run your code above like that, I see the error message
Traceback (most recent call last):
File "bilateral.py", line 20, in <module>
print listLength
NameError: name 'listLength' is not defined
which by accident (because I guess you didn't send in that) was the error the Spotify puzzle checker discovered. You have probably just misspelled a variable somewhere.
I'm fairly new to Perl and have been searching the interwebs for documentation for what I'm trying to do. I'm not having any luck.
I have a program that outputs information to stdout with prompts throughout. I need to make a Perl script to pipe that information to a file.
I thought I could use Expect but there seems to be a problem with the pipe after the first prompt.
Here is the part of my code:
# Run program and compare the output to the BASE file
$cmd = "./program arg1 arg2 arg3 arg4 > $outfile";
my $exp = new Expect;
$exp->spawn($cmd);
BAIL_OUT("couldn't create expect object") if (! defined $exp);
$exp->expect(2);
$exp->send("\n");
For this case there is only a single prompt for the user to press "enter". This program is small and very fast - 2 seconds is plenty of time to reach the first prompt.
The output file only contains the first half of the information.
Does anyone have any suggestions on how I can grab the second half as well?
UPDATE:
I've verified that this works with Expect by using a simple script:
spawn ./program arg1 arg2 arg3 arg4
expect "<Return>"
send "\r"
interact
Where "< Return >" is a verbose expression that the Perl script could look for.
Note: I've tried writing my Perl script to expect "< Return >"...it makes no difference.
i.e.
$exp->expect(2, '-re', "<Return>")
Any thoughts?
UPDATE2:
Hazaah! I've found a solution to my problem...completely by accident.
So, I had a mistype in some test code I made...
$exp->expect(2);
$exp->send("\r");
$exp->expect(2);
Note the trailing expect(2)...I accidentally left that in and it worked!
So, I'm trying to understand what is happening. Unix expect does not seem work this way! It appears Expect implemented in Perl "expects" anything...not just prompts?
So, I provided expect another 2 seconds to collect stdout and I am able to get everything.
If anyone can offer some more detailed information as to what is going on here I'd love to understand what is going on.
Try sending \r instead of \n - you're trying to emulate a carriage return, not a newline, and the tty settings might not be translating them.
ALSO:
A suggestion from the Expect docs FAQ section, which seems likely given your accidental solution:
My script fails from time to time without any obvious reason. It
seems that I am sometimes loosing output from the spawned program.
You could be exiting too fast without giving the spawned program
enough time to finish. Try adding $exp->soft_close() to terminate the
program gracefully or do an expect() for 'eof'.
Alternatively, try adding a 'sleep 1' after you spawn() the program.
It could be that pty creation on your system is just slow (but this is
rather improbable if you are using the latest IO-Tty).
Standard unix/tcl expect doesn't exit in interactive mode, which could give your program enough time to finish running.
It's been a while since I've used Expect, but I'm pretty sure you need to provide something for Expect to match the prompt against:
$exp->expect( 2, 'Press enter' );
for example.