I want to capture and plot the results from 5 or so timeit calls with logarithmically increasing sizes of N to show how methodX() scales with input.
So far I have tried:
output = %timeit -r 10 results = methodX(N)
It does not work...
Can't find info in the docs either. I feel like you should be able to at least intercept the string that is printed. After that I can parse it to extract my info.
Has anyone done this or tried?
PS: this is in an ipython notebook if that makes a diff.
This duplicate question Capture the result of an IPython magic function has an answer demonstrating that this has since been implemented.
Calling the %timeit magic with the -o option like:
%timeit -o <statement>
returns a TimeitResult object which is a simple object with all information about the %timeit run as attributes. For example:
In [1]: result = %timeit -o 1 + 2
Out[1]: 10000000 loops, best of 3: 23.2 ns per loop
In [2]: result.best
Out[2]: 2.3192405700683594e-08
PS: this is in an ipython notebook if that makes a diff.
No it does not.
On dev there is te %%capture cell magic.
The other way would be to modify the timeit magic to return value instead of printing, or use the timeit module itself. Patches welcomed.
Related
-- Edit : Resolved. See answer.
Background:
I'm writing a shell that will perform some extra actions required on our system when someone resizes a database.
The shell is written in ksh (requirement), the OS is Solaris 5.10 .
The problem is with one of the checks, which verifies there's enough free space on the underlying OS.
Problem:
The check reads the df -k line for root, which is what I check in this step, and prints it to a file. I then "read" the contents into variables which I use in calculations.
Unfortunately, when I try to run an arithmetic operation on one of the variables, I get an error indicating it is null. And a debug output line I've placed after that line verifies that it is null... It lost it's value...
I've tried every method of doing this I could find online, they work when I run it manually, but not inside the shell file.
(* The file does have #!/usr/bin/ksh)
Code:
df -k | grep "rpool/ROOT" > dftest.out
RPOOL_NAME=""; declare -i TOTAL_SIZE=0; USED_SPACE=0; AVAILABLE_SPACE=0; AVAILABLE_PERCENT=0; RSIGN=""
read RPOOL_NAME TOTAL_SIZE USED_SPACE AVAILABLE_SPACE AVAILABLE_PERCENT RSIGN < dftest.out
\rm dftest.out
echo $RPOOL_NAME $TOTAL_SIZE $USED_SPACE $AVAILABLE_SPACE $AVAILABLE_PERCENT $RSIGN
((TOTAL_SIZE=$TOTAL_SIZE/1024))
This is the result:
DBResize.sh[11]: TOTAL_SIZE=/1024: syntax error
I'm pulling hairs at this point, any help would be appreciated.
The code you posted cannot produce the output you posted. Most obviously, the error is signalled at line 11 but you posted fewer than 11 lines of code. The previous lines may matter. Always post complete code when you ask for help.
More concretely, the declare command doesn't exist in ksh, it's a bash thing. You can achieve the same result with typeset (declare is a bash equivalent to typeset, but not all options are the same). Either you're executing this script with bash, or there's another error message about declare, or you've defined some additional commands including declare which may change the behavior of this code.
None of this should have an impact on the particular problem that you're posting about, however. The variables created by read remain assigned until the end of the subshell, i.e. until the code hits a ), the end of a pipe (left-hand side of the pipe only in ksh), etc.
About the use of declare or typeset, note that you're only declaring TOTAL_SIZE as an integer. For the other variables, you're just assigning a value which happens to consist exclusively of digits. It doesn't matter for the code you posted, but it's probably not what you meant.
One thing that may be happening is that grep matches nothing, and therefore read reads an empty line. You should check for errors. Use set -e in scripts to exit at the first error. (There are cases where set -e doesn't catch errors, but it's a good start.)
Another thing that may be happening is that df is splitting its output onto multiple lines because the first column containing the filesystem name is too large. To prevent this splitting, pass the option -P.
Using a temporary file is fragile: the code may be executed in a read-only directory, another process may want to access the same file at the same time... Here a temporary file is useless. Just pipe directly into read. In ksh (unlike most other sh variants including bash), the right-hand side of a pipe runs in the main shell, so assignments to variables in the right-hand side of a pipe remain available in the following commands.
It doesn't matter in this particular script, but you can use a variable without $ in an arithmetic expression. Using $ substitutes a string which can have confusing results, e.g. a='1+2'; $((a*3)) expands to 7. Not using $ uses the numerical value (in ksh, a='1+2'; $((a*3)) expands to 9; in some sh implementations you get an error because a's value is not numeric).
#!/usr/bin/ksh
set -e
typeset -i TOTAL_SIZE=0 USED_SPACE=0 AVAILABLE_SPACE=0 AVAILABLE_PERCENT=0
df -Pk | grep "rpool/ROOT" | read RPOOL_NAME TOTAL_SIZE USED_SPACE AVAILABLE_SPACE AVAILABLE_PERCENT RSIGN
echo $RPOOL_NAME $TOTAL_SIZE $USED_SPACE $AVAILABLE_SPACE $AVAILABLE_PERCENT $RSIGN
((TOTAL_SIZE=TOTAL_SIZE/1024))
Strange...when I get rid of your "declare" line, your original code seems to work perfectly well (at least with ksh on Linux)
The code :
#!/bin/ksh
df -k | grep "/home" > dftest.out
read RPOOL_NAME TOTAL_SIZE USED_SPACE AVAILABLE_SPACE AVAILABLE_PERCENT RSIGN < dftest.out
\rm dftest.out
echo $RPOOL_NAME $TOTAL_SIZE $USED_SPACE $AVAILABLE_SPACE $AVAILABLE_PERCENT $RSIGN
((TOTAL_SIZE=$TOTAL_SIZE/1024))
print $TOTAL_SIZE
The result :
32962416 5732492 25552588 19% /home
5598
Which are the value a simple df -k is returning. The variables seem to last.
For those interested, I have figured out that it is not possible to use "read" the way I was using it.
The variable values assigned by "read" simply "do not last".
To remedy this, I have applied the less than ideal solution of using the standard "while read" format, and inside the loop, echo selected variables into a variable file.
Once said file was created, I just "loaded" it.
(pseudo code:)
LOOP START
echo "VAR_A="$VAR_A"; VAR_B="$VAR_B";" > somefile.out
LOOP END
. somefile.out
I'm writing some functions in a Jupyter notebook to prepare data, but they can take quite a while to run.
In [*]: print "hello"
df['expensive_col'] = df.apply(myfunc, axis=1)
It would be useful if myfunc would print status information such that I see how far into the execution it is. The downside of this is that it would seriously clutter the Jupyter output cell.
In [*]: print hello
df['expensive_col'] = df.apply(myfunc, axis=1)
hello
ix0
ix1
ix2
...
ix999
...
noisy output
...
Is there a way to print status updates from the long-running Jupyter cell in such a way to keep from cluttering the output cell?
For example, I could wipe everything printed by calling IPython.display.clear_output() but I might want to do a more targeted wipe, keeping stuff from before my expensive (hello, in the above example).
One simple thing you could do is to invoke the print function with the end parameter set to carriage return instead of newline, like this:
for i in range(100):
print('progress %i' % i, end='\r')
# things happen here
It effectively writes over the previous line, without cluttering the output field.
For a more advanced approach (and fancy looking progress bars!) check out the tqdm project.
I'm translating this Zsh function to Fish
function random_quote() {
QUOTE_FILES=( $PREFS_ROOT/quotes/* )
cat $QUOTE_FILES[$RANDOM%$#QUOTE_FILES+1]
}
Here's what I've got so far:
function random_quote
set QUOTE_FILES $PREFS_ROOT/quotes/*
cat $QUOTE_FILES[$RANDOM%$#QUOTE_FILES+1]
end
The cat line needs fixing still. I know that RANDOM should be replaced by random, but I'm not sure how to do the rest.
Just found out it's really simple, random itself supports getting a random entry from a list!
just use cat (random choice $QUOTE_FILES)
it also works without a variable random choice /path/to/some/folder/*
or with a bunch of arguments random choice option1 option2 option3
you can find the documentation for random here:
https://fishshell.com/docs/current/cmds/random.html
How about
function random_quote
set -l QUOTE_FILES $PREFS_ROOT/quotes/*
set -l n (math 'scale=0;'(random)'%'(count $QUOTE_FILES)'+1')
cat $QUOTE_FILES[$n]
end
This answer is outdated. Please follow #Niklas's correct one.
In Octave I can suppress or hide the output of an instruction adding a semicolon to the end of a line:
octave:1> exp([0 1])
ans = [ 1.0000 2.7183 ]
octave:2> exp([0 1]);
octave:3>
Now, how can I suppress the output if the function displays text (e.g. using disp() or print()) before returning its value? In other words, I want to be able to do this:
disp("Starting...");
% hide text the may get displayed after this point
% ...
% show all text again after this point
disp("Done!");
You can modify the PAGER variable (which is now a function) to redirect standard output. On Unix systems, you can redirect it to /dev/null. On Windows, I tried simply redirecting to a Python program that does nothing, and it works decently. (Basically, any program that ignores the input will do)
PAGER('/dev/null');
page_screen_output(1);
page_output_immediately(1);
You can just change it back after you're done. And maybe encapsulate this whole procedure in a function.
oldpager = PAGER('/dev/null');
oldpso = page_screen_output(1);
oldpoi = page_output_immediately(1);
% Call function here
PAGER(oldpager);
page_screen_output(oldpso);
page_output_immediately(oldpoi);
You can also simply run your scripts non-interactively, and redirect the output normally.
octave script.m > /dev/null
A quick hack of your problem and maybe not even worth mentioning is overloading the disp function like so:
function disp(x)
end
Then the original disp function is not called but yours instead in which no output is generated.
I also tried to somehow redirect stdout of octave, but unsuccessful. I hope that this dirty solution maybe will suffice in your situation^^
It's a very old question, but still, I've encountered the same problem and here is the trick that can help. One can use evalc to wrap a problematic function call. E.g. you have a code:
[a, b] = verbose_func(x,y);
Now you can do it:
evalc('[a, b] = verbose_func(x,y)');
and make it silent.
Funny, but it even works with other eval inside. I mean we can have:
code_str = '[a, b] = verbose_func(x,y)';
eval(code_str);
which is verbose. Now:
code_str = '[a, b] = verbose_func(x,y)';
evalc('eval(code_str)');
and this is not.
I am writing a program where if no command line arguments are supplied i.e #ARGV == 0, the program takes in three inputs. But, the program has the feature to read any files given as arguments, thus
calculate input1 input2
runs the formula on the numbers found in file1 and file2.
The problem I am running into is when I run
calculate < input1
#ARGV returns 0, thus it runs the code for user input.
How do I get around this so that the program can read input1 and use the values inside for calculations?
calculate < input1 is equivalent to cat input1 | calculate.
You need to read from <STDIN> and not look for command line arguments.
That should not be a problem. If you read reading from <> (which is really <ARGV>), then there is no difference.
You must be doing something wrong if redirection changes things. Are you actually opening files yourself???
You might consider using a module like Getopt::Euclid or Getopt::Long to make the argument passing more explicit. That might make the program easier to understand for other users too.