I really like to use system shell commands in iPython. But I was wondering if it is possible to loop over the returned values from a call to e.g. !ls. This works:
files = !ls ./*_subcell_cooc.txt
for f in files:
print f
But this does not:
for f in ( !ls ./*_subcell_cooc.txt):
print f
Error is:
File "<ipython-input-1-df2bc72907d7>", line 5
for f in ( !ls $ROOT/*_subcell_cooc.txt):
^
SyntaxError: invalid syntax
No it is not possible, the syntax var = !something is special cased in IPython. It is not valid python syntax, and we will not extend for loops and so on to work with it.
You can do assignment as you show in your first example, but using glob,os and other real python module to do that will be more robust, not much harder, and also work outside of IPython...
For the anecdote Guido was really not happy with IPython half-shell syntax when he saw it last time at SciPy2013.
(Also it uppercase I in IPython please.)
Related
I am trying to identify misspelled words with Aspell via Perl. I am working on a Linux server without administrator privileges which means I have access to Perl and Aspell but not, for example, Text::Aspell which is a Perl interface for Aspell.
I want to do the very simple task of passing a list of words to Aspell and having it return the words that are misspelled. If the words I want to check are "dad word lkjlkjlkj" I can do this through the command line with the following commands:
aspell list
dad word lkjlkjlkj
Aspell requires CTRL + D at the end to submit the word list. It would then return "lkjlkjlkj", as this isn't in the dictionary.
In order to do the exact same thing, but submitted via Perl (because I need to do this for thousands of documents) I have tried:
my $list = q(dad word lkjlkjlkj):
my #arguments = ("aspell list", $list, "^D");
my $aspell_out=`#arguments`;
print "Aspell output = $aspell_out\n";
The expected output is "Aspell output = lkjlkjlkj" because this is the output that Aspell gives when you submit these commands via the command line. However, the actual output is just "Aspell output = ". That is, Perl does not capture any output from Aspell. No errors are thrown.
I am not an expert programmer, but I thought this would be a fairly simple task. I've tried various iterations of this code and nothing works. I did some digging and I'm concerned that perhaps because Aspell is interactive, I need to use something like Expect, but I cannot figure out how to use it. Nor am I sure that it is actually the solution to my problem. I also think ^D should be an appropriate replacement for CTRL+D at the end of the commands, but all I know is it doesn't throw an error. I also tried \cd instead. Whatever it is, there is obviously an issue in either submitting the command or capturing the output.
The complication with using aspell out of a program is that it is an interactive and command-line driver tool, as you suspect. However, there is a simple way to do what you need.
In order to use aspell's command list one needs to pass it words via STDIN, as its man page says. While I find the GNU Aspell manual a little difficult to get going with, passing input to a program via its STDIN is easy enough and we can rewrite the invocation as
echo dad word lkj | aspell list
We get lkj printed back, as due. Now this can run out of a program just as it stands
my $word_list = q(word lkj good asdf);
my $cmd = qq(echo $word_list | aspell list);
my #aspell_out = qx($cmd);
print for #aspell_out;
This prints lines lkj and asdf.
I assemble the command in a string (as opposed to an array) for specific reasons, explained below. The qx is the operator form of backticks, which I prefer for its far superior readability.
Note that qx can return all output in a string, if in scalar context (assigned to a scalar for example), or in a list when in list context. Here I assign to an array so you get each word as an element (alas, each also comes with a newline, so may want to do chomp #aspell_out;).
Comment on a list vs string form of a command
I think that it's safe to recommend to use a list-form for a command, in general. So we'd say
my #cmd = ('ls', '-l', $dir); # to be run as an external command
instead of
my $cmd = "ls -l $dir"; # to be run as an external command
The list form generally makes it easier to manage the command, and it avoids the shell altogether.
However, this case is a little different
The qx operator doesn't really behave differently -- the array gets concatenated into a string, and that runs. The very fact that we can pass it an array is incidental, and not even documented
We need to pipe input to aspell's STDIN, and shell does that for us simply. We can use a shell with command's LIST form as well, but then we'd need to invoke it explicitly. We can also go for aspell's STDIN by means other than the shell but that's more complex
With a command in a list the command name must be the first word, so that "aspell list" from the question is wrong and it should fail (there is no command named that) ... except that in this case it wouldn't (if the rest were correct), since for qx the array gets collapsed into a string
Finally, apsell nicely exposes its API in a C library and that's been utilized for the module you mention. I'd suggest to install it as a user (no privileges needed) and use that.
You should take a step back and investigate if you can install Text::Aspell without administrator privilige. In most cases that's perfectly possible.
You can install modules into your home directory. If there is no C-compiler available on the server you can install the module on a compatible machine, compile and copy the files.
When I run a file which is begin with #!/usr/bin/perl -w, I get a error:
syntax error at line 153, near "=~ ?"
I try to add "#!/bin/bash", this error is not append, but I get another
error:
"line 34: syntax error near unexpected token `('"
line 153 in my file:
($output_volume =~ ?^([\S]+).mnc?) && ($base_name = $1) ||
die "sharpen_volume failed: output volume does not appear to be"
." a minc volume.\n";
line34 in my file:
use MNI::Startup qw(nocputimes);
$output_volume =~ ?^([\S]+).mnc?
This used to be valid perl and thus might appear in old code and instructional material.
From perlop:
In the past, the leading m in m?PATTERN? was optional, but omitting it would produce a deprecation warning. As of v5.22.0, omitting it produces a syntax error. If you encounter this construct in older code, you can just add m.
That is Perl code so the first error message is meaningful.
With delimiters other than // in the match operator you must have the explicit m for it, so
$output_volume =~ m?^([\S]+).mnc?
It is only with // delimiters that the m may be omitted; from Regex Quote-Like Operators (perlop)
If "/" is the delimiter then the initial m is optional.
See perlretut for a tutorial introduction to regex and perlre for reference.
Also note that the particular delimiters of ? trigger specific regex behavior in a special case. This is discussed by the end of the documentation section in perlop linked above.
You already have two answers that explain the problem.
? ... ? is no longer valid syntax for a match operator. You need m? ... ? instead.
Until Perl 5.22, your syntax generated a warning. Now it's a fatal error (which is what you are seeing). So I assume you're now running this on a more recent version of Perl.
There are, however, a few other points it is probably worth making.
You say you tried to investigate this by changing the first line of your file from #!/usr/bin/perl -w to #!/bin/bash. I'm not sure how you think this was going to help. This line defines the program that is used to run your code. As you have Perl code, you need to run it with Perl. Trying to run it with bash is very unlikely to be useful.
The m? ... ? (or, formerly, ? ... ?) syntax triggers an obscure and specialised behaviour. It seems to me that this behaviour isn't required in your case, so you can probably change it to the more usual / ... /.
Your regex contains an unescaped dot character. Given that you seem to be extracting the basename from a filename that has an extension, it seems likely that this should be escaped (using \.) so that it matches an actual dot (rather than any character).
If you are using this code to extract a file's basename, then using a regex probably isn't the best approach. Perhaps take a look at File::Basename instead.
-- Edit : Resolved. See answer.
Background:
I'm writing a shell that will perform some extra actions required on our system when someone resizes a database.
The shell is written in ksh (requirement), the OS is Solaris 5.10 .
The problem is with one of the checks, which verifies there's enough free space on the underlying OS.
Problem:
The check reads the df -k line for root, which is what I check in this step, and prints it to a file. I then "read" the contents into variables which I use in calculations.
Unfortunately, when I try to run an arithmetic operation on one of the variables, I get an error indicating it is null. And a debug output line I've placed after that line verifies that it is null... It lost it's value...
I've tried every method of doing this I could find online, they work when I run it manually, but not inside the shell file.
(* The file does have #!/usr/bin/ksh)
Code:
df -k | grep "rpool/ROOT" > dftest.out
RPOOL_NAME=""; declare -i TOTAL_SIZE=0; USED_SPACE=0; AVAILABLE_SPACE=0; AVAILABLE_PERCENT=0; RSIGN=""
read RPOOL_NAME TOTAL_SIZE USED_SPACE AVAILABLE_SPACE AVAILABLE_PERCENT RSIGN < dftest.out
\rm dftest.out
echo $RPOOL_NAME $TOTAL_SIZE $USED_SPACE $AVAILABLE_SPACE $AVAILABLE_PERCENT $RSIGN
((TOTAL_SIZE=$TOTAL_SIZE/1024))
This is the result:
DBResize.sh[11]: TOTAL_SIZE=/1024: syntax error
I'm pulling hairs at this point, any help would be appreciated.
The code you posted cannot produce the output you posted. Most obviously, the error is signalled at line 11 but you posted fewer than 11 lines of code. The previous lines may matter. Always post complete code when you ask for help.
More concretely, the declare command doesn't exist in ksh, it's a bash thing. You can achieve the same result with typeset (declare is a bash equivalent to typeset, but not all options are the same). Either you're executing this script with bash, or there's another error message about declare, or you've defined some additional commands including declare which may change the behavior of this code.
None of this should have an impact on the particular problem that you're posting about, however. The variables created by read remain assigned until the end of the subshell, i.e. until the code hits a ), the end of a pipe (left-hand side of the pipe only in ksh), etc.
About the use of declare or typeset, note that you're only declaring TOTAL_SIZE as an integer. For the other variables, you're just assigning a value which happens to consist exclusively of digits. It doesn't matter for the code you posted, but it's probably not what you meant.
One thing that may be happening is that grep matches nothing, and therefore read reads an empty line. You should check for errors. Use set -e in scripts to exit at the first error. (There are cases where set -e doesn't catch errors, but it's a good start.)
Another thing that may be happening is that df is splitting its output onto multiple lines because the first column containing the filesystem name is too large. To prevent this splitting, pass the option -P.
Using a temporary file is fragile: the code may be executed in a read-only directory, another process may want to access the same file at the same time... Here a temporary file is useless. Just pipe directly into read. In ksh (unlike most other sh variants including bash), the right-hand side of a pipe runs in the main shell, so assignments to variables in the right-hand side of a pipe remain available in the following commands.
It doesn't matter in this particular script, but you can use a variable without $ in an arithmetic expression. Using $ substitutes a string which can have confusing results, e.g. a='1+2'; $((a*3)) expands to 7. Not using $ uses the numerical value (in ksh, a='1+2'; $((a*3)) expands to 9; in some sh implementations you get an error because a's value is not numeric).
#!/usr/bin/ksh
set -e
typeset -i TOTAL_SIZE=0 USED_SPACE=0 AVAILABLE_SPACE=0 AVAILABLE_PERCENT=0
df -Pk | grep "rpool/ROOT" | read RPOOL_NAME TOTAL_SIZE USED_SPACE AVAILABLE_SPACE AVAILABLE_PERCENT RSIGN
echo $RPOOL_NAME $TOTAL_SIZE $USED_SPACE $AVAILABLE_SPACE $AVAILABLE_PERCENT $RSIGN
((TOTAL_SIZE=TOTAL_SIZE/1024))
Strange...when I get rid of your "declare" line, your original code seems to work perfectly well (at least with ksh on Linux)
The code :
#!/bin/ksh
df -k | grep "/home" > dftest.out
read RPOOL_NAME TOTAL_SIZE USED_SPACE AVAILABLE_SPACE AVAILABLE_PERCENT RSIGN < dftest.out
\rm dftest.out
echo $RPOOL_NAME $TOTAL_SIZE $USED_SPACE $AVAILABLE_SPACE $AVAILABLE_PERCENT $RSIGN
((TOTAL_SIZE=$TOTAL_SIZE/1024))
print $TOTAL_SIZE
The result :
32962416 5732492 25552588 19% /home
5598
Which are the value a simple df -k is returning. The variables seem to last.
For those interested, I have figured out that it is not possible to use "read" the way I was using it.
The variable values assigned by "read" simply "do not last".
To remedy this, I have applied the less than ideal solution of using the standard "while read" format, and inside the loop, echo selected variables into a variable file.
Once said file was created, I just "loaded" it.
(pseudo code:)
LOOP START
echo "VAR_A="$VAR_A"; VAR_B="$VAR_B";" > somefile.out
LOOP END
. somefile.out
I can execute Maxima code from Octave like this and it works:
mm=maxima("diff(a*x^3-b*x^2+x+d,x,1)")
but how can I execute multi line commands?
Example code below that works in Maxima
kill(all)$
numer:true$
ratprint:false$
angle_in_bits:3779$
total_fs:18136$
s:solve(angle_deg=(angle_in_bits/total_fs*360),angle_deg)$
round(s);
[round(angle_deg)=75]
When I try the code below in Octave I get syntax errors
mm=maxima("kill(all)$
numer:true$
ratprint:false$
angle_in_bits:3779$
total_fs:18136$
s:solve(angle_deg=(angle_in_bits/total_fs*360),angle_deg)$
round(s);")
Errors that I get:
>>> mm=maxima("kill(all)$
numer:true$
ratprint:false$
angle_in_bits:3779$
total_fs:18136$
s:solve(angle_deg=(angle_in_bits/total_fs*360),angle_deg)$
round(s);")
error: unterminated character string constant
parse error:
syntax error
>>> mm=maxima("kill(all)$
^
>>> _ide_reload_variables_list( whos() );
error: 'numer' undefined near line 1 column 1
error: invalid base value in colon expression
error: 'ratprint' undefined near line 1 column 1
error: invalid base value in colon expression
parse error:
syntax error
>>> angle_in_bits:3779$
^
parse error:
syntax error
>>> total_fs:18136$
^
parse error:
syntax error
>>> s:solve(angle_deg=(angle_in_bits/total_fs*360),angle_deg)$
^
error: unterminated character string constant
parse error:
syntax error
>>> round(s);")
^
Thanks to Fred Senese and rayryeng for the assist.
I know someone may need this so here's some example code. This bit of code allows you to directly access maxima's symbolic solver from octave (allows you to execute multiple lines of maxima's commands). Since octave doesn't have a good symbolic solver yet this will come in handy for another person down the line.
mm=maxima("(kill(all), numer:true, ratprint:false, angle_in_bits:3779, total_fs:18136, s:solve(angle_deg=(angle_in_bits/total_fs*360),angle_deg),(s))")
%mm = '[angle_deg = 75.01323334803705]';
[si ei xt mt] = regexp(mm, '(\d)*(\.)?(\d)*');
number = str2num(mt{1})
>>>number = 75.013
I will suppose here that you are using QtOctave which I am guessing from googling your error message "_ide_reload_variables_list( whos() );"
If this is not so, none of the following may apply to your question.
typing help maxima at the prompt points me to a file /usr/share/qtoctave/scripts_octave/maxima.m
with this contents:
function result=maxima(command)
in="";
in=sprintf("echo \"string(%s);\"|maxima --very-quiet", command);
[status,result]=system(in);
%if(status!=0) result=""; endif;
result = deblank ( strjust ( strrep (result, "%", "") ,"left") );
endfunction
Which tells me that maxima is called via octave's function system in a very special way that is not allowing for multiple commands in maxima.
modifying the assignment of in in the way below would allow you to call the function maxima now with a cell array of commands maxima({command_1,command_2}) where command_i are strings.
in=['echo ', sprintf('\"%s;\" ',command{:}), '| maxima --very-quiet'];
Please note that the function system still returns only one output, the one that is sent to standard out by maxima.
This may also be of interest for you as it describes methods of octave's interaction with subprocesses.
I am not sure if this helping much as I think the modification provided by me is only of very superficial use, but maybe it helps you to understand better what octave is doing if you tell it maxima(something). It helped me.
Last but not least as far as I know there is no real interface between octave (or matlab) and maxima. I hope someone will correct me if I am wrong about that.
I have Octave and Maxima in my Linux laptop (Ubuntu). There exist system -function in Octave, which could be used to run terminal -commands.
In terminal it is possible to call maxima functions by using pipe
(add quit(); to the end of maxima command) :
$ echo "factor(12345); quit();" | maxima
Maxima 5.41.0 http://maxima.sourceforge.net
using Lisp GNU Common Lisp (GCL) GCL 2.6.12
Distributed under the GNU Public License. See the file COPYING.
Dedicated to the memory of William Schelter.
The function bug_report() provides bug reporting information.
(%i1) (%o1) 3 5 823
$
In Octave' system -commad use double "" inside "-marks to get " :
[status,output]=system("echo ""factor(565);quit();""|maxima")
status = 0
output =
Maxima 5.41.0 http://maxima.sourceforge.net
using Lisp GNU Common Lisp (GCL) GCL 2.6.12
Distributed under the GNU Public License. See the file COPYING.
Dedicated to the memory of William Schelter.
The function bug_report() provides bug reporting information.
(%i1) (%o1) 5 113
Extra txt could be cutted out from the output -string in Octave. Use Maxima's properties to run it's commands from a script file, and the script could be created in Octave.
Br. Juha (juhap.karjalainen#mail.suomi.net)
GCC version 4.6
The Problem: To find a way to feed in parameters to the executable, say a.out, from the command line - more specifically feed in an array of double precision numbers.
Attempt: Using the READ(*,*) command, which is older in the standard:
Program test.f -
PROGRAM MAIN
REAL(8) :: A,B
READ(*,*) A,B
PRINT*, A+B, COMMAND_ARGUMENT_COUNT()
END PROGRAM MAIN
The execution -
$ gfortran test.f
$ ./a.out 3.D0 1.D0
This did not work. On a bit of soul-searching, found that
$./a.out
3.d0,1.d0
4.0000000000000000 0
does work, but the second line is an input prompt, and the objective of getting this done in one-line is not achieved. Also the COMMAND_ARGUMENT_COUNT() shows that the numbers fed into the input prompt don't really count as 'command line arguments', unlike PERL.
If you want to get the arguments fed to your program on the command line, use the (since Fortran 2003) standard intrinsic subroutine GET_COMMAND_ARGUMENT. Something like this might work
PROGRAM MAIN
REAL(8) :: A,B
integer :: num_args, ix
character(len=12), dimension(:), allocatable :: args
num_args = command_argument_count()
allocate(args(num_args)) ! I've omitted checking the return status of the allocation
do ix = 1, num_args
call get_command_argument(ix,args(ix))
! now parse the argument as you wish
end do
PRINT*, A+B, COMMAND_ARGUMENT_COUNT()
END PROGRAM MAIN
Note:
The second argument to the subroutine get_command_argument is a character variable which you'll have to parse to turn into a real (or whatever). Note also that I've allowed only 12 characters in each element of the args array, you may want to fiddle around with that.
As you've already figured out read isn't used for reading command line arguments in Fortran programs.
Since you want to read an array of real numbers, you might be better off using the approach you've already figured out, that is reading them from the terminal after the program has started, it's up to you.
The easiest way is to use a library. There is FLAP or f90getopt available. Both are open source and licensed under free licenses.
The latter is written by Mark Gates and me, just one module and can be learned in minutes but contains all what is needed to parse GNU- and POSIX-like command-line options. The first is more sophisticated and can be used even in closed-source projects. Check them out.
Furthermore libraries at https://fortranwiki.org/fortran/show/Command-line+arguments
What READ (*,*) does is that it reads from the standard input. For example, the characters entered using the keyboard.
As the question shows COMMAND_ARGUMENT_COUNT() can be used to get the number of the command line arguments.
The accepted answer by High Performance Mark show how to retrieve the individual command line arguments separated by blanks as individual character strings using GET_COMMAND_ARGUMENT(). One can also get the whole command line using GET_COMMAND(). One then has to somehow parse that character-based information into the data in your program.
I very simple cases you just need the program requires, for example, two numbers, so you read one number from arg 1 and another form arg 2. That is simple. Or you can read a triplet of numbers from a single argument if they are comma-separated like 1,2,3 using a simple read(arg,*) nums(1:3).
For general complicated command line parsing one uses libraries such as those mentioned in the answer by Hani. You have set them up so that the library knows the expected syntax of the command line arguments and the data it should fill with the values.
There is a middle ground, that is still relatively simple, but one already have multiple arguments, that correspond to Fortran variables in the program, that may or may not be present. In that case one can use the namelist for the syntax and for the parsing.
Here is an example, the man point is the namelist /cmd/ name, point, flag:
implicit none
real :: point(3)
logical :: flag
character(256) :: name
character(1024) :: command_line
call read_command_line
call parse_command_line
print *, point
print *, "'",trim(name),"'"
print *, flag
contains
subroutine read_command_line
integer :: exenamelength
integer :: io, io2
command_line = ""
call get_command(command = command_line,status = io)
if (io==0) then
call get_command_argument(0,length = exenamelength,status = io2)
if (io2==0) then
command_line = "&cmd "//adjustl(trim(command_line(exenamelength+1:)))//" /"
else
command_line = "&cmd "//adjustl(trim(command_line))//" /"
end if
else
write(*,*) io,"Error getting command line."
end if
end subroutine
subroutine parse_command_line
character(256) :: msg
namelist /cmd/ name, point, flag
integer :: io
if (len_trim(command_line)>0) then
msg = ''
read(command_line,nml = cmd,iostat = io,iomsg = msg)
if (io/=0) then
error stop "Error parsing the command line or cmd.conf " // msg
end if
end if
end subroutine
end
Usage in bash:
> ./command flag=T name=\"data.txt\" point=1.0,2.0,3.0
1.00000000 2.00000000 3.00000000
'data.txt'
T
or
> ./command flag=T name='"data.txt"' point=1.0,2.0,3.0
1.00000000 2.00000000 3.00000000
'data.txt'
T
Escaping the quotes for the string is unfortunately necessary, because bash eats the first quotes.