how to use output of one file as input to another file in python - import

like I have a code
def poly (x,a):
j=0
ans=[0]*len(x)
while (j<len(x)):
i=0
while(i<len(a)):
ans[j] = ans[j] + a[i]*((x[j])**i)
i=i+1
else:
j=j+1
else:
print ans
a=[1,1,1]
x=[1,1,1]
print poly(x,a)
now I want x in this file from another file containg program like:
def mult(x,q):
i=0
while(i<len(x)):
x[i] = x[i]*q
i=i+1
else:
print x
x=[1,1,1]
q=2
print mult(x,q)

Maybe this is a possible solution.
First: In your function definitions where you have "print ans" and "print x", I think you want return statements. i.e. "return ans" and "return x".
e.g.
def poly (x,a):
j=0
ans=[0]*len(x)
while (j<len(x)):
i=0
while(i<len(a)):
ans[j] = ans[j] + a[i]*((x[j])**i)
i=i+1
else:
j=j+1
else:
return ans # changed print to return
Then, assuming you have two separate program files (let's call them poly.py and mult.py) in which the given functions are separately defined you could write a new program file (e.g. polymult.py) which looks something like this:
#start code
# import programs poly.py and mult.py
# this will execute commands in these files
# but gives access to functions defined within
from poly import *
from mult import *
# Main
# define/reset the variables we need/want to use
a=[1,1,1]
x=[1,1,1]
q=2
# pass mult which returns new x into poly
print poly(mult(x,q),a)
#end code
A caveat to the way I wrote the above is that the files poly.py and mult.py have to be in the same directory/folder as the program importing them. (A simple alternative would be to simply define the functions (paste them in) in place of the import statements)
Also, if you just want to use them as imported functions as in my example you could remove all the code other than the actual function def's. Otherwise, when you execute polymult.py you will see the extra screen prints executed during the import of poly.py and mult.py

Related

Can operations on a numpy.memmap be deferred?

Consider this example:
import numpy as np
a = np.array(1)
np.save("a.npy", a)
a = np.load("a.npy", mmap_mode='r')
print(type(a))
b = a + 2
print(type(b))
which outputs
<class 'numpy.core.memmap.memmap'>
<class 'numpy.int32'>
So it seems that b is not a memmap any more, and I assume that this forces numpy to read the whole a.npy, defeating the purpose of the memmap. Hence my question, can operations on memmaps be deferred until access time?
I believe subclassing ndarray or memmap could work, but don't feel confident enough about my Python skills to try it.
Here is an extended example showing my problem:
import numpy as np
# create 8 GB file
# np.save("memmap.npy", np.empty([1000000000]))
# I want to print the first value using f and memmaps
def f(value):
print(value[1])
# this is fast: f receives a memmap
a = np.load("memmap.npy", mmap_mode='r')
print("a = ")
f(a)
# this is slow: b has to be read completely; converted into an array
b = np.load("memmap.npy", mmap_mode='r')
print("b + 1 = ")
f(b + 1)
Here's a simple example of an ndarray subclass that defers operations on it until a specific element is requested by indexing.
I'm including this to show that it can be done, but it almost certainly will fail in novel and unexpected ways, and require substantial work to make it usable.
For a very specific case it may be easier than redesigning your code to solve the problem in a better way.
I'd recommend reading over these examples from the docs to help understand how it works.
import numpy as np
class Defered(np.ndarray):
"""
An array class that deferrs calculations applied to it, only
calculating them when an index is requested
"""
def __new__(cls, arr):
arr = np.asanyarray(arr).view(cls)
arr.toApply = []
return arr
def __array_ufunc__(self, ufunc, method, *inputs, **kwargs):
## Convert all arguments to ndarray, otherwise arguments
# of type Defered will cause infinite recursion
# also store self as None, to be replaced later on
newinputs = []
for i in inputs:
if i is self:
newinputs.append(None)
elif isinstance(i, np.ndarray):
newinputs.append(i.view(np.ndarray))
else:
newinputs.append(i)
## Store function to apply and necessary arguments
self.toApply.append((ufunc, method, newinputs, kwargs))
return self
def __getitem__(self, idx):
## Get index and convert to regular array
sub = self.view(np.ndarray).__getitem__(idx)
## Apply stored actions
for ufunc, method, inputs, kwargs in self.toApply:
inputs = [i if i is not None else sub for i in inputs]
sub = super().__array_ufunc__(ufunc, method, *inputs, **kwargs)
return sub
This will fail if modifications are made to it that don't use numpy's universal functions. For instance percentile and median aren't based on ufuncs, and would end up loading the entire array. Likewise, if you pass it to a function that iterates over the array, or applies an index to substantial amounts the entire array will be loaded.
This is just how python works. By default numpy operations return a new array, so b never exists as a memmap - it is created when + is called on a.
There's a couple of ways to work around this. The simplest is to do all operations in place,
a += 1
This requires loading the memory mapped array for reading and writing,
a = np.load("a.npy", mmap_mode='r+')
Of course this isn't any good if you don't want to overwrite your original array.
In this case you need to specify that b should be memmapped.
b = np.memmap("b.npy", mmap+mode='w+', dtype=a.dtype, shape=a.shape)
Assigning can be done by using the out keyword provided by numpy ufuncs.
np.add(a, 2, out=b)

Why does Matlab's cellfun function not honor imports? How to propagate import statement to anonymous functions?

I want to test a cell array of objects for objects of a certain class.
I organize my files in packages and use import statements to make my code shorter.
The problem: cellfun(#(o) isa(o,'MyClassName'),myCellArray) seems to ignore the import declaration, returning false for cell array elements where invoking isa(myCellArray{i}, 'MyClassName') would yield true.
How can I make cellfun honor my import statements?
EDIT: Might the import statement not propagate to the anonymous function inside cellfun? If yes, how can I archieve this?
Minimal (Not) Working Example
Files:
myfun.m
+pkg/
+pkg/MyClass.m
myfun.m:
% cat <<"%EOF" > myfun.m # paste into shell if on UNIX
function myfun()
import pkg.MyClass;
o{1} = MyClass();
x = cellfun(#(d) isa(d,'MyClass'),o);
fprintf('cellfun: %d\n',x);
b = isa(o{1},'MyClass');
fprintf('direct: %d\n',b);
fprintf('classes of o: \n');
cellfun(#class, o,'UniformOutput',false)
end
%EOF
+pkg/MyClass.m:
% mkdir "+pkg"; cat <<"%EOF" >"+pkg/MyClass.m" # paste to shell if on UNIX
classdef MyClass
end
%EOF
My Output is:
>> myfun
cellfun: 0
direct: 1
classes of o:
ans =
'pkg.MyClass'
If I move MyClass.m to the same directory as myfun.m and remove the import ... line in myfun.m:
>> myfun
cellfun: 1
direct: 1
classes of o:
ans =
'MyClass'
>>

SAS IML use of Mattrib with Macro (symget) in a loop

In an IML proc I have several martices and several vectors with the names of columns:
proc IML;
mydata1 = {1 2 3, 2 3 4};
mydata2 = {1 2, 2 3};
names1 = {'red' 'green' 'blue'};
names2 = {'black' 'white'};
To assign column names to columns in matrices one can copypaste the mattrib statement enough times:
/* mattrib mydata1 colname=names1;*/
/* mattrib mydata2 colname=names2;*/
However, in my case the number of matrices is defined at execution, thus a do loop is needed. The following code
varNumb=2;
do idx=1 to varNumb;
call symputx ('mydataX', cat('mydata',idx));
call symputx ('namesX', cat('names',idx));
mattrib (symget('mydataX')) colname=(symget('namesX'));
end;
print (mydata1[,'red']) (mydata2[,'white']);
quit;
however produces the "Expecting a name" error on the first symget.
Similar question Loop over names in SAS-IML? offers the macro workaround with symget, what produces an error here.
What is the correct way of using mattrib with symget? Is there other way of making a variable from a string than macro?
Any help would be appreciated.
Thanks,
Alex
EDIT1
The problem is in the symget function. The &-sign resolves the name of the matrix contained in the macro variable, the symget only returns the name of the macro.
proc IML;
mydata1 = {1 2 3};
call symputx ('mydataX', 'mydata1');
mydataNew = (symget('mydataX'));
print (&mydataX);
print (symget("mydataX"));
print mydataNew;
quit;
results in
mydata1 :
1 2 3
mydata1
mydataNew :
mydata1
Any ideas?
EDIT2
Function value solves the symget problem in EDIT1
mydataNew = value(symget('mydataX'));
print (&mydataX);
print (value(symget("mydataX")));
print mydataNew;
The mattrib issue but persists.
SOLVED
Thanks Rick, you have opened my eyes to CALL EXECUTE() statement.
When you use CALL SYMPUTX, you should not use quotes for the second argument. Your statement
call symputx ('mydataX', 'mydata1');
assigns the string 'mydata1' to the macro variable.
In general, trying to use macro variables in SAS/IML loops often results in complicated code. See the article Macros and loops in the SAS/IML language for an indication of the issues caused by trying to combine a macro preprocessor with an interactive language. Because the MATTRIB statement expects a literal value for the matrix name, I recomend that you use CALL EXECUTE rather than macro substitution to execute the MATTRIB statement.
You are also having problems because a macro variable is always a scalar string, whereas the column name is a vector of strings. Use the ROWCAT function to concatenate the vector of names into a single string.
The following statements accomplish your objective without using macro variables:
/* Use CALL EXECUTE to set matrix attributes dynamically.
Requires that matrixName and varNames be defined at main scope */
start SetMattrib;
cmd = "mattrib " + matrixName + " colname={" + varNames + "};";
*print cmd; /* for debugging */
call execute(cmd);
finish;
varNumb=2;
do idx=1 to varNumb;
matrixName = cat('mydata',idx);
varNames = rowcat( value(cat('names',idx)) + " " );
run SetMattrib;
end;

SAS IML pass reference to variable defined in macro

In SAS/IML I try to pass to a user defined module a reference to a variable that is defined in macro. The module changes the variable value.
Since call of the function is in the do-loop I cannot use &-sign. However use of symget does not work. Here is my code.
proc iml;
start funcReference(argOut);
print "funcReference " argOut;
argOut = 5;
finish funcReference;
store module=funcReference;
quit;
proc IML;
mydata1 = {1 2 3};
call symput ('macVar', 'mydata1');
load module=funcReference;
run funcReference(symget('macVar'));
print mydata1;
quit;
The output shows that variable mydata1 have not changed:
argOut
funcReference mydata1
mydata1
1 2 3
Any ideas?
SOLVED
Thanks a lot!
You are sending in a temporary scalar matrix (the result of the SYMGET call). That temporary variable is being updated and promptly vanishes, as explained in the article "Oh, those pesky temporary variables!"
Instead of macro variables (which are text strings), you should use the VALUE and VALSET functions, as described in the article "Indirect assignment: How to create and use matrices named x1, x2,..., xn" You need to send in a real matrix in order for the values to be updated correctly, as follows:
proc IML;
load module=funcReference;
mydata1 = {1 2 3};
call symput('macVar', 'mydata1');
matrixName = symget('macVar'); /* matrix named &mydata1 */
z = value(matrixName); /* z contains data */
run funcReference(z); /* update values in z */
call valset(matrixName, z); /* update data in &mydata1 */
print mydata1;

Loop through values of a SPSS variable inside of a Macro

How can I pass the values of a specific variable to a list-processing loop inside a macro?
Let's say, as an simplified example, I've got a variable foo which contains the values 1,4,12,33 and 51.
DATA LIST FREE / foo (F2) .
BEGIN DATA
1
4
12
33
51
END DATA.
And a macro that does some stuff with those values.
For testing reasons this Macro will just echo those values.
I'd like to find a way to run a routine that works like the following:
DEFINE !testmacro (list !CMDEND)
!DO !i !IN (!list)
ECHO !QUOTE(!i).
!DOEND.
!ENDDEFINE.
!testmacro list = 1 4 12 33 51. * <- = values from foo.
This is a situation where using the Python apis would be a good choice.
I made myself a little bit familiar with Python recently :-)
So this is what I worked out.
If the variable is a numeric:
BEGIN PROGRAM PYTHON.
import spss,spssdata
foolist = [element[0] for element in spssdata.Spssdata('foo').fetchall()]
foostring = " ".join(str(int(i)) for i in foolist)
spss.Submit("!testmacro list = %(foostring)s." %locals())
END PROGRAM.
If the variable is a string:
BEGIN PROGRAM PYTHON.
import spss,spssdata
foolist = [element[0].strip() for element in spssdata.Spssdata('bar').fetchall()]
foostring = " ".join(foolist)
spss.Submit("!testmacro list = %(foostring)s." %locals())
END PROGRAM.
Variants
Duplicates removed and list is orderd
BEGIN PROGRAM PYTHON.
import spss,spssdata
foolist = sorted(set([element[0] for element in spssdata.Spssdata('foo').fetchall()]))
foostring = " ".join(str(int(i)) for i in foolist)
spss.Submit("!testmacro list = %(foostring)s." %locals())
END PROGRAM.
Duplicates removed and items in order of first appearance in the dataset
Here, I use a function which I retrieved from Peter Bengtsson's Homepage (peterbe.com)
BEGIN PROGRAM PYTHON.
import spss,spssdata
def uniquify(seq, idfun=None):
# order preserving
if idfun is None:
def idfun(x): return x
seen = {}
result = []
for item in seq:
marker = idfun(item)
if marker in seen: continue
seen[marker] = 1
result.append(item)
return result
foolist = uniquify([element[0] for element in spssdata.Spssdata('foo').fetchall()])
foostring = " ".join(str(int(i)) for i in foolist)
spss.Submit("!testmacro list = %(foostring)s." %locals())
END PROGRAM.
Non-Python Solution
Not that I recommend it, but there is even a way to do this without Python.
I got the basic Idea from a SPSS programming book, which goes as follows:
Use the WRITE command to create a text file with the wanted command and variable values and include it with the insert command.
DATASET COPY foolistdata.
DATASET ACTIVATE foolistdata.
AGGREGATE OUTFILE=* MODE=ADDVARIABLES
/BREAK
/NumberOfCases=N.
* Variable which contains the command as string in the first case.
STRING macrocommand (A18).
IF ($casenum=1) macroCommand = "!testmacro list = ".
EXECUTE.
* variable which contains a period (.) in the last case,
* for the ending of the command string.
STRING commandEnd (A1).
IF ($casenum=NumberOfCases) commandEnd = ".".
* Write the 'table' with the command and variable values into a textfile.
WRITE OUTFILE="macrocommand.txt" /macrocommand bar commandEnd.
EXECUTE.
* Macrocall.
INSERT FILE ="macrocommand.txt".