julia metaprogramming and nloops variable evaluation - macros

I am a noob at metaprogramming so maybe I am not understanding this. I thought the purpose of the #nloops macro in Base.Cartesian was to make it possible to code an arbitrary number of nested for loops, in circumstances where the dimension is unknown a priori. In the documentation for the module, the following example is given:
#nloops 3 i A begin
s += #nref 3 A i
end
which evaluates to
for i_3 = 1:size(A,3)
for i_2 = 1:size(A,2)
for i_1 = 1:size(A,1)
s += A[i_1,i_2,i_3]
end
end
end
Here, the number 3 is known a priori. For my purposes, however, and for the purposes that I thought nloops was created, the number of nested levels is not known ahead of time. So I would not be able to hard code the integer 3. Even in the documentation, it is stated:
The (basic) syntax of #nloops is as follows:
The first argument must be an integer (not a variable) specifying the number of loops.
...
If I assign an integer value - say the dimension of an array that is passed to a function - to some variable, the nloops macro no longer works:
b = 3
#nloops b i A begin
s += #nref b A i
end
This returns an error:
ERROR: LoadError: MethodError: no method matching _nloops(::Symbol, ::Symbol, ::Symbol, ::Expr)
Closest candidates are:
_nloops(::Int64, ::Symbol, ::Symbol, ::Expr...) at cartesian.jl:43
...
I don't know how to have nloops evaluate the b variable as an integer rather than a symbol. I have looked at the documentation and tried various iterations of eval and other functions and macros but it is either interpreted as a symbol or an Expr. What is the correct, julian way to write this?

See supplying the number of expressions:
julia> A = rand(4, 4, 3) # 3D array (Array{Int, 3})
A generated function is kinda like a macro, in that the resulting expression is not returned, but compiled and executed on invocation/call, it also sees the type (and their type parameters of course) of the arguments, ie:
inside the generated function, A is Array{T, N}, not the value of the array.
so T is Int and N is 3!
Here inside the quoted expression, N is interpolated into the expression, with the syntax $N, which evaluates to 3:
julia> #generated function mysum(A::Array{T,N}) where {T,N}
quote
s = zero(T)
#nloops $N i A begin
s += #nref $N A i
end
s
end
end
mysum (generic function with 1 method)
julia> mysum(A)
23.2791638775186
You could construct the expression and then evaluate it, ie.:
julia> s = 0; n = 3;
julia> _3loops = quote
#nloops $n i A begin
global s += #nref $n A i
end
end
quote
#nloops 3 i A begin
global s += #nref(3, A, i)
end
end
julia> eval(_3loops)
julia> s
23.2791638775186
I have scrubbed manually the LineNumberNodes from the AST for readability (there is also MacroTools.prettify, that does it for you).
Running this example in the REPL needs to declare s as global inside the loop in Julia 1.0.

Related

Calling macro from within generated function in Julia

I have been messing around with generated functions in Julia, and have come to a weird problem I do not understand fully: My final goal would involve calling a macro (more specifically #tullio) from within a generated function (to perform some tensor contractions that depend on the input tensors). But I have been having problems, which I narrowed down to calling the macro from within the generated function.
To illustrate the problem, let's consider a very simple example that also fails:
macro my_add(a,b)
return :($a + $b)
end
function add_one_expr(x::T) where T
y = one(T)
return :( #my_add($x,$y) )
end
#generated function add_one_gen(x::T) where T
y = one(T)
return :( #my_add($x,$y) )
end
With these declarations, I find that eval(add_one_expr(2.0)) works just as expected and returns and expression
:(#my_add 2.0 1.0)
which correctly evaluates to 3.0.
However evaluating add_one_gen(2.0) returns the following error:
MethodError: no method matching +(::Type{Float64}, ::Float64)
Doing some research, I have found that #generated actually produces two codes, and in one only the types of the variables can be used. I think this is what is happening here, but I do not understand what is happening at all. It must be some weird interaction between macros and generated functions.
Can someone explain and/or propose a solution? Thank you!
I find it helpful to think of generated functions as having two components: the body and any generated code (the stuff inside a quote..end). The body is evaluated at compile time, and doesn't "know" the values, only the types. So for a generated function taking x::T as an argument, any references to x in the body will actually point to the type T. This can be very confusing. To make things clearer, I recommend the body only refer to types, never to values.
Here's a little example:
julia> #generated function show_val_and_type(x::T) where {T}
quote
println("x is ", x)
println("\$x is ", $x)
println("T is ", T)
println("\$T is ", $T)
end
end
show_val_and_type
julia> show_val_and_type(3)
x is 3
$x is Int64
T is Int64
$T is Int64
The interpolated $x means "take the x from the body (which refers to T) and splice it in.
If you follow the approach of never referring to values in the body, you can test generated functions by removing the #generated, like this:
julia> function add_one_gen(x::T) where T
y = one(T)
quote
#my_add(x,$y)
end
end
add_one_gen
julia> add_one_gen(3)
quote
#= REPL[42]:4 =#
#= REPL[42]:4 =# #my_add x 1
end
That looks reasonable, but when we test it we get
julia> add_one_gen(3)
ERROR: UndefVarError: x not defined
Stacktrace:
[1] macro expansion
# ./REPL[48]:4 [inlined]
[2] add_one_gen(x::Int64)
# Main ./REPL[48]:1
[3] top-level scope
# REPL[49]:1
So let's see what the macro gives us
julia> #macroexpand #my_add x 1
:(Main.x + 1)
It's pointing to Main.x, which doesn't exist. The macro is being too eager, and we need to delay its evaluation. The standard way to do this is with esc. So finally, this works:
julia> macro my_add(a,b)
return :($(esc(a)) + $(esc(b)))
end
#my_add
julia> #generated function add_one_gen(x::T) where T
y = one(T)
quote
#my_add(x,$y)
end
end
add_one_gen
julia> add_one_gen(3)
4

Macros vs functions that accept expressions

When I try the following code snippet, I got that the variable i is not found. Why is that ?
function evalMyExpr(expr,n)
for i in 1:n
eval(expr)
end
end
expr1 = Meta.parse("println(\"hello\")")
expr2 = Meta.parse("println(string(i))")
evalMyExpr(expr1,2) # ok
evalMyExpr(expr2,2) # UndefVarError: i not defined
Note that if I transform it in a macro it works:
macro evalMyExprMacro(expr,n)
quote
for i in 1:$n
$expr
end
end
end
#evalMyExprMacro println(string(i)) 2 # ok
More in general, which is the difference between a function that accepts expressions as parameters and a macro?
Expressions passed to functions are just normal values that are processed at run time. The reason why code fails when it is passed expr2 is because eval evaluates expressions in global scope (in general it is not recommended to use eval in functions). Therefore, as probably variable i is not defined in global scope in your case, you get an error. See an example when i is defined in global scope:
julia> i = 1000
1000
julia> function evalMyExpr(expr,n)
for i in 1:n
eval(expr)
end
end
evalMyExpr (generic function with 1 method)
julia> expr2 = Meta.parse("println(string(i))")
:(println(string(i)))
julia>
julia> evalMyExpr(expr2,2)
1000
1000
Now - in marcos expressions are processed in compile time (before the code is run) so the expression you use is injected into the code generated by the macro that is executed afterwards. You can see the effect by using #macroexpand:
julia> macro evalMyExprMacro(expr,n)
quote
for i in 1:$n
$expr
end
end
end
#evalMyExprMacro (macro with 1 method)
julia> #macroexpand #evalMyExprMacro println(string(i)) 2
quote
#= REPL[23]:3 =#
for #6#i = 1:2
#= REPL[23]:4 =#
(Main.println)((Main.string)(#6#i))
end
end
Observe that the variable name was changed by the macro processing mechanizm to #6#i and it matches the name of the variable that is used in the for loop.

Interpolating an expression into an expression inside of a quote

This question builds off of a previous SO question which was for building expressions from expressions inside of of a macro. However, things got a little trucker when quoting the whole expression. For example, I want to build the expression :(name=val). The following:
macro quotetest(name,val)
quote
nm = Meta.quot($(QuoteNode(name)))
v = Meta.quot($(QuoteNode(val)))
println(nm); println(typeof(nm))
println(v); println(typeof(val))
end
end
#quotetest x 5 # Test case: build :(x=5)
prints out
:x
Expr
$(Expr(:quote, 5))
Expr
showing that I am on the right path: nm and val are the expressions that I want inside of the quote. However, I can't seem to apply the previous solution at this point. For example,
macro quotetest(name,val)
quote
nm = Meta.quot($(QuoteNode(name)))
v = Meta.quot($(QuoteNode(val)))
println(nm); println(typeof(nm))
println(v); println(typeof(v))
println(:($(Expr(:(=),$(QuoteNode(nm)),$(QuoteNode(val))))))
end
end
fails, saying nm is not defined. I tried just interpolating without the QuoteNode, escaping the interpolation $(esc(nm)), etc. I can't seem to find out how to make it build the expression.
I think you are using $ signs more than you need to. Is this what you're looking for?
julia> macro quotetest(name,val)
quote
expr = :($$(QuoteNode(name)) = $$(QuoteNode(val)))
println(expr)
display(expr)
println(typeof(expr))
end
end
#quotetest (macro with 1 method)
julia> #quotetest test 1
test = 1
:(test = 1)
Expr

Why does this Julia macro _not_ require `esc`?

I found an example of an unless macro in Julia here written as follows:
macro unless(test, branch)
quote
if !$test
$branch
end
end
end
However, when I try to use it, it fails (apparently there is a hygiene problem, but I can't figure it out exactly). Here is the test that I used:
x, y = 0, 1
#unless (x == 5) begin # should execute
y = 3
end
#unless (x == 0) begin # should not execute
y = 5
end
#assert y == 3 # FAILS! SAYS y is 0
Now I can make this work by escaping only the branch, not the test:
macro unless(test, branch)
quote
if !$test
$(esc(branch))
end
end
end
My question is: why does it suffice to only escape the branch but not the test? Now I did try macroexpanding. In the first case, without the esc, I get this:
julia> macroexpand(:(#unless (x == 5) begin y = 3 end))
quote # none, line 3:
if !(x == 5) # none, line 4:
begin # none, line 1:
#2#y = 3
end
end
end
Now even though neither macro parameter was escaped, ONLY the y was gensymed! Can anyone explain why this was the case? (I know that the second version works because when I escape the branch, The y doesn't get gensymed and the macro expands to y = 3 as expected. But I'm totally at a loss as to why the x was not gensymed even though there was no use of esc.)
Refer to Julia doc:
Variables within a macro result are classified as either local or
global. A variable is considered local if it is assigned to (and not
declared global), declared local, or used as a function argument name.
Otherwise, it is considered global....
So in this case test part does not assign anything therefore it's variables considered global, but in branch part, y got assigned thus it is considered local and assigning new value to it do not change y in the module scope.

SAS IML use of Mattrib with Macro (symget) in a loop

In an IML proc I have several martices and several vectors with the names of columns:
proc IML;
mydata1 = {1 2 3, 2 3 4};
mydata2 = {1 2, 2 3};
names1 = {'red' 'green' 'blue'};
names2 = {'black' 'white'};
To assign column names to columns in matrices one can copypaste the mattrib statement enough times:
/* mattrib mydata1 colname=names1;*/
/* mattrib mydata2 colname=names2;*/
However, in my case the number of matrices is defined at execution, thus a do loop is needed. The following code
varNumb=2;
do idx=1 to varNumb;
call symputx ('mydataX', cat('mydata',idx));
call symputx ('namesX', cat('names',idx));
mattrib (symget('mydataX')) colname=(symget('namesX'));
end;
print (mydata1[,'red']) (mydata2[,'white']);
quit;
however produces the "Expecting a name" error on the first symget.
Similar question Loop over names in SAS-IML? offers the macro workaround with symget, what produces an error here.
What is the correct way of using mattrib with symget? Is there other way of making a variable from a string than macro?
Any help would be appreciated.
Thanks,
Alex
EDIT1
The problem is in the symget function. The &-sign resolves the name of the matrix contained in the macro variable, the symget only returns the name of the macro.
proc IML;
mydata1 = {1 2 3};
call symputx ('mydataX', 'mydata1');
mydataNew = (symget('mydataX'));
print (&mydataX);
print (symget("mydataX"));
print mydataNew;
quit;
results in
mydata1 :
1 2 3
mydata1
mydataNew :
mydata1
Any ideas?
EDIT2
Function value solves the symget problem in EDIT1
mydataNew = value(symget('mydataX'));
print (&mydataX);
print (value(symget("mydataX")));
print mydataNew;
The mattrib issue but persists.
SOLVED
Thanks Rick, you have opened my eyes to CALL EXECUTE() statement.
When you use CALL SYMPUTX, you should not use quotes for the second argument. Your statement
call symputx ('mydataX', 'mydata1');
assigns the string 'mydata1' to the macro variable.
In general, trying to use macro variables in SAS/IML loops often results in complicated code. See the article Macros and loops in the SAS/IML language for an indication of the issues caused by trying to combine a macro preprocessor with an interactive language. Because the MATTRIB statement expects a literal value for the matrix name, I recomend that you use CALL EXECUTE rather than macro substitution to execute the MATTRIB statement.
You are also having problems because a macro variable is always a scalar string, whereas the column name is a vector of strings. Use the ROWCAT function to concatenate the vector of names into a single string.
The following statements accomplish your objective without using macro variables:
/* Use CALL EXECUTE to set matrix attributes dynamically.
Requires that matrixName and varNames be defined at main scope */
start SetMattrib;
cmd = "mattrib " + matrixName + " colname={" + varNames + "};";
*print cmd; /* for debugging */
call execute(cmd);
finish;
varNumb=2;
do idx=1 to varNumb;
matrixName = cat('mydata',idx);
varNames = rowcat( value(cat('names',idx)) + " " );
run SetMattrib;
end;