I was following along this notebook (originally written in Julia 0.x, I am using Julia 1.7.1). One of the cells defines the following macro.
macro twice(ex)
quote
$ex
$ex
end
end
On the very next cell, this macro is invoked.
x = 0
#twice println(x += 1)
Replicating this in the REPL (for brevity) results in the following error.
ERROR: syntax: invalid assignment location "Main.x" around REPL[1]:3
Stacktrace:
[1] top-level scope
# REPL[4]:1
So, I understand that x += 1 is somehow causing this problem, but after going through the docs (metaprogramming), I could not figure out why exactly this is an invalid assignment or how to fix this.
#macroexpand #twice println(x += 1) successfully returns the following.
quote
#= REPL[1]:3 =#
Main.println(Main.x += 1)
#= REPL[1]:4 =#
Main.println(Main.x += 1)
end
So, I tried evaling this in the top-level without the Main.s, and it evaluates successfully.
x = 0
eval(quote
println(x += 1)
println(x += 1)
end)
Output:
1
2
But, if I add the module name explicitly, it throws a different error.
eval(quote
Main.println(Main.x += 1)
Main.println(Main.x += 1)
end)
ERROR: cannot assign variables in other modules
Stacktrace:
[1] setproperty!(x::Module, f::Symbol, v::Int64)
# Base ./Base.jl:36
[2] top-level scope
# REPL[14]:3
[3] eval
# ./boot.jl:373 [inlined]
[4] eval(x::Expr)
# Base.MainInclude ./client.jl:453
[5] top-level scope
# REPL[14]:1
I tried a few other things, but these are the only things that I think might be getting somewhere.
Why exactly is the assignment in the first macro code block invalid? Is it for the same reason that evaling in the top-level with the module Main specified fails?
How can this invalid assignment be circumvented, OR how do I port this to Julia 1.7?
Use esc (This "prevents the macro hygiene pass from turning embedded variables into gensym variables"):
julia> macro twice(ex)
esc(quote
$ex
$ex
end)
end;
julia> x=1
1
julia> #twice println(x += 1)
2
3
Related
I have been messing around with generated functions in Julia, and have come to a weird problem I do not understand fully: My final goal would involve calling a macro (more specifically #tullio) from within a generated function (to perform some tensor contractions that depend on the input tensors). But I have been having problems, which I narrowed down to calling the macro from within the generated function.
To illustrate the problem, let's consider a very simple example that also fails:
macro my_add(a,b)
return :($a + $b)
end
function add_one_expr(x::T) where T
y = one(T)
return :( #my_add($x,$y) )
end
#generated function add_one_gen(x::T) where T
y = one(T)
return :( #my_add($x,$y) )
end
With these declarations, I find that eval(add_one_expr(2.0)) works just as expected and returns and expression
:(#my_add 2.0 1.0)
which correctly evaluates to 3.0.
However evaluating add_one_gen(2.0) returns the following error:
MethodError: no method matching +(::Type{Float64}, ::Float64)
Doing some research, I have found that #generated actually produces two codes, and in one only the types of the variables can be used. I think this is what is happening here, but I do not understand what is happening at all. It must be some weird interaction between macros and generated functions.
Can someone explain and/or propose a solution? Thank you!
I find it helpful to think of generated functions as having two components: the body and any generated code (the stuff inside a quote..end). The body is evaluated at compile time, and doesn't "know" the values, only the types. So for a generated function taking x::T as an argument, any references to x in the body will actually point to the type T. This can be very confusing. To make things clearer, I recommend the body only refer to types, never to values.
Here's a little example:
julia> #generated function show_val_and_type(x::T) where {T}
quote
println("x is ", x)
println("\$x is ", $x)
println("T is ", T)
println("\$T is ", $T)
end
end
show_val_and_type
julia> show_val_and_type(3)
x is 3
$x is Int64
T is Int64
$T is Int64
The interpolated $x means "take the x from the body (which refers to T) and splice it in.
If you follow the approach of never referring to values in the body, you can test generated functions by removing the #generated, like this:
julia> function add_one_gen(x::T) where T
y = one(T)
quote
#my_add(x,$y)
end
end
add_one_gen
julia> add_one_gen(3)
quote
#= REPL[42]:4 =#
#= REPL[42]:4 =# #my_add x 1
end
That looks reasonable, but when we test it we get
julia> add_one_gen(3)
ERROR: UndefVarError: x not defined
Stacktrace:
[1] macro expansion
# ./REPL[48]:4 [inlined]
[2] add_one_gen(x::Int64)
# Main ./REPL[48]:1
[3] top-level scope
# REPL[49]:1
So let's see what the macro gives us
julia> #macroexpand #my_add x 1
:(Main.x + 1)
It's pointing to Main.x, which doesn't exist. The macro is being too eager, and we need to delay its evaluation. The standard way to do this is with esc. So finally, this works:
julia> macro my_add(a,b)
return :($(esc(a)) + $(esc(b)))
end
#my_add
julia> #generated function add_one_gen(x::T) where T
y = one(T)
quote
#my_add(x,$y)
end
end
add_one_gen
julia> add_one_gen(3)
4
All these following lines of code are Julia expressions:
x = 10
1 + 1
println("hi")
if you want to pass an expression to a macro, it works like this. Macro foo just returns the given expression, which will be executed:
macro foo(ex)
return ex
end
#foo println("yes") # prints yes
x = #foo 1+1
println(x) # prints 2
If you want to convert a string into an expression, you can use Meta.parse():
string = "1+1"
expr = Meta.parse(string)
x = #foo expr
println(x) # prints 1 + 1
But, obviously, the macro treats expr as a symbol. What am i getting wrong here?
Thanks in advance!
Macro hygiene is important "macros must ensure that the variables they introduce in their returned expressions do not accidentally clash with existing variables in the surrounding code they expand into." There is a section in the docs. It is easiest just to show a simple case:
macro foo(x)
return :($x)
end
When you enter an ordinary expression in the REPL, it is evaluated immediately. To suppress that evaluation, surround the expression with :( ).
julia> 1 + 1
2
julia> :(1 + 1)
:(1 + 1)
# note this is the same result as you get using Meta.parse
julia> Meta.parse("1 + 1")
:(1 + 1)
So, Meta.parse will convert an appropriate string to an expression. And if you eval the result, the expression will be evaluated. Note that printing a simple expression removes the outer :( )
julia> expr = Meta.parse("1 + 1")
:(1 + 1)
julia> print(expr)
1 + 1
julia> result = eval(expr)
2
Usually, macros are used to manipulate things before the usual evaluation of expressions; they are syntax transformations, mostly. Macros are performed before other source code is compiled/evaluated/executed.
Rather than seeking a macro that evaluates a string as if it were typed directly into the REPL (without quotes), use this function instead.
evalstr(x::AbstractString) = eval(Meta.parse(x))
While I do not recommend this next macro, it is good to know the technique.
A macro named <name>_str is used like this <name>"<string contents>" :
julia> macro eval_str(x)
:(eval(Meta.parse($x)))
end
julia> eval"1 + 1"
2
(p.s. do not reuse Base function names as variable names, use str not string)
Please let me know if there is something I have not addressed.
When I try the following code snippet, I got that the variable i is not found. Why is that ?
function evalMyExpr(expr,n)
for i in 1:n
eval(expr)
end
end
expr1 = Meta.parse("println(\"hello\")")
expr2 = Meta.parse("println(string(i))")
evalMyExpr(expr1,2) # ok
evalMyExpr(expr2,2) # UndefVarError: i not defined
Note that if I transform it in a macro it works:
macro evalMyExprMacro(expr,n)
quote
for i in 1:$n
$expr
end
end
end
#evalMyExprMacro println(string(i)) 2 # ok
More in general, which is the difference between a function that accepts expressions as parameters and a macro?
Expressions passed to functions are just normal values that are processed at run time. The reason why code fails when it is passed expr2 is because eval evaluates expressions in global scope (in general it is not recommended to use eval in functions). Therefore, as probably variable i is not defined in global scope in your case, you get an error. See an example when i is defined in global scope:
julia> i = 1000
1000
julia> function evalMyExpr(expr,n)
for i in 1:n
eval(expr)
end
end
evalMyExpr (generic function with 1 method)
julia> expr2 = Meta.parse("println(string(i))")
:(println(string(i)))
julia>
julia> evalMyExpr(expr2,2)
1000
1000
Now - in marcos expressions are processed in compile time (before the code is run) so the expression you use is injected into the code generated by the macro that is executed afterwards. You can see the effect by using #macroexpand:
julia> macro evalMyExprMacro(expr,n)
quote
for i in 1:$n
$expr
end
end
end
#evalMyExprMacro (macro with 1 method)
julia> #macroexpand #evalMyExprMacro println(string(i)) 2
quote
#= REPL[23]:3 =#
for #6#i = 1:2
#= REPL[23]:4 =#
(Main.println)((Main.string)(#6#i))
end
end
Observe that the variable name was changed by the macro processing mechanizm to #6#i and it matches the name of the variable that is used in the for loop.
I am a noob at metaprogramming so maybe I am not understanding this. I thought the purpose of the #nloops macro in Base.Cartesian was to make it possible to code an arbitrary number of nested for loops, in circumstances where the dimension is unknown a priori. In the documentation for the module, the following example is given:
#nloops 3 i A begin
s += #nref 3 A i
end
which evaluates to
for i_3 = 1:size(A,3)
for i_2 = 1:size(A,2)
for i_1 = 1:size(A,1)
s += A[i_1,i_2,i_3]
end
end
end
Here, the number 3 is known a priori. For my purposes, however, and for the purposes that I thought nloops was created, the number of nested levels is not known ahead of time. So I would not be able to hard code the integer 3. Even in the documentation, it is stated:
The (basic) syntax of #nloops is as follows:
The first argument must be an integer (not a variable) specifying the number of loops.
...
If I assign an integer value - say the dimension of an array that is passed to a function - to some variable, the nloops macro no longer works:
b = 3
#nloops b i A begin
s += #nref b A i
end
This returns an error:
ERROR: LoadError: MethodError: no method matching _nloops(::Symbol, ::Symbol, ::Symbol, ::Expr)
Closest candidates are:
_nloops(::Int64, ::Symbol, ::Symbol, ::Expr...) at cartesian.jl:43
...
I don't know how to have nloops evaluate the b variable as an integer rather than a symbol. I have looked at the documentation and tried various iterations of eval and other functions and macros but it is either interpreted as a symbol or an Expr. What is the correct, julian way to write this?
See supplying the number of expressions:
julia> A = rand(4, 4, 3) # 3D array (Array{Int, 3})
A generated function is kinda like a macro, in that the resulting expression is not returned, but compiled and executed on invocation/call, it also sees the type (and their type parameters of course) of the arguments, ie:
inside the generated function, A is Array{T, N}, not the value of the array.
so T is Int and N is 3!
Here inside the quoted expression, N is interpolated into the expression, with the syntax $N, which evaluates to 3:
julia> #generated function mysum(A::Array{T,N}) where {T,N}
quote
s = zero(T)
#nloops $N i A begin
s += #nref $N A i
end
s
end
end
mysum (generic function with 1 method)
julia> mysum(A)
23.2791638775186
You could construct the expression and then evaluate it, ie.:
julia> s = 0; n = 3;
julia> _3loops = quote
#nloops $n i A begin
global s += #nref $n A i
end
end
quote
#nloops 3 i A begin
global s += #nref(3, A, i)
end
end
julia> eval(_3loops)
julia> s
23.2791638775186
I have scrubbed manually the LineNumberNodes from the AST for readability (there is also MacroTools.prettify, that does it for you).
Running this example in the REPL needs to declare s as global inside the loop in Julia 1.0.
I found an example of an unless macro in Julia here written as follows:
macro unless(test, branch)
quote
if !$test
$branch
end
end
end
However, when I try to use it, it fails (apparently there is a hygiene problem, but I can't figure it out exactly). Here is the test that I used:
x, y = 0, 1
#unless (x == 5) begin # should execute
y = 3
end
#unless (x == 0) begin # should not execute
y = 5
end
#assert y == 3 # FAILS! SAYS y is 0
Now I can make this work by escaping only the branch, not the test:
macro unless(test, branch)
quote
if !$test
$(esc(branch))
end
end
end
My question is: why does it suffice to only escape the branch but not the test? Now I did try macroexpanding. In the first case, without the esc, I get this:
julia> macroexpand(:(#unless (x == 5) begin y = 3 end))
quote # none, line 3:
if !(x == 5) # none, line 4:
begin # none, line 1:
#2#y = 3
end
end
end
Now even though neither macro parameter was escaped, ONLY the y was gensymed! Can anyone explain why this was the case? (I know that the second version works because when I escape the branch, The y doesn't get gensymed and the macro expands to y = 3 as expected. But I'm totally at a loss as to why the x was not gensymed even though there was no use of esc.)
Refer to Julia doc:
Variables within a macro result are classified as either local or
global. A variable is considered local if it is assigned to (and not
declared global), declared local, or used as a function argument name.
Otherwise, it is considered global....
So in this case test part does not assign anything therefore it's variables considered global, but in branch part, y got assigned thus it is considered local and assigning new value to it do not change y in the module scope.