I get an error by numba, it is complaining it can't resolve a function.
The minimal code to reproduce my errors is
import numba
import numpy as np
#numba.jit("float64(float64)", nopython=True)
def get_cut_tight_nom(eta):
binning_abseta = np.array([0., 10., 20.])
return np.digitize(eta, binning_abseta)
I don't understand the error message
TypingError: Failed in nopython mode pipeline (step: nopython frontend)
No implementation of function Function(<function digitize at 0x7fdb8c11dee0>) found for signature:
>>> digitize(float64, array(float64, 1d, C))
There are 2 candidate implementations:
- Of which 2 did not match due to:
Overload of function 'digitize': File: numba/np/arraymath.py: Line 3939.
With argument(s): '(float64, array(float64, 1d, C))':
No match.
During: resolving callee type: Function(<function digitize at 0x7fdb8c11dee0>)
During: typing of call at /tmp/ipykernel_309793/3917220133.py (8)
it seems it wants to resolve digitize(float64, array(float64, 1d, C)) and a function with the same signature is not matching?
It's indeed due to the signatures. The Numpy function (without Numba) of np.digitize returns an int64 (scalar or array), not a float64 as you've specified the return type of your function to be.
It seems like the Numba implementation of it requires both to always be arrays, which you'll also have to explicitly add to the signature.
So this for example works for me:
#numba.jit("int64[:](float64[:])", nopython=True)
def get_cut_tight_nom(eta):
binning_abseta = np.array([0., 10., 20.])
return np.digitize(eta, binning_abseta)
Resulting in:
But do you really need the signature in this case? Numba is able to figure it out itself as well, like:
#numba.njit
def get_cut_tight_nom(eta):
...
A signature can still add something if you for example want to explicitly cast float32 inputs to float64 etc.
You can also inspect what signatures Numba comes up with if you run it with some differently typed input. Running it twice with float32 & float64 as the input shows. It can help to highlight where issues like this might arise from.
Related
The numba allows eager compilation by telling the function signature. But, I can not find some information about the list type. My test code is:
import numba as nb
import numpy as np
# #nb.jit(nb.ListType())
def Cal():
return [1, np.zeros(shape=[5, 5]), 3]
a = Cal()
What's the function signature for Cal?
In addition, what if there are three outputs? How to provide the function signature? For example:
def TwoOutput():
return 1, 2
Any suggestion is appreciated
You cannot return the list [1, np.zeros(shape=[5, 5]), 3] from a Numba jitted function. In fact, if you try, Numba will throw few error saying that the "compilation is falling back to object mode WITH looplifting enabled because Function "Cal" failed type inference" due to the output type being not well defined. Indeed, the type of the items in the list needs to be all the same and not objects (called typed list). This is not the case here (reflected list). Reflected lists are not supported by Numba (anymore). What makes Numba fast is the type inference enabling it to generate a fast native code. With Python dynamic objects there is no way to generate a fast code (due to many heavy overhead like type checking, reference counting, allocations, etc.).
Note that you can return a tuple if the output always have the same small number of item defined at compile-time. Also note that Numba can automatically infer the output type so you can use #nb.jit(()) here to enable the eager compilation of the target function.
Put it shortly, Numba is not meant to support/speed-up this use-case. Note that Cython can and be slightly faster. You need not to use reflected lists (nor dynamic objects) if you want to get a fast code.
After updating Beam from 2.33 to 2.35, started getting this error:
def estimate_size(self, unused_value, nested=False):
estimate = 4 # 4 bytes for int32 size prefix
> for key, value in unused_value.items():
E AttributeError: 'list' object has no attribute 'items' [while running 'MyGeneric ParDo Job']
../../../python3.8/site-packages/apache_beam/coders/coder_impl.py:677: AttributeError
This is a method of MapCoderImpl. I don't know Beam enough to know when it's called.
Any thoughts on what might be causing it?
Beam uses Coder to encode and decode a PCollection. From the error message you got, Beam tried to use MapCoder to decode your input data. It expected a dict but received a list instead, hence the error.
Additionally, Beam uses the transform functions' type hints to infer the Coder for output PCollection's elements. My guess is that you might use a wrong return type for your function. Assuming you are implementing a DoFn's process, you yield a list in the function body, then you 'd see the error above if you define the function like this:
def process(self, element, **kwargs) -> List[Dict[A, B]]:
Beam sees the output element's type hint, Dict[A, B], and decides to use MapCoder. You might want to change the type hint to the one below, so that Beam could actually use ListCoder:
def process(self, element, **kwargs) -> List[List[Dict[A, B]]]:
More about benefits of using type hint is describe here.
I'm trying to perform the equivalent of the eval method from Python in nim.
I was under the impression that parseStmt from the macros package should help me with this, but I'm facing a compilation issue that I don't understand.
import macros
echo parseStmt("1 + 2")
I would have expected this to print 3 when executed, but instead the compilation complains that
Error: request to generate code for .compileTime proc: $
I found this thread, and the examples there work, and following this, I was able to make the following program that works as I would expect:
import macros
import strformat
macro eval(value: string): untyped =
result = parseStmt fmt"{value}"
echo eval("1+2")
But I don't undertand why it needs to be written in this way at all. If I inline the statement, let value = "1 + 2"; echo parseStmt fmt"{value}", I get the same compile error as above.
Also, why is parseStmt value different from parseStmt fmt"{value}", in the context of the eval macro above?
What am I missing here?
Thank you in advance for any clarifications!
Unlike Python which is an interpreted language, Nim is compiled. This means that all code is parsed and turned into machine code on compile-time and the program that you end up with doesn't really know anything about Nim at all (at least as long as you don't import the Nim compiler as a module, which is possible). So parseStmt and all macro/template expansion stuff in Nim is done completely during compilation. The error, although maybe a bit hard to read, is trying to tell you that what was passed to $ (which is the convert-to-string operator in Nim, and called by echo on all its arguments) is a compile-time thing that can't be used on runtime. In this case it's because parseStmt doesn't return "3", it returns something like NimNode(kind: nnkIntLit, intVal: 3), and the NimNode type is only available during compile-time. Nim however allows you to run code on compile-time to return other code, this is what a macro does. The eval macro you wrote there takes value which is any statement that resolves to a string during runtime, passed as a NimNode. This is also why result = parseStmt value won't work in your case, because value is not yet a string, but could be something like reading a string from standard input which would result in a string during runtime. However the use of strformat here is a bit confusing and overkill. If you change your code to:
import macros
macro eval(value: static[string]): untyped =
result = parseStmt value
echo eval("1+2")
It will work just fine. This is because we have now told Nim that value needs to be a static i.e. known during compile-time. In this case the string literal "1+2" is obviously known at compile-time, but this could also be a call to a compile-time procedure, or even the output of staticRead which reads a file during compilation.
As you can see Nim is very powerful, but the barrier between compile-time and run-time can sometimes be a bit confusing. Note also that your eval procedure doesn't actually evaluate anything at compile-time, it simply returns the Nim code 1 + 2 so your code ends up being echo 1 + 2. If you want to actually run the code at compile-time you might want to look into compile-time procedures.
Hope this helps shed some light on your issue.
Note: while this answer outlines why this happens, keep in mind that what you're trying to do probably won't result in what you want (which I assumed to be runtime evaluation of expressions).
You're trying to pass a NimNode to parseStmt which expects a string. The fmt macro automatically stringifies anything in the {}, you can omit the fmt by doing $value to turn the node into a string.
As I already noted, this will not work as it does in Python: Nim does not have runtime evaluation. The expression in the string is going to be evaluated at compile time, so a simple example like this will not do what you want:
import std/rdstdin
let x = readLineFromStdin(">")
echo eval(x)
Why?
First of all, because you're stringifying the AST you pass to the eval, it's not the string behind the x variable that's going to get passed to the macro - it's going to be the symbol that denotes the x variable. If you stringify a symbol, you get the underlying identifier, which means that parseStmt will receive "x" as its parameter. This will effect in the string stored in x being printed out, which is wrong.
What you want instead is the following:
import std/rdstdin
import std/macros
macro eval(value: static string): untyped =
result = parseStmt(value)
echo eval("1 + 2")
This prevents runtime-known values from being passed to the macro. You can only pass consts and literals to it now, which is the correct behavior.
While writing a library to read image values, I have the following problem:
I defined a new type called realimage. Within this type a function is referenced, which returns an array as a result.
module typedefinition
implicit none
type realimage
integer :: byteorder = 0
contains
procedure :: initialize => initializereal
procedure :: pxvalues => pxvaluesreal ! Array valued function
end type realimage
contains
function pxvaluesreal(this, x, y) result(val)
implicit none
type(realimage) this
real val(5)
integer :: x, y
...
end function
end module
Compiling the module with gfortran and calling the function with image1%pxvalues(x,y), I always get the following error message:
main.f95: In function ‘testtype’:
main.f95:15: internal compiler error
If I directly call the function in the main program (pxvaluesreal(image1,x,y)), everything works fine.
Is it possible to define the array dimension in the type definition in order to tell the compiler, which are the dimesions of the return value of the function?
Internal compiler errors are always due to a compiler bug. If you are using a recent version of gfortran you should consider reviewing their list of open bugs and perhaps filing a bug report.
Beyond that - your code is not standard compliant - the passed object 'this' must be polymorphic (declare it with CLASS rather than TYPE). Otherwise your specification of the size of the array function result is correct - when you reference the pxvalues binding the compiler knows that the size of the function result is 5 as it 'knows' the interface of the specific procedure pxvaluesreal that the binding is associated with.
I am working in a Scala embedded DSL and macros are becoming a main tool for achieving my purposes. I am getting an error while trying to reuse a subtree from the incoming macro expression into the resulting one. The situation is quite complex, but (I hope) I have simplified it for its understanding.
Suppose we have this code:
val y = transform {
val x = 3
x
}
println(y) // prints 3
where 'transform' is the involved macro. Although it could seem it does absolutely nothing, it is really transforming the shown block into this expression:
3 match { case x => x }
It is done with this macro implementation:
def transform(c: Context)(block: c.Expr[Int]): c.Expr[Int] = {
import c.universe._
import definitions._
block.tree match {
/* {
* val xNam = xVal
* xExp
* }
*/
case Block(List(ValDef(_, xNam, _, xVal)), xExp) =>
println("# " + showRaw(xExp)) // prints Ident(newTermName("x"))
c.Expr(
Match(
xVal,
List(CaseDef(
Bind(xNam, Ident(newTermName("_"))),
EmptyTree,
/* xExp */ Ident(newTermName("x")) ))))
case _ =>
c.error(c.enclosingPosition, "Can't transform block to function")
block // keep original expression
}
}
Notice that xNam corresponds with the variable name, xVal corresponds with its associated value and finally xExp corresponds with the expression containing the variable. Well, if I print the xExp raw tree I get Ident(newTermName("x")), and that is exactly what is set in the case RHS. Since the expression could be modified (for instance x+2 instead of x), this is not a valid solution for me. What I want to do is to reuse the xExp tree (see the xExp comment) while altering the 'x' meaning (it is a definition in the input expression but will be a case LHS variable in the output one), but it launches a long error summarized in:
symbol value x does not exist in org.habla.main.Main$delayedInit$body.apply); see the error output for details.
My current solution consists on the parsing of the xExp to sustitute all the Idents with new ones, but it is totally dependent on the compiler internals, and so, a temporal workaround. It is obvious that the xExp comes along with more information that the offered by showRaw. How can I clean that xExp for allowing 'x' to role the case variable? Can anyone explain the whole picture of this error?
PS: I have been trying unsuccessfully to use the substitute* method family from the TreeApi but I am missing the basics to understand its implications.
Disassembling input expressions and reassembling them in a different fashion is an important scenario in macrology (this is what we do internally in the reify macro). But unfortunately, it's not particularly easy at the moment.
The problem is that input arguments of the macro reach macro implementation being already typechecked. This is both a blessing and a curse.
Of particular interest for us is the fact that variable bindings in the trees corresponding to the arguments are already established. This means that all Ident and Select nodes have their sym fields filled in, pointing to the definitions these nodes refer to.
Here is an example of how symbols work. I'll copy/paste a printout from one of my talks (I don't give a link here, because most of the info in my talks is deprecated by now, but this particular printout has everlasting usefulness):
>cat Foo.scala
def foo[T: TypeTag](x: Any) = x.asInstanceOf[T]
foo[Long](42)
>scalac -Xprint:typer -uniqid Foo.scala
[[syntax trees at end of typer]]// Scala source: Foo.scala
def foo#8339
[T#8340 >: Nothing#4658 <: Any#4657]
(x#9529: Any#4657)
(implicit evidence$1#9530: TypeTag#7861[T#8341])
: T#8340 =
x#9529.asInstanceOf#6023[T#8341];
Test#14.this.foo#8339[Long#1641](42)(scala#29.reflect#2514.`package`#3414.mirror#3463.TypeTag#10351.Long#10361)
To recap, we write a small snippet and then compile it with scalac, asking the compiler to dump the trees after the typer phase, printing unique ids of the symbols assigned to trees (if any).
In the resulting printout we can see that identifiers have been linked to corresponding definitions. For example, on the one hand, the ValDef("x", ...), which represents the parameter of the method foo, defines a method symbol with id=9529. On the other hand, the Ident("x") in the body of the method got its sym field set to the same symbol, which establishes the binding.
Okay, we've seen how bindings work in scalac, and now is the perfect time to introduce a fundamental fact.
If a symbol has been assigned to an AST node,
then subsequent typechecks will never reassign it.
This is why reify is hygienic. You can take a result of reify and insert it into an arbitrary tree (that possibly defines variables with conflicting names) - the original bindings will remain intact. This works because reify preserves the original symbols, so subsequent typechecks won't rebind reified AST nodes.
Now we're all set to explain the error you're facing:
symbol value x does not exist in org.habla.main.Main$delayedInit$body.apply); see the error output for details.
The argument of the transform macro contains both a definition and a reference to a variable x. As we've just learned, this means that the corresponding ValDef and Ident will have their sym fields synchronized. So far, so good.
However unfortunately the macro corrupts the established binding. It recreates the ValDef, but doesn't clean up the sym field of the corresponding Ident. Subsequent typecheck assigns a fresh symbol to the newly created ValDef, but doesn't touch the original Ident that is copied to the result verbatim.
After the typecheck, the original Ident points to a symbol that no longer exists (this is exactly what the error message was saying :)), which leads to a crash during bytecode generation.
So how do we fix the error? Unfortunately there is no easy answer.
One option would be to utilize c.resetLocalAttrs, which recursively erases all symbols in a given AST node. Subsequent typecheck will then reestablish the bindings granted that the code you generated doesn't mess with them (if, for example, you wrap xExp in a block that itself defines a value named x, then you're in trouble).
Another option is to fiddle with symbols. For example, you could write your own resetLocalAttrs that only erases corrupted bindings and doesn't touch the valid ones. You could also try and assign symbols by yourself, but that's a short road to madness, though sometimes one is forced to walk it.
Not cool at all, I agree. We're aware of that and intend to try and fix this fundamental issue sometimes. However right now our hands are full with bugfixing before the final 2.10.0 release, so we won't be able to address the problem in the nearest future. upd. See https://groups.google.com/forum/#!topic/scala-internals/rIyJ4yHdPDU for some additional information.
Bottom line. Bad things happen, because bindings get messed up. Try resetLocalAttrs first, and if it doesn't work, prepare yourself for a chore.