from typing import Dict, List, Any, AnyStr, TypeVar
def abc(xyz: str) -> Dict[AnyStr, Any]:
return {"abc": 1}
And I use mypy to check this file. It's giving an Error.
Below is the Error message
"Dict entry 0 has incompatible type "str": "int"; expected "bytes":
"Any""
But I don't know why
The issue is that AnyStr is actually an alias for a typevar. This means your program is actually exactly equivalent to writing:
from typing import Dict, Any, AnyStr, TypeVar
T = TypeVar('T', str, bytes)
def abc(xyz: str) -> Dict[T, Any]:
return {"abc": 1}
This, however, presents us with a problem: how is mypy supposed to infer which of the two possible alternatives you wanted for T?
There are three possible fixes. You can either...
Find some way of using AnyStr at least two or more times within your type signature. For example, perhaps you decide this is really more what you meant?
def abc(xyz: AnyStr) -> Dict[AnyStr, Any]:
# etc
Use Union[str, bytes] instead of AnyStr:
from typing import Union, Dict, Any
def abc(xyz: str) -> Dict[Union[str, bytes], Any]:
# etc
If the type signature is starting to get uncomfortably long, you can shorten it by using a type alias:
from typing import Union, Dict, Any
# Note: this isn't a great type alias name, but whatever
UnionStr = Union[str, bytes]
def abc(xyz: str) -> Dict[UnionStr, Any]:
# etc
Related
What is the correct way to add type hints to the following function?
from typing import Callable
def format_callback(f: Callable) -> Callable:
"""Function to wrap a function to use as a click callback.
Taken from https://stackoverflow.com/a/42110044/8056572
"""
return lambda _, __, x: f(x)
Now mypy is complaining with Missing type parameters for generic type "Callable"
The code needs to be compatible with both Python 3.9 and 3.10. I can use typing_extensions if needed.
Edit:
The following passes mypy but has too many Any's for my taste. Is there a better way?
from typing import Any
from typing import Callable
import click
def format_callback(f: Callable[[Any], Any]) -> Callable[[click.Context, dict[str, Any], Any], Any]:
"""Function to wrap a function to use as a click callback.
Taken from https://stackoverflow.com/a/42110044/8056572
"""
return lambda _, __, x: f(x)
Without looking at click, the immediate fix you can do is to provide type variables that match f(x):
from typing import Any, Callable, TypeVar
import click
ArgT = TypeVar("ArgT")
ReturnT = TypeVar("ReturnT")
def format_callback(
f: Callable[[ArgT], ReturnT]
) -> Callable[[click.Context, dict[str, Any], ArgT], ReturnT]:
return lambda _, __, x: f(x)
This will guard you against bad typing in the internal body of format_callback.
A brief scan of click seems to indicate that you want to pass the return value of format_callback to one of the following class constructors or their subclass constructors:
click.core.Parameter.__init__::callback:
callback: t.Optional[t.Callable[[Context, "Parameter", t.Any], t.Any]]
click.core.Command.__init__::callback
callback: t.Optional[t.Callable[..., t.Any]]
Now, based on the other answer that you linked to, which passes the keyword argument callback to #click.argument and #click.option, it seems like you'd actually want to use click.core.Parameter.__init__, because the decorators' fallback classes are ArgumentClass = click.core.Argument and OptionClass = click.core.Option, respectively, which are both subclasses of click.Core.Parameter. This means that the second argument to the return Callable type cannot be dict[str, Any], according to the type annotation for click.core.Parameter.__init__::callback. Then, you really should have this instead:
def format_callback(
f: Callable[[ArgT], ReturnT]
) -> Callable[[click.Context, click.Parameter, ArgT], ReturnT]:
return lambda _, __, x: f(x)
Since you're discarding click.Context and click.Parameter as _, __ in your lambda, providing these are only for documentation purposes, of course. They could easily be object, object instead.
I wanted to verify substring existence in a string and came up with:
if "haystack".find("needle") != -1:
But I would have prefered:
if "needle" in "haystack": as in python.
Though we get:
Error: type mismatch: got (string, string) but expected one of: proc
contains[T](s: Slice[T]; value: T): bool proc contains[T](x: set[T];
y: T): bool proc contains[T](a: openArray[T]; item: T): bool
note that if you import strutils it starts to work.
You already gave the answer yourself, import strutils to get the contains proc for strings. Nim automatically used the contains proc for the in operator.
I am building a GraphQL endpoint for an API using Finch, Circe and Sangria. The variables that come through in a GraphQL query are basically an arbitrary JSON object (let's assume there's no nesting). So for example, in my test code as Strings, here are two examples:
val variables = List(
"{\n \"foo\": 123\n}",
"{\n \"foo\": \"bar\"\n}"
)
The Sangria API expects a type for these of Map[String, Any].
I've tried a bunch of ways but have so far been unable to write a Decoder for this in Circe. Any help appreciated.
The Sangria API expects a type for these of Map[String, Any]
This is not true. Variables for an execution in sangria can be of an arbitrary type T, the only requirement that you have an instance of InputUnmarshaller[T] type class for it. All marshalling integration libraries provide an instance of InputUnmarshaller for correspondent JSON AST type.
This means that sangria-circe defines InputUnmarshaller[io.circe.Json] and you can import it with import sangria.marshalling.circe._.
Here is a small and self-contained example of how you can use circe Json as a variables:
import io.circe.Json
import sangria.schema._
import sangria.execution._
import sangria.macros._
import sangria.marshalling.circe._
val query =
graphql"""
query ($$foo: Int!, $$bar: Int!) {
add(a: $$foo, b: $$bar)
}
"""
val QueryType = ObjectType("Query", fields[Unit, Unit](
Field("add", IntType,
arguments = Argument("a", IntType) :: Argument("b", IntType) :: Nil,
resolve = c ⇒ c.arg[Int]("a") + c.arg[Int]("b"))))
val schema = Schema(QueryType)
val vars = Json.obj(
"foo" → Json.fromInt(123),
"bar" → Json.fromInt(456))
val result: Future[Json] =
Executor.execute(schema, query, variables = vars)
As you can see in this example, I used io.circe.Json as variables for an execution. The execution would produce following result JSON:
{
"data": {
"add": 579
}
}
Here's a decoder that works.
type GraphQLVariables = Map[String, Any]
val graphQlVariablesDecoder: Decoder[GraphQLVariables] = Decoder.instance { c =>
val variablesString = c.downField("variables").focus.flatMap(_.asString)
val parsedVariables = variablesString.flatMap { str =>
val variablesJsonObject = io.circe.jawn.parse(str).toOption.flatMap(_.asObject)
variablesJsonObject.map(j => j.toMap.transform { (_, value: Json) =>
val transformedValue: Any = value.fold(
(),
bool => bool,
number => number.toDouble,
str => str,
array => array.map(_.toString),
obj => obj.toMap.transform((s: String, json: Json) => json.toString)
)
transformedValue
})
}
parsedVariables match {
case None => left(DecodingFailure(s"Unable to decode GraphQL variables", c.history))
case Some(variables) => right(variables)
}
}
We basically parse the JSON, turn it into a JsonObject, then transform the values within the object fairly simplistically.
Although the above answers work for the specific case of Sangria, I'm interested in the original question: What's the best approach in Circe (which generally assumes that all types are known up front) for dealing with arbitrary chunks of Json?
It's fairly common when encoding/decoding Json that 95% of the Json is specified, but the last 5% is some type of "additional properties" chunk which can be any Json object.
Solutions I've played with:
Encode/Decode the free-form chunk as Map[String,Any]. This means you'll have to introduce implicit encoders/decoders for Map[String, Any], which can be done, but is dangerous as that implicit can be pulled into places you didn't intend.
Encode/Decode the free-form chunk as Map[String, Json]. This is the easiest approach and is supported out of the box in Circe. But now the Json serialization logic has leaked out into your API (often you'll want to keep the Json stuff completely wrapped, so you can swap in other non-json formats later).
Encode/Decode to a String, where the string is required to be a valid Json chunk. At least you haven't locked your API into a specific Json library, but it doesn't feel very nice to have to ask your users to create Json chunks in this manual way.
Create a custom trait hierarchy to hold the data (e.g sealed trait Free; FreeInt(i: Int) extends Free; FreeMap(m: Map[String, Free] extends Free; ...). Now you you can create specific encoders/decoders for it. But what you've really done is replicate the Json type hierarchy that already exists in Circe.
I'm leaning more toward option 3. Since it's the most flexible, and will introduce the least dependencies in the API. But none of them are entirely satisfying. Any other ideas?
I am currently extracting some metrics from different data sources and storing them in a map of type Map[String,Any] where the key corresponds to the metric name and the value corresponds to the metric value. I need this to be more or less generic, which means that values types can be primitive types or lists of primitive types.
I would like to serialize this map to a JSON-formatted string and for that I am using json4s library. The thing is that it does not seem possible and I don't see a possible solution for that. I would expect something like the following to work out of the box :)
val myMap: Map[String,Any] = ... // extract metrics
val json = myMap.reduceLeft(_ ~ _) // create JSON of metrics
Navigating through source code I've seen json4s provides implicit conversions in order to transform primitive types to JValue's and also to convert Traversable[A]/Map[String,A]/Option[A] to JValue's (under the restriction of being available an implicit conversion from A to JValue, which I understand it actually means A is a primitive type). The ~ operator offers a nice way of constructing JObject's out of JField's, which is just a type alias for (String, JValue).
In this case, map values type is Any, so implicit conversions don't take place and hence the compiler throws the following error:
value ~ is not a member of (String, Any)
[error] val json = r.reduceLeft(_ ~ _)
Is there a solution for what I want to accomplish?
Since you are actually only looking for the JSON string representation of myMap, you can use the Serialization object directly. Here is a small example (if using the native version of json4s change the import to org.json4s.native.Serialization):
EDIT: added formats implicit
import org.json4s.jackson.Serialization
implicit val formats = org.json4s.DefaultFormats
val m: Map[String, Any] = Map(
"name "-> "joe",
"children" -> List(
Map("name" -> "Mary", "age" -> 5),
Map("name" -> "Mazy", "age" -> 3)
)
)
// prints {"name ":"joe","children":[{"name":"Mary","age":5},{"name":"Mazy","age":3}]}
println(Serialization.write(m))
json4s has method for it.
pretty(render(yourMap))
By dictionary I mean a lightweight map from names to values that can be used as the return value of a method.
Options that I'm aware of include making case classes, creating anon objects, and making maps from Strings -> Any.
Case classes require mental overhead to create (names), but are strongly typed.
Anon objects don't seem that well documented and it's unclear to me how to use them as arguments since there is no named type.
Maps from String -> Any require casting for retrieval.
Is there anything better?
Ideally these could be built from json and transformed back into it when appropriate.
I don't need static typing (though it would be nice, I can see how it would be impossible) - but I do want to avoid explicit casting.
Here's the fundamental problem with what you want:
def get(key: String): Option[T] = ...
val r = map.get("key")
The type of r will be defined from the return type of get -- so, what should that type be? From where could it be defined? If you make it a type parameter, then it's relatively easy:
import scala.collection.mutable.{Map => MMap}
val map: MMap[String, (Manifest[_], Any) = MMap.empty
def get[T : Manifest](key: String): Option[T] = map.get(key).filter(_._1 <:< manifest[T]).map(_._2.asInstanceOf[T])
def put[T : Manifest](key: String, obj: T) = map(key) = manifest[T] -> obj
Example:
scala> put("abc", 2)
scala> put("def", true)
scala> get[Boolean]("abc")
res2: Option[Boolean] = None
scala> get[Int]("abc")
res3: Option[Int] = Some(2)
The problem, of course, is that you have to tell the compiler what type you expect to be stored on the map under that key. Unfortunately, there is simply no way around that: the compiler cannot know what type will be stored under that key at compile time.
Any solution you take you'll end up with this same problem: somehow or other, you'll have to tell the compiler what type should be returned.
Now, this shouldn't be a burden in a Scala program. Take that r above... you'll then use that r for something, right? That something you are using it for will have methods appropriate to some type, and since you know what the methods are, then you must also know what the type of r must be.
If this isn't the case, then there's something fundamentally wrong with the code -- or, perhaps, you haven't progressed from wanting the map to knowing what you'll do with it.
So you want to parse json and turn it into objects that resemble the javascript objets described in the json input? If you want static typing, case classes are pretty much your only option and there are already libraries handling this, for example lift-json.
Another option is to use Scala 2.9's experimental support for dynamic typing. That will give you elegant syntax at the expense of type safety.
You can use approach I've seen in the casbah library, when you explicitly pass a type parameter into the get method and cast the actual value inside the get method. Here is a quick example:
case class MultiTypeDictionary(m: Map[String, Any]) {
def getAs[T <: Any](k: String)(implicit mf: Manifest[T]): T =
cast(m.get(k).getOrElse {throw new IllegalArgumentException})(mf)
private def cast[T <: Any : Manifest](a: Any): T =
a.asInstanceOf[T]
}
implicit def map2multiTypeDictionary(m: Map[String, Any]) =
MultiTypeDictionary(m)
val dict: MultiTypeDictionary = Map("1" -> 1, "2" -> 2.0, "3" -> "3")
val a: Int = dict.getAs("1")
val b: Int = dict.getAs("2") //ClassCastException
val b: Int = dict.getAs("4") //IllegalArgumetExcepton
You should note that there is no real compile-time checks, so you have to deal with all exceptions drawbacks.
UPD Working MultiTypeDictionary class
If you have only a limited number of types which can occur as values, you can use some kind of union type (a.k.a. disjoint type), having e.g. a Map[Foo, Bar | Baz | Buz | Blargh]. If you have only two possibilities, you can use Either[A,B], giving you a Map[Foo, Either[Bar, Baz]]. For three types you might cheat and use Map[Foo, Either[Bar, Either[Baz,Buz]]], but this syntax obviously doesn't scale well. If you have more types you can use things like...
http://cleverlytitled.blogspot.com/2009/03/disjoint-bounded-views-redux.html
http://svn.assembla.com/svn/metascala/src/metascala/OneOfs.scala
http://www.chuusai.com/2011/06/09/scala-union-types-curry-howard/