object IO {
def getHtmlFromWebsiteViaHttp(link: String, apiKey: String = ""): String = {
Http(link)
.param("access_token", apiKey)
.asString
.body
}
}
class SongService {
private def retrieveSongId(songName: String): Option[JsValue] = {
val formattedSongName = songName.replace(" ", "%20")
val searchLink = "https://api.genius.com/search?q=" + formattedSongName
//impure call
val geniusStringResponse = IO.getHtmlFromWebsiteViaHttp(searchLink, apiKey)
//Extra processing on geniusStringResponse
}
}
My current design is I would have a service class which is responsible for getting some information via an external API. Now I understand that it's impossible to have 100% pure functions.
My question: What is the best way to handle situations where you need to connect to an external API in Scala/FP?. The aim is to have the most adequate 'functional programming style' by minimising impure functions
Currently, I am encapsulating all API calls in IO object. Is this suitable enough? I see examples of monads for situations. Should I incorporate a monad style in this case?
This isn't so much an FP problem, as I don't see any problems with your code in terms of FP, but what you should do, in my opinion is use dependency injection, such that, for testing, you can substitute a test class for IO that has a guaranteed response. Something like this:
abstract class IO {
def getHtmlFromWebsiteViaHttp(link: String, apiKey: String = ""): String
}
class IOImpl extends IO {
def getHtmlFromWebsiteViaHttp(link: String, apiKey: String = ""): String = {
Http(link)
.param("access_token", apiKey)
.asString
.body
}
}
class IOTestImpl extends IO {
def getHtmlFromWebsiteViaHttp(link: String, apiKey: String = ""): String = ??? //some test HTML
}
And then in your service:
class SongService(io: IO) {
private def retrieveSongId(songName: String): Option[JsValue] = {
val formattedSongName = songName.replace(" ", "%20")
val searchLink = "https://api.genius.com/search?q=" + formattedSongName
val geniusStringResponse = io.getHtmlFromWebsiteViaHttp(searchLink, apiKey)
//Extra processing on geniusStringResponse
}
}
Then when you instantiate your SongService, pass it IOTestImpl in testing and IOImpl otherwise. You might find some relevant information on dependency injection and database access objects.
Related
I am trying to write unit test for a function under object1.
object Object1 {
def main(sysArgs: Array[String]): Unit = {
val inputDF: DataFrame = UtilObject.getInput()
}
}
object UtilObject {
def getInput(){
...
}
}
To write Unit test, I am using MockitoSugar.
"object1Main" should "should make correct calls" in {
val inputArgs = Array("abc")
val util = mock[UtilObject.type]
when(util.getInput().thenReturn(inputData))
Object1.main(inputArgs)
}
While running the test, it doesn't consider the util mock and just execute the getInput() function.
I think I am missing some sort of injection here. Any ideas?
Thanks in advance!
Mocking Scala objects should be impossible conceptually speaking. An object in Scala is a pure singleton. That means there can only be one member of that type at any time.
mockito-scala can mock Scala objects via reflection. I'll use a result type of String, instead of a DataFrame, but the idea is the same:
object UtilObject {
def getInput(): String = {
// ...
"done"
}
}
object Object1 {
def main(sysArgs: Array[String]): String = {
val inputDF: String = UtilObject.getInput()
inputDF
}
}
// in test file:
"object1Main" should {
"should make correct calls" in {
val inputArgs = Array("abc")
withObjectMocked[UtilObject.type] {
UtilObject.getInput() returns "mocked!"
Object1.main(inputArgs) shouldBe "mocked!"
}
Object1.main(inputArgs) shouldBe "done"
}
}
This mocks the singleton's method only inside the block of withObjectMocked.
Usually such powerful techniques often tend to be overused or misused, so I don't generally recommend them, unless the design cannot be refactored.
Luckily, yours can: the easiest way is to use Dependency Injection with a class or a function. For DI with a class you require to convert the object being mocked into a class:
class UtilObject {
def getInput(): String = {
// ...
"done"
}
}
object Object1 {
def main(sysArgs: Array[String], ut: UtilObject): String = {
val inputDF: String = ut.getInput()
inputDF
}
}
// in test file:
"object1Main" should {
"should make correct calls" in {
val inputArgs = Array("abc")
val util = mock[UtilObject]
when(util.getInput()).thenReturn("mocked!")
Object1.main(inputArgs, util) shouldBe "mocked!"
}
}
For DI with a function you need to lift the method you want to mock into a function:
object UtilObject {
def getInput(): String = {
// ...
"done"
}
}
object Object1 {
def main(sysArgs: Array[String], f: () => String = UtilObject.getInput): String = {
val inputDF: String = f()
inputDF
}
}
// in test file:
"object1Main" should {
"should make correct calls" in {
val inputArgs = Array("abc")
val f = mock[() => String]
when(f()).thenReturn("mocked!")
Object1.main(inputArgs, f) shouldBe "mocked!"
}
}
Since the function takes no arguments, you can convert it into a by-name parameter. I'll leave that to you.
Lastly, another way is to create a trait with the method you want to mock and extend that with the object. But now Object1 requires being a class and have a reference to the object being mocked:
object UtilObject extends Utils {
def getInput(): String = {
// ...
"done"
}
}
trait Utils {
def getInput(): String
}
class Object1 {
val uo: Utils = UtilObject
def main(sysArgs: Array[String]): String = {
val inputDF: String = uo.getInput()
inputDF
}
}
// in test file:
"object1Main" should {
"should make correct calls" in {
val classUnderTest = new Object1 {
override val uo = mock[Utils]
}
val inputArgs = Array("abc")
when(classUnderTest.uo.getInput()).thenReturn("mocked!")
classUnderTest.main(inputArgs) shouldBe "mocked!"
}
}
As you can see, there are a few ways to go. Neither is inherently wrong. It mostly depends on your requirements(e.g. you can't afford adding a dependency just for one UT), needs(e.g. does the object I am testing really needs to be an object or can I make it a class?), guidelines (e.g. your team decided to avoid using powerful testing frameworks that rely on reflection and use DI as much as possible instead) and personal preferences.
To give you a minimal example:
object Main extends JSApp
{
val someThing: String = determineSomething("test")
def main(): Unit =
{
println(someThing)
}
}
Now, two possibilities here:
private def determineSomething(s: String): String = "succeeded"
If the project is executed like this, well, I get a log entry saying
succeeded
But when I use the more functional syntax:
private val determineSomething: (s: String) => "succeeded"
I get
TypeError: this.determineSomething$1 is null
I am curious as to the why this happens as in the (JVM) repl, both ways work perfectly fine.
I think what you want is something like this:
object Main extends JSApp {
private val determineSomething: String => String = (s: String) => "succeeded"
val someThing: String = determineSomething("test")
def main(): Unit = {
println(someThing)
}
}
The declaration of determineSomething needs to come before the declaration of something, otherwise the former will be uninitialized when the compiler attempts to initialize the latter.
Is there any specification of scala compilator that can explain that behaviour?
scala version: 2_10_6
code example
trait Service {
def process(s: String)
}
object ServiceImpl extends Service{
override def process(s: String): Unit = {
println(s)
}
}
object Register {
var serviceInst : Service = ServiceImpl
}
object Client1 {
def process1(l: List[String]): Unit ={
l.foreach(x => Register.serviceInst.process(x))
}
}
object Client2 {
def process1(l: List[String]): Unit ={
l.foreach(Register.serviceInst.process)
}
}
I assume that process1 and process2 should have the similar behaviour. However, after comilation / decom
public final class Client1$$anonfun$process$1$$anonfun$apply$1 extends AbstractFunction1<String, BoxedUnit> implements Serializable {
public static final long serialVersionUID = 0L;
public final void apply(final String x$1) {
Register$.MODULE$.serviceInst().process(x$1);
}
}
public static final class Client2$$anonfun$process$1 extends AbstractFunction1<String, BoxedUnit> implements Serializable {
public static final long serialVersionUID = 0L;
private final Service eta$0$1$1;
public final void apply(final String s) {
this.eta$0$1$1.process(s);
}
}
It's because Scala compiler performs eta-expansion on method given in Client2, which works by generating Function that calls process directly on a concrete Service instance.
Here is an example how these functions look like before they are turned into bytecode:
object Client1 {
def process1(l: List[String]): Unit = {
l.foreach(new Function1[String, Unit] {
def apply(x: String) = Register.serviceInst.process(x)
})
}
}
object Client2 {
def process1(l: List[String]): Unit = {
l.foreach(new Function1[String, Unit] {
val eta = Register.serviceInst
def apply(x: String) = eta.process(x)
})
}
}
It's become more interesting if we rewrite serviceInst a bit:
object Register {
def serviceInst : Service = {
println("get service instance!!!")
ServiceImpl
}
}
And then execute:
Client1.process1(List("a","b"))
Client2.process1(List("a","b"))
Obviously results are different:
1.
get service instance!!!
a
get service instance!!!
b
res0: Unit = ()
2.
get service instance!!!
a
b
res1: Unit = ()
Explanation is behind parameter of foreach function:
Client1 contains function as below, that executes each invocation x => Register.serviceInst.process(x)
Client2 has function process that's going to be executed, but firstly serviceInst is about to be initialized.
The line below
l.foreach(x => Register.serviceInst.process(x))
is operationally equivalent to
l.foreach(Register.serviceInst.process)
The first one is called "point-ful style" while the second is "point-free style", or more specifically "eta-conversion", with the term "point" referring to the named argument which doesn't exist in the second case. They are two different concepts and thus compile differently. You can write code in point-free style and the Scala compiler is eta-expanding it internally, which is what you're seeing in the decompiled code for Client2.
eta conversion is a term from Lambda Calculus. If the sole purpose of a lambda abstraction is to pass its argument to another function, then the lambda is redundant and can be stripped via eta conversion/reduction. Java's Lambda Expressions vs Method References is another example.
I have some working jackson scala module code for roundtripping scala case classes. Jackson worked great for flat case classes but when I made one which contains a list of other case classes the amount of code I seemed to need was a lot. Consider:
abstract class Message
case class CardDrawn(player: Long, card: Int, mType: String = "CardDrawn") extends Message
case class CardSet(cards: List[CardDrawn], mType: String = "CardSet") extends Message
To get the CardSet to roundtrip to/from json with jackson scala module I used a custom serializer/deserializer written in java:
object ScrumGameMashaller {
val mapper = new ObjectMapper()
val module = new SimpleModule("CustomSerializer")
module.addSerializer(classOf[CardSet], new CardSetSerializer)
module.addDeserializer(classOf[CardSet], new CardSetDeserializer)
val scalaModule = DefaultScalaModule
mapper.registerModule(scalaModule)
mapper.registerModule(module)
def jsonFrom(value: Any): String = {
import java.io.StringWriter
val writer = new StringWriter()
mapper.writeValue(writer, value)
writer.toString
}
private[this] def objectFrom[T: Manifest](value: String): T =
mapper.readValue(value, typeReference[T])
private[this] def typeReference[T: Manifest] = new TypeReference[T] {
override def getType = typeFromManifest(manifest[T])
}
private[this] def typeFromManifest(m: Manifest[_]): Type = {
if (m.typeArguments.isEmpty) { m.runtimeClass }
else new ParameterizedType {
def getRawType = m.runtimeClass
def getActualTypeArguments = m.typeArguments.map(typeFromManifest).toArray
def getOwnerType = null
}
}
with serializer:
public class CardSetSerializer extends JsonSerializer<CardSet> {
#Override
public void serialize(CardSet cardSet, JsonGenerator jgen, SerializerProvider provider) throws IOException, JsonProcessingException {
jgen.writeStartObject();
jgen.writeArrayFieldStart("cards");
List<CardDrawn> cardsDrawn = cardSet.cards();
scala.collection.Iterator<CardDrawn> iter = cardsDrawn.iterator();
while(iter.hasNext()){
CardDrawn cd = iter.next();
cdSerialize(jgen,cd);
}
jgen.writeEndArray();
jgen.writeStringField("mType", "CardSet");
jgen.writeEndObject();
}
private void cdSerialize(JsonGenerator jgen, CardDrawn cd) throws IOException, JsonProcessingException {
jgen.writeStartObject();
jgen.writeNumberField("player", cd.player());
jgen.writeNumberField("card", cd.card());
jgen.writeEndObject();
}
}
and matching deserializer:
public class CardSetDeserializer extends JsonDeserializer<CardSet> {
private static class CardDrawnTuple {
Long player;
Integer card;
}
#Override
public CardSet deserialize(JsonParser jsonParser, DeserializationContext cxt) throws IOException, JsonProcessingException {
ObjectCodec oc = jsonParser.getCodec();
JsonNode root = oc.readTree(jsonParser);
JsonNode cards = root.get("cards");
Iterator<JsonNode> i = cards.elements();
List<CardDrawn> cardObjects = new ArrayList<>();
while( i.hasNext() ){
CardDrawnTuple t = new CardDrawnTuple();
ObjectNode c = (ObjectNode) i.next();
Iterator<Entry<String, JsonNode>> fields = c.fields();
while( fields.hasNext() ){
Entry<String,JsonNode> f = fields.next();
if( f.getKey().equals("player")) {
t.player = f.getValue().asLong();
} else if( f.getKey().equals("card")){
t.card = f.getValue().asInt();
} else {
System.err.println(CardSetDeserializer.class.getCanonicalName()+ " : unknown field " + f.getKey());
}
}
CardDrawn cd = new CardDrawn(t.player, t.card, "CardDrawn");
cardObjects.add(cd);
}
return new CardSet(JavaConversions.asScalaBuffer(cardObjects).toList(), "CardSet");
}
}
This seems like a lot code to deal with something fairly vanilla in the scala. Can this code be improved (what did I miss that jackson has to make this easy)? Else is there a library which will do structured case classes automatically? The jerkson examples looked easy but that seems to have been abandoned.
Argonaut does a great job. Mark Hibbard helped me out with getting the example below working. All that is needed is to create a codec for the types and it will implicitly add an asJson to your objects to turn them into strings. It will also add a decodeOption[YourClass] to strings to extract an object. The following:
package argonaut.example
import argonaut._, Argonaut._
abstract class Message
case class CardDrawn(player: Long, card: Int, mType: String = "CardDrawn") extends Message
case class CardSet(cards: List[CardDrawn], mType: String = "CardSet") extends Message
object CardSetExample {
implicit lazy val CodecCardSet: CodecJson[CardSet] = casecodec2(CardSet.apply, CardSet.unapply)("cards","mType")
implicit lazy val CodecCardDrawn: CodecJson[CardDrawn] = casecodec3(CardDrawn.apply, CardDrawn.unapply)("player", "card", "mType")
def main(args: Array[String]): Unit = {
val value = CardSet(List(CardDrawn(1L,2),CardDrawn(3L,4)))
println(s"Got some good json ${value.asJson}")
val jstring =
"""{
| "cards":[
| {"player":"1","card":2,"mType":"CardDrawn"},
| {"player":"3","card":4,"mType":"CardDrawn"}
| ],
| "mType":"CardSet"
| }""".stripMargin
val parsed: Option[CardSet] =
jstring.decodeOption[CardSet]
println(s"Got a good object ${parsed.get}")
}
}
outputs:
Got some good json {"cards":[{"player":"1","card":2,"mType":"CardDrawn"},{"player":"3","card":4,"mType":"CardDrawn"}],"mType":"CardSet"}
Got a good object CardSet(List(CardDrawn(1,2,CardDrawn), CardDrawn(3,4,CardDrawn)),CardSet)
The question is old but maybe someone could still find it helpful. Apart from Argonaut, Scala has several Json libraries. Here you can find a list of them updated to the beginning of 2016 (and it still gives you a good overall picture).
Most of them (probably all) should allow you to come up with a drier version of your custom serializer/deserailizer. My preference goes to json4s which aims to provide a single AST across multiple libraries including Jackson (a bit like slf4j does for logging libraries). In this post you can find a working example of a Json custom serializer/deserializer using Json4s and Akka Http.
Working with JAXB, the standard way of dealing with a list of "nested" resource representations (e.g. <products><product>X</product><product>Y</product></products> is to create a wrapper object, which in Java might look like this (borrowed from Jhopify):
#XmlType(name = "")
#XmlRootElement(name = "products")
public class ProductList {
List<Product> products = new ArrayList<Product>();
#XmlElement(name = "product", required = true)
public List<Product> getProducts() { return products; }
public void setProducts(List<Product> products) { this.products = products; }
}
However I'm struggling to determine exactly which collection objects to use when translating this to Scala. There's a good introductory post to doing this on the Mostly Blather blog, which uses a Scala Iterable implicitly converted (using JavaConversions) to and from a JCollection.
This works great for marshalling a JAXB class to XML but unfortunately when unmarshalling this throws UnsupportedOperationException on each add attempt. Based on the last paragraph on this Scala documentation page it looks like this happens because Java does not distinguish between mutable and immutable collections in their type.
To deal with the unmarshalling, I've tried an alternative approach specifically using mutable objects:
#XmlType(name = "")
#XmlRootElement(name = "products")
class ProductList {
private var products: Buffer[Product] = new ArrayBuffer[Product]
#XmlElement(name = "product", required = true)
def getProducts: JList[Product] = products
def setProducts(products: JList[Product]) {
this.products = products
}
}
But unfortunately with this approach, unmarshalling gives me an exception:
java.lang.NoSuchMethodError: ProductList.getProducts()Ljava/util/Collection;
Edit: as per Travis' request, here is my unmarshalling code:
val jaxbContext = JAXBContext.newInstance(ProductList.getClass())
val unmarshaller = jaxbContext.createUnmarshaller()
val root = unmarshaller.unmarshal(new StreamSource(new StringReader(responseString)), ProductList.getClass())
val r = root.getValue().asInstanceOf[ProductList]
val representations = r.getProducts.asScala.toList // Uses scalaj
So I'm a bit stumped... I've looked at scalaj's available conversions too but nothing obvious jumps out. Any help much appreciated!
Could you post your unmarshalling code? I've done something similar with JAXB from Scala, and what you have looks like it should work. Here's a complete working example:
import javax.xml.bind.annotation._
class Thing {
#scala.reflect.BeanProperty var name: String = _
}
#XmlRootElement(name = "things")
class Things {
import scala.collection.JavaConversions._
import scala.collection.mutable.Buffer
private var things: Buffer[Thing] = Buffer[Thing]()
#XmlElement(name = "thing", required = true)
def getThings: java.util.List[Thing] = this.things
def setThings(things: java.util.List[Thing]) {
this.things = things
}
}
I'll write the test code in Scala as well, but it would work identically in Java.
object Things {
import java.io.StringReader
import java.io.StringWriter
import javax.xml.bind.JAXBContext
def main(args: Array[String]) {
val thing1 = new Thing
val thing2 = new Thing
thing1.setName("Thing 1")
thing2.setName("Thing 2")
val list: java.util.List[Thing] = new java.util.ArrayList[Thing]
list.add(thing1)
list.add(thing2)
val things = new Things
things.setThings(list)
val context = JAXBContext.newInstance(classOf[Things])
val writer = new StringWriter
context.createMarshaller.marshal(things, writer)
println(writer.toString)
val readThings = context.createUnmarshaller().unmarshal(
new StringReader(writer.toString)
).asInstanceOf[Things]
println("Size: " + readThings.getThings.size)
println("Name of first: " + readThings.getThings.get(0).getName)
}
}
This compiles and produces the output you'd expect.