I have a generator that creates a very compelx object. I cannot create this object through something like
val myGen = for{
a <- Gen.choose(-10,10)
...
} yield new MyClass(a,b,c,...)
I tried an approach of creating a custom generator like this
val myComplexGen :Gen[ComplexObject] = {
...
val myTempVariable = Gen.choose(-10,10)
val otherTempVal = Gen.choose(100,2000)
new MyComplexObject(myTempVariable,otherTempVal,...)
}
and then
test("myTest") {
forAll(myComplexGen){ complexObj =>
... // Here, complexObj.myTempVariable is always the same through all the iterations
}
}
While this works, the values generated are always the same. The inner Gen.choose yield always the same value.
Is there any way I can write a custom Gen with its own logic, and use inner Gen.choose inside, that would be random ?
I've been able to workaround the problem. The solution is definitely not elegant but that's the only way I could work it out.
I have transformed myComplexGen into a def, and called it inside another gen with dummy variables
def myComplexGen :ComplexObject = {
...
val myTempVariable = Gen.choose(-10,10)
val otherTempVal = Gen.choose(100,2000)
new MyComplexObject(myTempVariable,otherTempVal,...)
}
val realComplexGen :Gen[ComplexObject] = for {
i <- Gen.choose(0,10) // Not actually used, but for cannot be empty
} yield myComplexGen()
Now I can use realComplexGenin a forAll and the object is really random.
Related
I have a pipeline with a set of PTransforms and my method is getting very long.
I'd like to write my DoFns and my composite transforms in a separate package and use them back in my main method. With python it's pretty straightforward, how can I achieve that with Scio? I don't see any example of doing that. :(
withFixedWindows(
FIXED_WINDOW_DURATION,
options = WindowOptions(
trigger = groupedWithinTrigger,
timestampCombiner = TimestampCombiner.END_OF_WINDOW,
accumulationMode = AccumulationMode.ACCUMULATING_FIRED_PANES,
allowedLateness = Duration.ZERO
)
)
.sumByKey
// How to write this in an another file and use it here?
.transform("Format Output") {
_
.withWindow[IntervalWindow]
.withTimestamp
}
If I understand your question correctly, you want to bundle your map, groupBy, ... transformations in a separate package, and use them in your main pipeline.
One way would be to use applyTransform, but then you would end up using PTransforms, which are not scala-friendly.
You can simply write a function that receives an SCollection and returns the transformed one, like:
def myTransform(input: SCollection[InputType]): Scollection[OutputType] = ???
But if you intend to write your own Source/Sink, take a look at the ScioIO class
You can use map function to map your elements example.
Instead of passing a lambda, you can pass a method reference from another class
Example .map(MyClass.MyFunction)
I think one way to solve this could be to define an object in another package and then create a method in that object that would have the logic required for your transformation. For example:
def main(cmdlineArgs: Array[String]): Unit = {
val (sc, args) = ContextAndArgs(cmdlineArgs)
val defaulTopic = "tweets"
val input = args.getOrElse("inputTopic", defaulTopic)
val output = args("outputTopic")
val inputStream: SCollection[Tweet] = sc.withName("read from pub sub").pubsubTopic(input)
.withName("map to tweet class").map(x => {parse(x).extract[Tweet]})
inputStream
.flatMap(sentiment.predict) // object sentiment with method predict
}
object sentiment {
def predict(tweet: Tweet): Option[List[TweetSentiment]] = {
val data = tweet.text
val emptyCase = Some("")
Some(data) match {
case `emptyCase` => None
case Some(v) => Some(entitySentimentFile(data)) // I used another method, //not defined
}
}
Please also this link for an example given in the Scio examples
I have below relationship in my entity classes,
customer -> * Invoice
now I have to implement a method which returns customers with their invoices
type CustomerWithInvoices = (Custimer,Seq[Invoice])
def findCustomerWitnInvoices:Future[Seq[CustomerWithInvoices]] = {
for{
customers <- findCustomers
eventualInvoices: Seq[Future[Seq[Invoice]]] = customers.map(customer => findInvoicesByCustomer(customer))
} yield ???
}
using existing repository methods as below
def findCustomers:Future[Seq[Customers]] = {...}
def findInvoicesByCustomer(customer:Customer):Future[Seq[Invoice]] = {...}
I try to use for expression as above but I can't figure the proper way to do it, as I'm fairly new to Scala, highly appreciate any help..
i would use Future.sequence, the simplified method signature is
sequence takes M[Future[A]] and returns Future[M[A]]
That is what we need to solve your problem, here's the code i would write:
val eventualCustomersWithInvoices: Future[Seq[(Customer, Seq[Invoice])]] = for {
customers <- findCustomers()
eventualInvoices <- Future.sequence(customers.map(customer => findInvoicesByCustomer(customer)))
} yield eventualInvoices
note that the type of eventualInvoices is Future[Seq[(Customer, Seq[Invoice])]] hence Future[Seq[CustomerWithInvoices]]
I want to create a list of Test class.
case class Person(name:String)
case class Test (desc:String)
val list =Seq(Person("abc"),Person("def"))
val s = Option(list)
private val elems = scala.collection.mutable.ArrayBuffer[Test]()
val f =for{
l<-s
}yield {
for{
e <-l
} yield elems+=tranform(e)
}
f.toSeq
def tranform(p:Person):Test= {
Test(desc = "Hello "+p.name)
}
can anyone please help with the following
better way to avoid multiple for
I want to get List(Test("Hello abc"),Test("Hello def")) instead of using ArrayBuffer
I don't know why you're wrapping a Seq in an Option; Seq represents the no Persons case perfectly well. Is there a difference between None and Some(Seq.empty[Person]) in your application?
Assuming that you can get by without an Option[Seq[Person]]:
list.map(transform).toList
I want to return a wrapper/holder for a result that I want to compute only once and only if the result is actually used. Something like:
def getAnswer(question: Question): Lazy[Answer] = ???
println(getAnswer(q).value)
This should be pretty easy to implement using lazy val:
class Lazy[T](f: () => T) {
private lazy val _result = Try(f())
def value: T = _result.get
}
But I'm wondering if there's already something like this baked into the standard API.
A quick search pointed at Streams and DelayedLazyVal but neither is quite what I'm looking for.
Streams do memoize the stream elements, but it seems like the first element is computed at construction:
def compute(): Int = { println("computing"); 1 }
val s1 = compute() #:: Stream.empty
// computing is printed here, before doing s1.take(1)
In a similar vein, DelayedLazyVal starts computing upon construction, even requires an execution context:
val dlv = new DelayedLazyVal(() => 1, { println("started") })
// immediately prints out "started"
There's scalaz.Need which I think you'd be able to use for this.
Here is the function,
def checkSupport(request: checkSupportRequest): Future[checkSupportResponse] = {
val lookupClient = Thrift.newIface[lookupClient[Future]](/*some initializers*/)
val req = lookupIPRequest(request.getIps, /*Unimportant variables*/)
var result = new checkSupportResponse()
lookupClient.performLookup(req).map { resp =>
// Returns a map of IP to Country Code (String:String)
val ip_cc_map = for {
(ip, lxn) <- resp.found
sr <- lxn.simpleResult
cc <- sr.countryCode
} yield (ip, cc)
// Returns a map of IP to Boolean (String:Boolean)
val not_found_map = for {
(ip) <- resp.notFound
} yield (ip, false)
val op_supp_temp_map = ip_cc_map.map {
case (ip, cc) => someLookup(cc) //This returns a boolean value
}
val op_supp_map = op_supp_temp_map.toList //List of Booleans
val ips_found = ip_cc_map.map { // Intended to become a list of IPs
case (ip, cc) => ip
}
val final_op_supp_map = ips_found zip op_supp_map // Converted to Map (String:Boolean)
final_map = final_op_supp_map ++ not_found_map
result.set_supported(final_result.toList.toMap.mapValues(Boolean.box))
}
Future.value(result)
}
The performLookup is a futurized call. Is it kosher to do things as above i.e. take the output of the futurized call and map it. Perform two back-to-back for comprehensions on it to obtain the necessary maps and then subsequently in the same for, combine the two results and return the same. For some reason, this compiles but when I'm trying to unit-test it, it appears as if the performLookup never ran. Any help would be appreciated.
Also I am in my unit-test doing an Await.result() on the checkSupport() call.
I'm assuming that by Future.value(result) you actually mean Future.successful(result).
The result variable inside the closure is different from the result variable outside of it, so you're never actually modifying the result variable that you're declaring at the beginning of your function. So this is why it looks like your Future never ran.
As a side-note, even if you mutate the state of result, you don't need to declare it as var, val is good here (var would only be necessary if you were reassigning it by doing result = ...).
So you can modify your function like this:
def checkSupport(request: checkSupportRequest): Future[checkSupportResponse] = {
// declare lookupClient and req as before, but not res
lookupClient.performLookup(req).map { resp =>
// do all your processing like before
val result = new checkSupportResponse()
result.setOperator_supported(final_result.toList.toMap.mapValues(Boolean.box))
// We want "result" to be the result of the computation
result
}
}