I have the following code that iterates recursively and does something over the network. As it goes over the network, I would like to do some optimizations, where the very first optimization would be to avoid going over the network for certain elements that I have already tried.
For example., in the case below, I call a URL, extract the HREF's found in that URL and call those URL's and report the status. As it could be possible that certain URL's might be fetched again, for those URL's that failed, I would like to add them to a global state so that when I encounter this URL the next time, I will avoid those network calls.
Here is the code:
def callURLWithCache(url: String): Task[HttpResult] = {
Task {
Http(url).timeout(connTimeoutMs = 1000, readTimeoutMs = 3000).asString
}.attempt.map {
case Left(err) =>
println(s"ERR happened ----------------- $url ************************ ${err.getMessage}")
// Add to the cache
val httpResult = HttpResult(source = url, isSuccess = false, statusCode = 1000, errorMessage = Some(err.getMessage))
val returnnnn: Try[Any] = httpResultErrorCache.put(url)(httpResult)
httpResult
case Right(doc) =>
if (doc.isError) {
HttpResult(source = url, isSuccess = doc.isSuccess, statusCode = doc.code)
} else {
val hrefs = (browser.parseString(doc.body) >> elementList("a[href]") >?> attr("href"))
.distinct.flatten.filter(_.startsWith("http"))
HttpResult(source = url, isSuccess = doc.isSuccess, statusCode = doc.code, elems = hrefs)
}
}
}
You can see in the case Left(....) block that I add the failed case class to the cache which I define globally on the enclosing class of this function as:
val underlyingCaffeineCache: cache.Cache[String, Entry[HttpResult]] = Caffeine.newBuilder().maximumSize(10000L).build[String, Entry[HttpResult]]
implicit val httpResultErrorCache: Cache[HttpResult] = CaffeineCache(underlyingCaffeineCache)
Here is the function that I do a recursive operation:
def parseSimpleWithFilter(filter: ParserFilter): Task[Seq[HttpResult]] = {
def parseInner(depth: Int, acc: HttpResult): Task[Seq[HttpResult]] = {
import cats.implicits._
if (depth > 0) {
val batched = acc.elems.collect {
case elem if httpResultErrorCache.get(elem).toOption.exists(_.isEmpty) =>
callURLWithCache(elem).flatMap(newElems => parseInner(depth - 1, newElems))
}.sliding(30).toSeq
.map(chunk => Task.parSequence(chunk))
Task.sequence(batched).map(_.flatten).map(_.flatten)
} else Task.pure(Seq(acc))
}
callURLWithCache(filter.url).map(elem => parseInner(filter.recursionDepth, elem)).flatten
}
It can be seen that I'm checking if the url that I have as my current elem is already in the cache, meaning that I have already tried it and failed, so I would like to avoid doing a HTTP call once again for it.
But what happens is that, the httpResultErrorCache turns up always empty. I'm not sure if the Task chunk is causing this behavior. Any ideas on how to get the cache to work?
Related
For some reason when I input a string with double spaces such as " ", the function does not remove them from the string, nor does it remove them when they are generated by two WUB's in a row
For example:
songDecoder("WUBCATWUBWUBBALLWUB") outputs "CAT_ _BALL" (underscores represent spaces)
I could fix this by other means, but since I have no idea why my current code isn't working I figured I should ask to patch my understanding.
def songDecoder(song:String):String = {
val l = song.indexOf("WUB")
if (song.contains(" ")) {
val e = song.indexOf(" ")
songDecoder(song.patch(e,Nil,1))
}
if (l==0) {
val c = song.patch(l,Nil,3)
songDecoder(c)
}
if (l== -1)
song.trim
else {
val c = song.patch(l,Nil,2)
val b = c.patch(l," ",1)
songDecoder(b)
}
}
The reason it doesn't work is because when you call a recursive method it eventually returns with its result. The code that clears out the double-whitespace doesn't save that result.
if (song.contains(" ")) {
val e = song.indexOf(" ")
songDecoder(song.patch(e,Nil,1)) //send patched song to decoder
} //don't save returned string
//continue with unpatched song
The 2nd if block also recurses without saving the result.
if (l==0) {
val c = song.patch(l,Nil,3)
songDecoder(c) //send patched song to decoder
} //don't save returned string
//continue with unpatched song
You can remove both of those if blocks and you'll get the same results from your method. The only code that effects the output is the final if/else and that's because it is at the end of the method's code block. So whatever the if/else produces that's what the method returns.
if (l== -1)
song.trim //return the final result string
else {
val c = song.patch(l,Nil,2) //remove one WUB
val b = c.patch(l," ",1) //replace with space
songDecoder(b) //return whatever the next recursion returns
}
Just as an FYI, here's a different approach.
def songDecoder(song:String):String =
"(WUB)+".r.replaceAllIn(song, " ").trim
How about something like:
song.split(“(WUB)+”).mkString(“ “).trim
I'm not seeing anything ever get returned by the scan. I know it's because the mutableList gets returned right away, but how do I block the return until the time expires?
Basically, all I want to do is fill up the mutable list for as long as the take() permits then return that mutableList to the calling function.
This is what I have tried.
private val timeoutScheduler: Scheduler = Schedulers.computation()
fun scanForAllDevicesStartingWith(devicePrefix: String): List<String> {
Log.d(TAG, "Scanning for devices starting with $devicePrefix")
val mutableList = mutableListOf<String>()
val result = scanForDevices()
.take(3, TimeUnit.SECONDS, timeoutScheduler)
.subscribe { scanResult ->
val name = scanResult.bleDevice.name
Logger.d(TAG, "Potential device named $name found")
if(name != null) {
if(name.startsWith(prefix = devicePrefix)) {
Logger.d(TAG, "Match found $name")
mutableList.plus(name)
}
}
}
return mutableList
}
private fun scanForDevices(): Observable<ScanResult>
= rxBleClient.scanBleDevices(
ScanSettings.Builder()
.setScanMode(ScanSettings.SCAN_MODE_LOW_LATENCY)
.setCallbackType(ScanSettings.CALLBACK_TYPE_ALL_MATCHES)
.build(),
ScanFilter.Builder()
.build())
}
OK, here it is boiled down for the next person who wants to do this kind of thing. In Rx, they have Singles which are Observables that just emit one value. In my case I needed a list of String values, so just need to use a Single of type List of type String. That gets just one element emitted that happens to be a list of Strings. The code looks like this...
fun returnAllDevicesStartingWith(devicePrefix: String): Single<List<String>> {
return scanForDevices()
.take(3, TimeUnit.SECONDS, timeoutScheduler)
.map { it.bleDevice.name }
.filter { it.startsWith(devicePrefix) }
.toList()
}
The function that calls it (written in Java instead of Kotlin) looks like this:
List<String> devices = bleUtility.returnAllDevicesStartingWith(prefix).blockingGet();
I tested it using a mocked function like this:
//Begin test code
var emittedList: List<String> = listOf("dev1-1", "dev1-2", "dev2-1", "dev2-2", "dev3-1", "dev3-2")
private fun scanForRoomDevices(): Observable<FoundDevice> = Observable
.intervalRange(0, emittedList.size.toLong(), 0, 1, TimeUnit.SECONDS, timeoutScheduler)
.map { index -> FoundDevice(emittedList[index.toInt()], BleDevice(emittedList[index.toInt()])) }
data class FoundDevice(val controllerId: String, val bleDevice: BleDevice)
data class BleDevice(val name: String)
Hope this helps others.
For exercise I'm trying to implement a LinkedList in Scala.
Main problem is about Null reference.
But first some code:
class Node(xkey: String, xnext: Option[Node], xinfo: Int) {
val key: String = xkey;
var next = xnext.getOrElse(None);
var info: Int = xinfo;
def this(xkey: String, xinfo: Int) {
this(xkey, None, xinfo);
}
def this(xkey: String) {
this(xkey, None, -1);
}
#Override
override def toString: String = key + ":" + info
}
At this point, I'm already concerned about things.
I declare xnext in construct as a Option[Node], because the tail in this linkedList does not have a next.
In my first try, it was just a Node, but had problem with null object because compilator just told me that "null can't cast to Node" (or something like that, I do not remember now) - And so I switch to this Option.
But, is it ok? Because, you know, for me next should be a Node, not a Option, otherwise, I don't know, in the linkedList how to reference to next Node.
Whatever, second class (i.e. my Linked List)
class LinkedNode {
private var first: Option[Node] = None;
private var last: Option[Node] = None;
def addNode(newNode: Node) = {
if (first == null) {
first = Some(newNode);
last = Some(newNode);
first.next = last;
}
else {
last.next = newNode;
newNode.next = null;
last = newNode
}
}
def size(): Long = {
var currentNode : = first;
var size = 0L;
while (currentNode != null) {
size+=1;
currentNode = currentNode.next;
}
size
}
def findNodeByKey(key: String) : Node = {
var currentNode = first;
while(currentNode != null) {
if (currentNode.key.equals(key))
currentNode
else {
currentNode = currentNode.next;
}
}
currentNode;
}
def delNodeByKey(key : String) : Boolean = {
var currentNode = first;
var previousNode = first;
while(currentNode != null) {
if (currentNode.key.equals(key)) {
previousNode = currentNode.next;
return true;
}
previousNode = currentNode;
currentNode = currentNode.next;
}
return false;
}
}
And nothing. I'm already block to my constructor because first and last.
How should I declare them? Node? Or Option[Node]?
Problems are also in Add method.
When I add a node, I want to add a Node object, not an Option[Node].
And I don't get how to achieve things I want with all Option, Some and None classes.
I know I should not be so vague with my request, but any help?
P.S. I've already read this Q/A and it didn't help me
At this point, I'm already concerned about things. I declare xnext in construct as a Option[Node], because the tail in this linkedList does not have a next.
[...]
But, is ok? because, you know, for me next should be a Node, not a Option, otherwise, I don't know, in the linkedList how to reference to next Node.
This is a good solution to replacing null, which you definitely want to do to prevent null-pointer exceptions and the like. An Option[Node] is simply a Node wrapped in a container (or None). You can check whether or not it has a value with isEmpty or get its value with get (which will throw an exception if the Option is empty).
The only difference to null, as you'd use it in Java, is that you need to check if it isEmpty instead of checking for null, and that you need to unwrap (option.get) it explicitly when you're sure that it is not None.
A more paradigmatic (scala-typical) way of retrieving the value from an option is pattern matching:
option match {
case Some(x) => println(x)
case None => println("Whoops, no value :(")
}
Regarding your other questions, they are indeed a little vague.
How should I declere them? Node? or Option[Node]?
Use Option[Node] if the possibility exists that there's no value for the variable (i.e., if you'd set it to null sometimes in Java).
When I add a node, I want to add a Node object, not a Option[Node].
No, you want to add an Option[Node] internally, because you will need to check later on if a node is set or not. In Scala it is preferrable to do this via Option[Node].isEmpty compared to setting things to null. You're already doing this in some parts of your code (e.g., addNode), where you do Some(newNode) (I'd call this "wrapping the node in an Option", but I'm not entirely sure if that's the correct terminology).
And I don't get how to achieve things I want with all Option, Some and None class.
What you're doing in your addNode does seem correct to a degree, but you somehow try to use null again in the else branch. What about:
// We don't need Option[Node] for the parameter, because there
// _must_ be a Node value to be added
def addNode(newNode: Node) = {
if (first.isEmpty) {
first = Some(newNode)
last = Some(newNode)
first.next = last
} else {
newNode.next = None
last.next = Some(newNode)
last = Some(newNode)
}
}
(I didn't run that code, nor did I do an thorough check of your logic)
I want to make file download from a database using Play framework.
But when I use this code I get this message:
value as is not a member of Array[Byte]
And if I change Ok(bytOfImage.as("image/jpg")) to Ok(bytOfImage) it works good but I get a file with a name: secondindex without .jpg
Here's my controller:
def secondindex(number: Int) = Action {
var bytOfImage = Array[Byte](1)
val conn = DB.getConnection()
try {
val stmt = conn.createStatement
val rs = stmt.executeQuery("SELECT image from images where id = " + number)
while(rs.next()) {
var blob = rs.getBlob("image")
bytOfImage = blob.getBytes(1, blob.length().toInt)
blob.free()
}
} finally {
conn.close() }
Ok(bytOfImage.as("image/jpg"))
}
You are calling the as method on the wrong object. It should look as follows:
Ok(bytOfImage).as("image/jpg")
If you need download images from browser, you can use method SimpleResult
and add in header"Content-Disposition" -> "attachment"
For example change row Ok(bytOfImage.as("image/jpg")) in you code on
val enumImg: Enumerator[Array[Byte]] = { Enumerator(bytOfImage) }
SimpleResult (
header = ResponseHeader(200, Map("Content-Disposition" -> "attachment")),
body = enumImg
)
I am using ws-lite to automate web service testing, and I want to have more flexible control over the xm l request generated.
Basically the request body is a groovy closure that will be passed to MarkupBuilder in order to create the SOAP message.
Here is an example of what I am trying to achieve (example taken from https://github.com/jwagenleitner/groovy-wslite):
#Grab(group='com.github.groovy-wslite', module='groovy-wslite', version='1.1.0')
import wslite.soap.*
def client = new SOAPClient('http://www.holidaywebservice.com/Holidays/US/Dates/USHolidayDates.asmx')
def month = ["Feb"]
def response = client.send(SOAPAction:'http://www.27seconds.com/Holidays/US/Dates/GetMothersDay') {
body {
GetMothersDay('xmlns':'http://www.27seconds.com/Holidays/US/Dates/') {
year(2011)
month.each{ a = null ->
if (a != null){
"month"(a)
}
}
}
}
}
assert "2011-05-08T00:00:00" == response.GetMothersDayResponse.GetMothersDayResult.text()
assert 200 == response.httpResponse.statusCode
assert "ASP.NET" == response.httpResponse.headers['X-Powered-By']
The above example, I can create month tag fine with value/values specified.
But if I change it to be:
#Grab(group='com.github.groovy-wslite', module='groovy-wslite', version='1.1.0')
import wslite.soap.*
def client = new SOAPClient('http://www.holidaywebservice.com/Holidays/US/Dates/USHolidayDates.asmx')
def month_cl = { a -> "month"(a) }
def response = client.send(SOAPAction:'http://www.27seconds.com/Holidays/US/Dates/GetMothersDay') {
body {
GetMothersDay('xmlns':'http://www.27seconds.com/Holidays/US/Dates/') {
year(2011)
month_cl("Feb")
}
}
}
assert "2011-05-08T00:00:00" == response.GetMothersDayResponse.GetMothersDayResult.text()
assert 200 == response.httpResponse.statusCode
assert "ASP.NET" == response.httpResponse.headers['X-Powered-By']
I will have a missing method exception.
I don't quite understand why I can't just invoke a groovy closure like that?
Delegate of month_cl closure has to be set to the current/parent delegate (in this case it is closure passed as param to GetMothersDay). Try with:
body {
GetMothersDay('xmlns':'http://www.27seconds.com/Holidays/US/Dates/') {
year(2011)
month_cl.delegate = delegate
month_cl("Feb")
}
}
It's quite normal to get MissingMethodException, because the closure month_cl is calling month method which does not exist. To do this (in your way), you should pass a Closure c to month_cl and call it on the argument a, like this:
def month_cl = { a, Closure c -> c(a) }
and using the new implementation, month_cl("Feb") becomes month_cl("Feb") { "month" } which results to "month"("Feb").
Here is a working example:
#Grab(group='com.github.groovy-wslite', module='groovy-wslite', version='1.1.0')
import wslite.soap.*
def client = new SOAPClient('http://www.holidaywebservice.com/Holidays/US/Dates/USHolidayDates.asmx')
def months = ["Feb"]
def month_cl = { m, Closure c -> return c(m) }
def response = client.send(SOAPAction:'http://www.27seconds.com/Holidays/US/Dates/GetMothersDay') {
body {
GetMothersDay('xmlns':'http://www.27seconds.com/Holidays/US/Dates/') {
year(2011)
month_cl("Feb") { "month" } // -> month("Feb")
}
}
}
assert "2011-05-08T00:00:00" != response.GetMothersDayResponse.GetMothersDayResult.text()
assert 200 == response.httpResponse.statusCode
assert "ASP.NET" == response.httpResponse.headers['X-Powered-By']