How to reserve ZIO response inside a custom method in - scala

I have this method
import ClientServer.*
import zio.http.{Client, *}
import zio.json.*
import zio.http.model.Method
import zio.{ExitCode, URIO, ZIO}
import sttp.capabilities.*
import sttp.client3.Request
import zio.*
import zio.http.model.Headers.Header
import zio.http.model.Version.Http_1_0
import zio.stream.*
import java.net.InetAddress
import sttp.model.sse.ServerSentEvent
import sttp.client3._
object fillFileWithLeagues:
def fill = for {
openDotaResponse <- Client.request("https://api.opendota.com/api/leagues")
bodyOfResponse <- openDotaResponse.body.asString
listOfLeagues <- ZIO.fromEither(bodyOfResponse.fromJson[List[League]].left.map(error => new Exception(error)))
save = FileStorage.saveToFile(listOfLeagues.toJson) //Ok
}yield ()
println("Im here fillFileWithLeagues.fill ")
and when I try use
fillFileWithLeagues.fill
nothing happens
I'm trying fill file with data from target api using
fillFileWithLeagues.fill
def readFromFileV8(path: Path = Path("src", "main", "resources", "data.json")): ZIO[Any, Throwable, String] =
val zioStr = (for bool <- Files.isReadable(path) yield bool).flatMap(bool =>
if (bool) Files.readAllLines(path, Charset.Standard.utf8).map(_.head)
else {
fillFileWithLeagues.fill
wait(10000)
println("im here readFromFileV8")
readFromFileV8()})
zioStr
I'm expecting data.json file must created from
Client.request("https://api.opendota.com/api/leagues")
but there is nothing happens
Maybe I should use some sttp, or some other tools?

If we fix indentation of the code we'll find this:
object fillFileWithLeagues {
def fill = {
for {
openDotaResponse <- Client.request("https://api.opendota.com/api/leagues")
bodyOfResponse <- openDotaResponse.body.asString
listOfLeagues <- ZIO.fromEither(bodyOfResponse.fromJson[List[League]].left.map(error => new Exception(error)))
save = FileStorage.saveToFile(listOfLeagues.toJson) //Ok
} yield ()
}
println("Im here fillFileWithLeagues.fill ")
}
As you see the println is part of fillFileWithLeagues, not of fill.
Another potential problem is that an expression like fillFileWithLeagues.fill only returns a ZIO instance, it is not yet evaluated. To evaluate it, it needs to be run. For example as follows:
import zio._
object MainApp extends ZIOAppDefault {
def run = fillFileWithLeagues.fill
}

Related

How to fix ZIO server Internal Server Error 500 generated buy custom method

I have this 2 methods
import org.h2.store.fs.FilePath
import zio.*
import zio.Console.printLine
import zio.http.Client
import zio.nio.file.*
import zio.nio.charset.Charset
import zio.stream.*
import java.io.IOException
object FileStorage:
def saveToFile(data: String = "", filePath: String = "src/main/resources/data.json"): Unit =
lazy val logic = for {
encoded <- Charset.Standard.utf8.encodeString(data)
path = Path(filePath.split("/").head, filePath.split("/").tail: _*)
notExists <- Files.notExists(path)
- <- if (notExists) Files.createFile(path) else ZIO.attempt(())
_ <- Files.writeBytes(path, encoded)
_ <- Console.printLine(s"written to $path")
} yield ()
def unsafeF = (unsafeVal: Unsafe) => {
implicit val unsafe: Unsafe = unsafeVal
Runtime.default.unsafe.run(logic)
}
Unsafe.unsafe(unsafeF)
def readFromFile: ZIO[Any, Throwable, String] = {
val path = Path("src", "main", "resources", "data.json")
val bool = for bool <- Files.isReadable(path) yield bool
val zioStr = bool.flatMap(bool =>
if (bool) Files.readAllLines(path, Charset.Standard.utf8).map(fileLines => fileLines.head)
else {
saveToFile()
readFromFile})
zioStr
}
In def readFromFile i try to make empty an empty file if file don't exists
File generation working fine
then I'm trying to read that empty file and return it like a ZIO Response like that
import zio.http.{Client, *}
import zio.json.*
import zio.http.model.Method
import zio.{Scope, Task, ZIO, ZIOAppDefault}
import zio.http.Client
import zhttp.http.Status.NotFound
import zhttp.http.Status
import scala.language.postfixOps
import zio._
import scala.collection.immutable.List
import zio.{ExitCode, URIO, ZIO}
object ClientServer extends ZIOAppDefault {
val app: Http[Client, Throwable, Request, Response] = Http.collectZIO[Request]
case Method.GET -> !! / "readLeagues" =>
FileStorage.readFromFile.map(str => Response.json(str))
BUT in this case I getting Internal Server Error 500 on postman in http://localhost:8080/readLeagues
If at first I feed prefilled json file to
def readFromFile
It works fine Status: 200
And I getting a nice looking json as a body
Maybe I should set another default strings for data to prefill
def saveToFile
so json can be parsable?
or smth else

How to produce a Traversable with cats-effect's IO given an async that will call multiple times

What I'm really trying to do is monitor multiple files and when any of them is modified I'd like to update some state and produce a side effect using this state. I imagine what I want is a scan over a Traversable that produces a Traversable[IO[_]]. But I don't see the path there.
as a minimal attempt to produce this I wrote
package example
import better.files.{File, FileMonitor}
import cats.implicits._
import com.monovore.decline._
import cats.effect.IO
import java.nio.file.{Files, Path}
import scala.concurrent.ExecutionContext.Implicits.global
object Hello extends CommandApp(
name = "cats-effects-playground",
header = "welcome",
main = {
val filesOpts = Opts.options[Path]("input", help = "input files")
filesOpts.map { files =>
IO.async[File] { cb =>
val watchers = files.map { path =>
new FileMonitor(path, recursive = false) {
override def onModify(file: File, count: Int) = cb(Right(file))
}
}
watchers.toList.foreach(_.start)
}
.flatMap(f => IO { println(f) })
.unsafeRunSync
}
}
)
but this has two major flaws. One it creates a thread for each file I'm watching, which is a little heavy. But more importantly the program finishes as soon as a single file is modified, even though onModify would be called more times if the program stayed running.
I'm not married to using better-files, it just seemed like the path of least resistance. But I do require using Cats IO.
This solution doesn't solve the issue of creating a bunch of threads, and it doesn't strictly produce a Traversable, but it solves the underlying use case. I'm very open to this being critiqued and a better solution provided.
package example
import better.files.{File, FileMonitor}
import cats.implicits._
import com.monovore.decline._
import cats.effect.IO
import java.nio.file.{Files, Path}
import java.util.concurrent.LinkedBlockingQueue
import scala.concurrent.ExecutionContext.Implicits.global
object Hello extends CommandApp(
name = "cats-effects-playground",
header = "welcome",
main = {
val filesOpts = Opts.options[Path]("input", help = "input files")
filesOpts.map { files =>
val bq: LinkedBlockingQueue[IO[File]] = new LinkedBlockingQueue()
val watchers = files.map { path =>
new FileMonitor(path, recursive = false) {
override def onModify(file: File, count: Int) = bq.put(IO(file))
}
}
def ioLoop(): IO[Unit] = bq.take()
.flatMap(f => IO(println(f)))
.flatMap(_ => ioLoop())
watchers.toList.foreach(_.start)
ioLoop.unsafeRunSync
}
}
)

Scala: Parallel execution with ListBuffer appends doesn't produce expected outcome

I know I'm doing something wrong with mutable.ListBuffer but I can't figure out how to fix it (and a proper explanation of the issue).
I simplified the code below to reproduce the behavior.
I'm basically trying to run functions in parallel to add elements to a list as my first list get processed. I end up "losing" elements.
import java.util.Properties
import scala.collection.mutable.ListBuffer
import scala.concurrent.duration.Duration
import scala.concurrent.{Await, Future}
import scala.concurrent.{ExecutionContext}
import ExecutionContext.Implicits.global
object MyTestObject {
var listBufferOfInts = new ListBuffer[Int]() // files that are processed
def runFunction(): Int = {
listBufferOfInts = new ListBuffer[Int]()
val inputListOfInts = 1 to 1000
val fut = Future.traverse(inputListOfInts) { i =>
Future {
appendElem(i)
}
}
Await.ready(fut, Duration.Inf)
listBufferOfInts.length
}
def appendElem(elem: Int): Unit = {
listBufferOfInts ++= List(elem)
}
}
MyTestObject.runFunction()
MyTestObject.runFunction()
MyTestObject.runFunction()
which returns:
res0: Int = 937
res1: Int = 992
res2: Int = 997
Obviously I would expect 1000 to be returned all the time. How can I fix my code to keep the "architecture" but make my ListBuffer "synchronized" ?
I don't know what exact problem is as you said you simplified it, but still you have an obvious race condition, multiple threads modify a single mutable collection and that is very bad. As other answers pointed out you need some locking so that only one thread could modify collection at the same time. If your calculations are heavy, appending result in synchronized way to a buffer shouldn't notably affect the performance but when in doubt always measure.
But synchronization is not needed, you can do something else instead, without vars and mutable state. Let each Future return your partial result and then merge them into a list, in fact Future.traverse does just that.
import scala.concurrent.duration._
import scala.concurrent.{Await, Future}
import scala.concurrent.ExecutionContext.Implicits.global
def runFunction: Int = {
val inputListOfInts = 1 to 1000
val fut: Future[List[Int]] = Future.traverse(inputListOfInts.toList) { i =>
Future {
// some heavy calculations on i
i * 4
}
}
val listOfInts = Await.result(fut, Duration.Inf)
listOfInts.size
}
Future.traverse already gives you an immutable list with all your results combined, no need to append them to a mutable buffer.
Needless to say, you will always get 1000 back.
# List.fill(10000)(runFunction).exists(_ != 1000)
res18: Boolean = false
I'm not sure the above shows what you are trying to do correctly. Maybe the issue is that you are actually sharing a var ListBuffer which you reinitialise within runFunction.
When I take this out I collect all the events I'm expecting correctly:
import java.util.Properties
import scala.collection.mutable.ListBuffer
import scala.concurrent.duration.Duration
import scala.concurrent.{ Await, Future }
import scala.concurrent.{ ExecutionContext }
import ExecutionContext.Implicits.global
object BrokenTestObject extends App {
var listBufferOfInts = ( new ListBuffer[Int]() )
def runFunction(): Int = {
val inputListOfInts = 1 to 1000
val fut = Future.traverse(inputListOfInts) { i =>
Future {
appendElem(i)
}
}
Await.ready(fut, Duration.Inf)
listBufferOfInts.length
}
def appendElem(elem: Int): Unit = {
listBufferOfInts.append( elem )
}
BrokenTestObject.runFunction()
BrokenTestObject.runFunction()
BrokenTestObject.runFunction()
println(s"collected ${listBufferOfInts.length} elements")
}
If you really have a synchronisation issue you can use something like the following:
import java.util.Properties
import scala.collection.mutable.ListBuffer
import scala.concurrent.duration.Duration
import scala.concurrent.{ Await, Future }
import scala.concurrent.{ ExecutionContext }
import ExecutionContext.Implicits.global
class WrappedListBuffer(val lb: ListBuffer[Int]) {
def append(i: Int) {
this.synchronized {
lb.append(i)
}
}
}
object MyTestObject extends App {
var listBufferOfInts = new WrappedListBuffer( new ListBuffer[Int]() )
def runFunction(): Int = {
val inputListOfInts = 1 to 1000
val fut = Future.traverse(inputListOfInts) { i =>
Future {
appendElem(i)
}
}
Await.ready(fut, Duration.Inf)
listBufferOfInts.lb.length
}
def appendElem(elem: Int): Unit = {
listBufferOfInts.append( elem )
}
MyTestObject.runFunction()
MyTestObject.runFunction()
MyTestObject.runFunction()
println(s"collected ${listBufferOfInts.lb.size} elements")
}
Changing
listBufferOfInts ++= List(elem)
to
synchronized {
listBufferOfInts ++= List(elem)
}
Make it work. Probably can become a performance issue? I'm still interested in an explanation and maybe a better way of doing things!

Error testing DAL in slick scala

Pretty new to scala and play and I have been assigned a task to test someone else app,which is running fine btw.Please check if my tests are right and what is the error.
This is employeeEntry.scala file in models
package models
import models.database.Employee
import play.api.db.slick.Config.driver.simple._
import play.api.db.slick._
import play.api.Play.current
case class EmployeeEntry(eid :Int, ename: String, eadd: String, emob: String)
object Employee {
val DBemp = TableQuery[Employee]
def savedat(value: EmployeeEntry):Long = {
DB.withSession { implicit session =>
DBemp+=EmployeeEntry(eid=value.eid,ename=value.ename,eadd=value.eadd,emob=value.emob)
}}
/*val query = for (c <- Employee) yield c.ename
val result = DB.withSession {
implicit session =>
query.list // <- takes session implicitly
}*/
//val query = for (c <- Employee) yield c.ename
def getPersonList: List[EmployeeEntry] = DB.withSession { implicit session => DBemp.list }
def Update: Int = DB.withSession { implicit session =>
(DBemp filter (_.eid === 1) map (s => (s.ename,s.eadd))) update ("test","khair")}
def delet :Int =DB.withSession {
implicit session => (DBemp filter (_.eid === 1)).delete
}
}
And this is file Employee.scala in models/database
package models.database
import models._
import models.EmployeeEntry
import play.api.db.slick.Config.driver.simple._
import scala.slick.lifted._
class Employee(tag:Tag) extends Table[EmployeeEntry](tag,"employee")
{
//val a = "hello"
def eid = column[Int]("eid", O.PrimaryKey)
def ename = column[String]("name", O.DBType("VARCHAR(50)"))
def emob = column[String]("emob",O.DBType("VARCHAR(10)"))
def eadd =column[String]("eadd",O.DBType("VARCHAR(100)"))
def * = (eid, ename, emob, eadd) <> (EmployeeEntry.tupled, EmployeeEntry.unapply)
}
Finally this is my test I am running,which is failing :
import play.api.db.slick.Config.driver.simple._
import play.api.db.slick._
import play.api.Play.current
import org.scalatest.FunSpec
import org.scalatest.matchers.ShouldMatchers
import models.database.Employee
import scala.slick.lifted._
import models._
import models.EmployeeEntry
//import scala.slick.driver.H2Driver.simple._
class databasetest extends FunSpec with ShouldMatchers{
describe("this is to check database layer"){
it("can save a row"){
val a = EmployeeEntry(1006,"udit","schd","90909090")
Employee.savedat(a) should be (1)
}
it("getpersonlist"){
Employee.getPersonList.size should be (1)
}
}
}
The test is failing and error is
java.lang.RuntimeException: There is no started application
at scala.sys.package$.error(package.scala:27)
at play.api.Play$$anonfun$current$1.apply(Play.scala:71)
at play.api.Play$$anonfun$current$1.apply(Play.scala:71)
at scala.Option.getOrElse(Option.scala:120)
at play.api.Play$.current(Play.scala:71)
at models.Employee$.getPersonList(EmployeeEntry.scala:27)
at databasetest$$anonfun$1$$anonfun$apply$mcV$sp$2.apply$mcV$sp(databasetest.scala:39)
at databasetest$$anonfun$1$$anonfun$apply$mcV$sp$2.apply(databasetest.scala:39)
at databasetest$$anonfun$1$$anonfun$apply$mcV$sp$2.apply(databasetest.scala:39)
at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
By default play provides spec2 testing framework.so no need to add scalatest framework for unit testing.For database access layer testing required a connection with database so start a fake application.(this is not running code, just an idea to write unit test)
for more detail take a look play doc : https://www.playframework.com/documentation/2.3.x/ScalaFunctionalTestingWithSpecs2
import org.specs2.mutable.Specification
import models.database.Employee
import models._
import models.EmployeeEntry
import play.api.test.FakeApplication
import play.api.test.Helpers.running
import play.api.Play.current
class databasetest extends Specification {
"database layer" should {
"save a row" in {
running(FakeApplication()) {
val a = EmployeeEntry(1006,"udit","schd","90909090")
Employee.savedat(a) must be equalTo (1)
}
}
"get list" in {
running(FakeApplication()) {
Employee.getPersonList.size must be equalTo (1)
}
}
}

How do you set the viewport, width, with phantomjs with play 2 framework

I have been struggling with this for a while, I can't find a way to instruct phantomjs about the viewport. I am using play 2.2 (code is based on: Set Accept-Language on PhantomJSDriver in a Play Framework Specification )
package selenium
import org.specs2.mutable.Around
import org.specs2.specification.Scope
import play.api.test.{TestServer, TestBrowser, FakeApplication}
import org.openqa.selenium.remote.DesiredCapabilities
import org.specs2.execute.{Result, AsResult}
import play.api.test.Helpers._
import scala.Some
import org.openqa.selenium.phantomjs.PhantomJSDriver
import org.openqa.selenium.NoSuchElementException
import java.io.File
import java.io.PrintWriter
import java.util.concurrent.TimeUnit
import collection.JavaConversions._
abstract class WithPhantomJS(val additionalOptions: Map[String, String] = Map()) extends Around with Scope {
implicit def app = FakeApplication()
implicit def port = play.api.test.Helpers.testServerPort
// for phantomjs
lazy val browser: TestBrowser = {
val defaultCapabilities = DesiredCapabilities.phantomjs
print(defaultCapabilities.toString)
val additionalCapabilities = new DesiredCapabilities(mapAsJavaMap(additionalOptions))
val capabilities = new DesiredCapabilities(defaultCapabilities, additionalCapabilities)
val driver = new PhantomJSDriver(capabilities)
val br = TestBrowser(driver, Some("http://localhost:" + port))
br.webDriver.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS)
br
}
override def around[T: AsResult](body: => T):Result = {
try {
running(TestServer(port, FakeApplication()))(AsResult.effectively(body))
} catch {
case e: Exception => {
val fn: String = "/tmp/test_" + e.getClass.getCanonicalName;
browser.takeScreenShot(fn + ".png")
val out = new PrintWriter(new File(fn + ".html"), "UTF-8");
try {
out.print(browser.pageSource())
} finally {
out.close
}
throw e;
}
} finally {
browser.quit()
}
}
}
and the actual test looks like this:
class ChangeName extends Specification {
"User" should {
"be able to change his name in account settings" in new WithPhantomJS() {
// ... this throws
I'd like the exception handler to render the site, which works, however, i can see the with matching a mobile device (narrow).
How can I change the width?
This seems to work:
import org.openqa.selenium.Dimension
...
browser.webDriver.manage().window().setSize(new Dimension(1280, 800))