Objective:-
Retrieve json object from aws s3 bucket. Retry on failure.
If retry attempts are exhausted, send the error message to azure's event hub.
//Retry function
def retry[T](n: Int, id: String)(fn: => T): Option[T] = {
Try(fn) match {
case Success(x) => Some(x)
case Failure(e) => {
Thread.sleep(1000)
if (n > 1) {
retry(n - 1, id)(fn)
} else {
val eMessage = "TransactionId =" + id + " : ".concat(e.toString())
SendMessage(eMessage, event_hub_client)
None
}
}
}
}
//Main function (with above retry) to retrieve objects from aws s3 bucket
def getS3Object(s3ObjectName: String, s3Client: AmazonS3, evtClient: EventHubClient): String = {
retry(2,s3ObjectName) {
val inputS3Stream = s3Client.getObject("my_s3_bucket", s3ObjectName).getObjectContent
val inputS3String = IOUtils.toString(inputS3Stream, "UTF-8")
return inputS3String
}
}
I get the below compilation failed message:-
error: type mismatch
found : Option[Nothing]
required: String
retry2(2,s3ObjectName)
How to declare the return types of the retry and main functions so that they match?
In terms of types retry look's okay. You need to adapt your other method.
Either make it return a Option[String] or use getOrElse(..) if you want to fallback to some other value.
def getS3Object(s3ObjectName: String, s3Client: AmazonS3, evtClient: EventHubClient): Option[String] = {
retry(2,s3ObjectName) {
val inputS3Stream = s3Client.getObject("my_s3_bucket", s3ObjectName).getObjectContent
val inputS3String = IOUtils.toString(inputS3Stream, "UTF-8")
inputS3String
}
}
Notice the removal of return keyword which is useless and doesn't do what you think it does (I'll let you search on this topic by yourself).
Related
I have an incoming JSON data that looks like below:
{"id":"1000","premium":29999,"eventTime":"2021-12-22 00:00:00"}
Now, I have created a class that will accept this record and will check whether the data type of the incoming record is according to the data types defined in the case class. However, when I am calling the method it is always calling the Failure part of the match case.
case class Premium(id: String, premium: Long, eventTime: String)
class Splitter extends ProcessFunction[String, Premium] {
val outputTag = new OutputTag[String]("failed")
def fromJson[T](json: String)(implicit m: Manifest[T]): Either[String, T] = {
Try {
println("inside")
lazy val mapper = new ObjectMapper() with ScalaObjectMapper
mapper.registerModule(DefaultScalaModule)
mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
mapper.readValue[T](json)
} match {
case Success(x) => {
Right(x)
}
case Failure(err) => {
Left(json)
}
}
}
override def processElement(i: String, context: ProcessFunction[String, Premium]#Context, collector: Collector[Premium]): Unit = {
fromJson(i) match {
case Right(data) => {
collector.collect(data)
println("Good Records: " + data)
}
case Left(json) => {
context.output(outputTag, json)
println("Bad Records: " + json)
}
}
}
}
Based on the sample record above, it should pass the Success value but no matter what I pass, it always enters the Failure part. What else is missing?
I am using Scala 2.11.12 and I tried examples from this link and this link but no luck.
In the following code, I am reading no. of lines from a file. If something goes wrong, I'll like to close the file pointer. But how can I find out if f contains valid pointer or not?
def countLines(filename:String):Option[Int] = {
try{
val f = Source.fromFile(filename)
println(s"no. of lines ${f.getLines().size}")
Some(f.getLines.size)
} catch {
case ex: FileNotFoundException => {
println(s"file ${filename} not found")
None
}
} finally {
//f might not be a valid pointer depending on when the error occured
}
}
The book I am reading uses var to maintain state (if f is valid or not) but I am trying to avoid it for sake of using only immutable variables.
def countLines(filename:String):Option[Int] = {
var f:Option[Source] = None
try{
f = Some(Source.fromFile(filename))
println(s"no. of lines ${f.get.getLines().size}")
Some(f.get.getLines.size)
} catch {
case ex: FileNotFoundException => {
println(s"file ${filename} not found")
None
}
} finally {
for(i<-f){
println("closing file")
i.close()
}
}
}
A double Try(). This closes the io resource even if the getLines() fails, but only if the fromFile() succeeds.
import scala.util.Try
def countLines(filename: String): Option[Int] =
Try(io.Source.fromFile(filename)).fold(_ => None, {f =>
val count = Try(f.getLines().length)
f.close()
count.toOption
})
What do you think about this?
If you want Scala-way - i think it's good example for your task:
def countLines(filename: String): Try[Int] = Try(Source.fromFile(filename).getLines.toList.size)
def test() = {
val filename = "/etc/passwd"
countLines(filename) match {
case Success(n) => println(n)
case Failure(f) => println(f)
}
}
When n - is a number of our lines, and f - is a Throwable.
How about this:
def countLines(filename: String): Option[Int] = {
val file = Try(Source.fromFile(filename))
val count = file.map(_.getLines().size)
(for {
_ <- count.recoverWith { case _ => file.map(_.close()) }
lineCount <- count
} yield lineCount).toOption
}
Let's analyze it:
If file does not exist we will have failed Try instance and method returns None. In this case you do not need to clear any resources as no actual stream was created.
If getLines fails for any reason or anything else during processing goes south we will close created stream in first line of for comprehension
Hope it helps
Simply, how about this:
def numLines(fileName:String):Option[Int] = {
try {
val f = scala.io.Source.fromFile(fileName)
try { Some(f.getLines.size) }
catch { case ex: IOException =>
Console.err.println("i/o excetion")
None
}
finally { f.close() }
}
catch {
case ex: FileNotFoundException =>
Console.err.println("file not found")
None
}
}
I got the error
found : scala.concurrent.Future[Option[models.ProcessTemplatesModel]]
required: Option[models.ProcessTemplatesModel]
My function is below
def createCopyOfProcessTemplate(processTemplateId: Int): Future[Option[ProcessTemplatesModel]] = {
val action = processTemplates.filter(_.id === processTemplateId).result.map(_.headOption)
val result: Future[Option[ProcessTemplatesModel]] = db.run(action)
result.map { case (result) =>
result match {
case Some(r) => {
var copy = (processTemplates returning processTemplates.map(_.id)) += ProcessTemplatesModel(None, "[Copy of] " + r.title, r.version, r.createdat, r.updatedat, r.deadline, r.status, r.comment, Some(false), r.checkedat, Some(false), r.approvedat, false, r.approveprocess, r.trainingsprocess)
val composedAction = copy.flatMap { id =>
processTemplates.filter(_.id === id).result.headOption
}
db.run(composedAction)
}
}
}
}
what is my problem in this case?
edit:
my controller function looks like this:
def createCopyOfProcessTemplate(processTemplateId: Int) = Action.async {
processTemplateDTO.createCopyOfProcessTemplate(processTemplateId).map { process =>
Ok(Json.toJson(process))
}
}
Is there my failure?
According to the your code - there are the following issues:
You use two db.run which return futures, but inner future will
not complete. For resolving it you should compose futures with
flatMap or for-comprehension.
You use only one partial-function case Some(_) => for pattern matching
and don't handle another value None.
You can use only one db.run and actions composition.
Your code can be like as:
def createCopyOfProcessTemplate(processTemplateId: Int): Future[Option[ProcessTemplatesModel]] = {
val action = processTemplates.filter(...).result.map(_.headOption)
val composedAction = action.flatMap {
case Some(r) =>
val copyAction = (processTemplates returning processTemplates...)
copyAction.flatMap { id =>
processTemplates.filter(_.id === id).result.headOption
}
case _ =>
DBIO.successful(None) // issue #2 has been resolved here
}
db.run(composedAction) // issue #3 has been resolved here
}
We get rid of issue #1 (because we use actions composition).
I am just new to learning Scala and the related technologies.I am coming across the problem where the loadUser should return a record but its coming empty.
I am getting the following error:
java.util.NoSuchElementException: None.get
I appreciate this is not ideal Scala, so feel free to suggest me improvements.
class MongoDataAccess extends Actor {
val message = "Hello message"
override def receive: Receive = {
case data: Payload => {
val user: Future[Option[User]] = MongoDataAccess.loadUser(data.deviceId)
val twillioApiAccess = context.actorOf(Props[TwillioApiAccess], "TwillioApiAccess")
user onComplete {
case Failure(exception) => println(exception)
case p: Try[Option[User]] => p match {
case Failure(exception) => println(exception)
case u: Try[Option[User]] => twillioApiAccess ! Action(data, u.get.get.phoneNumber, message)
}
}
}
case _ => println("received unknown message")
}
}
object MongoDataAccess extends MongoDataApi {
def connect(): Future[DefaultDB] = {
// gets an instance of the driver
val driver = new MongoDriver
val connection = driver.connection(List("192.168.99.100:32768"))
// Gets a reference to the database "sensor"
connection.database("sensor")
}
def props = Props(new MongoDataAccess)
def loadUser(deviceId: UUID): Future[Option[User]] = {
println(s"Loading user from the database with device id: $deviceId")
val query = BSONDocument("deviceId" -> deviceId.toString)
// By default, you get a Future[BSONCollection].
val collection: Future[BSONCollection] = connect().map(_.collection("profile"))
collection flatMap { x => x.find(query).one[User] }
}
}
Thanks
There is no guaranty the find-one (.one[T]) matches at least one document in your DB, so you get an Option[T].
Then it's up to you to consider (or not) that having found no document is a failure (or not); e.g.
val u: Future[User] = x.find(query).one[User].flatMap[User] {
case Some(matchingUser) => Future.successful(matchingUser)
case _ => Future.failed(new MySemanticException("No matching user found"))
}
Using .get on Option is a bad idea anyway.
On the first line below, I want to know if there's anything I can or should put to the left of the = that will future-proof against bad editing. I had one Action.async, but now I want to have two choices (run a single MongoDB query or run multiple MongoDB queries) to arrive at the answer to the web query q. My code compiles as-is, but I think I should have to put something before the = to make the code more type-safe.
def q(arg: String) =
if (wantMultipleQueriesByDataSource)
runMultipleQueriesByDataSource(arg)
else
runSingleQuery(arg)
def runSingleQuery(arg: String) = Action.async {
if (oDb.isDefined) {
val collection: MongoCollection[Document] = oDb.get.getCollection(collectionName)
val fut = getMapData(collection, arg)
fut.map { docs: Seq[Document] =>
val docsSources = List(DocsSource(docs, "*"))
val pickedDocs = pickDocs(args, docsSources)
Ok(buildQueryAnswer(pickedDocs))
} recover {
case e => BadRequest("FAIL(rect): " + e.getMessage)
}
}
else Future.successful(Ok(buildQueryAnswer(Nil)))
}
def runMultipleQueriesByDataSource(arg: String) = Action.async {
if (oDb.isDefined) {
val collection: MongoCollection[Document] = oDb.get.getCollection(collectionName)
val dataSources = List("apples", "bananas", "cherries")
val futureCollections = dataSources map { getMapDataByDataSource(collection, _, arg) }
Future.sequence(futureCollections) map {
case docsGroupedByDataSource: Seq[Seq[Document]] =>
val docsSources = (docsGroupedByDataSource zip dataSources) map {x => DocsSource(x._1, x._2)}
val pickedDocs = pickDocs(args, docsSources)
Ok(buildQueryAnswer(pickedDocs))
case _ => Ok(buildQueryAnswer(Nil))
} recover {
case e => BadRequest("FAIL(rect): " + e.getMessage)
}
}
else Future.successful(Ok(buildQueryAnswer(Nil)))
}
You don't need to because Scala has type inference in this case. But if you want, you can change the method signature to the following:
def q(arg: String): Action[AnyContent]
I'm assuming here that you are using Playframework.
Edit: In fact, in this case, Action.async return an Action[AnyContent] and it receives a block that returns a Future[Result].