how to catch a exception and do something in catch block when I save data into hdfs fail. like this:
try {
item.map(
r => doSome(r).saveAsTextFiles(outputPath + "/data")
} catch {
case e: Exception => {
val failMessage = "Exception from output part" + e.getClass + "\t" + e.getMessage
println("The exception is executed")
update(aaa)
}
} finally {
mc.close()
}
I want to update some status when the save action throw exception. how to do it ?
Use NonFatal hope it will solve your issue
import scala.util.control.NonFatal
try {
item.map(
r => doSome(r).saveAsTextFiles(outputPath + "/data")
} catch {
case NonFatal(error) => {
val failMessage = s"Exception from output part $error"
println("The exception is executed")
update(aaa)
}
} finally {
mc.close()
}
Related
Flux.<Integer>push(sink -> {
try {
for (int i = 0; i < 10 && !sink.isCancelled(); i++) {
sink.next(i);
}
sink.complete();
} catch (Exception e) {
System.out.println("Error in wrong place " + e);
}
})
.doOnNext(i -> { if (i % 5 == 0) throw new RuntimeException("Error"); })
.subscribe(i -> System.out.println("i = " + i));
Output of this code is
`Error in wrong place reactor.core.Exceptions$ErrorCallbackNotImplemented: java.lang.RuntimeException: Error`
it means that exception is rethrown by sink.next(i); in push method and not push to error reactive channel. Why?
It is pushed to flux error channel, but You have no error subscriber on this channel, You can add:
.doOnError(e -> System.out.println("Now error in good place = " + e))
or
.subscribe(i -> System.out.println("i = " + i), e -> System.out.println("Now error in good place = " + e));
The full code then is:
Flux.<Integer>push(sink -> {
try {
for (int i = 0; i < 10 && !sink.isCancelled(); i++) {
sink.next(i);
}
sink.complete();
} catch (Exception e) {
System.out.println("Error in wrong place " + e);
}
})
.doOnNext(i -> { if (i % 5 == 0) throw new RuntimeException("Error"); })
.doOnError(e -> System.out.println("Now error in good place = " + e)) //it's missing
.subscribe(i -> System.out.println("i = " + i));
Also see this answer, why You need subscriber and standard (fox next()) subscriber is not throwing an error: What is correct way to generate and handle exceptions when using Flux from projectreactor
I have tried many things, but no matter what I do in my tests (which simply sends a PUT request to "create a user") the logs do not enter any of the pathPrefix and just go to the end and fail to match anything.
Can anyone offer insight? Below is the class I have written as well as the simple test (which doesn't even check anything yet)
I know overall it is rather rudimentary and I am not doing things great, but it's just something I threw together to the point where I felt i could do a few quick tests on the routing
package system
import akka.actor.{Props, ActorRef, ActorLogging}
import akka.util.Timeout
import spray.http.HttpHeader
import spray.routing.directives.OnCompleteFutureMagnet
import spray.routing.{RequestContext, HttpServiceActor, Route}
import spray.http.StatusCodes._
import scala.concurrent.duration._
import akka.pattern.ask
import system.F_BackBone._
import scala.util.{Failure, Success}
import language.postfixOps
class F_Listener(backbone: ActorRef) extends HttpServiceActor with ActorLogging {
import context.dispatcher
implicit val timeout = Timeout(5 seconds)
log.debug("beginning user log\n")
//TODO write the main listener spawner that acts as the main spray server and binds these listeners to connections
//TODO must watch the spray side http server actor and die with it
val route: Route = { uri =>
log.debug("request received " + uri.unmatchedPath)
pathPrefix("data") { req =>
log.debug("data path detected " + req.unmatchedPath + "\n")
get {
genericGet(req, GetImage) //URIs for actual data like pictures
} ~
path("uploadimage") {
put { req =>
genericPut(PutImage(req.request.entity))
}
}
} ~
pathPrefix("user") { req =>
log.debug("user path detected" + req.unmatchedPath + "\n")
get {
genericGet(req, GetUserInfo)
} ~
post {
genericPost(req, req.request.headers, UpdateUserData)
} ~
delete {
genericDelete(req, DeleteUser)
}
pathPrefix("newuser") {
put { req =>
genericPut(CreateUser(req.request))
}
}
} ~
pathPrefix("page") { req =>
log.debug("page path detected" + "\n")
get {
genericGet(req, GetPageInfo)
} ~
post {
genericPost(req, req.request.headers, UpdatePageData)
} ~
delete {
genericDelete(req, DeletePage)
}
path("newpage") {
put { req =>
genericPut(CreatePage(req.request))
}
}
} ~
pathPrefix("profile") { req =>
log.debug("profile path detected" + "\n")
get {
genericGet(req, GetProfileInfo)
} ~ //no need for "createprofile" they are created with the user and can be accessed through the JSON returned with that creation
post {
genericPost(req, req.request.headers, UpdateProfileData)
} //no need to delete profile, deleted with user
} ~
pathPrefix("picture") { req =>
log.debug("picture path detected" + "\n")
get {
genericGet(req, GetPictureInfo)
} ~ //same as for profile, when you upload an image the picture JSON is created
post {
genericPost(req, req.request.headers, UpdateImageData)
} ~
delete {
genericDelete(req, DeletePicture)
}
} ~
pathPrefix("album") { req =>
log.debug("album path detected" + "\n")
get {
genericGet(req, GetAlbumInfo)
} ~
post {
genericPost(req, req.request.headers, UpdateAlbumData)
} ~
delete {
genericDelete(req, DeleteAlbum)
}
path("createalbum") {
put { req =>
genericPut(CreateAlbum(req.request))
}
}
}
log.error("no path matched" + "\n")
complete(NotFound, "That resource does not exist")
}
/**
* goes inside of a spray routing "get" and completes the passed message to the backbone given the id
* it internally converts the remaining request path to a bigint, if this fails it completes with a failure
* #param req the reques who contains the string to be converted to bigint as an ID
* #param messageConstructor the case class message (constructor but can be passed with sugar) to compose the bigint into
* #return returns a route for thed spray routing to go through and side effects the complete needed
*/
def genericGet(req: RequestContext, messageConstructor: (BigInt) => GetInfo): Route = {
val id = req.unmatchedPath.toString
if (!id.contains("/")) { //if this is the last element only
try {
val idBig = BigInt(id)
onComplete(OnCompleteFutureMagnet((backbone ? messageConstructor.apply(idBig)).mapTo[String])) {
case Success(entityJson) =>
log.info("get completed successfully: " + messageConstructor + " " + "for " + idBig)
complete(entityJson)
case Failure(ex) =>
log.error(ex, "get failed: " + messageConstructor + " for " + idBig)
complete(InternalServerError, "Request could not be completed: \n" + ex.getMessage)
}
} catch {
case e: NumberFormatException =>
log.info("illegally formatted id requested: " + id)
complete(BadRequest, "Numbers are formatted incorrectly")
}
}
else reject
}
/**
* same as above but for posts, I treid to write a more generic function to repeat rewriting code but it ended up just not being worth the thought
* #param req request who contains id to parse to bigint
* #param args arguments to change
* #param messageConstructor the message to send
* #return
*/
def genericPost(req: RequestContext, args: List[HttpHeader], messageConstructor: (BigInt, List[HttpHeader]) => PostInfo) = {
val id = req.unmatchedPath.toString
if (!id.contains("/")) {
try {
val idBig = BigInt(id)
onComplete(OnCompleteFutureMagnet((backbone ? messageConstructor.apply(idBig, args)).mapTo[String])) {
case Success(newEntityJson) =>
log.info("post completed successfully: " + messageConstructor + " for " + idBig)
complete(newEntityJson)
case Failure(ex) =>
log.error(ex, "get failed: " + messageConstructor + " for " + idBig)
complete(InternalServerError, "Request could not be completed: \n" + ex.getMessage)
}
} catch {
case e: NumberFormatException =>
log.info("illegally formatted id requested: " + id)
complete(BadRequest, "Numbers are formatted incorrectly")
}
}
else reject
}
/**
* same as above except no need to parse a special message since the path tells all and this is for putting, so the fully constructed message gets passed here
* #param message constructed message to send to the backbone for handling
* #return
*/
def genericPut(message: PutInfo) = {
pathEnd {
onComplete(OnCompleteFutureMagnet((backbone ? message).mapTo[String])) {
case Success(newEntityJson) =>
log.info("put completed successfully: " + message)
complete(newEntityJson)
case Failure(ex) =>
log.error(ex, "put failed: " + message)
complete(InternalServerError, "Error putting entity: " + ex.getMessage)
}
}
}
/**
* almost identical to get, should probobly only be one function
* #param req identical
* #param messageConstructor identical
* #return route
*/
def genericDelete(req: RequestContext, messageConstructor: (BigInt) => DeleteInfo) = {
val id = req.unmatchedPath.toString
if (!id.contains("/")) {
try {
val idBig = BigInt(id)
onComplete(OnCompleteFutureMagnet((backbone ? messageConstructor.apply(idBig)).mapTo[String])) {
case Success(newEntityJson) =>
log.info("delete completed successfully: " + messageConstructor + " for " + idBig)
complete(newEntityJson)
case Failure(ex) =>
log.error(ex, "get failed: " + messageConstructor + " for " + idBig)
complete(InternalServerError, "Request could not be completed: \n" + ex.getMessage)
}
} catch {
case e: NumberFormatException =>
log.info("illegally formatted id requested: " + id)
complete(BadRequest, "Numbers are formatted incorrectly")
}
}
else reject
}
override def receive = runRoute(route)
}
object F_Listener {
def props(backbone: ActorRef) = Props(new F_Listener(backbone))
}
/*ideas for speed improvements:
parse out arguments before passing to backbone (might help with scaling to distributed system)
genericDelete, Post, Put, and Get are all pretty similar, some functional composition is probobly possible, esp for delete and get
*/
import akka.testkit.TestActorRef
import org.scalatest.{WordSpec}
import spray.http.HttpHeaders
import spray.testkit.ScalatestRouteTest
import system.F_Listener
class FakebookRoutingTests extends WordSpec with ScalatestRouteTest {
val route = TestActorRef(new F_Listener(null)).underlyingActor.route
//implicit val host = DefaultHostInfo(HttpHeaders.Host("fake.fakebook.com"), false)
"the route" should {
"succeed on path to create a user" in {
Put("/user/newuser") ~> route ~> check {
}
}
}
}
Removing req => helps. If I remember correctly, put, get, post etc. don't provide any arguments for directives. Same goes for path. Removing uri => may also be needed.
Update:
If req => is to stay, then all calls finalizing the request should be called on that object. So, for example, instead of complete(entityJson) there should be req.complete(entityJson) on the top-most context object.
For given code top-most block with context is the first one containing uri so somewhere in the code there should be a call to uri.complete.
In my REST API service layer, I have a class ProductService.
The following logic exists in all my functions: Do Validate, if validation fails i throw invalid exception, if passes, i continue to the next try-catch and throw general-error in case of failure:
def addProduct(request:AddProductRequest): BaseResponse[String] =
{
try
{
request.validate
}
catch
{
case ex: Exception => {
Logger.error("Failed to add product, Validation failed", ex);
val errorResponse:ErrorResponse[String] = new ErrorResponseList().InvalidParameters
errorResponse.addMessage(ex.getMessage)
return errorResponse
}
}
try
{
val addedProductId = productRepository.addProduct(request.language, request.tenantId, request.product)
DTOResponse(addedProductId)
}
catch
{
case ex: Exception => {
Logger.error("Failed to add product to tenant Id="+request.tenantId+" language="+request.language, ex);
val errorResponse:ErrorResponse[String] = new ErrorResponseList().GeneralError
errorResponse.addMessage(ex.getMessage())
return errorResponse
}
}
}
Now, instead of repeating the request.validate with the same try and catch for all functions, i created a base class with the following function:
abstract class ServiceBase {
def validate[T](request:BaseRequest)
{
try
{
request.validate
}
catch
{
case ex: Exception => {
Logger.error("Validation failed", ex);
val errorResponse:ErrorResponse[String] = new ErrorResponseList().InvalidParameters
errorResponse.addMessage(ex.getMessage)
return errorResponse
}
}
}
So now, my addProduct(..) will look like:
validate(request)
..the rest of the code - the 2nd try-catch
This saves alot of lines.
The problem is that if validation fails, it will never return. I get the following errors in ServiceBase:
Multiple markers at this line
- enclosing method validate has result type Unit: return value discarded
- enclosing method validate has result type Unit: return value discarded
- a pure expression does nothing in statement position; you may be omitting necessary
parentheses
validate has no return type (and thus defaults to returning Unit), in ServiceBase your signature for validate should look like this:
def validate[T](request:BaseRequest): BaseResponse[String] =
(assuming you want to return a BaseResponse[String])
this may be useful to someone, someday, functional programming.. Did we say ^_^
Changed the ServiceBase validate to:
def validate[T](request:BaseRequest):Option[BaseResponse[T]] =
{
try
{
request.validate
None
}
catch
{
case ex: Exception => {
Logger.error("Validation failed", ex);
val errorResponse:ErrorResponse[T] = new ErrorResponseList().InvalidParameters
errorResponse.addMessage(ex.getMessage)
return Some(errorResponse)
}
}
}
And now i do:
def getProducts(request:GetProductsRequest) :BaseResponse[ProductSearchResults] =
{
validate[ProductSearchResults](request).getOrElse(
{
try
{
val products = productRepository.getProducts(request.language,request.tenantId,request.productIds)
val foundProducts = for (product <- products) yield (product.id)
val notFoundProducts = request.productIds filterNot (foundProducts.toSet)
val responseWrapper = new ProductSearchResults(products, notFoundProducts)
DTOResponse(responseWrapper)
}
catch
{
case ex: Exception => {
Logger.error("Failed to get products from tenant Id="+request.tenantId+" language="+request.language, ex);
val errorResponse:ErrorResponse[ProductSearchResults] = new ErrorResponseList().GeneralError
errorResponse.addMessage(ex.getMessage())
return errorResponse
}
}})
}
I validate my function, if an exception is thrown during validation, i wish to stop the method in the catch and return, from some reason, it will continue and only caught in the main try /catch.
Code:
def updateProduct(request: UpdateProductRequest): BaseResponse[String] =
{
try
{
try
{
ValidateUpdateProductRequest(request)
}
catch
{
case ex: Exception => {
val errorResponse:ErrorResponse[String] = ErrorResponse(ErrorCode.InvalidParameters, ex.getMessage, 500)
errorResponse // <=- This does not return from function.. In debug i get here
}
}
val deleteProductResult = productRepository.updateProduct(request) //I dont want to get here !!
DTOResponse(deleteProductResult)
}
catch
{
case ex: Exception => {
Logger.error("Failed to update product Id = " +request.product.id, ex);
var errorResponse:ErrorResponse[String] = ErrorResponse(ErrorCode.GeneralError, ex.getMessage, 500)
errorResponse
}
}
}
I Understand in scala the last line of function is the only place function will return, so how do i return from the catch ?
The reason is i want to use different ErrorCode in the BaseResponse[string]
Thanks!
Whenever you have an inner expression that you want to propagate to the outermost level to be the result, you can either assign it to a temporary variable in an outer expression, or use return. So, for example:
def foo: Int = {
try { bar }
catch { case ikte: IKnowTheAnswerException => return 42 }
lotsOfMath
}
def foo: Int = {
val iKnowIt = {
try { bar }
catch { case ikte: IKnowTheAnswerException => Some(42) }
}
iKnowIt.getOrElse( lotsOfMath )
}
Even though the second pattern seems uselessly wordy, keep in mind that jumping out of the method with a return is not always obvious, especially in longer methods. So the second can, in some cases, be clearer to read (especially when you know to expect the pattern).
I put together the code below; the intent was to have a non-blocking server accept a connection and then pass off this connection to an actor for further processing. This works the first time through, but on the subsequent request the program freezes at conServ ! servSoc.accept. Any ideas why this is happening?
import java.net._
import java.io._
import java.nio._
import java.nio.channels._
import java.util._
import scala.actors.Actor
import scala.actors.Actor._
def test() = {
var channel: ServerSocketChannel = null
val isa: InetSocketAddress = new InetSocketAddress(23)
val conServ = actor {
react {
case conn: Socket => {
try {
var pw: PrintWriter = new PrintWriter(conn.getOutputStream(), true)
pw.println("Current time: " + new Date)
pw.close
conn.close
} catch {
case ioe: IOException => println("IOException: " + ioe.getMessage)
case e: Exception => println("Exception: " + e.getMessage)
}
}
}
}
try {
channel = ServerSocketChannel.open
channel.configureBlocking(false)
channel.socket().bind(isa)
var selector: Selector = Selector.open
channel.register(selector, SelectionKey.OP_ACCEPT)
println("** Server ready for requests **")
while (true) {
if (selector.select > 0) {
var selKeys: Set[SelectionKey] = selector.selectedKeys
var selIt: Iterator[SelectionKey] = selKeys.iterator
while (selIt.hasNext) {
var key: SelectionKey = selIt.next.asInstanceOf[SelectionKey]
selIt.remove
if (key.isAcceptable) {
var ssc: ServerSocketChannel = key.channel.asInstanceOf[ServerSocketChannel]
var servSoc: ServerSocket = ssc.socket
try {
conServ ! servSoc.accept
} catch {
case ioe: IOException => println(ioe.printStackTrace)
}
}
}
} else {
continue
}
}
} catch {
case ioe: IOException => println("Could not listen to port 23. " + ioe.printStackTrace)
case e: Exception => println("Error: " + e.printStackTrace)
}
}
test
Enclose your react in a loop block like this:
val conServ = actor {
loop {
react {
// ...
}
}
}
What happens now, is that your actor is started, processes the first message and is not "reacting" again to process additional message from its queue.
See An actor's act method that uses loop.
This is what an actor do, treating one message at the time. What you want is a separate thread to handle each request. For this you can try using Futures.