I have written a lambda code that tracks login to AWS console, and send notification to user email about it.
The initial code I have written was in Java, and worked. After converting the code to Scala, I came to the following code:
class SNSHandler {
private val creds: AWSCredentials = new BasicAWSCredentials("xxx", "999/xyz12345")
private val eventType: String = "ConsoleLogin"
private val topicArn: String = "arn:aws:sns:us-east-1:1111111111:CTInterestingEvents"
def processLoginRecord(loginRecord: String, lambdaLogger: LambdaLogger) = {
val userName = JsonPath.read(loginRecord.asInstanceOf[Object], "$.userIdentity.type").asInstanceOf[String] match {
case "Root" => "Root"
case _ => JsonPath.read(loginRecord.asInstanceOf[Object], "$.userIdentity.userName")
}
val accountId = JsonPath.read(loginRecord.asInstanceOf[Object], "$.userIdentity.accountId")
new AmazonSNSClient(creds).publish(topicArn, "This is an auto notification message.\nUser " + userName +
" has logged in to AWS account id " + accountId + ".\n You are receiving this email because someone has subscribed your" +
" email address to this event.")
}
def processCloudTrailBulk(event: String, logger: LambdaLogger) = {
JsonPath.read(event.asInstanceOf[Object], "$.Records[?(#.eventName == '" + eventType + "' && #.responseElements.ConsoleLogin == 'Success')]").
asInstanceOf[java.util.List[String]].asScala.map(loginRecord => processLoginRecord(loginRecord, logger))
}
def processS3File(bucketName: String, file: String, logger: LambdaLogger) = {
Source.fromInputStream(new GZIPInputStream(new AmazonS3Client(creds).
getObject(new GetObjectRequest(bucketName, file)).getObjectContent),"UTF-8").getLines().
foreach(line => processCloudTrailBulk(line,logger))
}
def processSNSRecord(notification: SNSRecord, logger: LambdaLogger) = {
val bucketName: String = JsonPath.read(notification.getSNS.getMessage.asInstanceOf[Object], "$.s3Bucket")
logger.log("Notifications arrived.\nBucket: " + bucketName)
JsonPath.read(notification.getSNS.getMessage.asInstanceOf[Object], "$.s3ObjectKey[*]").asInstanceOf[java.util.List[String]].
asScala.map(file => processS3File(bucketName,file,logger))
}
def handleCloudTrailIncoming(event: SNSEvent, context: Context) = {
event.getRecords.asScala.map(record => processSNSRecord(record,context.getLogger))
}
}
Now, the addition of .asInstanceOf[Object] to the first param of every 'read' call wasn't there initially, however I had the famous compiler error of ambiguous reference to overloaded function, and after taking a look at: ambiguous reference to overloaded definition, from a Java library I added it, and indeed my code now compiles.
The problem is however that in runtime, the read now fails to detect the fields, and I get the following error:
Property ['s3Bucket'] not found in path $: com.jayway.jsonpath.PathNotFoundException com.jayway.jsonpath.PathNotFoundException: Property ['s3Bucket'] not found in path $ at com.jayway.jsonpath.internal.token.PropertyPathToken.evaluate(PropertyPathToken.java:41) ........
Related
I am Writing a scala method that reads in a Yaml file and returns a Map of the content of the Yaml file. I can do this successfully, but working with the data structure is very cumbersome as I will demonstrate below.
Note I can and have used jackson in scala to take a yaml file and constitute it into a case class. That works great and is not cumbersome to use. In this problem the yaml is dynamic so we need to put it into a dynamic data structure i.e Map or List of Map
In Java there is no issue in solving the problem. The data structure that is returned back is easy to work with.
Java example:
public Map readMapYml(String fullFileName) {
ObjectMapper mapper = new ObjectMapper(new YAMLFactory());
try {
return mapper.readValue(new File(fullFileName), Map.class);
} catch (Exception e) {
throw new RuntimeException("JavaParser->writeYml:
Unable to write yaml file: " + e.getMessage());
}
}
My equivalent scala code. (I have also tried many variation of the code below)
def readMapYml(fullFileName: String): Map[String,Any] = {
val mapper = new ObjectMapper(new YAMLFactory())
mapper.registerModule(DefaultScalaModule)
try {
mapper.readValue(new File(fullFileName), classOf[Map[String,Any]])
}
catch {
case e: Exception =>
throw new RuntimeException("Parser->readMapYml: Unable to read yaml
file to map. filename: " + fullFileName + " Message: " + e.getMessage)
}
}
So this works and I can parse through the data, but it is really cumbersome.
Example of how cumbersome:
result.get("groups").get.asInstanceOf[List[Map[String,Any]]](0).get("group").get.asInstanceOf[Map[String,Any]].get("colors").get.asInstanceOf[List[Map[String,Any]]](0).get("color").get
Btw interop works just fine I can write this in Java and call it from scala. However, in this case we need to get our scala code working
My question: I would like fasterxml Jackson to return a data structure that is much easier to use similar to the data structure I get back when done through Java. How do I do that?
One of the difficulties with the following approach is that it requires the datatype to be defined every time we extract the data it takes up datatype Any - thereby forcing us to define the data type for values.
mapper.readValue(new File(fullFileName), classOf[Map[String,Any]])
Since the YAML file is expected to be dynamic, it is better to use the much more developed JsonNode data type from Jackson, leveraging following approach:
val yaml = mapper.readValue(new File(fullFileName), classOf[Any])
val jsonString = mapper.writerWithDefaultPrettyPrinter().writeValueAsString(yaml)
val jsonObj = mapper.readTree(jsonString)
The resulting jsonObj is of JsonNode data type, which will have a dynamic schema & supports data conversion/typecasting needs with its in-built methods
The following code does a good job navigating a kv map returned from Jackson.
/**
* Purpose is to parse through a generic kv map of data returned from Jackson.
* #param structure data return from Jackson or a sub-structure of data returned from
Jackson
* #param path A path to the data we want to return. A stack so order is leaf to
branch to branch .... to root
* #return the section requested. The first item added to your stack. In other words
the last item pop.
*/
def getStructure(structure: Any, path: mutable.Stack[String]): Any = {
var retVal: Any = structure
if (path.nonEmpty) {
structure match {
case map: Map[String, Any] =>
retVal = map.get(path.pop())
case some: Some[Any] =>
retVal = some.get
case list: List[Any] =>
retVal = list(path.pop().toInt)
case None =>
throw new IllegalStateException("DataHelper->getStructure: Bad path keyword does not exist in section of path. remaining stack: " + path)
case _ =>
throw new IllegalStateException("DataHelper->getStructure: Structure type is unexpected. Type: " + structure.getClass.getName)
}
if (path.nonEmpty) {
retVal = getStructure(retVal, path)
}
}
retVal match {
case some: Some[Any] =>
retVal = some.get //If the last item is a some get the content of the some.
case _ =>
}
retVal
}
Test code:
test("testMyDataHelper") {
val mapParser = new MapParser
val result = mapParser.readMapYml("test.yaml")
var path = mutable.Stack[String]()
path.push("name","0","groups")
println(DataHelper.getStructure(result, path))//Joe
path.push("name","1","groups")
println(DataHelper.getStructure(result, path))//Bill
path.push("part2","0","items","0","groups")
println(DataHelper.getStructure(result,path))//dash
path.push("part2","2","items","0","groups")
println(DataHelper.getStructure(result,path))//line
//Example of getting a subsection of yaml
path.push("items","0","groups")
val subsection = DataHelper.getStructure(result,path)
//use the subsection
path.push("part1","2")
println(DataHelper.getStructure(subsection,path))//green
path.push("part2","0")
println(DataHelper.getStructure(subsection,path))//dash
}
yaml file
document: "0.0.1"
groups:
- version: "0.0.0"
type: "A"
name: "Joe"
agency: "skjenco"
number: 8
items:
- part1: "red"
part2: "dash"
- part1: "brown"
part2: "underline"
- part1: "green"
part2: "line"
- version: "0.0.1"
type: "B"
name: "Bill"
agency: "billco"
number: 3
items:
- part1: "orange"
part2: "line"
- part1: "pink"
part2: "dash"
- part1: "blue"
part2: "line"
Here is the code:
def readEntityMultipleTimes(entityName: String, pathPrefix: String = "") = {
val plural = entityName + "s"
exec(http(s"Geting all $plural")
.get(pathPrefix + plural)
.check(status is 200)
.check(jsonPath("$[*].id").findAll.saveAs("entityIds"))
).exec(s => {
if (logLevel >= 2) println("\nids:\n" + s("entityIds"))
s
})
.pause(interval millis)
.foreach("${entityIds}", "entityId") {
repeat(readEntityNumber) {
exec(http(s"Getting one $entityName")
.get(pathPrefix + plural + "/${entityId}")
.check(status is 200)
)
}
}
}
The issue is that entityId may contain a space and it fails the HTTP GET request. I need the spaces to be replaced with %20.
I tried the gatling EL ${entityId.replaceAll(\" \", \"%20\")}"
or ${java.net.URLEncoder.encode(entityId)}
I guess the suggested way is to get the entityId from the session and do the stuff in Scala, but this variable is dynamically created for each loop iteration, so I am not sure where to put the "session lambda" (session => ...)
Gatling EL syntax is limited and you can't place any Scala code in there.
You indeed have to pass a function.
.get(session => pathPrefix + plural + URLEncoder.encode(session("entityId").as[String])))
I am working with the Lift framework and Scala. I have a form to sign up to my application, and I want to validate all the fields in it. I have a snippet where I access my form values, and one validation class where I wrote my validation functions. The following code is what I've tried so far. In my Snippet:
if(validationClassObject.validateName(first_name)){
if(validationClassObject.validateName(last_name)){
if(validationClassObject.validateEmail(email)){
if(validationClassObject.validateUserName(name)){
// Adding values to the DB
S.redirectTo("/")
}
else{
S.notice("Invalid User Name")
}
}
else{
S.notice("Invalid Mail Id")
}
}
else{
S.notice("Invalid Last name")
}
}
else{
S.notice("Invalid First Name")
}
In the validationClass I wrote the validation code looks like:
//function for validating mail address
def validateEmail(email: String): Boolean =
"""(\w+)#([\w\.]+)""".r.unapplySeq(email).isDefined
//code for validating remaining fileds like above
This is working, but I know this is not the best way of coding this operation in Scala. How could I modify my code in a more scalable way? How can I use case classes here?
You could do:
def av[T,V](validationFunction: => Boolean, error: => T)(f: => V)={
if(!validationFunction) error
else f
}
def v[V](validationFunction: => Boolean, error: => String)(f: => V)=av(validationFunction,S.notice(error))(f)
import validationCalssObject._
v(validateName(last_name),"Invalid Last name"){v(validateName(name),"Invalid User Name"){...}}
av is a abstract method with T and V as result types for the error function and continue function f. v is the more specific function what expects a string for error and encapsulates the notice() call. we give f as the part in the curly braces v(validation, errormsg){/*todo when there is no problem*/}.
I can't do formatting in comments so I'll post a new answer.
def badName() = if ("name" == "") Some("bad name") else None
def badEmail() = if ("email" == "") Some("bad email") else None
val verifications = List[() => Option[String]](badName, badEmail)
val failed = verifications.flatMap(_())
if (failed.nonEmpty) {
// handle failed
} else {
// your custom logic here
}
if (badName) S.notice
else if (badEmail) S.notice
else if (badDay) S.notice
else { // everything OK...
// return a JsCmd or what else do you wanted here
}
An alternative solution can be written using Option and flatMap, without these all "if"-s hardcoded. If you're interested in that -- ask..
I am trying to upload multiple large files in the play framework using scala. I'm still a scala and play noob.
I got some great code from here which got me 90% of the way, but now I'm stuck again.
The main issue I have now is that I can only read the file data, not any other data that's been uploaded, and after poking around the play docs I'm unclear as to how to get at that from here. Any Suggestions appreciated!
def directUpload(projectId: String) = Secured(parse.multipartFormData(myFilePartHandler)) { implicit request =>
Ok("Done");
}
def myFilePartHandler: BodyParsers.parse.Multipart.PartHandler[MultipartFormData.FilePart[Result]] = {
parse.Multipart.handleFilePart {
case parse.Multipart.FileInfo(partName, filename, contentType) =>
println("Handling Streaming Upload:" + filename + "/" + partName + ", " + contentType);
//Set up the PipedOutputStream here, give the input stream to a worker thread
val pos: PipedOutputStream = new PipedOutputStream();
val pis: PipedInputStream = new PipedInputStream(pos);
val worker: UploadFileWorker = new UploadFileWorker(pis,contentType.get);
worker.start();
//Read content to the POS
play.api.libs.iteratee.Iteratee.fold[Array[Byte], PipedOutputStream](pos) { (os, data) =>
os.write(data)
os
}.mapDone { os =>
os.close()
worker.join()
if( worker.success )
Ok("uplaod done. Size: " + worker.size )
else
Status(503)("Upload Failed");
}
}
}
You have to handle the data part. As you can guess (or look up in the documentation) the function to handle the data part ist called: handleFilePart.
def myFilePartHandler: BodyParsers.parse.Multipart.PartHandler[MultipartFormData.FilePart[Result]] = {
parse.Multipart.handleFilePart {
// ...
}
parse.Multipart.handleFilePart {
// ...
}
}
Another way would be the handlePart method. Check the documentation for more details.
I am trying to do a simple case of /author/ and get the Lift to build a Person object based on the id passed in.
Currently i have an Author snippet
class Author(item: Person) {
def render = {
val s = item match { case Full(item) => "Name"; case _ => "not found" }
" *" #> s;
}
}
object Author{
val menu = Menu.param[Person]("Author", "Author", authorId => findPersonById(authorId), person => getIdForPerson(person)) / "author"
def findPersonById(id:String) : Box[Person] = {
//if(id == "bob"){
val p = new Person()
p.name="Bobby"
p.age = 32
println("findPersonById() id = " +id)
Full(p)
//}else{
//return Empty
//}
}
def getIdForPerson(person:Person) : String = {
return "1234"
}
}
What i am attempting to do is get the code to build a boxed person object and pass it in to the Author class's constructor. In the render method i want determine if the box is full or not and proceed as appropriate.
If i change
class Author(item: Person) {
to
class Author(item: Box[Person]) {
It no longer works but if i leave it as is it is no longer valid as Full(item) is incorrect. If i remove the val s line it works (and replace the s with item.name). So how do i do this. Thanks
The Box returned from findPersonById(id:String) : Box[Person] is evaluated and if the Box is Full, the unboxed value is passed into your function. If the Box is Empty or Failure the application will present a 404 or appropriate error page instead.
You can try double boxing your return if you want to handle this error checking yourself (so that the result of this method is always a Full Box).
def findPersonById(id:String) : Box[Box[Person]] = {
if(id == "bob"){
val p = new Person()
p.name="Bobby"
p.age = 32
println("findPersonById() id = " +id)
Full(Full(p))
}else{
return Full(Empty)
}
}
and then this should work:
class Author(item: Box[Person])