SignatureDoesNotMatch Aws CloudSearch scala - scala

I keep getting:
"#SignatureDoesNotMatch","error":{"message":"[Deprecated: Use the
outer message field] The request signature we calculated does not
match the signature you provided. Check your AWS Secret Access Key and
signing method. Consult the service documentation for details.
from trying to do a get request to cloudsearch. I verified that my Canonical String and String-to-Sign match the ones sent back from the error message everytime now, but I keep getting the error. Im assuming my signature itself isn't being processed correctly. But hard to nail it down.
def getHash(key:Array[Byte]): String = {
try
{
val md = MessageDigest.getInstance("SHA-256").digest(key)
md.map("%02x".format(_)).mkString.toLowerCase()
}
catch
{
case e: Exception => ""
}
}
.
def HmacSHA256(data:String, key:Array[Byte]): Array[Byte] = {
val algorithm="HmacSHA256";
val mac = Mac.getInstance(algorithm);
mac.init(new SecretKeySpec(key, algorithm));
mac.doFinal(data.getBytes("UTF8"));
}
.
...
val algorithm = "AWS4-HMAC-SHA256"
val credential_scope = date + "/us-west-1/cloudsearch/aws4_request"
val string_to_sign = algorithm + "\n" + dateTime + "\n" + credential_scope + "\n" + getHash(canonical_request)
val kSecret = ("AWS4" + config.getString("cloud.secret")).getBytes("utf-8")
val kDate = HmacSHA256(date.toString, kSecret)
val kRegion = HmacSHA256("us-west-1",kDate)
val kService = HmacSHA256("cloudsearch",kRegion)
val kSigning = HmacSHA256("aws4_request",kService)
val signing_key = kSigning
val signature = getHash(HmacSHA256(string_to_sign, kSigning))
val authorization_header = algorithm + " " + "Credential=" + config.getString("cloud.key") + "/" + credential_scope + ", " + "SignedHeaders=" + signed_headers + ", " + "Signature=" + signature
val complexHolder = holder.withHeaders(("x-amz-date",dateTime.toString))
.withHeaders(("Authorization",authorization_header))
.withRequestTimeout(5000)
.get()
val response = Await.result(complexHolder, 10 second)

I just released a helper library to sign your HTTP requests to AWS: https://github.com/ticofab/aws-request-signer . Hope it helps!

Related

Mallet Api in scala Akka throwing error -Request timeout encountered for request [GET /mallet Empty]

Please pardon me if the question sound naive as i am pretty new in akka.
I am trying to use the mallet api in scala akka in rest API
But getting error Request timeout encountered
below is the snapshot of my Server
Configuration.parser.parse(args,Configuration.ConfigurationOptions()) match {
case Some(config) =>
val serverBinding: Future[Http.ServerBinding] = Http().bindAndHandle(routes, config.interface, config.port)
serverBinding.onComplete {
case Success(bound) =>
println(s"Server online at http://${bound.localAddress.getHostString}:${bound.localAddress.getPort}/")
case Failure(e) =>
log.error("Server could not start: ", e)
system.terminate()
}
case None =>
system.terminate()
}
Await.result(system.whenTerminated, Duration.Inf)
snapshot of Router
lazy val apiRoutes: Route =
ignoreTrailingSlash {
pathSingleSlash {
complete(HttpEntity(ContentTypes.`text/html(UTF-8)`, "<html><body>Hello world!</body></html>"))
} ~
path("health") {
complete(HttpEntity(ContentTypes.`text/html(UTF-8)`, "ok"))
} ~
pathPrefix("mallet") {
parameter('malletFile) { malletFile =>
{val modelFile: Future[MalletModel] =
(malletActor ? GetMalletOutput(malletFile)).mapTo[MalletModel]
complete(modelFile)
}
}
}
}
and finally the snapshot of MalletActor
class MalletActor(implicit val uaCache: Cache[String, MalletModel],
implicit val executionContext: ExecutionContext)
extends Actor with ActorLogging with JsonSupport {
import MalletActor._
def receive: Receive = {
case GetMalletOutput(malletFile) => sender() ! createMalletResult2(malletFile)
}
def createMalletResult2(malletFile: String): MalletModel = {
logger.debug("run count...")
val res = MalletResult(malletFile)
val converted = res.Score.parseJson.convertTo[MalletRepo]
val fileName = converted.ContentId
val fileTemp = new File("src/main/resources/new_corpus/" + fileName)
if (fileTemp.exists) {
fileTemp.delete()
}
val output = new BufferedWriter(new FileWriter("src/main/resources/new_corpus/" + fileName))
output.write(converted.ContentText)
output.close()
//runMalletInferring()
val tmpDir = "src/main/resources/"
logger.debug("Import all documents to mallet...")
Text2Vectors.main(("--input " + tmpDir + "new_corpus/ --keep-sequence --remove-stopwords " + "--output " + tmpDir + "new_corpus.mallet --use-pipe-from " + tmpDir + "corpus.mallet").split(" "))
logger.debug("Run training process...")
InferTopics.main(("--input " + tmpDir + "new_corpus.mallet --inferencer " + tmpDir + "inferencer " + "--output-doc-topics " + tmpDir + "doc-topics-new.txt --num-iterations 1000").split(" "))
logger.debug("Inferring process finished.")
I am the getting error while calling text2Vector.main , the new vectorized file is getting created in new_corpus directory and new_corpus is getting generated as well. however after that I am getting the below error
Server online at http://127.0.0.1:9000/
12:45:38.622 [Sophi-Mallet-Api-akka.actor.default-dispatcher-3] DEBUG io.sophi.api.mallet.actors.MalletActor$ - run count...
12:45:38.634 [Sophi-Mallet-Api-akka.actor.default-dispatcher-3] DEBUG io.sophi.api.mallet.actors.MalletActor$ - Import all documents to mallet...
Couldn't open cc.mallet.util.MalletLogger resources/logging.properties file.
Perhaps the 'resources' directories weren't copied into the 'class' directory.
Continuing.
May 26, 2019 12:45:38 PM cc.mallet.classify.tui.Text2Vectors main
INFO: Labels =
May 26, 2019 12:45:38 PM cc.mallet.classify.tui.Text2Vectors main
INFO: src/main/resources/new_corpus/
May 26, 2019 12:45:46 PM cc.mallet.classify.tui.Text2Vectors main
INFO: rewriting previous instance list, with ID = 4e0a5a65540221c3:d508579:14b2ca15a26:-7ff7
[INFO] [05/26/2019 12:45:58.476] [Sophi-Mallet-Api-akka.actor.default-dispatcher-4] [akka.actor.ActorSystemImpl(Sophi-Mallet-Api)] Request timeout encountered for request [GET /mallet Empty]
in web browser as well I am getting the error
The server was not able to produce a timely response to your request.
Please try again in a short while!
I figure out the solution myself. Please correct me if it still is vague.
There were couple of change needed.
First in application configuration file (application.conf)
below code needed to be updated.
server{
idle-timeout = infinite
request-timeout = infinite
}
settings {
akka-workers-count = 100
akka-workers-count = ${?AKKA_WORKERS_COUNT}
actor-timeout = 100
actor-timeout = ${?ACTOR_TIMEOUT}
}
And the issue is needed to be handled in server code
by putting the api call inside timeout response
val timeoutResponse = HttpResponse(
StatusCodes.EnhanceYourCalm,
entity = "Running Mallet modelling.")
lazy val apiRoutes: Route =
ignoreTrailingSlash {
pathSingleSlash {
complete(HttpEntity(ContentTypes.`text/html(UTF-8)`, "<html><body>Hello world!</body></html>"))
} ~
path("health") {
complete(HttpEntity(ContentTypes.`text/html(UTF-8)`, "ok"))
} ~
pathPrefix("mallet") {
withRequestTimeout(5.minute, request => timeoutResponse) {
parameter('malletFile) { malletFile => {
val modelFile: Future[MalletModel] =
(malletActor ? GetMalletOutput(malletFile)).mapTo[MalletModel]
complete(modelFile)
}
}
}
}
}

slick db.run is not called

I want to insert a record in database but the db.run is not called
my code looks like this
val insertQueryStep = processStepTemplates returning processStepTemplates.map(_.id) into ((processStep, id) => processStep.copy(id = Some(id)))
/**
* Generates a new ProcessStepTemplate
*
* #param step
* #return
*/
def addProcessStepTemplateToProcessTemplate(step: ProcessStepTemplatesModel, processId: Int): Future[Some[ProcessStepTemplatesModel]] = {
println("In DTO: " + step + ", processtemplate: " + processId)
//val p = processStepTemplates returning processStepTemplates.map(_.id) += step
val p = insertQueryStep += step
db.run(p).map(id => {
println("Die Query lautet: " + p)
println("Die erzeugte ID lautet: " + id)
//Update the foreign key
val q = for { p <- processStepTemplates if p.id == id } yield p.processtemplate
val updateAction = q.update(Some(processId))
db.run(updateAction).map(id => {
println("Der neue Prozesschritt lautet: " + step)
Some(step)
})
Some(step)
})
}
What could be a problem in this case?
You should compose your futures as monads (with flatMap). Because the inner future will not complete.
Try to change your code in the following way (see comments #1, #2):
def addProcessStepTemplateToProcessTemplate(step: ProcessStepTemplatesModel, processId: Int): Future[Some[ProcessStepTemplatesModel]] = {
println("In DTO: " + step + ", processtemplate: " + processId)
//val p = processStepTemplates returning processStepTemplates.map(_.id) += step
val p = insertQueryStep += step
db.run(p).flatMap(id => { // #1 change map to flatMap
println("Die Query lautet: " + p)
println("Die erzeugte ID lautet: " + id)
//Update the foreign key
val q = for { p <- processStepTemplates if p.id == id } yield p.processtemplate
val updateAction = q.update(Some(processId))
val innerFuture = db.run(updateAction).map(id => {
println("Der neue Prozesschritt lautet: " + step)
Some(step)
})
innerFuture // # 2 return inner future
})
}
Also use a logging for detecting another issues (connected with db-schema, queries, etc).

Saving KMeansModel on HDFS

I want to save a KMeans Model on Hdfs. To do this I use the method save and create the otuput directory during runtime(See code). I get the Error Exception metadata already exists. How can I solve this problem?
val lastUrbanKMeansModel = KMeansModel.load(spark, defaultPath + "UrbanRoad/201692918")
val newUrbanKMeansObject = new KMeans()
.setK(7)
.setMaxIterations(20)
.setInitialModel(lastUrbanKMeansModel)
val vectorUrbanRoad = typeStreet.filter(k => k._2 == 1).map(_._1)
if (!vectorUrbanRoad.isEmpty()) {
val newUrbanModel = newUrbanKMeansObject.run(vectorUrbanRoad)
newUrbanModel.save(spark, defaultPath + "UrbanRoad/" +
Calendar.getInstance().get(Calendar.YEAR).toString
+ (Calendar.getInstance().get(Calendar.MONTH) + 1).toString +
Calendar.getInstance().get(Calendar.DAY_OF_MONTH).toString +
Calendar.getInstance().get(Calendar.HOUR_OF_DAY).toString)
}

support and lift for fp-growth rules in mllib spark/scala

I would like to extract support and lift for generated association rules with fp-growth. Having found the rules with the code below I manually go through the transactions and calculate support and lift. I wonder if there is a more legant way to extract this info. thanks!
val fpg = new FPGrowth()
.setMinSupport(0.2)
.setNumPartitions(10)
val model = fpg.run(transactions)
model.freqItemsets.collect().foreach { itemset =>
println(itemset.items.mkString("[", ",", "]") + ", " + itemset.freq)
}
val minConfidence = 0.8
model.generateAssociationRules(minConfidence).collect().foreach { rule =>
println(
rule.antecedent.mkString("[", ",", "]")
+ " => " + rule.consequent .mkString("[", ",", "]")
+ ", " + rule.confidence)
}
mm not elegant but this is what I do
val freqs = fpgrowth_model(transactions, min_supp=supp)
val supps = freqs.withColumn("support", $"freq" / total_transactions)
val rules = get_rules(transactions, min_supp=supp, min_confidence=conf)
val cross_df = supps.join(rules, $"items" === $"consequent")
.withColumn("lift",$"confidence" / $"support")

Scala - Twitter API returns 401 for oauth

I'm making my first steps in Scala and trying to implement application which uses Twitter streaming API. Below is my code (user tokens are hidden). From main function, I call getStreamData function, which calls makeAPIrequest.
package com.myname.myapp
import java.net.URL
import javax.net.ssl.HttpsURLConnection
import java.io.InputStream
import java.io.OutputStream;
import scala.io.Source
import java.net.URLEncoder
import java.util.Base64
import java.nio.charset.StandardCharsets
import scala.collection.immutable.HashMap
import java.util.Calendar
import java.io.Serializable
import scala.collection.immutable.TreeMap
import javax.crypto
import java.security.SecureRandom
import java.math.BigInteger
import scala.util.Random
object TwitterConnector {
private val AUTH_URL: String = "https://api.twitter.com/oauth2/token"
private val CONSUMER_KEY: String = "mykey"
private val CONSUMER_SECRET: String = "mysecret"
private val STREAM_URL: String = "https://stream.twitter.com/1.1/statuses/filter.json"
private var TOKEN: String = "mytoken"
private var TOKEN_SECRET: String = "mytokensecret"
def getStreamData {
val data = "track=" + "twitter"
makeAPIrequest(HTTPmethod("POST"), "https://stream.twitter.com/1.1/statuses/filter.json", None, Option(data))
}
private def makeAPIrequest(method: HTTPmethod, url:String, urlParams:Option[String], data:Option[String]){
//form oauth parameters
val oauth_nonce = Random.alphanumeric.take(32).mkString
val oauth_signature_method: String = "HMAC-SHA1"
val oauth_version: String = "1.0"
val oauth_timestamp = (Calendar.getInstance.getTimeInMillis/1000).toString()
var signatureData = scala.collection.mutable.Map(("oauth_consumer_key", CONSUMER_KEY), ("oauth_token", TOKEN), ("oauth_signature_method", oauth_signature_method), ("oauth_nonce", oauth_nonce), ("oauth_timestamp", oauth_timestamp), ("oauth_version", oauth_version))
//find keys for parameters
val getParams = (parameter: String) => {
val arr = parameter.split("=")
if(arr.length == 1) return
val key = arr(0).asInstanceOf[String]
val value = arr(1).asInstanceOf[String]
signatureData(key) = value
}
val params = urlParams match {
case Some(value) => {
val result = urlParams.get
result.split("&").foreach {getParams}
result
}
case None => ""
}
val postData = data match {
case Some(value) => {
val result = data.get
result.split("&").foreach {getParams}
result
}
case None => ""
}
//url-encode headers data
signatureData.foreach { elem => {
signatureData.remove(elem._1)
signatureData(urlEnc(elem._1)) = urlEnc(elem._2)
}
}
println(signatureData)
//sort headers data
val sortedSignatureData = TreeMap(signatureData.toSeq:_*)
println("Sorted: " + sortedSignatureData)
//form output string
var parameterString = ""
sortedSignatureData.foreach(elem => {
if(parameterString.length() > 0){
parameterString += "&"
}
parameterString += elem._1 + "=" + elem._2
})
val outputString = method.method.toUpperCase() + "&" + urlEnc(url) + "&" + urlEnc(parameterString)
val signingKey = urlEnc(CONSUMER_SECRET) + "&" + urlEnc(TOKEN_SECRET)
println(outputString)
println(signingKey)
val SHA1 = "HmacSHA1";
val key = new crypto.spec.SecretKeySpec(bytes(signingKey), SHA1)
val oauth_signature = {
val mac = crypto.Mac.getInstance(SHA1)
mac.init(key)
new String(base64(mac.doFinal(bytes(outputString)).toString()))
}
println("Signature: " + oauth_signature)
val authHeader: String = "OAuth oauth_consumer_key=\"" + urlEnc(CONSUMER_KEY) + "\", oauth_nonce=\"" + urlEnc(oauth_nonce) + "\", oauth_signature=\"" + urlEnc(oauth_signature) + "\", oauth_signature_method=\"HMAC-SHA1\", oauth_timestamp=\"" + urlEnc(oauth_timestamp) + "\", oauth_token=\"" + urlEnc(TOKEN) + "\", oauth_version=\"1.0\""
println(authHeader)
var text = url
if(params.length > 0){
text += "?"
}
val apiURL: URL = new URL(text + params)
val apiConnection: HttpsURLConnection = apiURL.openConnection.asInstanceOf[HttpsURLConnection]
apiConnection.setRequestMethod(method.method)
apiConnection.setRequestProperty("Authorization", authHeader)
apiConnection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded;charset=UTF-8")
if(method.method == "POST" && postData.length() > 0){
println("POSTING ", postData)
apiConnection.setDoOutput(true)
val outStream: OutputStream = apiConnection.getOutputStream
outStream.write(postData.getBytes())
}
val inStream: InputStream = apiConnection.getInputStream
val serverResponse = Source.fromInputStream(inStream).mkString
println(serverResponse)
}
private def bytes(str: String) = str.getBytes("UTF-8")
private def urlEnc(str: String) = URLEncoder.encode(str, "UTF-8").replace(" ", "%20")
private def base64(str: String) = Base64.getEncoder.encodeToString(str.getBytes(StandardCharsets.UTF_8))
}
Twitter returns me 401 code response.
Obviously, I'm doing something wrong. Could you point me where my error is?
I recommend using a better library for making web requests, such as the WS library from the Play Framework. Right now, you're sort of writing Java in Scala. Here's a sample usage of the WS library:
val clientConfig = new DefaultWSClientConfig()
val secureDefaults: com.ning.http.client.AsyncHttpClientConfig = new NingAsyncHttpClientConfigBuilder(clientConfig).build()
val builder = new com.ning.http.client.AsyncHttpClientConfig.Builder(secureDefaults)
builder.setCompressionEnabled(true)
val secureDefaultsWithSpecificOptions: com.ning.http.client.AsyncHttpClientConfig = builder.build()
implicit val implicitClient = new play.api.libs.ws.ning.NingWSClient(secureDefaultsWithSpecificOptions)
val oauthCalc = OAuthCalculator(ConsumerKey(TwitterConfig.consumerKey, TwitterConfig.consumerSecret), RequestToken(TwitterConfig.accessKey, TwitterConfig.accessSecret))
def lookup(ids: List[String]): Future[List[Tweet]] =
WS.clientUrl(`statuses/lookup`)
.withQueryString("id" -> ids.mkString(","))
.sign(oauthCalc)
.get()
.map { r =>
JsonHelper.deserialize[List[Tweet]](r.body)
}
You should be able to modify this example pretty easily to work with the streaming API.