I'm trying to connect to an IAM controlled ElasticSearch domain, I've created a request, and signed it, and everything works fine for method GET, but on method POST I get a 400 Bad Request
This clearly has something to do with the payload. If I provide a payload empty string ("") it works apporpriately, but anything else results in a bad request.
What am I missing?
val url = s"https://$host/TEST/article/_search"
val serviceName = "es"
val regionName = "us-east-1"
val request = new DefaultRequest(serviceName)
val payload =
"""{"1":"1"}""".trim
val payloadBytes = payload.getBytes(StandardCharsets.UTF_8)
val payloadStream = new ByteArrayInputStream(payloadBytes)
request.setContent(payloadStream)
val endpointUri = URI.create(url)
request.setEndpoint(endpointUri)
request.setHttpMethod(HttpMethodName.POST)
val credProvider = new EnvironmentVariableCredentialsProvider
val credentials = credProvider.getCredentials
val signer = new AWS4Signer
signer.setRegionName(regionName)
signer.setServiceName(serviceName)
signer.sign(request, credentials)
val context = new ExecutionContext(true)
val clientConfiguration = new ClientConfiguration()
val client = new AmazonHttpClient(clientConfiguration)
val rh = new MyHttpResponseHandler
val eh = new MyErrorHandler
val response =
client.execute(request, rh , eh, context);
note: if you run into this problem, inspect the actual content of the response, it may be a result of a mismatch between the index and your query.
My problem was that the specific query I was using was inappropriate for the specified index, and that resulted in a 400
Related
So I had a url link for a file, and the file is encrypted using "AES/CBC/PKCS5Padding", I have the cipher key and the iv string. I am using amazon ec2 to download the data and store to s3. what I am doing currently is creating an input stream using HTTPCLIENT() and GETMETHOD() and then javax.crypto library on the input stream and then finally putting this inputstream in a s3 location. Thia works for one file; but I need to scale up and do the same thing for multiple such url. How or can I take help of parallel download?
the download time for say n files is same if I don't parallelize the array of url, if I use 1 node or 4 nodes(1 master and 3 slaves). And if I use .par, it gives out of memory error. Here is the code for the data download and upload(the function downloadAndProcessDataFunc)
val client = new HttpClient()
val s3Client = AmazonS3ClientBuilder.standard().withRegion(Regions.EU_WEST_1).build()
val method = new GetMethod(url)
val metadata = new ObjectMetadata()
val responseCode = client.executeMethod(method);
println("FROM SCRIPT -> ", destFile, responseCode)
val istream = method.getResponseBodyAsStream();
val base64 = new Base64();
val key = base64.decode(cipherKey.getBytes());
val ivKey = base64.decode(iv.getBytes());
val secretKey = new SecretKeySpec( key, "AES" );
val ivSpec = new IvParameterSpec(ivKey);
val cipher = Cipher.getInstance( algorithm )
cipher.init( Cipher.DECRYPT_MODE, secretKey, ivSpec );
val cistream = new CipherInputStream( istream, cipher );
s3Client.putObject(<bucket>, destFile, cistream, metadata)
this is the call I do, but isn't parallelized
manifestArray.foreach{row => downloadAndProcessDataFunc(row.split(",")(0), row.split(",")(1), row.split(",")(2), row.split(",")(3), row.split(",")(4), row.split(",")(5).toLong)
this runs out of memory
manifest.par.foreach{row => downloadAndProcessDataFunc(row.mkString(",").split(",")(0), row.mkString(",").split(",")(1), row.mkString(",").split(",")(2), row.mkString(",").split(",")(3), element.replace("s3://testbucketforemrupload/manifest/", "data/") + "/" + row.mkString(",").split(",")(4).split("/").last)
now if i change the downloadAndProcessDataFunc function to this, it just downloads around 30/1536 such url and kills the rest of tje executors for out of memory error
val client = new HttpClient()
val s3Client = AmazonS3ClientBuilder.standard().withRegion(Regions.EU_WEST_1).build()
val method = new GetMethod(url)
val metadata = new ObjectMetadata()
val responseCode = client.executeMethod(method);
println("FROM SCRIPT -> ", destFile, responseCode)
val istream = method.getResponseBodyAsStream();
val base64 = new Base64();
val key = base64.decode(cipherKey.getBytes());
val ivKey = base64.decode(iv.getBytes());
val secretKey = new SecretKeySpec( key, "AES" );
val ivSpec = new IvParameterSpec(ivKey);
val cipher = Cipher.getInstance( algorithm )
cipher.init( Cipher.DECRYPT_MODE, secretKey, ivSpec );
val cistream = new CipherInputStream( istream, cipher );
metadata.setContentLength(sizeInBytes);
s3Client.putObject(<bucket>, destFile, cistream, metadata)
the variable are self explanatory
I have a request generated by putting map of parameters
val reqUrl = url("http://example.com")
val req = reqUrl << Map("key" -> "value")
I need to get request body in order to calculate it's hash. I'm trying this way
val data = req.toRequest.getStringData
println(data)
but it results null.
The request you currently defined is a GET request which normally has no body. So null is the expected body value.
You could try using a POST as described here : http://dispatch.databinder.net/HTTP+methods+and+parameters.html.
val reqUrl = url("http://example.com")
val postReq = reqUrl.POST
val req = postReq << Map("key" -> "value")
req.toRequest.getStringData
I don't know how to send multipart data in play scala Test case(Specs2).
In my project, method receives multipart data, code look like this
def school_register() = Action(parse.multipartFormData) { implicit request =>
}
In my test case using Specs2 how to form multipart data manually and want hit the school_register method. Once method hitted values are inserted into DB. I have four parameters regno=100,name="xxxx", address="xxx", std=5. Without multipart its working but i have to use multipart data.
Without Multipart i'm using like this
"com.example.schooladmin" should {
"responds with 200 for addSchoolRegister action with all required parameters" in new WithApplication {
val controller = new TestController()
val result = controller.school_registerr() .apply(FakeRequest().withFormUrlEncodedBody(
“Reg No” -> “100”,“Name” -> “XXX”,“Address” -> “XXXXX”,“std” -> “5”))
status(result) must equalTo(OK)
}
}
Here is how I'm doing this, I create a temporary file and I post it with withMultipartFormDataBody:
val tempFile = TemporaryFile(new java.io.File("../server/idCards/5e7b7c6c-98b3-4245-a5fb-405c9cc904f4"))
val part = FilePart[TemporaryFile](key = "picture", filename = "the.file", contentType = Some("image/jpeg"), ref = tempFile)
val formData = MultipartFormData(dataParts = Map(), files = Seq(part), badParts = Seq(), missingFileParts = Seq())
val Some(result) = route(FakeRequest(userDomain.routes.UserController.createIdCard())
.withMultipartFormDataBody(formData))
status(result) mustEqual OK
I am writing tests for an application created using Scala/Play Framework. There is a route with takes file to upload. This is what I have written so far.
val dataFile: File = new File("../TestCSV/product.csv")
val tempFile = TemporaryFile(dataFile)
val part = FilePart[TemporaryFile](key = "dataFile", filename = "product.csv", contentType = Some("application/vnd.ms-excel"), ref = tempFile)
val formData: MultipartFormData[TemporaryFile] = MultipartFormData[TemporaryFile](dataParts = Map(), files = Seq(part), badParts = Seq(), missingFileParts = Seq())
val request: FakeRequest[MultipartFormData[TemporaryFile]] = FakeRequest[MultipartFormData[TemporaryFile]]("POST", "/api/core/v0.1/data-import/uploads/%s/product".format(sandboxId), headers, formData)
val response = route(request).get
status(response) mustBe OK
I am getting this error.
Cannot write an instance of play.api.mvc.MultipartFormData[play.api.libs.Files.TemporaryFile] to HTTP response. Try to define a Writeable[play.api.mvc.MultipartFormData[play.api.libs.Files.TemporaryFile]]
How do I make this class writable?
I can't seem to set a cookie using Dispatch. The server resends a new session ID implying that the one I tried to send didn't get sent in the right way. Here is the code:
val domain = "myhost.com"
val host_req = host(domain).secure
val request = host_req / "path" / "path"
def post = request << Map("key" -> "SomeValue")
val response: Either[Throwable, Map[String, String]] =
Http(post OK asHeaders).either()
//The cookie comes down in the form "Set-Cookie: SESSIONID=<somesession>; Path=/path/; HttpOnly"
//successfully retrieves the session id...
val sessionId = getSessionId(response)
println(sessionId)
val sessionCookie = new com.ning.http.client.Cookie(domain, "SESSIONID", sessionId, "/path/", -1, true)
request.POST.addCookie(sessionCookie)
def establishPost = request << Map("key" -> "SomeValue")
establishPost.addCookie(sessionCookie)
val establishResponse: Either[Throwable, Map[String, String]] =
Http(establishPost OK asHeaders).either()
//Server sends down new SESSIONID...
//sessionId != getSEssionId(establishPost)
This is using the newest version of Dispatch. I'm trying to learn Scala as I go, and the only thing I can't figure out how to do either is inspect the establishPost object for its headers before it is sent as a request.
This should be better:
def reqWithParams = request << Map("key" -> "SomeValue")
val reqWithCookies = reqWithParams.addCookie(sessionCookie)
addCookie method returns the new object (the one with a cookie), but your code didn't use it.