I am writing tests for an application created using Scala/Play Framework. There is a route with takes file to upload. This is what I have written so far.
val dataFile: File = new File("../TestCSV/product.csv")
val tempFile = TemporaryFile(dataFile)
val part = FilePart[TemporaryFile](key = "dataFile", filename = "product.csv", contentType = Some("application/vnd.ms-excel"), ref = tempFile)
val formData: MultipartFormData[TemporaryFile] = MultipartFormData[TemporaryFile](dataParts = Map(), files = Seq(part), badParts = Seq(), missingFileParts = Seq())
val request: FakeRequest[MultipartFormData[TemporaryFile]] = FakeRequest[MultipartFormData[TemporaryFile]]("POST", "/api/core/v0.1/data-import/uploads/%s/product".format(sandboxId), headers, formData)
val response = route(request).get
status(response) mustBe OK
I am getting this error.
Cannot write an instance of play.api.mvc.MultipartFormData[play.api.libs.Files.TemporaryFile] to HTTP response. Try to define a Writeable[play.api.mvc.MultipartFormData[play.api.libs.Files.TemporaryFile]]
How do I make this class writable?
Related
So I had a url link for a file, and the file is encrypted using "AES/CBC/PKCS5Padding", I have the cipher key and the iv string. I am using amazon ec2 to download the data and store to s3. what I am doing currently is creating an input stream using HTTPCLIENT() and GETMETHOD() and then javax.crypto library on the input stream and then finally putting this inputstream in a s3 location. Thia works for one file; but I need to scale up and do the same thing for multiple such url. How or can I take help of parallel download?
the download time for say n files is same if I don't parallelize the array of url, if I use 1 node or 4 nodes(1 master and 3 slaves). And if I use .par, it gives out of memory error. Here is the code for the data download and upload(the function downloadAndProcessDataFunc)
val client = new HttpClient()
val s3Client = AmazonS3ClientBuilder.standard().withRegion(Regions.EU_WEST_1).build()
val method = new GetMethod(url)
val metadata = new ObjectMetadata()
val responseCode = client.executeMethod(method);
println("FROM SCRIPT -> ", destFile, responseCode)
val istream = method.getResponseBodyAsStream();
val base64 = new Base64();
val key = base64.decode(cipherKey.getBytes());
val ivKey = base64.decode(iv.getBytes());
val secretKey = new SecretKeySpec( key, "AES" );
val ivSpec = new IvParameterSpec(ivKey);
val cipher = Cipher.getInstance( algorithm )
cipher.init( Cipher.DECRYPT_MODE, secretKey, ivSpec );
val cistream = new CipherInputStream( istream, cipher );
s3Client.putObject(<bucket>, destFile, cistream, metadata)
this is the call I do, but isn't parallelized
manifestArray.foreach{row => downloadAndProcessDataFunc(row.split(",")(0), row.split(",")(1), row.split(",")(2), row.split(",")(3), row.split(",")(4), row.split(",")(5).toLong)
this runs out of memory
manifest.par.foreach{row => downloadAndProcessDataFunc(row.mkString(",").split(",")(0), row.mkString(",").split(",")(1), row.mkString(",").split(",")(2), row.mkString(",").split(",")(3), element.replace("s3://testbucketforemrupload/manifest/", "data/") + "/" + row.mkString(",").split(",")(4).split("/").last)
now if i change the downloadAndProcessDataFunc function to this, it just downloads around 30/1536 such url and kills the rest of tje executors for out of memory error
val client = new HttpClient()
val s3Client = AmazonS3ClientBuilder.standard().withRegion(Regions.EU_WEST_1).build()
val method = new GetMethod(url)
val metadata = new ObjectMetadata()
val responseCode = client.executeMethod(method);
println("FROM SCRIPT -> ", destFile, responseCode)
val istream = method.getResponseBodyAsStream();
val base64 = new Base64();
val key = base64.decode(cipherKey.getBytes());
val ivKey = base64.decode(iv.getBytes());
val secretKey = new SecretKeySpec( key, "AES" );
val ivSpec = new IvParameterSpec(ivKey);
val cipher = Cipher.getInstance( algorithm )
cipher.init( Cipher.DECRYPT_MODE, secretKey, ivSpec );
val cistream = new CipherInputStream( istream, cipher );
metadata.setContentLength(sizeInBytes);
s3Client.putObject(<bucket>, destFile, cistream, metadata)
the variable are self explanatory
I am trying to write a file to Amazon S3.
val creds = new BasicAWSCredentials(AWS_ACCESS_KEY, AWS_SECRET_KEY)
val amazonS3Client = new AmazonS3Client(creds)
val filePath = "/service2/2019/06/30/21"
val fileContent = "{\"key\":\"value\"}"
val meta = new ObjectMetadata();
amazonS3Client.putObject(bucketName, filePath, new ByteArrayInputStream(fileContent.getBytes), meta)
The program is finished with no error, but no file is written into the bucket.
The key argument seems to have a typo. Try without the initial forward slash
val filePath = "service2/2019/06/30/21"
instead of
val filePath = "/service2/2019/06/30/21"
I'm trying to unit test a file upload through multipartFormData and returning status 200 if it returns back the uploaded url. However, I keep getting status 400 and unsure why its been unsucessful.
I've found online of the call needs to be writeable and have included a writeable class which i've imported as lazy val
"return a 200 if successful" in new SetUp {
val fakeFileName = "testImage.png"
val fakeUploadImageUrl = "https://s3.fakeImageUrl.png"
val imageFile = new File(fakeFileName)
val tempFile = TemporaryFile(imageFile)
val multipartFormData = new MultipartFormData(
dataParts = Map(),
files = Seq(FilePart[TemporaryFile]("file", fakeFileName, Some("text/plain"), tempFile)),
badParts = Seq()
)
when(imageService.uploadWebImage(any(), any())).thenReturn(Future.successful(fakeUploadImageUrl))
val request = FakeRequest(HttpVerbs.POST, s"/internal/challengesv2/v1/instances/image/$fakeFileName")
.withBody(multipartFormData)
val response = testController.uploadImage(fakeFileName).apply(request)
assert(status(response.run()) === 200)
assert(contentAsJson(response.run()) === Json.toJson(fakeUploadImageUrl))
}
expect 200 but returning 400
I'm trying to connect to an IAM controlled ElasticSearch domain, I've created a request, and signed it, and everything works fine for method GET, but on method POST I get a 400 Bad Request
This clearly has something to do with the payload. If I provide a payload empty string ("") it works apporpriately, but anything else results in a bad request.
What am I missing?
val url = s"https://$host/TEST/article/_search"
val serviceName = "es"
val regionName = "us-east-1"
val request = new DefaultRequest(serviceName)
val payload =
"""{"1":"1"}""".trim
val payloadBytes = payload.getBytes(StandardCharsets.UTF_8)
val payloadStream = new ByteArrayInputStream(payloadBytes)
request.setContent(payloadStream)
val endpointUri = URI.create(url)
request.setEndpoint(endpointUri)
request.setHttpMethod(HttpMethodName.POST)
val credProvider = new EnvironmentVariableCredentialsProvider
val credentials = credProvider.getCredentials
val signer = new AWS4Signer
signer.setRegionName(regionName)
signer.setServiceName(serviceName)
signer.sign(request, credentials)
val context = new ExecutionContext(true)
val clientConfiguration = new ClientConfiguration()
val client = new AmazonHttpClient(clientConfiguration)
val rh = new MyHttpResponseHandler
val eh = new MyErrorHandler
val response =
client.execute(request, rh , eh, context);
note: if you run into this problem, inspect the actual content of the response, it may be a result of a mismatch between the index and your query.
My problem was that the specific query I was using was inappropriate for the specified index, and that resulted in a 400
I don't know how to send multipart data in play scala Test case(Specs2).
In my project, method receives multipart data, code look like this
def school_register() = Action(parse.multipartFormData) { implicit request =>
}
In my test case using Specs2 how to form multipart data manually and want hit the school_register method. Once method hitted values are inserted into DB. I have four parameters regno=100,name="xxxx", address="xxx", std=5. Without multipart its working but i have to use multipart data.
Without Multipart i'm using like this
"com.example.schooladmin" should {
"responds with 200 for addSchoolRegister action with all required parameters" in new WithApplication {
val controller = new TestController()
val result = controller.school_registerr() .apply(FakeRequest().withFormUrlEncodedBody(
“Reg No” -> “100”,“Name” -> “XXX”,“Address” -> “XXXXX”,“std” -> “5”))
status(result) must equalTo(OK)
}
}
Here is how I'm doing this, I create a temporary file and I post it with withMultipartFormDataBody:
val tempFile = TemporaryFile(new java.io.File("../server/idCards/5e7b7c6c-98b3-4245-a5fb-405c9cc904f4"))
val part = FilePart[TemporaryFile](key = "picture", filename = "the.file", contentType = Some("image/jpeg"), ref = tempFile)
val formData = MultipartFormData(dataParts = Map(), files = Seq(part), badParts = Seq(), missingFileParts = Seq())
val Some(result) = route(FakeRequest(userDomain.routes.UserController.createIdCard())
.withMultipartFormDataBody(formData))
status(result) mustEqual OK