File upload, declaration via apiOperation (Swagger and Scalatra 2.6) - scala

There is an existing project that uses Scalatra (2.6) and Swagger:
scalaMajorVersion = '2.12'
scalaVersion = "${scalaMajorVersion}.8"
scalatraVersion = "${scalaMajorVersion}:2.6.4"
compile "org.scalatra:scalatra-swagger_${scalatraVersion}"
I easily could add a new end point like:
get ("/upload", op[String]( // op finally invokes apiOperation
name = "Test method",
params = List(
query[Long]("id" -> "ID"),
query[String]("loginName" -> "login name")
),
authorizations = List(Permission.xxxxx.name)
)) {
...
}
but I cannot upload a file.
I expect to see a file selector button, but instead I see a single-line edit field.
(There are numerous things I'm uncertain about: form or file, [String] or [FileItem], which trait(s), what kind of initialization, etc.)
In the existing code I found a comment that someone could not get swagger to handle file upload. At the same time, I read that Scalatra and Swagger can do that, not all versions of them, but it looks like the version used in the project should be able to do that.
I could find code examples with yml/json interface definitions, but in the project there is no yml, only the apiOperation-based stuff.
Is there a working example using Scalatra 2.6, Swagger, and apiOperation?

I managed to get the file chooser (file selector, "Browse") button; there was no predefined constant (like DataType.String) for that. After I used DataType("File"), everything else just worked.
https://docs.swagger.io/spec.html says:
4.3.5 File
The File (case sensitive) is a special type used to denote file
upload. Note that declaring a model with the name File may lead to
various conflicts with third party tools and SHOULD be avoided.
When using File, the consumes field MUST be "multipart/form-data", and
the paramType MUST be "form".
post ("/uploadfile", op[String]( // op finally invokes apiOperation
name = "Upload File",
params = List(
new Parameter(
`name` = "kindaName",
`description` = Some("kindaDescription2"),
`type` = DataType("File"), // <===== !!!!!!
`notes` = Some("kindaNotes"),
`paramType` = ParamType.Form, // <===== !!
`defaultValue` = None,
`allowableValues` = AllowableValues.AnyValue,
`required` = true
)
),
consumes = List("multipart/form-data"), // <===== !!
...
)) {
val file: FileItem = fileParams("kindaName") // exception if missing
println("===========")
println("file: " + file)
println("name: " + file.getName + " size:"+file.getSize+" fieldName:"+file.getFieldName+ " ContentType:"+file.getContentType+" Charset:"+file.getCharset)
println("====vvv====")
io.copy(file.getInputStream, System.out)
println("====^^^====")
val file2: Option[FileItem] = fileParams.get("file123") // None if missing, and it is missing
println("file2: " + file2)
PS the apiOperation stuff is called "annotations".

Related

Downloading csv file using Play Framework?

I want to add to my app a simple button that on click will call an Action that will create a csv file from two lists I have and download it to the user computer.
This is my Action:
def createAndDownloadFile = Action {
val file = new File("newFile.csv")
val writer = CSVWriter.open(file)
writer.writeAll(List(listOfHeaders, listOfValues))
writer.close()
Ok.sendFile(file, inline = false, _ => file.getName)
}
but this is now working for me, the file is not getting downloaded from the browser...
im expecting to see the file get downloaded by the browser, i thought Ok.sendFile should do the trick..
thanks!
You can use Enumerators and streams for that. It should work like this:
val enum = Enumerator.fromFile(...)
val source = akka.stream.scaladsl.Source.fromPublisher(play.api.libs.streams.Streams.enumeratorToPublisher(enum))
Result(
header = ResponseHeader(OK, Map(CONTENT_DISPOSITION → "attachment; filename=whatever.csv.gz")),
body = HttpEntity.Streamed(source.via(Compression.gzip), None, None)
)
This will actually pipe the download through gzip. Just remove the .via(Compression.gzip) part if that is not needed.

Compute file content hash with Scala

In our app, we are in need to compute file hash, so we can compare if the file was updated later.
The way I am doing it right now is with this little method:
protected[services] def computeMigrationHash(toVersion: Int): String = {
val migrationClassName = MigrationClassNameFormat.format(toVersion, toVersion)
val migrationClass = Class.forName(migrationClassName)
val fileName = migrationClass.getName.replace('.', '/') + ".class"
val resource = getClass.getClassLoader.getResource(fileName)
logger.debug("Migration file - " + resource.getFile)
val file = new File(resource.getFile)
val hc = Files.hash(file, Hashing.md5())
logger.debug("Calculated migration file hash - " + hc.toString)
hc.toString
}
It all works perfectly, until the code get's deployed into different environment and file file is located in a different absolute path. I guess, the hashing take the path into account as well.
What is the best way to calculate some sort of reliable hash of a file content that well produce the same result for as log as the content of a file stays the same?
Thanks,
Having perused the source code https://github.com/google/guava/blob/master/guava/src/com/google/common/io/Files.java - only the file contents are hashed - the path does not come into play.
public static HashCode hash(File file, HashFunction hashFunction) throws IOException {
return asByteSource(file).hash(hashFunction);
}
Therefore you need not worry about locality of the file. Now why you end up with a different hash on a different fs .. maybe you should compare the size/contents to ensure eg no compound eol's were introduced.

cgi.parse_multipart function throws TypeError in Python 3

I'm trying to make an exercise from Udacity's Full Stack Foundations course. I have the do_POST method inside my subclass from BaseHTTPRequestHandler, basically I want to get a post value named message submitted with a multipart form, this is the code for the method:
def do_POST(self):
try:
if self.path.endswith("/Hello"):
self.send_response(200)
self.send_header('Content-type', 'text/html')
self.end_headers
ctype, pdict = cgi.parse_header(self.headers['content-type'])
if ctype == 'multipart/form-data':
fields = cgi.parse_multipart(self.rfile, pdict)
messagecontent = fields.get('message')
output = ""
output += "<html><body>"
output += "<h2>Ok, how about this?</h2>"
output += "<h1>{}</h1>".format(messagecontent)
output += "<form method='POST' enctype='multipart/form-data' action='/Hello'>"
output += "<h2>What would you like to say?</h2>"
output += "<input name='message' type='text'/><br/><input type='submit' value='Submit'/>"
output += "</form></body></html>"
self.wfile.write(output.encode('utf-8'))
print(output)
return
except:
self.send_error(404, "{}".format(sys.exc_info()[0]))
print(sys.exc_info() )
The problem is that the cgi.parse_multipart(self.rfile, pdict) is throwing an exception: TypeError: can't concat bytes to str, the implementation was provided in the videos for the course, but they're using Python 2.7 and I'm using python 3, I've looked for a solution all afternoon but I could not find anything useful, what would be the correct way to read data passed from a multipart form in python 3?
I've came across here to solve the same problem like you have.
I found a silly solution for that.
I just convert 'boundary' item in the dictionary from string to bytes with an encoding option.
ctype, pdict = cgi.parse_header(self.headers['content-type'])
pdict['boundary'] = bytes(pdict['boundary'], "utf-8")
if ctype == 'multipart/form-data':
fields = cgi.parse_multipart(self.rfile, pdict)
In my case, It seems work properly.
To change the tutor's code to work for Python 3 there are three error messages you'll have to combat:
If you get these error messages
c_type, p_dict = cgi.parse_header(self.headers.getheader('Content-Type'))
AttributeError: 'HTTPMessage' object has no attribute 'getheader'
or
boundary = pdict['boundary'].decode('ascii')
AttributeError: 'str' object has no attribute 'decode'
or
headers['Content-Length'] = pdict['CONTENT-LENGTH']
KeyError: 'CONTENT-LENGTH'
when running
c_type, p_dict = cgi.parse_header(self.headers.getheader('Content-Type'))
if c_type == 'multipart/form-data':
fields = cgi.parse_multipart(self.rfile, p_dict)
message_content = fields.get('message')
this applies to you.
Solution
First of all change the first line to accommodate Python 3:
- c_type, p_dict = cgi.parse_header(self.headers.getheader('Content-Type'))
+ c_type, p_dict = cgi.parse_header(self.headers.get('Content-Type'))
Secondly, to fix the error of 'str' object not having any attribute 'decode', it's because of the change of strings being turned into unicode strings as of Python 3, instead of being equivalent to byte strings as in Python 3, so add this line just under the above one:
p_dict['boundary'] = bytes(p_dict['boundary'], "utf-8")
Thirdly, to fix the error of not having 'CONTENT-LENGTH' in pdict just add these lines before the if statement:
content_len = int(self.headers.get('Content-length'))
p_dict['CONTENT-LENGTH'] = content_len
Full solution on my Github:
https://github.com/rSkogeby/web-server
I am doing the same course and was running into the same problem. Instead of getting it to work with cgi I am now using the parse library. This was shown in the same course just a few lessons earlier.
from urllib.parse import parse_qs
length = int(self.headers.get('Content-length', 0))
body = self.rfile.read(length).decode()
params = parse_qs(body)
messagecontent = params["message"][0]
And you have to get rid of the enctype='multipart/form-data' in your form.
In my case I used cgi.FieldStorage to extract file and name instead of cgi.parse_multipart
form = cgi.FieldStorage(
fp=self.rfile,
headers=self.headers,
environ={'REQUEST_METHOD':'POST',
'CONTENT_TYPE':self.headers['Content-Type'],
})
print('File', form['file'].file.read())
print('Name', form['name'].value)
Another hack solution is to edit the source of the cgi module.
At the very beginning of the parse_multipart (around the 226th line):
Change the usage of the boundary to str(boundary)
...
boundary = b""
if 'boundary' in pdict:
boundary = pdict['boundary']
if not valid_boundary(boundary):
raise ValueError('Invalid boundary in multipart form: %r'
% (boundary,))
nextpart = b"--" + str(boundary)
lastpart = b"--" + str(boundary) + b"--"
...

How can I get the project path in Scala?

I'm trying to read some files from my Scala project, and if I use: java.io.File(".").getCanonicalPath() I find that my current directory is far away from them (exactly where I have installed Scala Eclipse). So how can I change the current directory to the root of my project, or get the path to my project? I really don't want to have an absolute path to my input files.
val PATH = raw"E:\lang\scala\progfun\src\examples\"
def printFileContents(filename: String) {
try {
println("\n" + PATH + filename)
io.Source.fromFile(PATH + filename).getLines.foreach(println)
} catch {
case _:Throwable => println("filename " + filename + " not found")
}
}
val filenames = List("random.txt", "a.txt", "b.txt", "c.txt")
filenames foreach printFileContents
Add your files to src/main/resources/<packageName> where <packageName> is your class package.
Change the line val PATH = getClass.getResource("").getPath
new File(".").getCanonicalPath
will give you the base-path you need
Another workaround is to put the path you need in an user environmental variable, and call it with sys.env (returns exception if failure) or System.getenv (returns null if failure), for example val PATH = sys.env("ScalaProjectPath") but the problem is that if you move the project you have to update the variable, which I didn't want.

Selenium IDE - always fail on any 500 error

Is there an easy way to tell Selenium IDE that any action that results in a http 500 response means the test failed?
I have tests that are 75 page requests long. Sometimes, I get a crash and burn somewhere in the middle, but the tests come back green.
Taking a look at selenium-api.js, I saw that there is a parameter ignoreResponseCode in the signature of the doOpen method in selenium-api.js :
Selenium.prototype.doOpen = function(url, ignoreResponseCode) {
This parameter is used by the browserbot object :
if (!((self.xhrResponseCode >= 200 && self.xhrResponseCode <= 399) || self.xhrResponseCode == 0)) {
// TODO: for IE status like: 12002, 12007, ... provide corresponding statusText messages also.
LOG.error("XHR failed with message " + self.xhrStatusText);
e = "XHR ERROR: URL = " + self.xhrOpenLocation + " Response_Code = " + self.xhrResponseCode + " Error_Message = " + self.xhrStatusText;
self.abortXhr = false;
self.isXhrSent = false;
self.isXhrDone = false;
self.xhrResponseCode = null;
self.xhrStatusText = null;
throw new SeleniumError(e);
}
I've tried calling the open function from selenium IDE with value = false and this results in an error (test failed).
My PHP test page was :
<?php
header('HTTP/1.1 500 Simulated 500 error');
?>
And this results in :
For me, this solves the problem of checking HTTP response status.
Make a JavaScript file called "user-extensions.js" and add it to the Selenium-IDE under Options > Options. If you are running Selenium RC, pass it into the parameter when starting up your server in the jar command. There should be a user extensions javascript file attribute.
Then close and restart Selenium-IDE. The User-Extensions file is cached when the IDE starts up.
Add this code to your Selenium user-extensions.js file to make a custom command called "AssertLocationPart". As you know "assertLocation" and "storeLocation" are standard commands. I tried to reduce the extra line of code to storeLocation just by getting the href in the custom function. I wasn't able to get the doAssertValue command to work. I'll have to post my own question for that. That's why it's commented out. For now, just use "this.doStore" instead. And add an extra line to your script after your custom AssertLocationPart command. Since we're not actually doing an assertion in the custom function/command, we should call it "storeLocationPart" (function would be named "doStoreLocationPart"), not "assertLocationPart" (function would be named "doAssertLocationPart"), and just pass in the first parameter. But if you can get the doAssert* to work, please let me know. I'll mess with it another day since I need to do this same thing for work.
Selenium.prototype.doAssertLocationPart = function(partName,assertTo) {
var uri = selenium.browserbot.getCurrentWindow().document.location.href;
//alert("URI = " + uri);
var partValue = parseUri(uri,partName);
//alert("Part '" + partName + "' = " + partValue);
//this.doAssertValue(partValue,assertTo);
this.doStore(partValue,"var_"+partName);
};
// Slightly modified function based on author's original:
// http://badassery.blogspot.com/2007/02/parseuri-split-urls-in-javascript.html
//
// parseUri JS v0.1, by Steven Levithan (http://badassery.blogspot.com)
// Splits any well-formed URI into the following parts (all are optional):
//
// - source (since the exec() method returns backreference 0 [i.e., the entire match] as key 0, we might as well use it)
// - protocol (scheme)
// - authority (includes both the domain and port)
// - domain (part of the authority; can be an IP address)
// - port (part of the authority)
// - path (includes both the directory path and filename)
// - directoryPath (part of the path; supports directories with periods, and without a trailing backslash)
// - fileName (part of the path)
// - query (does not include the leading question mark)
// - anchor (fragment)
//
function parseUri(sourceUri,partName){
var uriPartNames = ["source","protocol","authority","domain","port","path","directoryPath","fileName","query","anchor"];
var uriParts = new RegExp("^(?:([^:/?#.]+):)?(?://)?(([^:/?#]*)(?::(\\d*))?)?((/(?:[^?#](?![^?#/]*\\.[^?#/.]+(?:[\\?#]|$)))*/?)?([^?#/]*))?(?:\\?([^#]*))?(?:#(.*))?").exec(sourceUri);
var uri = {};
for(var i = 0; i < 10; i++){
uri[uriPartNames[i]] = (uriParts[i] ? uriParts[i] : "");
if (uriPartNames[i] == partName) {
return uri[uriPartNames[i]]; // line added by MacGyver
}
}
// Always end directoryPath with a trailing backslash if a path was present in the source URI
// Note that a trailing backslash is NOT automatically inserted within or appended to the "path" key
if(uri.directoryPath.length > 0){
uri.directoryPath = uri.directoryPath.replace(/\/?$/, "/");
if (partName == "directoryPath") {
return uri.directoryPath; // line added by MacGyver
}
}
return uri;
}
Then add this to your web.config file and make sure customErrors is turned off. Since you have a 500 error, it will redirect the user to the default page. Feel free to add a custom page for a 500 HTTP status code if you want to be specific in your Selenium scripts.
<customErrors mode="On" defaultRedirect="/ErrorHandler.aspx">
<error statusCode="401" redirect="/AccessDenied.aspx" />
<error statusCode="403" redirect="/AccessDenied.aspx" />
<error statusCode="404" redirect="/PageNotFound.aspx" />
</customErrors>
This is what your commands will look like in the IDE:
Make sure you're on this page (or something similar) before running the script:
https://localhost/ErrorHandler.aspx?aspxerrorpath=/path/pathyouweretryingtoviewinwebapp.aspx
Log shows that it passed!
[info] Executing: |storeLocation | var_URI | |
[info] Executing: |echo | ${var_URI} | |
[info] echo: https://localhost/ErrorHandler.aspx?aspxerrorpath=//path/pathyouweretryingtoviewinwebapp.aspx
[info] Executing: |assertLocationPart | fileName | ErrorHandler.aspx |
[info] Executing: |assertExpression | ${var_fileName} | ErrorHandler.aspx |
Using the error handler from my previous answer:
Command: assertLocation
Target: regexp:^(https://localhost/ErrorHandler.aspx).*$
Or (per your comment) it's inverse if you don't have error handling turned on, use AssertNotLocation. This may require more work on the person writing the scripts. You'd have to keep track of all pages.
More on pattern matching:
http://seleniumhq.org/docs/02_selenium_ide.html#matching-text-patterns
http://www.codediesel.com/testing/selenium-ide-pattern-matching/