I can't believe this isn't in the http4s documentation, and the example code I was able to dig up online (after poking around long enough to discover the UrlForm class) is not working for me.
The relevant bit of code looks like this:
case req # POST -> Root / "compose" =>
req.decode[UrlForm] { ps =>
println("ps.values: " + ps.values)
val content = ps.getFirstOrElse("content",
throw new IllegalStateException("No content given!"))
// Do something with `content`...
}
When submitting the associated form, the IllegalStateException is thrown. ps.values is an empty map (Map()).
I can see (using println) that the Content-Type is application/x-www-form-urlencoded, as expected, and I can see from my browser's Network tab that request "paramaters" (the encoded form values) are being sent properly.
The problem is that I had a filter (javax.servlet.Filter) in place that was calling getParameterMap on the HttpServletRequest. This was draining the InputStream for the request, and it was happening before the request got passed off to the servlet (BlockingHttp4sServlet) instance.
It seems to me the BlockingHttp4sServlet should raise an IllegalStateException (or something more descriptive) when it receives an InputStream with isFinished returning true. (I've filed an issue with the http4s project on Github.)
Related
I am using SOAP UI 5.7.0 to mock a REST service and it is working fine. Only, when I want to access the body of a POST request with mockRequest.requestContent, the correct content is returned only in the first call, but from then on, it always returns the empty string.
I tried the call in OnRequestScript and in the POST requests own Dispatch script but the behavior is the same in both cases. I have the suspicion, that the Stream of the request is already read somewhere else and so does not return any more content. But I don't know what to do about it.
I wonder what is the correct way to read the content of a POST request.
Thank you
Appears to be a known issue, see posts in the community forum here and here.
this seems to be an old bug of PUT operation in REST mocks.
Unfortunately, this is a bug. There is an internal defect of SOAP-2344 for this issue. This applies for the PUT and DELETE methods for a REST mock service.
I have the same issue with a PATCH request.
Use the following trick to get the body of the PUT and DELETE requests:
mockRequest.with {
if (method.toString() == 'PUT' || method.toString() == 'DELETE') {
InputStreamReader isr = new InputStreamReader(request.getInputStream(), "UTF-8")
BufferedReader br = new BufferedReader(isr)
StringBuilder sb = new StringBuilder()
while ((s=br.readLine())!=null) {
sb.append(s)
}
def requestBody = new groovy.json.JsonSlurper().parseText(sb.toString())
log.info "requestBody: " + requestBody
}
}
I use it on the project but I don't really remember how where I got the snippet from. I had to change some parts to make it work as far as I remember. Give it a try.
[EDITED: I realized after reading response that I oversimplified my question.]
I am new to Locust and not sure how to solve this problem.
I have function (call it "get_doc") that is passed a locust.HttpSession() and uses it to issue an HTTP request. It gets the response and parses it, returning it up several layers of call. One of these higher-level calls looks at the returned, parsed document to decide if the response was what was expected or not. If not, I want Locust to mark the request/response as failed. A code sketch would be:
class MyUser (HttpUser):
#task
def mytask(self):
behavior1 (self.client)
def bahavior1(session):
doc = get_doc(session, url1)
if not doc_ok (doc):
??? how to register a failure with Locust here...
doc2 = get_doc(session, url2)
...
def get_doc(http_session, url):
page = http_session.get(url)
doc = parse (page)
return doc
There may be several behavior[n] functions and several Locust users calling them.
A constraint is that I would like to keep Locust-specific stuff out of bahavior1() so that I can call it with an ordinary Requests session. I have tried to do something like this in get_doc() (the catch_response parameter and success/fail stuff is actually conditionalized on 'session' being an HttpSession object):
def get_doc (session, meth, url):
resp = session.request (meth, url, catch_response=True)
doc = parse (resp.content)
doc.logfns = resp.success, resp.failure
return doc
and then in behavior1() or some higher up-chain caller I can
doc.logfns[1]("Document not as expected")
or
doc.logfns[0] # Looks good!
Unfortunately this is not working; the calls to them produce no errors but Locust doesn't seem to record any successes or failures either. I am not sure if it should work or I bungled something in my code. Is this feasible? Is there a better way?
You can make get_doc a context manager, call .get with catch_response=True and yield instead of return inside it. Similar to how it is done here: https://github.com/SvenskaSpel/locust-plugins/blob/2cbbdda9ae37b6cbb0a11cf69aca80b164198aec/locust_plugins/users/rest.py#L22
And then use it like this
def mytask(self):
with get_doc(self.client, url) as doc:
if not doc_ok(doc):
doc.failure(”doc was not ok :(”)
If you want, you can add the parsed doc as a field on the response before yielding in your doc function, or call doc.failure() inside doc_ok.
scrapys duplication filter ignores already seen urls/requests. So far, so good.
The Problem
Even if a request is dropped I still want to keep the redirection history.
Example:
Request 1 : B
Request 2 : A --301--> B
In this case request 2 is dropped without letting me know that it is a 'hidden' duplicate of request 1.
Attempts
I already tried to catch the signal request_dropped. This works but I don't see a possibility to sent an item to the pipeline from the handler.
Best regards and thanks for your help :)
Raphael
You are probably looking for DUPEFILTER_DEBUG
Set it to True in settings.py file and you will see all URLs that were ignored because of being duplicate
I figured out a way to handle those 'hidden' redirects:
Catch the signal 'request_dropped' from 'from_crawler':
#classmethod
def from_crawler(cls, crawler, *args, **kwargs):
spider = super(YourSpider, cls).from_crawler(crawler, *args, **kwargs)
crawler.signals.connect(spider.on_request_dropped, signal=signals.request_dropped)
return spider
Use 'self.crawler.engine.scraper.enqueue_scrape' to route the response to a callback which can yield items. enqueue_scrape expects a response so you can simply create a dummy response from the dropped request (I used TextResponse for this). With this response you can also define the callback.
def on_request_dropped(self, request, spider):
""" handle dropped request (duplicates) """
request.callback = self.parse_redirection_from_dropped_request
response = TextResponse(url=request.url, request=request)
self.crawler.engine.scraper.enqueue_scrape(request=request,
response=response, spider=self)
Process the redirection history of the dropped request within the callback you defined. From here you can handle things exactly like within the regular parse callback.
def parse_redirection_from_dropped_request(self, response):
...
yield item
I hope this might help you if you stumble upon the same problem.
I'm trying to set up some tests for an API made by a coworker with spray.io, and I'm encountering some odd behavior. When a request results in an error for any reason, we want to return a JSON value along the lines of:
{"status":false,"message":"useful message here"}
This happens just fine in the actual browser. I have navigated to an unhandled route in the web browser, and I get the desired JSON value. So, I want to test this. Now, since I'm new to spray.io, I started off with the very simple test:
"leave GET requests to root path unhandled" in {
Get() ~> myRoute ~> check {
handled must beFalse
}
}
This went fine, no problems. Since it's my first time playing with spray.io, I looked at some of the sample tests for testing false routes, and wrapped myRoute with sealRoute() so I could check the response without failing tests:
"leave GET requests to root path unhandled" in {
Get() ~> sealRoute(myRoute) ~> check {
handled must beTrue
}
}
This also works fine. So, I decided to just make sure the text of the response was usable with this, before I went to the trouble of parsing JSON and verifying individual values:
"leave GET requests to root path unhandled" in {
Get() ~> sealRoute(myRoute) ~> check {
responseAs[String] contains "false"
}
}
This is failing. To investigate, I threw a simple line of code in to log the actual value of responseAs[String] to a file, and I got this:
The requested resource could not be found.
Can anyone tell me what I'm doing wrong? I'm thinking that one of the following is occurring:
responseAs[String] is doing more than taking the exact response and giving it back to me, applying some type of filter along the way
The framework itself is not fully evaluating the query, but rather making a mockup object for the test framework to evaluate, and therefore not executing the desired 'turn errors to json' methods that my co-worker has implemented
I have tried searching google and stack overflow specifically for similar issues, but I'm either not putting in the right queries, or most other people are content to have the default error messages and aren't trying to test them beyond checking handled must beFalse.
Edit - This is the relevant part of the RejectionHandler:
case MissingQueryParamRejection(paramName) :: _=>
respondWithMediaType(`application/json`) {
complete(BadRequest, toJson(Map("status" -> false, "message" -> s"Missing parameter $paramName, request denied")))
}
Okay, so with insight from here and a coworker, the problem has been found:
Basically, the custom RejectionHandler was defined within our custom Actor object, and it wasn't coming into scope in the tests. To resolve this, the following steps were taken:
Moved the definition for the custom RejectionHandler into its own object in a separate file (as it had to do its own imports, it was causing a "encountered unrecoverable cycle resolving import" error)
Imported this new object into both the original file and the test spec.
(fun fact - http://spray.io/documentation/1.2.2/spray-routing/key-concepts/rejections/#rejectionhandler seems to demonstrate the RejectionHandler as a top-level object but you can't have top-level implicit vals in Scala, hence the need for an object)
I'm currently trying to connect an Extjs form to a Play! Scala backend.
I have no problem receiving the request, but
FormName.bindFromRequest.get returns to me None .
I'm able to find the data generated via submitting from Extjs in the POST request in request.request.body.data (the first request object is generated by SocialSecure's controller), whereas normally the data bound to the form would be found in request.request.body.data.elems
I think Extjs's eschewal of using <form> in their inserted HTML is what causes me this problem, but I'd still like to take advantage of Extjs's nice form verification UI.
Does Play! or Scala have any resources for modifying a request after the server has received it?
More info
This is the method my /requestAudit cuurently points to after a POST request:
def requestAudit = SecuredAction(WithProvider("google")) { // SecureSocial syntax
implicit request => { // let's call this line 0'
println(request.request.body.asFormUrlEncoded) // let's call this line 1'
println(request.body.asText) // let's call this line 2'
newAuditForm.bindFromRequest.fold(
errors => BadRequest(views.html.error(newAuditForm))
success => { /*insert the object into my db*/ }
) } }
Ext.js request
When I'm debugging in Eclipse with an Ext.js form, the Variables window shows: (click for closeup)
where the form values are located in request.body.data.key1, request.body.data.key2, etc
Bootstrap form request
On the other hand, the request for Bootstrap has the values stored in request.body.data.elems
#2manyprojects 's suggestion set me on the right path:
newAuditForm.bindFromRequest(
(request.request.body.asFormUrlEncoded).getOrElse(Map()))
.fold( ... )
worked.
I was still getting form binding errors after changing my code to this, and then I discovered a typo in the name property of one of my Ext.js form fields. The name of the field must be the same on both the UI and the Play Form.