The WTForms documentation gives a great example for implementing CSRF with Flask:
class MyBaseForm(Form):
class Meta:
csrf = True
csrf_class = SessionCSRF
csrf_secret = app.config['CSRF_SECRET_KEY']
#property
def csrf_context(self):
return session
I'd like to do the same thing but with a webapp2 session instead of a Flask session.
The csrf_context is a shortcut so that you don't have to pass the session every time that you create a form. Does anyone know how to create such a shortcut for a webapp2 session?
Without this shortcut, you need to do something like this every time you create a form:
form = MyForm(meta={'csrf_context': self.session})
which is some pretty awkward syntax that I'd prefer to avoid.
I've come up with a solution that reduces the awkward syntax described in my question. I've modified __init__ of my subclass of wt.Form like this:
class MyBaseForm(wt.Form):
class Meta:
csrf = True
csrf_class = SessionCSRF
csrf_secret = settings.SESSION_KEY
csrf_time_limit = timedelta(minutes=60)
def __init__(self, *args, **kwargs):
if "session" in kwargs:
super(MyBaseForm, self).__init__(
*args, meta={'csrf_context': kwargs["session"]}, **kwargs)
else:
super(MyBaseForm, self).__init__(*args, **kwargs)
Now, when I create a form, I can do this:
form = MyForm(session=self.session)
instead of the awkward syntax shown in the question.
For processing POSTed form data, I've figured out another technique to simplify processing.
First, I create a form with no fields other than the CSRF field:
class NoFieldForm(MyBaseForm):
pass
Second, I create a decorator that is used to check CSRF:
def check_csrf(func):
def wrapper(*args, **kwargs):
handler = args[0]
session = handler.session
request = handler.request
f = forms.NoFieldForm(request.POST, session=session)
f.validate()
if f.csrf_token.errors:
msg = "The CSRF token expired. Please try again. "
self.session["msg"] = msg
self.redirect(self.request.path)
else:
func(*args, **kwargs)
return wrapper
Third, I decorate all my POST handlers:
class SomeHandler(webapp2.RequestHandler):
#check_csrf
def post(self):
pass
Let me give some explanation. The decorator (via the call to the post webhandler) is going to receive some form data, but for purposes of CSRF checking we are going to throw away all form data except for CSRF. We can make it generic by using the NoFieldForm which ignores any other form data that is present. I use wtforms to check to make sure the CSRF form field and session token match and that the session token isn't expired.
If the CSRF passes, then we call the handler for normal processing to handle the specific form. If the CSRF fails, then we don't call the handler at all, and in my example, we redirect back to where we came from with an error message.
Related
[EDITED: I realized after reading response that I oversimplified my question.]
I am new to Locust and not sure how to solve this problem.
I have function (call it "get_doc") that is passed a locust.HttpSession() and uses it to issue an HTTP request. It gets the response and parses it, returning it up several layers of call. One of these higher-level calls looks at the returned, parsed document to decide if the response was what was expected or not. If not, I want Locust to mark the request/response as failed. A code sketch would be:
class MyUser (HttpUser):
#task
def mytask(self):
behavior1 (self.client)
def bahavior1(session):
doc = get_doc(session, url1)
if not doc_ok (doc):
??? how to register a failure with Locust here...
doc2 = get_doc(session, url2)
...
def get_doc(http_session, url):
page = http_session.get(url)
doc = parse (page)
return doc
There may be several behavior[n] functions and several Locust users calling them.
A constraint is that I would like to keep Locust-specific stuff out of bahavior1() so that I can call it with an ordinary Requests session. I have tried to do something like this in get_doc() (the catch_response parameter and success/fail stuff is actually conditionalized on 'session' being an HttpSession object):
def get_doc (session, meth, url):
resp = session.request (meth, url, catch_response=True)
doc = parse (resp.content)
doc.logfns = resp.success, resp.failure
return doc
and then in behavior1() or some higher up-chain caller I can
doc.logfns[1]("Document not as expected")
or
doc.logfns[0] # Looks good!
Unfortunately this is not working; the calls to them produce no errors but Locust doesn't seem to record any successes or failures either. I am not sure if it should work or I bungled something in my code. Is this feasible? Is there a better way?
You can make get_doc a context manager, call .get with catch_response=True and yield instead of return inside it. Similar to how it is done here: https://github.com/SvenskaSpel/locust-plugins/blob/2cbbdda9ae37b6cbb0a11cf69aca80b164198aec/locust_plugins/users/rest.py#L22
And then use it like this
def mytask(self):
with get_doc(self.client, url) as doc:
if not doc_ok(doc):
doc.failure(”doc was not ok :(”)
If you want, you can add the parsed doc as a field on the response before yielding in your doc function, or call doc.failure() inside doc_ok.
scrapys duplication filter ignores already seen urls/requests. So far, so good.
The Problem
Even if a request is dropped I still want to keep the redirection history.
Example:
Request 1 : B
Request 2 : A --301--> B
In this case request 2 is dropped without letting me know that it is a 'hidden' duplicate of request 1.
Attempts
I already tried to catch the signal request_dropped. This works but I don't see a possibility to sent an item to the pipeline from the handler.
Best regards and thanks for your help :)
Raphael
You are probably looking for DUPEFILTER_DEBUG
Set it to True in settings.py file and you will see all URLs that were ignored because of being duplicate
I figured out a way to handle those 'hidden' redirects:
Catch the signal 'request_dropped' from 'from_crawler':
#classmethod
def from_crawler(cls, crawler, *args, **kwargs):
spider = super(YourSpider, cls).from_crawler(crawler, *args, **kwargs)
crawler.signals.connect(spider.on_request_dropped, signal=signals.request_dropped)
return spider
Use 'self.crawler.engine.scraper.enqueue_scrape' to route the response to a callback which can yield items. enqueue_scrape expects a response so you can simply create a dummy response from the dropped request (I used TextResponse for this). With this response you can also define the callback.
def on_request_dropped(self, request, spider):
""" handle dropped request (duplicates) """
request.callback = self.parse_redirection_from_dropped_request
response = TextResponse(url=request.url, request=request)
self.crawler.engine.scraper.enqueue_scrape(request=request,
response=response, spider=self)
Process the redirection history of the dropped request within the callback you defined. From here you can handle things exactly like within the regular parse callback.
def parse_redirection_from_dropped_request(self, response):
...
yield item
I hope this might help you if you stumble upon the same problem.
I'm load testing a local API that will redirect a user based on a few conditions. Locust is not redirecting the simulated users hitting the end points and I know this because the app logs all redirects. If I manually hit the end points using curl, I can see the status is 302 and the Location header is set.
According to the embedded clients.HttpSession.request object, the allow_redirects option is set to True by default.
Any ideas?
We use redirection in our locust test, especially during the login phase. The redirects are handled for us without a hitch. Print the status_code of the response that you get back. Is it 200, 3xx or something worse?
Another suggestion: Don't throw your entire testing workflow into the locust file. That makes it too difficult to debug problems. Instead, create a standalone python script that uses the python requests library directly to simulate your workflow. Iron out any kinks, like redirection problems, in that simple, non-locust test script. Once you have that working, extract what you did into a file or class and have the locust task use the class.
Here is an example of what I mean. FooApplication does the real work. He is consumed by the locust file and a simple test script.
foo_app.py
class FooApplication():
def __init__(self, client):
self.client = client
self.is_logged_in = False
def login(self):
self.client.cookies.clear()
self.is_logged_in = False
name = '/login'
response = self.client.post('/login', {
'user': 'testuser',
'password': '12345'
}, allow_redirects=True, name=name)
if not response.ok:
self.log_failure('Login failed', name, response)
def load_foo(self):
name = '/foo'
response = self.client.get('/foo', name=name)
if not response.ok:
self.log_failure('Foo request failed ', name, response)
def log_failure(self, message, name, response):
pass # add some logging
foo_test_client.py
# Use this test file to iron out kinks in your request workflow
import requests
from locust.clients import HttpSession
from foo_app import FooApplication
client = HttpSession('http://dev.foo.com')
app = FooApplication(client)
app.login()
app.load_foo()
locustfile.py
from foo_app import FooApplication
class FooTaskSet(TaskSet):
def on_start(self):
self.foo = FooApplication(self.client)
#task(1)
def login(self):
if not self.foo.is_logged_in:
self.foo.login()
#task(5) # 5x more likely to load a foo vs logging in again
def load_foo(self):
if self.foo.is_logged_in:
self.load_foo()
else:
self.login()
Since Locust uses the Requests HTTP library for Python, you might find your answer there.
The Response object can be used to evaluate if a redirect has happened and what the history of redirects contains.
is_redirect:
True if this Response is a well-formed HTTP redirect that could have been
processed automatically (by Session.resolve_redirects).
There might be an indication that the redirect is not well-formed.
Or more precisely...
I already have an error page route defined like so:
GET /error controllers.pages.ErrorController.page(msg: String, returnTo: String)
And a controller method like this:
object ErrorController extends Controller {
def page(msg: String, returnTo: String) = ReceiverRestricted { implicit req =>
val action = List(Button(F8, "Continue", Call("GET", returnTo)))
Results.Ok(views.html.base(Html("Oops"), List(Html(msg)), None, action))
}
}
If I programmatically call, say, ErrorController.page("You did something daft!", "/home") I get to a page that looks like I want, ie:
Oops
You did something daft!
F8 Continue
However the url is ugly:
http://localhost:9000/error?msg=You%20did%20something%20daft!&returnTo=/home
I want to change this so the msg= query parameter doesn't appear in the url. How can I accomplish this? I tried removing the query parameter and redirecting to the error page with the message passed in via the flash cookie - that worked but reloading the browser page loses the message. I can't use the session cookie because I already store other data in the session almost upto its limit.
You can use flash feature.
Here is a sample:
In your controller you can redirect the user to error page with:
Redirect("/error").flashing(
"reason" -> "The item has been created"
)
And in Error action:
def error = Action { implicit request =>
Ok {
val reason = flash.get("reason").getOrElse("General Error")
//DO your stuff with reason variable
}
}
Obviously you can have as many as flash variables you want.
Since Play is restful and stateless, I can't see an easy way to pass on an error message during a redirect without using Play's flash. Of course you could store the message in an temporary cookie in the browser. Another possibility could be to store it in your database (or whatever persistence technology you use), but this seems to be like cracking a nut with a sledgehammer.
I have a web application in Play. The web application consists of several pages. In every page there is a small flag that enables the user to change the language (locale) from german to english and back.
I handle this with a redirect to referer:
def referer(implicit request: Request[AnyContent]) =
request.headers.get(REFERER).getOrElse(mainUrl)
def locale(l: String) = Authenticated { user =>
implicit request =>
Redirect(referer).withCookies(Cookie(LANG, if (l == "de" || l == "en") l else "de"))
}
It is working fine. Well, at least for GET requests.
I have a specific page where the user has to input data in a form. This form is then POSTed to the server. Were errors found, the form is displayed again with the error messages, as usual. Now, if the user wants to change the language (by clicking on the flag), the redirect to referer does not work, because it tries to use a GET request, and Play complains that a GET route does not exist for this method (which is true).
I am solving this by caching the form and defining another method where the form is taken from the cache:
# User data is POSTed to the server
POST /create/insert controllers.MyCreate.insert()
# After a redirect the cached form is displayed again
GET /create/insert controllers.MyCreate.insertGet()
It works, but I don't like this solution. It does not seem normal to have to create another entry in the routes and another method just to adress this problem. I would need to add this hack for every POST route in my application!
Is there a more elegant solution to this?
You could change it into something like this (untested):
def changeLang(lang:String, returnUri:String) = Action {
Redirect(returnUri)
.withCookies(Cookie(LANG, if (lang == "de" || lang == "en") lang else "de"))
}
In you template you would output the route to changeLang in the link, you can get the uri via the request
#routes.Application.changeLang("en", request.uri).url
I suggest you make request implicit in your action and define it as implicit in your template so you don't need to pass it on to each template.
// in the controller
def myUrl = Action { implicit request =>
Ok(views.html.myTemplate("something"))
}
// in the template
#(title:String)(implicit request:play.api.mvc.RequestHeader)
Edit
As for the POST requests, it common (for these types of framework) to have POST requests simple handle stuff and then redirect to another page. The usual flow is like this:
Form submits to a handler
Handler does something with the form information
Handler redirects to a page
An example:
// Hooked up to a GET route
def edit(id:Long) = Action {
// render the view with a form that displays the element with given id
// if the flash scope contains validation information, use that in display
}
// Hooked up to a POST route
def editHandler = Action {
// validate the form
// if validation succeeds
// persist the object
// redirect to edit
// else
// put the form information into the flash scope
// put any validation messages into the flash scope
// redirect to edit
}
If you do not want to use this flow you need to have both a GET and POST route anyway. The user might do a page reload on the resulting page.