I use ajax calls against a thin tastypie layer to CRUD (using csrf tokens). Everything works like a charm until I run the site in e.g. Chrome incognito mode. I keep getting 401s on CUD requests.
Looking at the request cookies I find that the sessionid cookie is set but the csrftoken cookie is not (its properly set if I run in normal mode).
In my settings.py I have :
MIDDLEWARE_CLASSES = (
'django.middleware.common.CommonMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
)
Anyone ran into that issue and can save me some time here?
Thanks a lot,
Juergen
I found the reason for the cookie not being set in Django's middleware file csrf.py. The code below if kicked in when in incognito mode preventing the cookie to be set:
if not request.META.get("CSRF_COOKIE_USED", False):
return response
My workaround is to set this value for my ModelResources in tastypie's api.py file manually:
class MyModelResource( ModelResource ) :
[..]
def wrap_view(self, view):
def wrapper(request, *args, **kwargs):
request.META["CSRF_COOKIE_USED"] = True
wrapped_view = super(MyModelResource, self).wrap_view(view)
return wrapped_view(request, *args, **kwargs)
return wrapper
Related
I'm load testing a local API that will redirect a user based on a few conditions. Locust is not redirecting the simulated users hitting the end points and I know this because the app logs all redirects. If I manually hit the end points using curl, I can see the status is 302 and the Location header is set.
According to the embedded clients.HttpSession.request object, the allow_redirects option is set to True by default.
Any ideas?
We use redirection in our locust test, especially during the login phase. The redirects are handled for us without a hitch. Print the status_code of the response that you get back. Is it 200, 3xx or something worse?
Another suggestion: Don't throw your entire testing workflow into the locust file. That makes it too difficult to debug problems. Instead, create a standalone python script that uses the python requests library directly to simulate your workflow. Iron out any kinks, like redirection problems, in that simple, non-locust test script. Once you have that working, extract what you did into a file or class and have the locust task use the class.
Here is an example of what I mean. FooApplication does the real work. He is consumed by the locust file and a simple test script.
foo_app.py
class FooApplication():
def __init__(self, client):
self.client = client
self.is_logged_in = False
def login(self):
self.client.cookies.clear()
self.is_logged_in = False
name = '/login'
response = self.client.post('/login', {
'user': 'testuser',
'password': '12345'
}, allow_redirects=True, name=name)
if not response.ok:
self.log_failure('Login failed', name, response)
def load_foo(self):
name = '/foo'
response = self.client.get('/foo', name=name)
if not response.ok:
self.log_failure('Foo request failed ', name, response)
def log_failure(self, message, name, response):
pass # add some logging
foo_test_client.py
# Use this test file to iron out kinks in your request workflow
import requests
from locust.clients import HttpSession
from foo_app import FooApplication
client = HttpSession('http://dev.foo.com')
app = FooApplication(client)
app.login()
app.load_foo()
locustfile.py
from foo_app import FooApplication
class FooTaskSet(TaskSet):
def on_start(self):
self.foo = FooApplication(self.client)
#task(1)
def login(self):
if not self.foo.is_logged_in:
self.foo.login()
#task(5) # 5x more likely to load a foo vs logging in again
def load_foo(self):
if self.foo.is_logged_in:
self.load_foo()
else:
self.login()
Since Locust uses the Requests HTTP library for Python, you might find your answer there.
The Response object can be used to evaluate if a redirect has happened and what the history of redirects contains.
is_redirect:
True if this Response is a well-formed HTTP redirect that could have been
processed automatically (by Session.resolve_redirects).
There might be an indication that the redirect is not well-formed.
The WTForms documentation gives a great example for implementing CSRF with Flask:
class MyBaseForm(Form):
class Meta:
csrf = True
csrf_class = SessionCSRF
csrf_secret = app.config['CSRF_SECRET_KEY']
#property
def csrf_context(self):
return session
I'd like to do the same thing but with a webapp2 session instead of a Flask session.
The csrf_context is a shortcut so that you don't have to pass the session every time that you create a form. Does anyone know how to create such a shortcut for a webapp2 session?
Without this shortcut, you need to do something like this every time you create a form:
form = MyForm(meta={'csrf_context': self.session})
which is some pretty awkward syntax that I'd prefer to avoid.
I've come up with a solution that reduces the awkward syntax described in my question. I've modified __init__ of my subclass of wt.Form like this:
class MyBaseForm(wt.Form):
class Meta:
csrf = True
csrf_class = SessionCSRF
csrf_secret = settings.SESSION_KEY
csrf_time_limit = timedelta(minutes=60)
def __init__(self, *args, **kwargs):
if "session" in kwargs:
super(MyBaseForm, self).__init__(
*args, meta={'csrf_context': kwargs["session"]}, **kwargs)
else:
super(MyBaseForm, self).__init__(*args, **kwargs)
Now, when I create a form, I can do this:
form = MyForm(session=self.session)
instead of the awkward syntax shown in the question.
For processing POSTed form data, I've figured out another technique to simplify processing.
First, I create a form with no fields other than the CSRF field:
class NoFieldForm(MyBaseForm):
pass
Second, I create a decorator that is used to check CSRF:
def check_csrf(func):
def wrapper(*args, **kwargs):
handler = args[0]
session = handler.session
request = handler.request
f = forms.NoFieldForm(request.POST, session=session)
f.validate()
if f.csrf_token.errors:
msg = "The CSRF token expired. Please try again. "
self.session["msg"] = msg
self.redirect(self.request.path)
else:
func(*args, **kwargs)
return wrapper
Third, I decorate all my POST handlers:
class SomeHandler(webapp2.RequestHandler):
#check_csrf
def post(self):
pass
Let me give some explanation. The decorator (via the call to the post webhandler) is going to receive some form data, but for purposes of CSRF checking we are going to throw away all form data except for CSRF. We can make it generic by using the NoFieldForm which ignores any other form data that is present. I use wtforms to check to make sure the CSRF form field and session token match and that the session token isn't expired.
If the CSRF passes, then we call the handler for normal processing to handle the specific form. If the CSRF fails, then we don't call the handler at all, and in my example, we redirect back to where we came from with an error message.
I need to programmatically log into a backend server, which returns an auth token as a cookie, then use that cookie to make requests going forward. I'm working in Lift 2.4, and everything I read seems to recommend using http-dispatch, but that has been cumbersome to learn! :-/ I'm working in dispatch-classic because of my SBT version (0.1-SNAPSHOT) and scala version 2.9.1. So I'm currently loading the dispatch 0.8.6 libraries.
I found the following at
https://groups.google.com/forum/#!msg/dispatch-scala/m7oWv2YAtjQ/imnkYoCDVUcJ
For retrieving cookies:
To read a cookie from a response, you have to call the Response#getCookies method. For example, you could do something like this:
val res = Http(url("http://www.google.com/ig/api").addQueryParameter("weather", "Bonn, Germany"))
val response = for { r <- res } yield (r.getCookies, r.getResponseBody)
for adding cookies to subsequent requests:
url("http://www.google.com/ig/api").addCookie(cookie)
But I can't get this to work.
My preference is code that works with dispatch 0.8.6, but if you can make it work in another version and don't see what that version won't work with my SBT and scala/Lift versions, I'll try using your recommended library version.
To get a the cookie, you should be able to do something like this:
Http(url("http://www.google.com/ig/api") <<? List("weather" -> "Bonn, Germany") >:> ((h) => h.get("Set-Cookie")))
That will request the URL, append the weather param, and then pass the response headers to a handler function which looks for the Set-Cookie header and returns an Option with the value, or None if it was not present.
To set a cookie, you can do:
Http(url("http://www.google.com/ig/api") <<? List("weather" -> "Bonn, Germany") <:< Map("Set-Cookie" -> "something") >| )
This will add the headers in the Map following the <:< directive, which in the case above includes the cookie. The >| handler simply ignores the response, but you can use any handler you want.
This guide is a pretty good reference for the different functions and handlers available.
I checked this out with 0.8.8, as I don't have the earlier version, and everything seemed to work for me. I can't sure for sure, but I think it should be the same with 0.8.6.
I am a beginning Facebook app developer, but I'm an experienced developer. I'm using web.py as my web framework, and to make matters a bit worse, I'm new to Python.
I'm running into an issue, where when I try to switch over to using the newer "OAuth 2.0 for Canvas", I simply can't get anything to work. The only thing being returned in my Facebook app is "None".
My motivation for turning on OAuth 2.0 is because it sounds like Facebook is going to force it by July, and I might as well learn it now and now have to rewrite it in a few weeks.
I turned on "OAuth 2.0 for Canvas" in the Advanced Settings, and rewrote my code to look for "signed_request" that is POSTed to my server whenever my test user tries to access my app.
My code is the following (I've removed debugging statements and error checking for brevity):
#!/usr/bin/env python
import base64
import web
import minifb
import urllib
import json
FbApiKey = "AAAAAA"
FbActualSecret = "BBBBBB"
CanvasURL = "http://1.2.3.4/fb/"
RedirectURL="http://apps.facebook.com/CCCCCCCC/"
RegURL = 'https://graph.facebook.com/oauth/authorize?client_id=%s&redirect_uri=%s&type=user_agent&display=page' % (FbApiKey, RedirectURL)
urls = (
'/fb/', 'index',
)
app = web.application(urls, locals())
def authorize():
args = web.input()
signed_request = args['signed_request']
#split the signed_request via the .
strings = signed_request.split('.')
hmac = strings[0]
encoded = strings[1]
#since uslsafe_b64decode requires padding, add the proper padding
numPads = len(encoded) % 4
encoded = encoded + "=" * numPads
unencoded = base64.urlsafe_b64decode(str(encoded))
#convert signedRequest into a dictionary
signedRequest = json.loads(unencoded)
try:
#try to find the oauth_token, if it's not there, then
#redirect to the login page
access_token = signedRequest['oauth_token']
print(access_token)
except:
print("Access token not found, redirect user to login")
redirect = "<script type=\"text/javascript\">\ntop.location.href=\"" +_RegURL + "\";\n</script>"
print(redirect)
return redirect
# Do something on the canvas page
returnString = "<html><body>Hello</body></html>"
print(returnString)
class index:
def GET(self):
authorize()
def POST(self):
authorize()
if __name__ == "__main__":
app.run()
For the time being, I want to concentrate on the case where the user is already logged in, so assume that oauth_token is found.
My question is: Why is my "Hello" not being outputted, and instead all I see is "None"?
It appears that I'm missing something very fundamental, because I swear to you, I've scoured the Internet for solutions, and I've read the Facebook pages on this many times. Similarly, I've found many good blogs and stackoverflow questions that document precisely how to use OAuth 2.0 and signed_request. But the fact that I am getting a proper oauth_token, but my only output is "None" makes me think there is something fundamental that I'm doing incorrectly. I realize that "None" is a special word in python, so maybe that's the cause, but I can't pin down exactly what I'm doing wrong.
When I turn off OAuth 2.0, and revert my code to look for the older POST data, I'm able to easily print stuff to the screen.
Any help on this would be greatly appreciated!
How embarrassing!
In my authorize function, I return a string. But since class index is calling authorize, it needs to be returned from the class, not from authorize. If I return the return from authorize, it works.
I am using Scrapy to fetch some data from iTunes' AppStore database. I start with this list of apps: http://itunes.apple.com/us/genre/mobile-software-applications/id36?mt=8
In the following code I have used the simplest regex which targets all apps in the US store.
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.contrib.spiders import CrawlSpider, Rule
class AppStoreSpider(CrawlSpider):
domain_name = 'itunes.apple.com'
start_urls = ['http://itunes.apple.com/us/genre/mobile-software-applications/id6015?mt=8']
rules = (
Rule(SgmlLinkExtractor(allow='itunes\.apple\.com/us/app'),
'parse_app', follow=True,
),
)
def parse_app(self, response):
....
SPIDER = AppStoreSpider()
When I run it I receive the following:
[itunes.apple.com] DEBUG: Crawled (200) <GET http://itunes.apple.com/us/genre/mobile-software-applications/id6015?mt=8> (referer: None)
[itunes.apple.com] DEBUG: Filtered offsite request to 'itunes.apple.com': <GET http://itunes.apple.com/us/app/bloomberg/id281941097?mt=8>
As you can see, when it starts crawling the first page it says: "Filtered offsite request to 'itunes.apple.com'". and then the spider stops..
it also returns this message:
[ScrapyHTTPPageGetter,client] /usr/lib/python2.5/cookielib.py:1577: exceptions.UserWarning: cookielib bug!
Traceback (most recent call last):
File "/usr/lib/python2.5/cookielib.py", line 1575, in make_cookies
parse_ns_headers(ns_hdrs), request)
File "/usr/lib/python2.5/cookielib.py", line 1532, in _cookies_from_attrs_set
cookie = self._cookie_from_cookie_tuple(tup, request)
File "/usr/lib/python2.5/cookielib.py", line 1451, in _cookie_from_cookie_tuple
if version is not None: version = int(version)
ValueError: invalid literal for int() with base 10: '"1"'
I have used the same script for other website and I didn't have this problem.
Any suggestion?
When I hit that link in a browser, it automatically tries to open iTunes locally. That could be the "offsite request" mentioned in the error.
I would try:
1) Remove "?mt=8" from the end of the URL. It looks like it's not needed anyway and it could have something to do with the request.
2) Try the same request in the Scrapy Shell. It's a much easier way to debug your code and try new things. More details here: http://doc.scrapy.org/topics/shell.html?highlight=interactive
I see this post is pretty old, if you haven't figured out the cause yet, here it is.
I run into a similar issue working with itunesconnect using mechanize. After much frustration i found that there's a bug in cookielib that doesn't handle some cookies correctly. It's discussed here: http://bugs.python.org/issue3924
The fix at the bottom of that post worked for me. I'll repost here for convenience.
Basically you create a custom subclass of cookielib.CookieJar, override _cookie_from_cookie_tuple and use this CustomCookieJar in place of the cookielib jar
class CustomCookieJar(cookielib.CookieJar):
def _cookie_from_cookie_tuple(self, tup, request):
name, value, standard, rest = tup
version = standard.get("version", None)
if version is not None:
# Some servers add " around the version number, this module expects a pure int.
standard["version"] = version.strip('"')
return cookielib.CookieJar._cookie_from_cookie_tuple(self, tup,request)