I'm trying to extract some data from a craigslist HTML page, but I seem to be running into a strange bug- every once in a while, the page I try to load with an NSURLRequest comes back as some strange form of data, which when converted to a parseable string, returns null. However, I can't consistently reproduce it- it'll suddenly stop working, and then I'll try it again an hour later and it'll be working perfectly, and then some time later it'll stop working again. Anyone know what could be causing it? I'm using an NSURLRequest, asynchronous, with the 'didReceiveData' and 'didReceiveResponse' delegate methods. If I cast the NSURLResponse to an NSHTTPURLResponse and check the response code, I get 200, meaning there were no issues. But when I go to initialize a string with the response data, it returns null, and I obviously then can't parse it.
The URL that seems to do it most often is: http://sarasota.craigslist.org/app/
I've tried messing with the User-Agent header for the request, the cache policy, everything I can think of... but nothing seems to fix it.
If there is data but when you ask for a string it's null then I might suspect the string encoding you're using to decode the data? Is there an odd character that is only sometimes in craigs list advert titles?
Just out of interest, why don't you use the rss feed instead - it's probably more consistently formatted / strictly encoded as it's xml not html.
http://sarasota.craigslist.org/app/index.rss
Related
I would like to make a simple GET request via Spray with a few query parameters
Get("http://localhost/user?email=abc+a#abc.com")
However + means a space in application/x-www-form-urlencoded content resulting the call to http://localhost/user?email=abc a#abc.com (with a space instead of plus sign).
I could use a non-Spray java.net.URLEncoder to encode the URL before passing it to the GET request however I doing this every time seems like a hack.
Is there a Spray way of applying query parameters and encoding them?
Uri("http://localhost/").withQuery(Map("email"->"abc+a#abc.com")) is a nice way to construct a Uri but it doesn't encode the params as well...
Actually Uri("http://localhost/").withQuery(Map("email"->"abc+a#abc.com")) works fine as it encodes the special symbols.
However, Uri("http://localhost/").withQuery("email=abc+a#abc.com") doesn't.
I use java.net.URLEncoder. I believe that is the accepted method.
It would be nice if that happened automatically!
I have seen several posts addressing this issue or similar to this issue for requests or GETs. I am not having this problem getting the data from the server, its solely on the POST.
The Errors I get are
The JSON request was too large to be deserialized.
or either
Error during serialization or deserialization using the JSON JavaScriptSerializer. The length of the string exceeds the value set on the maxJsonLength property. Parameter name: input
I haven't been able to consistently determine which actions result in which error, but it is predominately the latter one.
In an effort to get the value of the MaxJsonSize value, on the Index method of the controller, I get this data and dump it into a viewbag to write to console on the client side. Every time it comes back at 10k (102400).
If I reduce the data package size, and still serialize as previously, I get no errors.
In fiddler I can inspect the package and all the JSON is deserializable in fiddler, so I don't see an issue in my JSON. Additionally if I console.log(data) chrome sees no problems with it either.
The VM in the controller is the same for both POST and GET. With the exception there is more data with the POST than the GET. To test this I got a huge data set from the server;
GeoJSON data for all 50 states. Following was the result.
GET Content-Length: 3229309 return 200
POST Content-Length: 2975244 return 500
The POST failed in this scenario and returned the second error listed previously.
I only changed the data minimally (one string) and don't know why when sent back its smaller, but the JSON for both the GET and the POST is virtually identical.
I've tried changing the web.config file:
<system.web.extensions>
<scripting>
<webServices>
<jsonSerialization maxJsonLength="2147483644"/>
</webServices>
</scripting>
</system.web.extensions>
I just added this to the end of my config file just prior to
I've also added a parameter in Settings.config
<add key="aspnet:MaxJsonDeserializerMembers" value="2147483644" />
I have also verified that this param loads as part of the application settings in IIS.
Is there something else I can try to change to allow for these large data sets to be sent in a POST.
As a last resort, I was going to pull all of the GeoJSON data out of the POST. However when a user navigates back and they haven't changed what they were mapping, we'd have to find all the GeoJSON data again, causing undue work on the server etc. I thought if I only had to fetch it once that would be best from an efficiency perspective.
I struggled with this too, nothing I changed in web.config helped, despite several SO answers looking relevant. They helped with returning large JSON data, but the large JSON post kept failing. In the end I found this:
increase maxJsonLength for JSON POST and used the solution there, and it worked for me.
Quoting from there :
the MVC json serializer does not look at the webconfig to get the max length (thats for asp.net web services). you need to use your own serializer. you override ExecuteResult and supply you own json serializer. to override the input, create a new JsonValueProviderFactory, then override ValueProvider in the controller to return your new json factory when its a json request.
I am using the ruby paypal rest api and everything I do seems to be right but PayPal keeps saying my date format is incorrect.
Here is my call:
Payment.all(:start_time => '2013-03-06T11:00:00Z', :end_time => '2013-03-06T11:00:00Z')
After many attempts I literally took the date example in the documentation: https://developer.paypal.com/webapps/developer/docs/api/
Because no matter what I do I keep getting the same response:
'{"name":"VALIDATION_ERROR","details":[{"field":"start_time","issue":"Must
be a date_time string of form yyyy-mm-ddThh:mm:ss(.sss)?Z"},{"field":"end_time","issue":"Must
be a date_time string of form yyyy-mm-ddThh:mm:ss(.sss)?Z"}],"message":"Invalid
request - see details","information_link":"https://developer.paypal.com/webapps/developer/docs/api/#VALIDATION_ERROR","debug_id":"dcd8a9ce6a5e7"}'
Am I blind or does it not make sense?
Looks like it's a bug in our API where it's expecting raw value rather than url encoded value. The SDK was always url encoding the value of the start_time and end_time (or as a matter of fact all url parameters as per the HTTP spec). We have filed a bug and get this fixed as soon as possible. Meanwhile if this is something you need to work, we can possibly make the SDK not url encode the params or you can just modify the code for now locally.
I think your issue might be that you're setting both the start time and the end time to the same exact value. If you want a single day's worth of transactions the end time would be 2013-03-06T23:59:59Z.
This is driving me nuts and it may be something simple, but every time I call the following from Fiddler script is throws a NullReferenceException:
if (oSession.oResponse.headers.ExistsAndContains("Content-Type", "html")){...
This is straight off the Fiddler script knowledge base and I've got the same thing happening on two different Win 8 machines. It seems that oSession.oResponse.headers is null and yes, I am calling this from within OnBeforeRequest!
Same deal with oSession.responseBodyBytes - it looks like the response is null inside OnBeforeRequest. Otherwise Fiddler is behaving normally, traffic proxies through just fine and everything else looks good.
Ideas?
The oResponse.headers and oSession.responseBodyBytes objects don't exist until after the request has been sent to the server and the response has been read back. If you want to interact with the Response headers and body, do so inside the OnBeforeResponse function instead.
The KB was mangled a bit when updated for the new site. I'll file a bug to have this made more explicit.
I have read around and this error seems to be from bad JSON. Easy enough, here is my JSON
{"year":"2012","wheels":"Standard","trans":"Unknown"}
My issue is, this appears to be correct, and when I run it through JSON lint it returns vaild. I have also used cURL to download this page and used json_decode() to read it...worked fine.
Here is an example page: http://drivingthenation.com/app/carlist/getVinDtn.php?v=JA3215H1C**M&f=v
I ran it through HTTPScoop and the only thing the response text returned was
{"year":"2012","wheels":"Standard","trans":"Unknown"}
On the objective-c end I am using NSURL and NSData to get the URL, and then NSJSONSerialization. I can print out before NSJSONSerialization and see that it is infact getting data, but this error only occurs when I try to format it into JSON. Any thoughts?
The NSJSONSerialization class, by default, expects the input to be not just valid JSON, but a valid JSON object. If you want to read something that's not an object, you need to specify the NSJSONReadingAllowFragments option to the reader.