I'm using Play Framework 2.3 and the WS API to download and parse HTML pages. For none-English pages (e.g Russian, Hebrew), I often get wrong encoding.
Here's an example:
def test = Action.async { request =>
WS.url("http://news.walla.co.il/item/2793388").get.map { response =>
Ok(response.body)
}
}
This returns the web page's HTML. English characters are received ok. The Hebrew letters appear as Gibberish. (Not just when rendering, at the internal String level). Like so:
<title>29 ×ר×××× ××פ××ת ×ש×××× ×× ×¤××, ××× ×©×××©× ×שר×××× - ×××××! ××ש×ת</title>
Other articles from the same web-site can appear ok.
using cURL with the same web-page returns perfectly fine which makes me believe the problem is within the WS API.
Any ideas?
Edit:
I found a solution in this SO question.
Parsing the response as ISO-8859-1 and then converting it to UTF-8 like-so:
Ok(new String(response.body.getBytes("ISO-8859-1") , response.header(CONTENT_ENCODING).getOrElse("UTF-8")))
display correctly. So I have a working solution, but why isn't this done internally?
Ok, here the solution I ended up using in production:
def responseBody = response.header(CONTENT_TYPE).filter(_.toLowerCase.contains("charset")).fold(new String(response.body.getBytes("ISO-8859-1") , "UTF-8"))(_ => response.body)
Explanation:
If the request returns a "Content-Type" header that also specifies a charset, simply return the response body sine the WS API will use it to decode correctly, otherwise, assume the response is ISO-8859-1 encoded and convert it to UTF-8
Related
I'm trying to pass a complete URL as a parameter to a java-based REST service (GET), but I'm not sure how to format it in order to avoid a "HTTP 400 Bad Request". I've tried Base64 encoding, but still get the 400 error. I think part of the problem is that the url contains a question mark, "?", since it seems to be fine if I remove that and pass the url as-is. I'm not sure what is the problem when its encoded.
example url - http://my.site.com/testing-service?some+parms
method annotations:
#GET
#Path("/{fullurl}")
#Produces("application/json")
public Response findByUrl(#PathParam("fullurl") String fullurl)
...
(I've updated the description with a little more detail per the first couple of comments)
Apparently the encoding approach was close, but Base64 (java or commons-code) didn't work for whatever reason (length perhaps?). I found switching to Base32 (commons-code) works for my situation.
I am currently trying to modify a script to use the requests library instead of the urllib2 library. I haven't really used it before and I am looking to do the equivalent of urlopen("http://www.example.org").read(), so I tried the requests.get("http://www.example.org").text function.
This works fine with normal everyday html, however when I fetch from this url (https://gtfsrt.api.translink.com.au/Feed/SEQ) it doesn't seem to work.
So I wrote the below code to print out the responses from the same url using both the requests and urllib2 libraries.
import urllib2
import requests
#urllib2 request
request = urllib2.Request("https://gtfsrt.api.translink.com.au/Feed/SEQ")
result = urllib2.urlopen(request)
#requests request
result2 = requests.get("https://gtfsrt.api.translink.com.au/Feed/SEQ")
print result2.encoding
#urllib2 write to text
open("Output.txt", 'w').close()
text_file = open("Output.txt", "w")
text_file.write(result.read())
text_file.close()
open("Output2.txt", 'w').close()
text_file = open("Output2.txt", "w")
text_file.write(result2.text)
text_file.close()
The openurl().read() works fine but the requests.get().text doesn't work for the given this url. I suspect it has something to do with encoding, but i don't know what. Any thoughts?
Note: The supplied url is a feed in the google protocol buffer format, once I receive the message i give the feed to a google library that interprets it.
Your issue is that you're making the requests module interpret binary content in a response as text.
A response from the requests library has two main way to access the body of the response:
Response.content - will return the response body as a bytestring
Response.text - will decode the response body as text and return unicode
Since protocol buffers are a binary format, you should use result2.content in your code instead of result2.text.
Response.content will return the body of the response as-is, in bytes. For binary content this is exactly what you want. For text content that contains non-ASCII characters this means the content must have been encoded by the server into a bytestring using a particular encoding that is indicated by either a HTTP header or a <meta charset="..." /> tag. In order to make sense of those bytes they therefore need to be decoded after receiving using that charset.
Response.text now is a convenience method that does exactly this for you. It assumes the response body is text, and looks at the response headers to find the encoding, and decodes it for you, returning unicode.
But if your response doesn't contain text, this is the wrong method to use. Binary content doesn't contain characters, because it's not text, so the whole concept of character encoding does not make any sense for binary content - it's only applicable to text composed of characters. (That's also why you're seeing response.encoding == None - it's just bytes, there is no character encoding involved).
See Response Content and Binary Response Content in the requests documentation for more details.
I am stuck with one of problem.
XML and JSON response of REST call don't display correct Chinese or any other foreign language that we we are expecting because post call send other than English language. But Reponse XML or JSON always displayed as question mark.
I tried to Accept-charset,content-language, vary header but i didn't able to get exact language content as response.
POST call url have body part as below
{
"assignmentActivityId": "b1ff695f-1693",
"assignmentActivityTitle":"Test",
"assignmentActivityType":"INDIVIDUAL",
"profileId": "158a1f0a9",
"earnedPoints":2,
"evaluationComments": "我能吞下玻璃而不伤身体 check 2 19/02/2015",
"reviewerComments" : "reviewer ",
"status": "DRAFT",
"documents" : [{"documentUri":"/ContentRepository"}]
}
Get call URL should receive evaluationComments of as Chinese but it display ? question mark on response xml or json.
How do you display it?
If you do a console.log or alert, is it still wrong?
If yes, then it is not a display problem, the response is really damaged.
Mihai
There was charset problem.
JSP display charset and UTF-8 but DB stored the data on Latin1.
So i have change the DB setting for Charset from latin1 to UTF-8.
Now working fine all the character are displaying according to internationalization .
I'm using the GMail API to retrieve an email contents. I am getting the following base64 encoded data for the body: http://hastebin.com/ovucoranam.md
But when I run it through a base64 decoder, it either returns an empty string (error) or something that resembles the HTML data but with a bunch of weird characters.
Help?
I'm not sure if you've solved it yet, but GmailGuy is correct. You need to convert the body to the Base64 RFC 4648 standard. The jist is you'll need to replace - with + and _ with /.
I've taken your original input and did the replacement: http://hastebin.com/ukanavudaz
And used base64decode.org to decode it, and it was fine.
You need to use URL (aka "web") safe base64 decoding alphabet (see rfc 4648), which it doesn't appear you're doing. Using the standard base64 alphabet may work sometimes but not always (2 of the characters are different).
Docs don't seem to consistently mention this important detail. Here's one where it does though:
https://developers.google.com/gmail/api/guides/drafts
Also, if your particular library doesn't support the "URL safe" alphabet then you can do string substitution on the string first ("-" with "+" and "_" with "/") and then do normal base64 decoding on it.
I had the same issue decoding the 'data' fields in the message object response from the Gmail API. The Google Ruby API library wasn't decoding the text correctly either. I found I needed to do a url-safe base64 decode:
#data = Base64.urlsafe_decode64(JSON.parse(#result.data.to_json)["payload"]["body"]["data"])
Hope that helps!
There is an example for python 2.x and 3.x:
decodedContents = base64.urlsafe_b64decode(payload["body"]["data"].encode('ASCII'))
If you only need to decode for displaying purposes, consider using atob to decode the messages in JavaScript frontend (see ref).
I found whilst playing with the API result, once I had drilled down to the body I was given an option to decode in the available methods.
val message = mService!!.users().messages().get(user, id).setFormat("full").execute()
println("Message snippet: " + message.snippet)
if(message.payload.mimeType == "text/plain"){
val body = message.payload.body.decodeData() // getValue("body")
Log.i("BODY", body.toString(Charset.defaultCharset()))
}
The result:-
com.example.quickstart I/BODY: ISOLATE NORMAL: 514471,Fap, South Point Rolleston, 55 Faringdon Boulevard , Rolleston, 30 May 2018 20:59:21
I coped the base64 test to a file (b64.txt), then base64-decoded it using base64 (from coreutils) with the -d option (see http://linux.die.net/man/1/base64) and I got text that was perfectly readable. The command I used was:
cat b64.txt | base64 -d
I'm using a Google Gears Worker to submt a POST httprequest (using var request = google.gears.factory.create('beta.httprequest'); )
with a parameter containing the string
"bford%20%24%23%26!%3F%40%20%E5%BE%B3%E5%8A%9B%E5%9F%BA%E5%BD%A6"
but the Django HttpRequest is receiving it as "bford $#&!?# å¾³å\u008a\u009bå\u009fºå½¦"
How do I specify to one or the other of the parties in the transaction to leave it untranslated?
Check the HttpRequest.encoding and the DEFAULT_CHARSET settings. Judging by the encoded value, this should be UTF-8 (which is indeed usually the right thing).
You can get the ‘untranslated’ (with %s still in) value by looking at the input stream (for POST) or environ QUERY_STRING (for GET) and decoding it manually, but it would be better to fix Django's incorrect string-to-unicode decoding really.
As I understand it, Django 1.0 should default to using UTF-8, so I'm not sure why it's not in your case.