UTF-8 encoded characters from REST-query not rendered properly - rest

Question

I'm consuming an external REST service that provides all content as UTF-8 encoded.
For some reason my application cannot properly handle the response. If I dump the response I will se things like LuleÃ¥ (should be Luleå).
EDIT:
The same behavior happens if i forward (without altering) the string to the UI, ex.:
flash.message = "Test" + integrationService.testEncoding()
What I did was to create a _Events.groovy file in the /script folder and specifying there that
eventConfigureTomcat = { tomcat ->
tomcat.connector.URIEncoding = "UTF-8"
tomcat.connector.useBodyEncodingForURI = true
}
I also have the following in my Config.groovy:
grails.views.gsp.encoding = "UTF-8"
grails.converters.encoding = "UTF-8"
But that changed nothing. The response is still wrongly shown. I'm not sure if this is a configuration issue with Grails, with the embedded tomcat or with something else. I'm currently running my test setup on windows 7, but the same issue happens on my server running on Centos. Please advice.
EDIT2:
If i consume the REST service using curl, everything is rendered correctly in the output.
EDIT3:
I'm using org.springframework.web.client.RestTemplate and HttpComponents to consume the service:
private static final HttpHeaders requestHeaders
static{
requestHeaders = new HttpHeaders()
requestHeaders.set(HttpHeaders.CONTENT_TYPE, "application/json")
requestHeaders.set(HttpHeaders.ACCEPT, "application/json")
requestHeaders.set("Accept-Encoding", "gzip")
}
private final static RestTemplate restTemplate = new RestTemplate(new HttpComponentsClientHttpRequestFactory(
HttpClientBuilder.create().build()))
...
...
public def testEncoding(){
ResponseEntity<String> response = restTemplate.exchange(
"https://www.url.com", HttpMethod.GET, new HttpEntity<Object>(requestHeaders),
String.class)
def gamesJson = JSON.parse(response.getBody())
//...
//parse value from gamesJson
//...
return testValue
}

Answer 1

Per my previous answer:
You just need to add the StringHttpMessageConverter to the template's message converters:
RestTemplate template = new RestTemplate();
template.getMessageConverters()
.add(0, new StringHttpMessageConverter(Charset.forName("UTF-8")));
ResponseEntity<Object> response = template.exchange(endpoint, method, entity,
Object.class);

Answer 2

The encoding type can be enforced in the environment itself.
JAVA_TOOL_OPTIONS -Dfile.encoding=UTF8 -Dclient.encoding.override=UTF-8
Just try setting the above encoding settings in windows/linux. I hope this should resolve the issue.
In this case, JVM will pickup the default encoding type from environment variable.

Answer 3

Our team experienced a similar issue before, we have a 3rd party service and they said their output is encoded in UTF-8. But the strings returned are still garbled. After a bit of testing, it turns out that they were returning ISO-8859-1 encoded strings. What we did was to decode/encode their input into UTF-8 encoded characters so we can use those properly.
For your case, I think this is a similar issue:
UTF-8: Luleå
ISO-8859-1: Luleå
In Java, we did something like this:
Charset initialEncoding = Charsets.ISO_8859_1;
Charset outputEncoding = Charsets.UTF_8;
byte[] byteArray = input.getBytes(initialEncoding);
String output = new String(new String(byteArray, outputEncoding));
In Groovy, I think you could do something like
import groovy.json.JsonSlurper;
def main = {
def response = '{"name":"Luleå"}'
def slurper = new JsonSlurper()
def result = slurper.parse(response.getBytes(), 'UTF-8')
println result.name // prints Luleå
}

Answer 4

The answer to my problem is already found on Stack Exchange.
You just need to add the StringHttpMessageConverter to the template's message converters:
restTemplate.getMessageConverters()
.add(0, new StringHttpMessageConverter(Charset.forName("UTF-8")));

Answer 5

In my case that i had the same problem with contents received from my REST web service from the server and not my local enviroment.
I had search a lot and finally i found a solution that resolved my issue.
In Windows I added a new environment variable with key: JAVA_TOOL_OPTIONS and set its value to: -Dfile.encoding=UTF8.
The (Java) System property will be set automatically every time a JVM is started. You will know that the parameter has been picked up because the following message will be posted to System.err:
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8