Spring RestTemplate POST file with UTF-8 filename - rest

I'm using Spring RestTemplate to perform POST request sending a PDF file. The filename contains some UTF-8 characters (e.g. é, è, à, ê, ë).
The problem is that after sending the request, on the other side where the request is received, the filename doesn't have the expected UTF-8 characters, and I have something like ?popi?.pdf instead.
I've tried to explicitly set UTF-8 charset in RestTemplate, but it still doesn't work.
Here is my code,
public SomeThing storeFile(InputStream content, String fileName) {
Charset utf8 = Charset.forName("UTF-8");
RestTemplate restTemplate = new RestTemplate();
HttpHeaders headersFile = new HttpHeaders();
headersFile.setContentType(MediaType.APPLICATION_OCTET_STREAM);
headersFile.setContentDispositionFormData("file", fileName);
List<Charset> listCharSet = new ArrayList<Charset>();
listCharSet.add(utf8);
headersFile.setAcceptCharset(listCharSet);
InputStreamResource inputStreamResource = new InputStreamResource(content);
HttpEntity<InputStreamResource> requestEntityFile = new HttpEntity<>(inputStreamResource, headersFile);
MultiValueMap<String, Object> multipartRequest = new LinkedMultiValueMap<>();
multipartRequest.add("file", requestEntityFile);
RestTemplate newRestTemplate = new RestTemplate();
MappingJackson2HttpMessageConverter mappingJackson2HttpMessageConverter = new MappingJackson2HttpMessageConverter();
HttpMessageConverter stringHttpMessageConverter = new StringHttpMessageConverter(Charset.forName("UTF-8"));
newRestTemplate.getMessageConverters().add(0, stringHttpMessageConverter);
newRestTemplate.getMessageConverters().add(mappingJackson2HttpMessageConverter);
FormHttpMessageConverter convForm = new FormHttpMessageConverter();
convForm.setCharset(utf8);
newRestTemplate.getMessageConverters().add(convForm);
HttpHeaders header = new HttpHeaders();
header.setContentType(MediaType.MULTIPART_FORM_DATA);
HttpEntity<MultiValueMap<String, Object>> requestEntity = new HttpEntity<>(multipartRequest, header);
ResponseEntity<String> result = newRestTemplate.postForEntity(env.getProperty("core.endpoint") + "/documents", requestEntity, String.class);
}

according to rfc7578 when you POST a file with multipart/form-data you should use "percent-encoding" instead of filename*
NOTE: The encoding method described in [RFC5987], which would add a
"filename*" parameter to the Content-Disposition header field, MUST NOT be used.
it could be easyly realesed:
RestTemplate restTemplate = new RestTemplate();
restTemplate.getMessageConverters().add(0, new FormHttpMessageConverter() {
#Override
protected String getFilename(Object part) {
if (part instanceof Resource) {
Resource resource = (Resource) part;
try {
return URLEncoder.encode(resource.getFilename(), "UTF-8");
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
return null;
}
} else {
return null;
}
}
});

Most likely the file encoding's system property of the used JVM hasn't been explicitly set, meanwhile the operating system where JVM runs is not using UTF-8 as the default charset. For instance, if JVM runs on Windows, and we don't specify the charset, the default value will be Windows-1252 instead.
Could you double check the JVM arguments of both applications that send and receive the file? Please ensure that it has -Dfile.encoding=UTF-8 argument before specifying the main class name.
Please also ensure that the application/service which receives the file has been configured to accept UTF-8 charset.
Feel free to also check the other possible related answers, if adding the file.encoding argument on JVMs doesn't help solving the problem,
How to get UTF-8 working in Java webapps?
Spring MVC UTF-8
Encoding

Related

Handle French Char in Response while getting using REST Template Spring Boot

I have below piece of code:
**RestTemplate restTemplate = new RestTemplate(getClientHttpRequestFactory());
restTemplate.getMessageConverters().add(0, new StringHttpMessageConverter(StandardCharsets.UTF_8));
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.APPLICATION_JSON_UTF8);
HttpEntity<?> requestObject = new HttpEntity<>(request,headers);
ResponseEntity<String> result = restTemplate.postForEntity(uri, requestObject, String.class);**
Now we are getting french char like 'Numéro'. while getting response in result variable it became 'Num�ro' .. I need same as we have in response ('Numéro').
Try setting content type in headers.
headers.setAccept(new ArrayList(MediaType.APPLICATION_JSON))

Encoded special characters in parameters causing spring boot contract testing query does not match problem

While writing the spring boot contract testing on consumer side, I having problem when request parameters contains special characters. They'll automatically encoding causing the test failed due to the spring consider that the "Query does not match"
"自动制动" has been encoded as "%E8%87%AA%E5%8A%A8%E5%88%B6%E5%8A%A8"
Check the log, i could see:
Query: word = 自动制动 | word: %E8%87%AA%E5%8A%A8%E5%88%B6%E5%8A%A8 <<<<< Query does not match
Here's my groovy file on producer side:
Contract.make {
description "Returns \"Auto hold\"'s canonical value_Mandarin"
name "getSynonym_AutoHold_canonical_Mandarin"
request {
urlPath( "/synonyms"){
headers {"accept: application/json;charset=UTF-8"}
queryParameters {
parameter("filter","canonical")
parameter("lang", "cmn-CHN")
parameter("word","自动制动")
}
}
method GET()
}
response {
status OK()
headers {
contentType applicationJson()
}
body '''
{
"canonical": "autohold",
"word": "自动制动"
}'''
}
}
And here's what I have in consumer side:
#Test
public void testSynonyms_Cmn(){
RestTemplate restTemplate = new RestTemplate();
String url = "http://localhost:8080/synonyms";
HttpHeaders httpHeaders = new HttpHeaders();
httpHeaders.setContentType(MediaType.APPLICATION_JSON_UTF8);
UriComponentsBuilder builder = UriComponentsBuilder.fromHttpUrl(url)
.queryParam("filter","canonical")
.queryParam("lang","cmn-CHN")
.queryParam("word","自动制动");
HttpEntity<?> entity = new HttpEntity<>(httpHeaders);
CentralizedSynonyms centralizedSynonyms = restTemplate.exchange(builder.toUriString(), HttpMethod.GET, entity
, CentralizedSynonyms.class).getBody();
assertThat(centralizedSynonyms.getWord()).isEqualTo("自动制动");
assertThat(centralizedSynonyms.getCanonical()).isEqualTo("autohold");
}
I had something similar and fixed it with this:
url value(consumer("/path1/path2/something%3Dsomethingelse"), producer("/path1/path2/something=somethingelse"))

Required String not present : Spring Rest Template

I am trying to consume a (GET) rest service.
http://localhost:7010/abc/status?configFilePath=config%2Fconfig.properties
I am trying to use that using the Spring RestTemplate service
Below is the code which I have used for the restTemplate:
String configFile = "config/config.properties";
Map<String,String> restvars = new HashMap<String,String>();
restvars.put("configFilePath", configFile);
RestTemplate restTemplate = new RestTemplate();
String restUrl = http://localhost:7010/abc/status?
String restCall = restTemplate.getForObject(restUrl, String.class, restvars);
System.out.println(restCall.toString());
It throws
Required String parameter 'configFilePath' is not present
Doesn't the Map actually pass the parameters ?
I was looking at it from wrong angle. The answer from this SO question helped.
HttpHeaders headers = new HttpHeaders();
headers.set("Accept", MediaType.APPLICATION_JSON_VALUE);
UriComponentsBuilder builder = UriComponentsBuilder.fromHttpUrl(url)
.queryParam("configFilePath", configFile);
HttpEntity<?> entity = new HttpEntity<>(headers);
HttpEntity<String> response = restTemplate.exchange(builder.toUriString(),String.class);

spring boot (mvc) response with different content type encoding on error

I need help with spring handling an error.
a client service is sending a request accepting two different content types - binary and json. when everything works fine I prefer communicating to my server with binary encoding to save bandwidth. but on error I would like serialise ResponseEntity to json as my binary serialiser do not know how to serialise it to binary format, plus it is better for logging, etc.
I configured instance of ResponseEntityExceptionHandler and I am handling different exceptions from that implementation. but spring always choses binary format as it is first on the accept (or produces) list.
all I get is (because spring do not know how to serialise ResponseEntity to my custom binary format. see AbstractMessageConverterMethodProcessor#writeWithMessageConverters)
org.springframework.web.HttpMediaTypeNotAcceptableException: Could not find acceptable representation
client sends
headers {Accept: [application/custom-binary, application/json]
server's controller is configured to
// pseudo code
#RequestMapping(method = GET, produces = {"application/custom-binary", APPLICATION_JSON_VALUE})
public BannerMetaCollection get(#RequestParam(value = "q") UUID[] q) {
if (q != null) {
return service.getAllDataWith(q);
} else {
throw new IllegalArgumentException("invalid data");
}
}
// pseudo code
public class RestExceptionResolverSupport extends ResponseEntityExceptionHandler {
#ExceptionHandler
public ResponseEntity<Object> illegalArgumentException(IllegalArgumentException ex, WebRequest request {
Object body = errorResponse()
.withCode(BAD_REQUEST)
.withDescription("Request sent is invalid")
.withMessage(ex.getMessage())
.build());
return new ResponseEntity<Object>(body, new HttpHeaders(), HttpStatus.BAD_REQUEST);
}
}
any hints?
What I do to get this to work is that a let my endpoint method return a ResponseEntity and I don't declare what content is produced in the #RequestMapping annotation. I then set the Content-type header myself before returning the response, e.g.
// pseudo code
#RequestMapping(method = GET)
public ResponseEntity<BannerMetaCollection> get(#RequestParam(value = "q") UUID[] q) {
if (q != null) {
HttpHeaders headers = new HttpHeaders();
headers.add(HttpHeaders.CONTENT_TYPE, "application/custom-binary");
return new ResponseEntity<>(service.getAllDataWith(q),
headers,
HttpStatus.OK);
} else {
throw new IllegalArgumentException("invalid data");
}
}

UTF-8 encoded characters from REST-query not rendered properly

I'm consuming an external REST service that provides all content as UTF-8 encoded.
For some reason my application cannot properly handle the response. If I dump the response I will se things like LuleÃ¥ (should be Luleå).
EDIT:
The same behavior happens if i forward (without altering) the string to the UI, ex.:
flash.message = "Test" + integrationService.testEncoding()
What I did was to create a _Events.groovy file in the /script folder and specifying there that
eventConfigureTomcat = { tomcat ->
tomcat.connector.URIEncoding = "UTF-8"
tomcat.connector.useBodyEncodingForURI = true
}
I also have the following in my Config.groovy:
grails.views.gsp.encoding = "UTF-8"
grails.converters.encoding = "UTF-8"
But that changed nothing. The response is still wrongly shown. I'm not sure if this is a configuration issue with Grails, with the embedded tomcat or with something else. I'm currently running my test setup on windows 7, but the same issue happens on my server running on Centos. Please advice.
EDIT2:
If i consume the REST service using curl, everything is rendered correctly in the output.
EDIT3:
I'm using org.springframework.web.client.RestTemplate and HttpComponents to consume the service:
private static final HttpHeaders requestHeaders
static{
requestHeaders = new HttpHeaders()
requestHeaders.set(HttpHeaders.CONTENT_TYPE, "application/json")
requestHeaders.set(HttpHeaders.ACCEPT, "application/json")
requestHeaders.set("Accept-Encoding", "gzip")
}
private final static RestTemplate restTemplate = new RestTemplate(new HttpComponentsClientHttpRequestFactory(
HttpClientBuilder.create().build()))
...
...
public def testEncoding(){
ResponseEntity<String> response = restTemplate.exchange(
"https://www.url.com", HttpMethod.GET, new HttpEntity<Object>(requestHeaders),
String.class)
def gamesJson = JSON.parse(response.getBody())
//...
//parse value from gamesJson
//...
return testValue
}
Per my previous answer:
You just need to add the StringHttpMessageConverter to the template's message converters:
RestTemplate template = new RestTemplate();
template.getMessageConverters()
.add(0, new StringHttpMessageConverter(Charset.forName("UTF-8")));
ResponseEntity<Object> response = template.exchange(endpoint, method, entity,
Object.class);
The encoding type can be enforced in the environment itself.
JAVA_TOOL_OPTIONS -Dfile.encoding=UTF8 -Dclient.encoding.override=UTF-8
Just try setting the above encoding settings in windows/linux. I hope this should resolve the issue.
In this case, JVM will pickup the default encoding type from environment variable.
Our team experienced a similar issue before, we have a 3rd party service and they said their output is encoded in UTF-8. But the strings returned are still garbled. After a bit of testing, it turns out that they were returning ISO-8859-1 encoded strings. What we did was to decode/encode their input into UTF-8 encoded characters so we can use those properly.
For your case, I think this is a similar issue:
UTF-8: Luleå
ISO-8859-1: Luleå
In Java, we did something like this:
Charset initialEncoding = Charsets.ISO_8859_1;
Charset outputEncoding = Charsets.UTF_8;
byte[] byteArray = input.getBytes(initialEncoding);
String output = new String(new String(byteArray, outputEncoding));
In Groovy, I think you could do something like
import groovy.json.JsonSlurper;
def main = {
def response = '{"name":"Luleå"}'
def slurper = new JsonSlurper()
def result = slurper.parse(response.getBytes(), 'UTF-8')
println result.name // prints Luleå
}
The answer to my problem is already found on Stack Exchange.
You just need to add the StringHttpMessageConverter to the template's message converters:
restTemplate.getMessageConverters()
.add(0, new StringHttpMessageConverter(Charset.forName("UTF-8")));
In my case that i had the same problem with contents received from my REST web service from the server and not my local enviroment.
I had search a lot and finally i found a solution that resolved my issue.
In Windows I added a new environment variable with key: JAVA_TOOL_OPTIONS and set its value to: -Dfile.encoding=UTF8.
The (Java) System property will be set automatically every time a JVM is started. You will know that the parameter has been picked up because the following message will be posted to System.err:
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8