Changing DynamicChannelBuffer in Netty to String and back to ChannelBuffer - scala

My web server is written in Scala using Twitter's Finagle library, which in turn relies on Netty. As such, the request content is returned as a DynamicChannelBuffer. If I upload an image to the server using curl from the Terminal like this:
curl -T "abc.jpg" http://127.0.0.1:8080/test/image
Then I can read, and forward the image to a backend webserver using a SOAP packet that looks like this:
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
<soap:Header>
<AuthHeader xmlns="http://www.testtesttest.co.za/">
<LogonID>testtesttest</LogonID>
<Password>testtesttest</Password>
</AuthHeader>
</soap:Header>
<soap:Body>
<uploadFile xmlns="http://www.testtesttest.co.za/">
<FileDetails>
<FileName>image.jpg</FileName>
<FileContents>
{(Base64.encode(request.getContent())).toString(UTF_8)
</FileContents>
</FileDetails>
</uploadFile>
</soap:Body>
</soap:Envelope>
In the example above, the code: (Base64.encode(request.getContent())).toString(UTF_8) converts the request content to a base 64 encoded string.
The problem is that I need to read the image content from Multipart Http request that is sent from a PhoneGap mobile app. PhoneGap gives me no option, to send only the image, and insists in doing the file upload as a multipart request.
To break the multipart request apart, I change the request.getContent() result into a string using toString(UTF_8), and then getting the image data part by splitting the http multipart message into it's separate chunks:
var requestParts = request.content.toString(UTF_8).split("\\Q--*****org.apache.cordova.formBoundary\\E")
val imageParts = requestParts(3).split("\\n\\s*\\n")
val imageHeader = imageParts(0)
val imageBody = imageParts(1)
This is crappy, I know (I'll improve later), but does the trick for now. imageBody now has the image content as a string.
Now, if I put the imageBody back into the SOAP packet, I have to encode it again using:
val encoder = new BASE64Encoder();
val encodedImage = encoder.encode(imageBody)
At this point the image is just garble. It's size looks right, but I'm messing something up with the string conversion or encoding. For the first example, I'm using Netty's encoder, but for the second example I'm using the standard java encoder. The reason is that Netty's encoder can only encode objects of type ChannelBuffer.
I don't want to say this too loud, but I've been struggling with this for more than a day. Any help here will be GREATLY appreciated.

So this works:
image --> [curl] ------> post1 --> [your code] --> soap msg 1 --> [back-end]
This does not:
image --> [phonegap] --> post2 --> [your code] --> soap msg 2 --> [back-end]
To solve this type of problem reliably you need to understand which encoding is used in each step.
Assuming you can use the same image, can you check the raw encoded content in post1 and post2 and infer which encoding is being used? Then when you understand that, log the content in your code as you decode and recode the message. That way you can ensure it's the same in soap msg1 and soap msg2.

Related

Karate - How can we see the exact request and input parameters sent when the request in feature file is called from a file

Karate - How can we see the exact request and input parameters sent when the request is called from a file and input parameters are coming through CSV file
In the HTML report - at the given request step,whole of the request with input parameters is displayed if the request has been put in the feature file.
HTML Report:
**Test 49 : Given request**
<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
<soap:Body>
<Add xmlns="http://tempuri.org/">
<intA>2</intA>
<intB>6</intB>
</Add>
</soap:Body>
</soap:Envelope>
whereas, if the request had been put into an XML file and then called by using the XML in the feature file then it does not display the whole of the request in the HTML report.
For Example- Feature File:
Given request read('classpath:RequestFiles/AddRequest.xml')
HTML Report just displays the test step as:
"Test 86 : When soap action 'http://tempuri.org/Add'"
Is anyone aware of the way to display the whole of request with input parameters if the request(in the feature file) is called from a XML file?
There is a simple way to put anything in what you refer to as the Doc String section of the report. Just print it.
* print someVarYouReadFromAFile
Also note that HTTP requests and responses will appear in the report by default, and most teams are fine with it. If you don't see this, there may be some other problem.
Finally a word of advice. I see many teams focus too much on time on making the reports "pretty". I would focus more on the question "am I able to test more scenarios and detect failures ?".

Encoding a GPX file such that it's accepted by the /matchroute endpoint of the Here API

I am trying to call the resource /matchroute via a GET request.
However, I can't figure out how to encode the GPX file so that the resource accepts my request: I always receive HTTP error 400 as a response from the Here server.
As exemplary data I used the following file:
<?xml version="1.0"?>
<gpx version="1.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://www.topografix.com/GPX/1/0"
xsi:schemaLocation="http://www.topografix.com/GPX/1/0
http://www.topografix.com/GPX/1/0/gpx.xsd">
<trk>
<trkseg>
<trkpt lat="51.10177" lon="0.39349"/>
<trkpt lat="51.10181" lon="0.39335"/>
<trkpt lat="51.10255" lon="0.39366"/>
<trkpt lat="51.10398" lon="0.39466"/>
<trkpt lat="51.10501" lon="0.39533"/>
</trkseg>
</trk>
</gpx>
that I got from the this example.
I encoded this file using MATLAB's function matlab.net.base64encode which yielded the following base64-encoded string:
PD94bWwgdmVyc2lvbj0iMS4wIj8+PGdweCB2ZXJzaW9uPSIxLjAieG1sbnM6eHNpPSJodHRwOi8vd3d3LnczLm9yZy8y
MDAxL1hNTFNjaGVtYS1pbnN0YW5jZSJ4bWxucz0iaHR0cDovL3d3dy50b3BvZ3JhZml4LmNvbS9HUFgvMS8wInhzaTpz
Y2hlbWFMb2NhdGlvbj0iaHR0cDovL3d3dy50b3BvZ3JhZml4LmNvbS9HUFgvMS8wIGh0dHA6Ly93d3cudG9wb2dyYWZp
eC5jb20vR1BYLzEvMC9ncHgueHNkIj48dHJrPjx0cmtzZWc+PHRya3B0IGxhdD0iNTEuMTAxNzciIGxvbj0iMC4zOTM0
OSIvPjx0cmtwdCBsYXQ9IjUxLjEwMTgxIiBsb249IjAuMzkzMzUiLz48dHJrcHQgbGF0PSI1MS4xMDI1NSIgbG9uPSIw
LjM5MzY2Ii8+PHRya3B0IGxhdD0iNTEuMTAzOTgiIGxvbj0iMC4zOTQ2NiIvPjx0cmtwdCBsYXQ9IjUxLjEwNTAxIiBs
b249IjAuMzk1MzMiLz48L3Rya3NlZz48L3Ryaz48L2dweD4=
However, as stated before, the HERE server consistently responds with HTTP-error 400 to my request
https://rme.api.here.com/2/matchroute.json?app_id={app_id}&app_code={app_code}&routemode=car&file=...
where "..." equals the above mentioned base64-encoded string.
Question: Could anyone please provide a code sample showing how to encode the above mentioned GPX file correctly (ideally in MATLAB language) so that the /matchroute resource is able to respond?
Remarks:
If I use the base64 string
UEsDBBQAAAAIANmztEQSwaeZzwAAAM8BAAAQAAAAc2FtcGxlLXRyYWNlLmdweIXPTQuCMBwG8HufQnZv%2F605S0k9dj
EIungdZjpSJ27kPn6%2BRBgYXcYYv2cPzzG2deU8805L1YSIYoLiaHMsWvv9uBlYowOrZYhKY9oAoO973DOsugJ2hFBI
z8k1K%2FNabGWjjWiy%2FJ36ShjVqqITd2lxpmo4XVKgMP6vZaCneKIyYabivzHnr4BhCbb6hoZRpnvMp86L%2BdIapx
ImRJxiSuh%2Bj5xq7CWY%2Bcz1EaypA10qxlfVjvOl8rxVxfzDQrk%2FFCfLRs7YpOCzA%2BZd49LoBVBLAQIUABQAAA
AIANmztEQSwaeZzwAAAM8BAAAQAAAAAAAAAAEAIAAAAAAAAABzYW1wbGUtdHJhY2UuZ3B4UEsFBgAAAAABAAEAPgAAAP
0AAAAAAA%3D%3D
from this example the GET request works. However, I couldn't figure out how to reproduce this encoding myself so that I am able to encode my own data accordingly.
Link to the Here API definition: https://developer.here.com/documentation/route-match/topics/resource-matchroute-request.html
Looking at the two base64 strings I can tell you the fundamental difference between them - the first one (which doesn't work) is unescaped whereas the second one (which works) is.
You can convert between the two formats manually using various online tools like this one. The escaped version of the non-working base64 string, in case you want to test it, is:
PD94bWwgdmVyc2lvbj0iMS4wIj8+PGdweCB2ZXJzaW9uPSIxLjAieG1sbnM6eHNpPSJodHRwOi8vd3d3LnczLm9yZy8y
%0AMDAxL1hNTFNjaGVtYS1pbnN0YW5jZSJ4bWxucz0iaHR0cDovL3d3dy50b3BvZ3JhZml4LmNvbS9HUFgvMS8wInhza
Tpz%0AY2hlbWFMb2NhdGlvbj0iaHR0cDovL3d3dy50b3BvZ3JhZml4LmNvbS9HUFgvMS8wIGh0dHA6Ly93d3cudG9wb2
dyYWZp%0AeC5jb20vR1BYLzEvMC9ncHgueHNkIj48dHJrPjx0cmtzZWc+PHRya3B0IGxhdD0iNTEuMTAxNzciIGxvbj0
iMC4zOTM0%0AOSIvPjx0cmtwdCBsYXQ9IjUxLjEwMTgxIiBsb249IjAuMzkzMzUiLz48dHJrcHQgbGF0PSI1MS4xMDI1
NSIgbG9uPSIw%0ALjM5MzY2Ii8+PHRya3B0IGxhdD0iNTEuMTAzOTgiIGxvbj0iMC4zOTQ2NiIvPjx0cmtwdCBsYXQ9I
jUxLjEwNTAxIiBs%0Ab249IjAuMzk1MzMiLz48L3Rya3NlZz48L3Ryaz48L2dweD4%3D
I'm not an expert on this, but as I understand, you need to URL-encode strings only when you want to paste them as-is into the web path of your browser (read about "URL Params"). If you construct your HTTP requests the right way™ (by this I mean specify the headers of the request and the key-value pairs correctly), you shouldn't have to worry about URL-encoding at all, since the tool that you're using (in this case, MATLAB) should take care of the conversion for you.
Unfortunately, I cannot test this theory, as I have no access to the discussed API - but I am fairly certain that this would solve your problem.
I had the exact same problem.
The documentation seems to be incomplete. You can check here for additional information. Several ways I solved this:
Use filetype='CSV' or filtetype='GPX' in parameter. It says the filetype is guessed if passed, that is actually not true. After passing an XML file the API told me my file didn't look like a 'CSV'
Compression is OPTIONAL, I suggest to avoid it completely I could not find a suitable compression either. It works fine with plain base64 encoding.
I suggest to actually use CSV because the XML actually returns parsing errors.
In python
data='''latitude,longitude
51.10177,0.39349
'''
r = requests.get('https://rme.api.here.com/2/matchroute.json?app_id={APP_ID}&app_code={APP_CODE}&routemode=car&file={file}&filetype={filetype}'.format(
APP_ID=os.getenv('HERE_APP_ID'),
APP_CODE=os.getenv('HERE_APP_CODE'),
filetype='CSV',
file=base64.b64encode(data.encode()).decode()
))

Scalatra -- byte array/image stream in request body

There's tons of documentation on Internet how to send a file stream in HTTP POST request in other languages, but not in Scalatra.
To the topic: I'd like to send an image as byte array or as a file stream (sorry for sloppy terminology, I'm an absolute newbie) via Scalatra post(). I already have backend Java functions that take byte array, convert it back to .jpg image and store it on the server. What I'm unclear on is the exact syntax how to do this in Scalatra.
That's what my post request looks like:
val imageInBytes = ... //obtain image in bytes
post("/images", ("image" -> imageInBytes))
However, Eclipse says that overloaded method post cannot be applied to (String, (String, Array[Byte]))
On the server side:
post("/images"){
contentType = "image/jpeg" //for displaying the image
val imInBytes = params("image") //obtain data from request body
//do something with it.
}
Any help would be greatly appreciated!
you would need to base64 url encode the image bytes then you can get it from the params bag.
You can also post a raw image to an endpoint and then read the inputStream.

iphone xml parsing error

Hello all in my iphone application after sending login xml request am getting following faulted xml... but i cant understand what the error is...
<?xml version="1.0" encoding="ISO-8859-1"?><SOAP-ENV:Envelope SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"
xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/">
<SOAP-ENV:Body><SOAP-ENV:Fault><faultcode xsi:type="xsd:string">SOAP-ENV:Client</faultcode><faultactor xsi:type="xsd:string"></faultactor><faultstring xsi:type="xsd:string">error in msg parsing:
Charset from HTTP Content-Type &apos;UTF-8&apos; does not match encoding from XML declaration &apos;ISO-8859-1&apos;</faultstring><detail xsi:type="xsd:string"></detail></SOAP-ENV:Fault></SOAP-ENV:Body></SOAP-ENV:Envelope>
Well, the error message is saying that you tried to parse a document where the document header said it was encoded in ISO 8859/1, while the HTTP header said it was in UTF-8. Parsers are allowed to resolve that conflict by giving the HTTP header precedence, but it seems this parser chose not to. The inconsistency means there's some kind of configuration problem and it needs to be investigated.

Get NSString/NSData object from hardcoded bytes saved as NSString object in webservice response

I am working with the iPhone SDK and I have to process a webservice response I receive from an external service.
The response data consists of an XML string that was UTF8-encoded to a byte array.
This byte array is converted to string
This string is put into a XML wrapper element
The wrapper is returned via an HTTP response
Therefore I need to know how to convert the response data back to the XML string it used to be. Unfortunately, I cannot change the way my response is created, so I have to deal with it somehow.
Example of the raw data I get from the webservice:
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
<s:Body><ExportBytesResponse xmlns="http://knowledgepark-ag.com/webservices/">
<ExportBytesResult>PERhdGE+PFJlY29yZCBEQj0iOTk5OTQiIEtleT0iMjA1Mjc0Ij48U2luZ2xlIEl0ZW1JZD0iQVJCUEFLRVQiIFZhbHVlPSIyOTAiIC8+PFNpbmdsZSBJdGVtSWQ9IkhET0tOUiIgVmFsdWU9IjIwNTI3NCIgLz48U2luZ2xlIEl0ZW1JZD0iVkVSQSIgVmFsdWU9IjEuNzEuMzIxMy4xNzU3NyIgLz48U2luZ2xlIEl0ZW1JZD0iRElTIiBWYWx1ZT0iR2VNLCBNw6RyIDEwIDIwMDggIDU6NDFQTTomI3hEO1ouenQuIGV4aXNpdGVydCBrZWluZSBNw7ZnbGljaGtlaXQgZGFzIEJlZW5kZW4gZWluZXMgTW9kdWxzIGR1cmNoIEFiYnJlY2hlbiBvZGVyIEhhcmQtQ2xvc2UtWCB6dSBlcmZhaHJlbi4gQW4gZGllc2VyIFN0ZWxsZSBtw7xzc3RlbiBtZWlzdCBVc2VydmFycyBkZXMgTW9kdWxzIGdlbMO2c2NodCB3ZXJkZW4uIE9kZXIgYW5kZXJlIEF1ZnLDpHVtYWt0aW9uIGR1cmNoZ2Vmw7xocnQgd2VyZGVuLiYjeEE7RUxLRUcgMDYuMDguMjAwOCAwOToyMToxNyYjeEE7JiN4QTsgQmVpc3BpZWxhdWZiYXUgOiYjeEQ7JiN4QTsmbHQ7Q2xv</ExportBytesResult>
</ExportBytesResponse></s:Body>
How do I get back my old xml string representation hidden inside these raw bytes?
Any help is highly appreciated, I feel just stupid right now for not being able to come up with a solution.
Best Regards,
David
ExportBytesResponse is Base64 encoded, so you must first decode that node using something like this.
That will give you an NSString containing the XML. Then you can use NSXMLParser to parse your data.