Prevent URL-encoding of values when adding headers to TRESTClient - rest

I'm working on a Delphi REST client for a public API that requires an HMAC256/Base64 signed string to be added to the headers of the request to authenticate. I've spent hours trying to figure out why it's not working, so I compared the raw request from my Delphi client to that of a working C# library (using Wireshark).
It turns out my request matches perfectly the request generated by the working C# library, except that Delphi's REST client is URL-encoding the values added to the request's header, therefore invalidating the carefully crafted signature.
This is how I'm adding the signature string to the header:
RESTRequest1.Params.AddHeader('SIGNATURE', FSignature);
The signature string may have slashes, plus signs, and/or equal signs that are being URL-encoded when they shouldn't. For example when the value of the signature string is...
FSignature = '8A1BgACL9kB6P/kXuPdm99s05whfkrOUnEziEtU+0OY=';
...then the request should should output raw headers like...
GET /resource HTTP/1.1
User-Agent: Embarcadero URI Client/1.0
Connection: Keep-Alive
<snip>
SIGNATURE: 8A1BgACL9kB6P/kXuPdm99s05whfkrOUnEziEtU+0OY=
<snip>
...but instead Wireshark shows this as the real value being sent...
SIGNATURE: 8A1BgACL9kB6P%2FkXuPdm99s05whfkrOUnEziEtU%2B0OY%3D
Is there a way to prevent the URL-encoding of values when using AddHeader? Or maybe another way to add raw headers to a TRESTClient request?
PS: I already tried both TRESTRequest.Params.AddHeader and TRESTClient.AddParameter with TRESTRequestParameterKind.pkHTTPHEADER as the Kind parameter. Both resulted in URL-encoded values.
PS2: Using Delphi RAD Studio 10.3.

You should include poDoNotEncode in the Options property of the TRESTRequestParameter.
This can be done using:
RESTClient1.AddParameter('SIGNATURE', FSignature, pkHTTPHEADER, [poDoNotEncode]);
or by using:
RESTClient1.Params.AddHeader('SIGNATURE', FSignature).Options := [poDoNotEncode];

Related

Implementing Put Block of Azure rest api?

I am following this MSDN Reference (https://learn.microsoft.com/en-us/rest/api/storageservices/fileservices/put-block) to implement a rest call for Put Block.
I am coding in Java and I formed below Authorisation string and URL before signing.
PUT
364070
x-ms-blob-type:BlockBlob
x-ms-date:Fri, 20 Jan 2017 12:57:06 GMT
x-ms-version:2016-05-31
/xyz/mycontainer/imageBlock1
comp:block
sun.net.www.protocol.https.DelegateHttpsURLConnection:https://xyz.blob.core.windows.net/mycontainer/imageBlock1?comp=block&blockid=YmxvY2stMQ==
Error I am getting:
403
Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
I read gaurav mantras post http://gauravmantri.com/2013/02/16/uploading-large-files-in-windows-azure-blob-storage-using-shared-access-signature-html-and-javascript/. But, its not working for me.
Is there anything wrong with the string I am sending to sign or URL or
below httpConn Request Header.
The Http Header I am setting is:
httpConn.setRequestMethod("PUT");
httpConn.setRequestProperty("x-ms-blob-type", blobType);
httpConn.setRequestProperty("x-ms-date", date);
httpConn.setRequestProperty("x-ms-version", storageServiceVersion);
httpConn.setRequestProperty("Authorization", authorizationHeader);
httpConn.setRequestProperty("Content-Length",String.valueOf(blobLength) );
System.out.println(httpConn);
DataOutputStream wr = new DataOutputStream(httpConn.getOutputStream());
wr.write(bytes);
wr.flush();
wr.close();
int response = httpConn.getResponseCode();
As I known, Put Block is a operation against Block Blobs. So, we do not need to specify x-ms-blob-type header. If you specify this in your HTTP header, you need to follow the following tutorial about constructing the Canonicalized Headers String:
Retrieve all headers for the resource that begin with x-ms-, including the x-ms-date header.
Convert each HTTP header name to lowercase.
Sort the headers lexicographically by header name, in ascending order. Each header may appear only once in the string.
Finally, append a new-line character to each canonicalized header in the resulting list. Construct the CanonicalizedHeaders string by concatenating all headers in this list into a single string.
So, based on your code, your canonicalized headers string looks like:
x-ms-blob-type:BlockBlob\nx-ms-date:Fri, 20 Jan 2017 12:57:06 GMT\nx-ms-version:2016-05-31\n
Moreover, the CanonicalizedResource you built is incorrect. Based on your code, it should look as follows:
/{account-name}/{container-name}/{blob-name}\nblockid:{block-id}\ncomp:block
Note: For more details about constructing the Canonicalized Resource String, you could refer to this official document.
The StringToSign would look like this:
And the traffic captured by Fiddler as follows:

Perl Change Content Type Of Response

I am calling a SOAP web service as client.
Following is content-type value of response
Content-Type: text/xml
I requested customer to add UTF-8 to response as follow:
Content-Type: text/xml;charset=utf-8
But customer says that it can be from client side. Is this possible? Can I determine content type of server as client?
PS: I noticed that the cited RFC2376 is obsolete by RFC3023 (conservative enough) and then RFC7303 that I'm omitting to evaluate now in involved current use and content, so the relevance of the following might not be that definitive, I'm feeling to delete it.
You have everything formal in RFC2376 XML Media Types: Section 3.1 text/xml Registration
See also Section 6 Examples of that RFC, particularly Section 6.4 text/xml with Omitted Charset
The server side (your customer) is STRONGLY RECOMMENDED to use charset parameter that they are not currently using.
And if charter is omitted XML processors MUST use the default charset value of "us-ascii"
You are right asking the customer to specify charset, the "MUST" in the RFC is a strong requirement that limits also your adaptability from client side when they are not sending us-ascii.

How to design REST API for export endpoint?

I am designing a REST API and am running into a design issue. I have alerts that I'd like the user to be able to export to one of a handful of file formats. So we're already getting into actions/commands with export, which feels like RPC and not REST.
Moreover, I don't want to assume a default file format. Instead, I'd like to require it to be provided. I don't know how to design the API to do that, and I also don't know what response code to return if the required parameter isn't provided.
So here's my first crack at it:
POST /api/alerts/export?format=csv
OR
POST /api/alerts/export/csv
Is this endpoint set up the way you would? And is it set up in the right way to require the file format? And if the required file format isn't provided, what's the correct status code to return?
Thanks.
In fact you should consider HTTP content negotiation (or CONNEG) to do this. This leverages the Accept header (see the HTTP specification: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.1) that specifies which is the expected media type for the response.
For example, for CSV, you could have something like that:
GET /api/alerts
Accept: text/csv
If you want to specify additional hints (file name, ...), the server could return the Content-Disposition header (see the HTTP specification: http://www.w3.org/Protocols/rfc2616/rfc2616-sec19.html#sec19.5.1) in the response, as described below:
GET /api/alerts
Accept: text/csv
HTTP/1.1 200 OK
Content-Disposition: attachment; filename="alerts.csv"
(...)
Hope it helps you,
Thierry

BigCommerce API Update Order with PUT

I need to update an order which is done via PUT method passing the order id as part of the https url string and a single parameter, the status_id.
https://mystore.mybigcommerce.com/orders/12345.json
I have tried several methods to pass the status_id value but no matter what I try "status_id=12" or formatted as JSON "{"status_id": 12,}" I always get the same response:
[{"status":415,"message":"The specified input content type is not valid."}]
I have also tried as a POST request passing the JSON or XML code as raw data but that method is not supported.
How am I supposed to pass that field=value pair? can I embed it in the url string?
I also tried it but it wouldn't work for me.
Any ideas?
In case you are wondering I am doing it within FileMaker with TROIUrl plugIn, not a very popular technology, but the GET method retrieving orders works like a charm
TURL_Put( ""; $url ;"status_id=12") (I have also tried other FM plugIns to no avail)
Don't get too caught up in the Filemaker part, I don't expect many people out there to be familiar with BigCommerce and Filemaker. I just need a generic answer.
Thanks
Commandline tool curl is worth a try. It supports put and https.
Mac OS X: curl already installed, call from FileMaker via AppleScript do shell script.
Windows: must be installed, call via Powershell.
It works for me using { "status_id": "3" } which means you probably need to put quotes around the actual number.
Also, it is a PUT operation and application/json which is part of the request content.
The error message received by the OP:
[{"status":415,"message":"The specified input content type is not valid."}]
Is saying that he did not supply the 'Content-Type' header in his request or that the header supplied is for a content type that is not allowed. For the OP's case using JSON he would need to include the header:
Content-Type: application/json
in his HTTPS request. This description can be found along with those of the other status codes you may see here:
https://developer.bigcommerce.com/api/status-codes

Problem with OAuth, POST with parameters

I'm using Jon Crosby's open source Objective-C OAuth library http://code.google.com/p/oauthconsumer/ for some basic http authentication that does not deal with tokens, only consumer key and consumer secret. My code works great for GET, GET with parameters in the URL, and POST. When I issue a POST request that has parameters in the URL, though, the request fails authorization. I'm trying to figure out why.
The server is using Apache Commons OAuth, so I'd like to compare my base string with that library. Here's a contrived example and the base string and signature produced by my library. Can anyone see what the problem is?
consumer key: abcdef
consumer secret: ghijkl
POST request: http://emptyrandomhost.com/a/uriwith/params?interesting=foo&prolific=bar
my base string: POST&http%3A%2F%2Femptyrandomhost.com%2Fa%2Furiwith%2Fparams&interesting%3Dfoo%26oauth_consumer_key%3Dabcdef%26oauth_nonce%3D1%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D2%26oauth_version%3D1.0%26prolific%3Dbar
This data produces the following OAuth header authorization:
Authorization: OAuth oauth_consumer_key="abcdef",
oauth_version="1.0",
oauth_signature_method="HMAC-SHA1",
oauth_timestamp="2",
oauth_nonce="1",
oauth_signature="Z0PVIz5Lo4eB7aZFT8FE3%2FFlbz0%3D"
And apparently my signature is wrong. The problem has to either be in the construction of the base string, in the way that the HMAC-SHA1 function is implemented (using Apple's CCHmac from CommonHMAC.h, so hopefully this isn't it), or with my Base64Transcoder, which is open source c. 2003 by Jonathan Wight/Toxic Software. I primarily suspect the base string, since the requests work for GET and POST and only fail with POST with URL parameters as above.
Can someone with lots of OAuth experience spot the problem above? Something else that would be very useful is the base string that is produced by Apache Commons OAuth in their authentication. Thanks.
As per RFC 5849 section 3.4.1.2, the OAuth base string URI does not include the query string or fragment. If either the client or the server does not remove the query parameters from the base string URI and add them to the normalized OAuth parameter list, the signatures won't match. Unfortunately, it's hard to tell which side is making this mistake. But it's easy to determine this is the problem: If it always works without query parameters but always fails with query parameters, you can be pretty sure that one side or the other is generating the wrong base string. (Be sure that it always happens though... intermittent errors would be something else. Similarly, if it never works with or without a query string, that would also be something else.) The other possibility is that normalization was done incorrectly — the parameter list must be sorted and percent encoded sequences must be upper-cased. If it's not normalized correctly on both sides, that will also cause a base string mismatch, and thus a signature mismatch.
you can build and check visually your request at this URL:
http://hueniverse.com/2008/10/beginners-guide-to-oauth-part-iv-signing-requests/
Open the boxes denoted by [+] signs and fill in your values, that way you may be able to see if the problem is at your code, or at the provider side.