So if I have prepared a REST Request, is there a way that I can view the output of it before sending for debugging purposes?
So:
LRequest.Client := LClient;
LRequest.Method := rmPOST;
LRequest.Resource := AUTH_PATH;
LRequest.Accept := 'application/json';
// required parameters
LRequest.AddParameter(ExCode, AuthCode, pkGETorPOST);
LRequest.AddParameter(ExRedirectURI, RedirectionEndPoint, pkGETorPOST);
LRequest.AddParameter(ExGrantType, ExAuthorizationCode, pkGETorPOST);
LRequest.AddParameter('Autorization', ExAuthParam, pkHTTPHEADER);
LRequest.Execute;
might yield:
POST https://oauth.platform.com/oauth2/v1/tokens/bearer HTTP/1.1
Accept: application/json
Authorization: Basic UTM0dVBvRDIwanp2OUdxNXE1dmlMemppcTlwM1d2
NzRUdDNReGkwZVNTTDhFRWwxb0g6VEh0WEJlR3dheEtZSlVNaFhzeGxma1l
XaFg3ZlFlRzFtN2szTFRwbw==
Content-Type: application/x-www-form-urlencoded
Host: oauth.platform.com
Body: grant_type=authorization_code&
code=L3114709614564VSU8JSEiPkXx1xhV8D9mv4xbv6sZJycibMUI&
redirect_uri=https://www.mydemoapp.com/oauth-redirect
Related
Please help. I'm trying to learn REST concepts and make the first programs using Delphi and REST objects. I came across a problem that I don’t know how to solve. In the database I have a test with special characters from my mother tongue (Bosnian): č, ć, đ, š, ž. When I pass this text via the GET method, objects that parse and display this text in objects on the form display these characters as "?", "æ", "æè" etc.
I researched and tried unsuccessfully to solve the problem. I tried to use the Utf8ToAnsi function and put iso-8859-2 instead of UTF-8 in the RESTClient, RESTRequest and RESTResponce object parameters.
Please for help, instruction or advice on how to solve or investigate this problem.
Source code on server side:
procedure TWebModule1.UsersGet(Request: TWebRequest; Response: TWebResponse);
var
a: TJSONArray;
o: TJSONObject;
i: Integer;
Q1: TADOQuery;
begin
Q1:=TADOQuery.Create(nil);
Q1.Connection:= BasicDBConn.Konekcija;
with Q1 do
begin
Active:=False;
SQL.Clear;
SQL.Add('Select USER_ID, USER_NAME, ROLE_NAME from USERS, ROLES ' +
' where USERS.ROLE_ID=ROLES.ROLE_ID ');
Active:= True;
end;
a := TJSONArray.Create;
if Q1.RecordCount > 0 then
begin
for i:=1 to Q1.RecordCount do
begin
o := TJSONObject.Create;
o.AddPair('USER_ID', Q1.Fields.Fields[0].Value);
o.AddPair('USER_NAME', Q1.Fields.Fields[1].Value);
o.AddPair('ROLE_NAME', Q1.Fields.Fields[2].Value);
Q1.Next;
a.AddElement(o);
end;
end;
Response.ContentType := 'application/json';
Response.Content := a.ToString;
a.DisposeOf;
Q1.Free;
end;
on client side:
procedure TGlavna.Button1Click(Sender: TObject);
begin
RESTRequest.Resource := 'Users';
RESTRequest.Method := TRESTRequestMethod.rmGet;
RESTRequest.Response := RESTResponse;
RESTRequest.Execute;
if assigned(fJSONArray) then fJSONArray.DisposeOf;
fJSONArray := TJSONObject.ParseJSONValue(RESTResponse.Content) as TJSONArray;
if RESTResponse.Content.IsEmpty then Application.MessageBox('Empty', 'Information', MB_OK)
else Memo1.Lines.Add(RESTResponse.Content);
end;
This is what a data in DB looks like
USER_ID USER_NAME ROLE_ID
u1 Test User 2
u2 T_č_ć_š_đ_ž 1
u3 t_Č_Ć_Ž_Đ_Š 1
u4 Bradić Kenan 1
In web browser, responce data is:
[{"USER_ID":"u1","USER_NAME":"Test User","ROLE_ID":"2"},{"USER_ID":"u3","USER_NAME":"t_È_Æ_Ž_Ð_Š","ROLE_ID":"1"},{"USER_ID":"u4","USER_NAME":"Bradiæ Kenan","ROLE_ID":"1"},{"USER_ID":"u2","USER_NAME":"T_è_æ_š_ð_ž","ROLE_ID":"1"}]
Headers in web browser:
General:
Request URL: http://localhost:8080/Users
Request Method: GET
Status Code: 200 OK
Remote Address: [::1]:8080
Referrer Policy: strict-origin-when-cross-origin
Response Headers:
Connection: close
Content-Length: 228
Content-Type: application/json; charset=ISO-8859-1
Date: Mon, 27 Dec 2021 17:07:54 GMT
Request Headers:
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Cache-Control: max-age=0
Connection: keep-alive
Host: localhost:8080
sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="96", "Google Chrome";v="96"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36
Thank you...
The HTTP response contains this line:
Content-Type: application/json; charset=ISO-8859-1
So the client handles interprets the Response content as ISO-85591 encoded instead of UTF-8.
Your server side codes contains this line:
Response.ContentType := 'application/json';
Your server side code should clear the charset=ISO-85591 part.
You can encode your value with base64, and create a JSON payload to translate the JSON payload with REST API.
Moreover, if the value is sensitive, you can encrypt it with AES, and then translate it with REST API.
What technology will you use in server side? PHP? or .NET or JAVA? All 3 technologies used AES in native code, but there are little differences between the technologies.
If you used PHP or PHP + Laravel, you might reference my sample project in my GitHub Repo, the description you can read from my article.
I have a question, for security purposes, it is sometimes very useful to make HTTP requests with illegal paths, such as: http://localhost/%x, in Golang it does not seem to be a way to do that, could anyone give some light to this issue?
What I have tried so far:
- .Get() function
response, err := http.DefaultClient.Get("http://localhost/%x")
if err != nil{
panic(err) // net/url.EscapeError
}
.Do() function
requestObject, err := http.NewRequest("GET", "http://localhost/%x", nil)
if err != nil {
panic(err) // panics!
}
And
requestObject, err := http.NewRequest("GET", "http://localhost", nil)
if err != nil {
panic(err) // Not panic
}
requestObject.URL.Path = "/%x"
response, err = netClient.goHttpClient.Do(requestObject)
if err != nil {
panic(err) // Not panic
}
This last one looks promising, but request is made to %25x instead of %x, can someone give some help please?
UPDATE:
Go can send %x in the request, in the query section though, not in the path, I want to send %x in the path, so my issue is not solved:
requestObject, err := http.NewRequest("GET", "http://localhost", nil)
if err != nil {
panic(err) // Not panic
}
requestObject.URL.Path = "/%x"
requestObject.URL.RawQuery = "%x"
Request is made to http://localhost/%25x?%x The %x in the path is encoded, in the query it is not.
UPDATE:
The answer given #Cerise is the one I use, and it works fine, but there are edge cases to take into account.
1 - Proxy: If you have to forward the requests to a proxy, for some reason, the HTTP framework does not know how to send HTTP requests that your proxy can understand, to fix this, in Burp suite, you have to enable invisible proxy as documented here: https://portswigger.net/support/using-burp-suites-invisible-proxy-settings-to-test-a-non-proxy-aware-thick-client-application
This still has an issue, I noticed it when I had to debug some HTTP requests, I could not make sense, of what Burp Suite was telling me, then inspecting the requests in Wireshark I knew there was something wrong, if you use .Opaque + BurpSuite, BurpSuite will modify some requests, this is where the second issue comes into play, keep reading.
2 - Opaque does not handle well the paths starting with "//", I don't think it is a bug though, it is well known that https://stackoverflow.com can also be rewritten as //stackoverflow.com, I think this the logic taken into account by go, anyway, if you do:
requestObject, err := http.NewRequest(methodStr, "http://localhost", nil)
requestObject.URL.Opaque = "//something"
This is what the HTTP request looks like with Wireshark:
GET http://something HTTP/1.1
Host: localhost
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.116 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Language: en-US,en;q=0.9,ca;q=0.8
Cache-Control: no-cache
Connection: close
Pragma: no-cache
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
Accept-Encoding: gzip
notice the first line, it is not what we may have though first, right?
I used to debug HTTP requests with Burp Suite, but let's use the same code, and forward the requests to Burp using:
http.DefaultTransport.(*http.Transport).Proxy = http.ProxyURL("http://localhost:8080")
Don't forget to enable invisible proxy on BurpSuite, otherwise it will reject the request. What Burp Suite shows is:
GET / HTTP/1.1
Host: localhost
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.116 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Language: en-US,en;q=0.9,ca;q=0.8
Cache-Control: no-cache
Connection: close
Pragma: no-cache
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
Accept-Encoding: gzip, deflate
outch!!, Burp has changed our request completely!!!
Takeaways:
- Do not use Your HTTP proxy to debug requests made with Opaque
- Do not use .Opaque with paths starting with double slashes, use the following:
if strings.HasPrefix(yourPath, "//") {
requestObject.URL.Path = fmt.Sprintf("%s", yourPath)
} else {
requestObject.URL.Opaque = fmt.Sprintf("%s", yourPath)
}
Set Request.URL.Opaque to the invalid path. The Opaque data is written to the network as is.
requestObject, err := http.NewRequest("GET", "http://localhost", nil)
if err != nil {
panic(err)
}
requestObject.URL.Opaque = "/%x"
response, err = netClient.goHttpClient.Do(requestObject)
if err != nil {
panic(err)
}
This code creates an invalid request. The net/http server handles the invalid request by responding with 400 Bad Request. Other servers and proxies may do something different.
Run it on the playground
I'm fighting a REST API call from Delphi. I want to send a PUT request to a server, with contenttype set to application/json:
aCustomRestClient.BaseURL := domein;
aCustomRestRequest.Client := aCustomRestClient;
aCustomRestRequest.Response := aCustomRestResponse;
aCustomRestRequest.Method := rmPUT;
aCustomRestRequest.Resource := '/service/API/1501/allowanceCategory';
aCustomRestRequest.Params.AddHeader('sessionId', SessionToken);
qryAllowanceCategory.First;
o := TJSONObject.Create;
arr:=TJSONarray.create;
try
o.AddPair('createdBy',qryAllowanceCategory.fieldbyname('createdBy').AsString);
{lot more pairs here}
o.AddPair('percentage', TJSONnumber.create(qryAllowanceCategory.fieldbyname('percentage').AsFloat));
o.AddPair('remark', 'JUR '+ qryAllowanceCategory.fieldbyname('remark').AsString);
arr.add(o);
aCustomRestRequest.ClearBody;
aCustomRestResponse.contenttype:='application/json';
aCustomRestClient.contenttype:='application/json';
aCustomRestRequest.AddBody(arr.ToString);
aCustomRestRequest.Execute;
However, when I view my request as sent with Fiddler:
PUT http://shadow.jdvretail.com/service/API/1501/allowanceCategory HTTP/1.1
Connection: Keep-Alive
Content-Type: application/x-www-form-urlencoded
Accept: application/json, text/plain; q=0.9, text/html;q=0.8,
Accept-Charset: utf-8, *;q=0.8
Authorization: Basic <<Authorization properly filled>>
User-Agent: Embarcadero RESTClient/1.0
sessionId: <<sessionID GUID properly filled here>>
Content-Length: 542
The Content-type remains wrong... how/where do I correctly set the contentType to 'application/json'?
TRestClient.AddBody(<string>) has an optional second parameter allowing to set the content type.
procedure AddBody(const ABodyContent: string; AContentType: TRESTContentType = ctNone); overload;
we try to develop an interface for Magento 2.1 in Delphi XE6.
Before Magento 2 we used SOAP but with changes in the API since Magento 2, we think this is not the best solution anymore.
Our problems:
We created and activated an integration in Magento with all necessary rights.
Now we get consumer key, consumer secret, access token and access token secret.
We have a RESTClient, RESTRequest, RESTResponse and a OAuth1Authenticator. The authenticator gets both tokens and both secrets.
We adjust the properties for Accept, AcceptCharset and AcceptEncoding.
If we put a GET-Request to http://192.168.200.108/index.php/rest/V1/orders/1 everything is fine and we get the first order, which was placed in the shop.
The IP 192.168.200.108 is a standard installation of Magento in a virtual machine in our local network.
If we want to get a filtered list of orders via
http://192.168.200.108/index.php/rest/V1/ordersindex.php/rest/V1/orders?searchCriteria[filter_groups][0][filters][0][field]=created_at&searchCriteria[filter_groups][0][filters][0][value]=2017-06-01%2000:00:00&searchCriteria[filter_groups][0][filters][0][condition_type]=gt, we get "invalid signature" as response.
But with the REST extension for Firefox we get a valid filtered list of orders with the same URL.
Signature method of the authenticator is HMAC_SHA1. We compared both requests (Firefox and Delphi) via Wireshark, but both look equal.
Delphi encodes the brackets via url encoding, the REST extension in Firefox doesn't. If we tell the RESTClient in Delphi, that the URL is already encoded, Delphi doesn't encode the brackets but we get an HTML error 400 with "Bad request".
OAuth1Authenticator1.ConsumerKey := sCONSUMER_KEY;
OAuth1Authenticator1.ConsumerSecrect := sCONSUMER_SECRET;
OAuth1Authenticator1.AccessToken := sACCESS_TOKEN;
OAuth1Authenticator1.AccessTokenSecret := sACCESS_TOKEN_SECRET;
RestClient2.AcceptEncoding := 'gzip, deflate';
RESTRequest2.AcceptEncoding := 'gzip, deflate';
RestClient2.Accept := '*/*';
RESTRequest2.Accept := '*/*';
RESTClient2.AcceptCharset := 'UTF-8';
RESTRequest2.AcceptCharset := 'UTF-8';
RESTClient2.BaseURL := 'http://192.168.200.108/';
RESTRequest2.Params.Clear;
RESTRequest2.Resource := 'index.php/rest/V1/orders?searchCriteria[filter_groups][0][filters][0][field]=created_at&searchCriteria[filter_groups][0][filters][0][value]=2016-07-01 00:00:00&searchCriteria[filter_groups][0][filters][0][condition_type]=gt';
RESTRequest2.Method := rmGET;
RESTRequest2.Params.Clear;
RESTRequest2.Params.AddItem;
RESTRequest2.Params[0].name := 'private_content_version';
RESTRequest2.Params[0].Value := 'c6f9e0919d7028c78ee63e79593ef689';
RESTRequest2.Params[0].ContentType := ctTEXT_PLAIN;
RESTRequest2.Params[0].Kind := pkCOOKIE;
RESTRequest2.Params[0].Options := RESTRequest2.Params[0].Options + [poDoNotEncode];
RESTRequest2.Params.AddItem;
RESTRequest2.Params[1].name := 'mage-messages';
RESTRequest2.Params[1].Value := '%5B%5D';
RESTRequest2.Params[1].ContentType := ctTEXT_PLAIN;
RESTRequest2.Params[1].Kind := pkCOOKIE;
RESTRequest2.Params[1].Options := RESTRequest2.Params[1].Options + [poDoNotEncode];
RESTRequest2.Params.AddItem;
RESTRequest2.Params[2].name := 'Accept-Language';
RESTRequest2.Params[2].Value := 'null';
RESTRequest2.Params[2].ContentType := ctTEXT_PLAIN;
RESTRequest2.Params[2].Kind := pkHTTPHEADER;
RESTRequest2.Params[2].Options := RESTRequest2.Params[2].Options + [poDoNotEncode];
RESTRequest2.Params.AddItem;
RESTRequest2.Params[3].name := 'DNT';
RESTRequest2.Params[3].Value := '1';
RESTRequest2.Params[3].ContentType := ctTEXT_PLAIN;
RESTRequest2.Params[3].Kind := pkHTTPHEADER;
RESTRequest2.Params[3].Options := RESTRequest2.Params[3].Options + [poDoNotEncode];
RESTRequest2.Params.AddItem;
RESTRequest2.Params[4].name := 'Connection';
RESTRequest2.Params[4].Value := 'keep-alive';
RESTRequest2.Params[4].ContentType := ctTEXT_PLAIN;
RESTRequest2.Params[4].Kind := pkHTTPHEADER;
RESTRequest2.Params[4].Options := RESTRequest2.Params[4].Options + [poDoNotEncode];
RESTRequest2.Execute;
Memo1.Lines.Add(RESTResponse2.Content);
Here the requests tracked with Wireshark.
Delphi:
GET /index.php/rest/V1/orders?searchCriteria%5Bfilter_groups%5D%5B0%5D%5Bfilters%5D%5B0%5D%5Bfield%5D=created_at&searchCriteria%5Bfilter_groups%5D%5B0%5D%5Bfilters%5D%5B0%5D%5Bvalue%5D=2016-07-01%2000:00:00&searchCriteria%5Bfilter_groups%5D%5B0%5D%5Bfilters%5D%5B0%5D%5Bcondition_type%5D=gt HTTP/1.1
Accept-Language: null
DNT: 1
Connection: keep-alive
Authorization: OAuth oauth_consumer_key="5cbx1us6n7tpn5iv0a33p6iyq0otupsp", oauth_nonce="D4F1CF6F5DBB4F833D6CE644D3C20D95", oauth_signature="iXk6yBvuasDKnbdcxfATSQRJ%2BEg%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1496824715", oauth_token="nvoankn2g8jraga3k22ax9eyf55lqk9e", oauth_version="1.0"
Host: 192.168.200.108
Accept: */*
Accept-Charset: UTF-8
Accept-Encoding: gzip, deflate, identity
User-Agent: Embarcadero RESTClient/1.0
Cookie: mage-messages=%5B%5D; private_content_version=c6f9e0919d7028c78ee63e79593ef689
Firefox:
GET /index.php/rest/V1/orders?searchCriteria[filter_groups][0][filters][0][field]=created_at&searchCriteria[filter_groups][0][filters][0][value]=2016-07-01%2000:00:00&searchCriteria[filter_groups][0][filters][0][condition_type]=gt HTTP/1.1
Host: 192.168.200.108
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:53.0) Gecko/20100101 Firefox/53.0
Accept: */*
Accept-Language: null
Accept-Encoding: gzip, deflate
Authorization: OAuth oauth_version="1.0", oauth_signature_method="HMAC-SHA1", oauth_nonce="5xayplNhc6EVeoM", oauth_timestamp="1496824768", oauth_consumer_key="5cbx1us6n7tpn5iv0a33p6iyq0otupsp", oauth_token="nvoankn2g8jraga3k22ax9eyf55lqk9e", oauth_signature="Vl8tCEx%2F3abTjQFcxSfWcpzQ%2FHE%3D"
Cookie: private_content_version=c6f9e0919d7028c78ee63e79593ef689; mage-messages=%5B%5D
DNT: 1
Connection: keep-alive
In Delphi we always got the Accept-Encoding "identity", which we aren't able to delete.
We deleted the Accept-Charset, which is in the request from Delphi but it doesn't make a difference.
Now we don't have any clues what else we can do. Maybe anyone of you had the same problem and could help us? Or maybe one of you has any ideas, what we could try?
how can I use Fiddler to pass 2 parameters to a method with a signature like this :
[WebInvoke(UriTemplate = "Login", Method = "POST", ResponseFormat = WebMessageFormat.Xml, BodyStyle = WebMessageBodyStyle.WrappedRequest)]
bool Login(string login, string password);
Any suggestion to change something in the method is welcome, but I have always to pass 2 string parameters.
It is much easier to the services with WCFTestClient. Anyway, Fiddler has composer functionality which can be used to test any http calls.
Request Headers should look like below. Just change SOAPAction accordingly.
Content-Type: text/xml; charset=utf-8
SOAPAction: "http://tempuri.org/IService/Login"
Host: localhost:2045
Content-Length: 179
Expect: 100-continue
Accept-Encoding: gzip, deflate
Proxy-Connection: Keep-Alive
And request body:
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"><s:Body><Login xmlns="http://tempuri.org/"><login>sdf</login><password>sdf</password></Login></s:Body></s:Envelope>