Attempting to get a spray endpoint negotiating a postman request for content. My XML marshaller seems to let me down, ie, it never resolves the correct marshaller option based on Accept header and charset.
I have the following:
object ResponseVO {
val NodeSeqMarshaller = Marshaller.delegate[ResponseVO, NodeSeq](ContentType(`text/xml`, `UTF-8`)) { respVO => <ussd><type>{ respVO.respType }</type><message>{ respVO.message }</message></ussd> }
val supportedContentTypes = List[ContentType](ContentType(`text/xml`, `UTF-8`))
implicit val marshaller = Marshaller[ResponseVO] { (respVO, ctx) =>
ctx.tryAccept(supportedContentTypes) match {
case Some(ContentType(`text/xml`, `UTF-8`)) => NodeSeqMarshaller(respVO, ctx)
case whatever => println(whatever); ctx.rejectMarshalling(supportedContentTypes);
}
}
}
and the following route:
trait USSDRoute {
this: SimpleRoutingApp with BootStrappedActorSystem =>
val ussdRoute = path("ussd") {
parameters('msisdn.as[Long], 'session.as[String], 'type.as[Int], 'msg.as[String], 'network.as[Int]) { (msisdn, session, reqType, msg, network) =>
complete {
val reqVO = RequestVO(msisdn, session, reqType, msg, network)
println(s"received $reqVO")
ResponseVO(1, "Spirit midget medium escapes from prison, headlines read: Small medium at large!")
}
}
}
}
Unfortunately I never seem to negotiate correctly, ie, my marshaller resolution will dive bomb into the "whatever" block and respond with a 406
"Resource representation is only available with these Content-Types:
text/xml; charset=UTF-8"
I am using Postman and my request headers read:
GET /ussd?msisdn=0794138690&type=1&network=4&msg=hello&session=99 HTTP/1.1
Host: localhost:9999
Connection: keep-alive
Accept: text/xml; charset=UTF-8
Cache-Control: no-cache
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML,
like Gecko) Chrome/40.0.2214.91 Safari/537.36
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8,nl;q=0.6
It's probably something small and silly - hopefully someone can help steer me in the right direction.
Thank you
Have a look at how the case class ContentType is defined:
case class ContentType(mediaType: MediaType, definedCharset: Option[HttpCharset])
Note that definedCharset is defined as an Option[HttpCharset], therefore your pattern match via
case Some(ContentType(`text/xml`, `UTF-8`)) => ...
never can succeed. So you need to use a Some to perform the pattern match successfully:
case Some(ContentType(`text/xml`, Some(`UTF-8`))) => ...
Related
I have a question, for security purposes, it is sometimes very useful to make HTTP requests with illegal paths, such as: http://localhost/%x, in Golang it does not seem to be a way to do that, could anyone give some light to this issue?
What I have tried so far:
- .Get() function
response, err := http.DefaultClient.Get("http://localhost/%x")
if err != nil{
panic(err) // net/url.EscapeError
}
.Do() function
requestObject, err := http.NewRequest("GET", "http://localhost/%x", nil)
if err != nil {
panic(err) // panics!
}
And
requestObject, err := http.NewRequest("GET", "http://localhost", nil)
if err != nil {
panic(err) // Not panic
}
requestObject.URL.Path = "/%x"
response, err = netClient.goHttpClient.Do(requestObject)
if err != nil {
panic(err) // Not panic
}
This last one looks promising, but request is made to %25x instead of %x, can someone give some help please?
UPDATE:
Go can send %x in the request, in the query section though, not in the path, I want to send %x in the path, so my issue is not solved:
requestObject, err := http.NewRequest("GET", "http://localhost", nil)
if err != nil {
panic(err) // Not panic
}
requestObject.URL.Path = "/%x"
requestObject.URL.RawQuery = "%x"
Request is made to http://localhost/%25x?%x The %x in the path is encoded, in the query it is not.
UPDATE:
The answer given #Cerise is the one I use, and it works fine, but there are edge cases to take into account.
1 - Proxy: If you have to forward the requests to a proxy, for some reason, the HTTP framework does not know how to send HTTP requests that your proxy can understand, to fix this, in Burp suite, you have to enable invisible proxy as documented here: https://portswigger.net/support/using-burp-suites-invisible-proxy-settings-to-test-a-non-proxy-aware-thick-client-application
This still has an issue, I noticed it when I had to debug some HTTP requests, I could not make sense, of what Burp Suite was telling me, then inspecting the requests in Wireshark I knew there was something wrong, if you use .Opaque + BurpSuite, BurpSuite will modify some requests, this is where the second issue comes into play, keep reading.
2 - Opaque does not handle well the paths starting with "//", I don't think it is a bug though, it is well known that https://stackoverflow.com can also be rewritten as //stackoverflow.com, I think this the logic taken into account by go, anyway, if you do:
requestObject, err := http.NewRequest(methodStr, "http://localhost", nil)
requestObject.URL.Opaque = "//something"
This is what the HTTP request looks like with Wireshark:
GET http://something HTTP/1.1
Host: localhost
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.116 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Language: en-US,en;q=0.9,ca;q=0.8
Cache-Control: no-cache
Connection: close
Pragma: no-cache
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
Accept-Encoding: gzip
notice the first line, it is not what we may have though first, right?
I used to debug HTTP requests with Burp Suite, but let's use the same code, and forward the requests to Burp using:
http.DefaultTransport.(*http.Transport).Proxy = http.ProxyURL("http://localhost:8080")
Don't forget to enable invisible proxy on BurpSuite, otherwise it will reject the request. What Burp Suite shows is:
GET / HTTP/1.1
Host: localhost
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.116 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Language: en-US,en;q=0.9,ca;q=0.8
Cache-Control: no-cache
Connection: close
Pragma: no-cache
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
Accept-Encoding: gzip, deflate
outch!!, Burp has changed our request completely!!!
Takeaways:
- Do not use Your HTTP proxy to debug requests made with Opaque
- Do not use .Opaque with paths starting with double slashes, use the following:
if strings.HasPrefix(yourPath, "//") {
requestObject.URL.Path = fmt.Sprintf("%s", yourPath)
} else {
requestObject.URL.Opaque = fmt.Sprintf("%s", yourPath)
}
Set Request.URL.Opaque to the invalid path. The Opaque data is written to the network as is.
requestObject, err := http.NewRequest("GET", "http://localhost", nil)
if err != nil {
panic(err)
}
requestObject.URL.Opaque = "/%x"
response, err = netClient.goHttpClient.Do(requestObject)
if err != nil {
panic(err)
}
This code creates an invalid request. The net/http server handles the invalid request by responding with 400 Bad Request. Other servers and proxies may do something different.
Run it on the playground
The code used to work.
The URL in question is
https://yobit.net/api/3/info
It works in IE. It used to work with webclient. It doesn't work in webclient now. And I wonder what the problem is
Suddenly it stops working. So I am checking
Try
Dim wc = New WebClient
wc.Headers.Add("Accept", "text/html, application/xhtml+xml, image/jxr, */*")
wc.Headers.Add("Accept-Encoding", "gzip, deflate")
wc.Headers.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko")
wc.Headers.Add("Accept-Language", "en-US,en;q=0.5")
wc.DownloadString(URL)
Catch ex As Exception
End Try
I also tried simple version. Not working
Try
Dim wc = New WebClient
wc.DownloadString(URL)
Catch ex As Exception
End Try
In both cases yobit throw 503 access denied exception
I use fiddler and try to use internetexplorer to access directly
It works fine
GET https://yobit.net/api/3/info HTTP/1.1
Accept: text/html, application/xhtml+xml, image/jxr, */*
Accept-Language: en-US,en;q=0.5
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko
Accept-Encoding: gzip, deflate
Host: yobit.net
Connection: Keep-Alive
Cookie: __cfduid=de63c60d603f271520b9ee58dfdd257061517932785; cf_clearance=7e58588df28b267842f753567dcdc475d29679a6-1517932789-86400; locale=en
If I use webclient this is the header
GET https://yobit.net/api/3/info HTTP/1.1
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.5
Accept: text/html, application/xhtml+xml, image/jxr, */*
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko
Host: yobit.net
Connection: Keep-Alive
Almost the exact same thing.
Let me try another URL
Say http://google.com
GET http://www.google.com/ HTTP/1.1
Accept: text/html, application/xhtml+xml, image/jxr, */*
Accept-Language: en-US,en;q=0.5
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko
Accept-Encoding: gzip, deflate
Host: www.google.com
Connection: Keep-Alive
Cookie: NID=121=GUd4VKHT_gcwUx-hK39mphuCg93Q_W2fL_yCc-JO3AJkgh74EGajif0537eraLK8ns2EdEQPexOOeBxSlOxVrj8t_AVn21FRme2hAxuLXz4F8aCZExIzME4jaYMBuUp_lnak5Q; OGPC=19004116-3:; 1P_JAR=2018-1-9-7
If I use webclient
GET http://google.com/ HTTP/1.1
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.5
Accept: text/html, application/xhtml+xml, image/jxr, */*
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko
Host: google.com
Connection: Keep-Alive
Both works.
There could be different issues here
Framework Version
You recently changed the framework version and that has some issue which causes it not to work. You can change the framework versions and see if it helps
Compression of Data
WebClient by default is not doing decompression and you add headers to request gzip data if available. Now there is a possibility that the site didn't enable gzip earlier which made it work for you and now they have enabled gzip responses. You can fix that by changing
wc.Headers.Add("Accept-Encoding", "gzip, deflate")
to
wc.Headers.Add("Accept-Encoding", "deflate")
Or if you want the data to come in compressed form only, then you can enable auto decompression as shown in below code
class Program
{
class MyWebClient : WebClient
{
protected override WebRequest GetWebRequest(Uri address)
{
var request = base.GetWebRequest(address) as HttpWebRequest;
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
return request;
}
}
public static void Main(string[] args)
{
var URL = "https://yobit.net/api/3/info";
var wc = new WebClient();
wc.Headers.Add("Accept", "text/html, application/xhtml+xml, image/jxr, */*");
wc.Headers.Add("Accept-Encoding", "gzip, deflate");
wc.Headers.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko");
wc.Headers.Add("Accept-Language", "en-US,en;q=0.5");
Console.WriteLine(wc.DownloadString(URL));
// TODO: Implement Functionality Here
Console.Write("Press any key to continue . . . ");
Console.ReadKey(true);
}
}
The above example is C# and not VB.NET but you can easily convert it
I have a problem with my requests using Edge/IE. I'm getting a code 400 code on
preflight OPTIONS.
I'm adding a 2 additional headers to all my requests in application with axios:
Authorization:
if ( token ) {
axios.defaults.headers.common[ 'Authorization' ] = 'Bearer ' + token;
} else {
delete axios.defaults.headers.common[ 'Authorization' ];
}
Accept-Language:
if ( lang ) {
axios.defaults.headers.common[ 'Accept-Language' ] = lang;
} else {
delete axios.defaults.headers.common[ 'Accept-Language' ];
}
Here's how my request headers works in Chrome:
Accept-Language: pl-PL
Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJOZXcgUGFzdG9yYWwgQXBwIiwiYXVkIjoiTmV3IHBhc3RvcmFsIHVzZXIiLCJpYXQiOjE1MTM1OTEzODUsImV4cCI6MTUxMzU5ODU4NSwidWlkIjoxMSwidHlwZSI6NCwibGFuZ3VhZ2UiOjEsImRpc3BsYXlOYW1lIjoiVGVzdG8gUHJvYm9zemN6eiJ9.qyFHZuCHC9-NpNqjY7keY7j4LE5zu5JKxK9VyAFwH60
Request headers in Edge:
Accept: */*
Accept-Encoding: gzip, deflate
Access-Control-Request-Headers: content-type
Access-Control-Request-Method: POST
Cache-Control: no-cache
Connection: Keep-Alive
Content-Length: 0
Host: 52.169.186.226
Origin: http://localhost:8080
Referer: http://localhost:8080/
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 Edge/16.16299
Response headers:
Access-Control-Allow-Headers: X-Requested-With, Content-Type, Accept, Origin, Authorization
Access-Control-Allow-Methods: OPTIONS,POST
Access-Control-Allow-Origin: *
Connection: close
Content-Length: 0
Content-Type: text/html; charset=UTF-8
Date: Mon, 18 Dec 2017 11:29:49 GMT
Server: Apache/2.4.25 (Ubuntu)
When I remove Accept-Language header the application start working in Edge, but problem still exists in IE.
The only messages from browser is code http error 400 and the console log of error is:
description: "Network Error"
message: "Network Error"
stack: "Error: Network Error at createError (http://localhost:8080/index_bundle.js:68811:3) at handleError (http://localhost:8080/index_bundle.js:68656:7)"
Rest of the headers is standard for axios. On the backend we're using Slimp PHP framework.
Any idea how to debug the problem, or similar problems using standard REST requestes with React + Axios? Or what should I check in first order?
I got a form in my application. I input there russian and chineese symbols. Http request from browser to application looks like this:
POST /payout/process HTTP/1.0
Host: my_host
X-Real-IP: 10.3.4.6
X-Forwarded-For: 10.3.4.6
X-Forwarded-Proto: https
Connection: close
Content-Length: 286
Accept: */*
Origin: https://my_host
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/43.0.2357.130 Chrome/43.0.2357.130 Safari/537.36
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Referer: https://my_host
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8,ru;q=0.6
Cookie: piwikExternalId=; COOKIEPOO="d14ef459332e84ab12f8187181e01e50df3abd61-5d635c8478334352d2865b11dd747d6d3fe84704=1&d6e2c1ad27a45b7d9b8ed7b039ab96c7d3ef8051=1&7cf1199693fa6d40bf3654db17ec4773e0d61f38=1"
group=27&qwipi.customer_name=%D1%84%D1%8B%D0%B2&qwipi.bank_code=%E6%8B%9B%E5%95%86%E9%93%B6%E8%A1%8C&qwipi.bank_branch=%D1%84%D0%B2%D0%B0&customer_purse=%D1%84%D1%8B%D0%B2%D0%B0&qwipi.bank_city=%D1%84%D0%B2%D1%8B%D0%B0&qwipi.bank_province=%E6%8B%9B%E5%95%86%E9%93%B6%E8%A1%8C&amount=546
Check out request body, parameter customer_name. Decoded it will give фыв. Chec out paramter bank_branch. Decoded it will give 招商银行.
I log the request to console and graylog like this:
def logRequest(request: Request[AnyContent]) {
request.body match {
case AnyContentAsFormUrlEncoded(data) => Logger.info(data.toString())
case AnyContentAsText(text) => Logger.info(text + ", ")
case other => Logger.info(other.toString())
}
}
and I get something like
body: 'qwipi.bank_code' => '??????', 'qwipi.bank_province' => '??????', 'qwipi.customer_name' => '??????', 'amount' => '654', 'qwipi.bank_branch' => '??????', 'qwipi.bank_city' => '??????'
i.e., all non-latin symbols turn to question signs.
echo $LANG outputs en_US.UTF-8, so I guess my application's encoding (whatever that means -- application's itself or java's or playframework's or smth else) is different.
This is what I tried:
def logRequest(request: Request[AnyContent]) {
Logger.info(new String(appendRequestData(request).getBytes(StandardCharsets.UTF_8), StandardCharsets.UTF_8))
}
i.e., got the byte array in UTF-8 and turn it into the String which is in UTF-8 as well. But nothing changed.
What I also tried was starting an application passing -Dfile.encoding=UTF8 key -- the result still the same.
So how can I make my stuff logged properly? What things I might have missed?
It didn't help converting my string from one encoding to another on the fly. It didn't help passing UTF-8 to scalac option in Build.scala like that
So I just passed -Dfile.encoding=UTF-8 when starting my java service.
how can I use Fiddler to pass 2 parameters to a method with a signature like this :
[WebInvoke(UriTemplate = "Login", Method = "POST", ResponseFormat = WebMessageFormat.Xml, BodyStyle = WebMessageBodyStyle.WrappedRequest)]
bool Login(string login, string password);
Any suggestion to change something in the method is welcome, but I have always to pass 2 string parameters.
It is much easier to the services with WCFTestClient. Anyway, Fiddler has composer functionality which can be used to test any http calls.
Request Headers should look like below. Just change SOAPAction accordingly.
Content-Type: text/xml; charset=utf-8
SOAPAction: "http://tempuri.org/IService/Login"
Host: localhost:2045
Content-Length: 179
Expect: 100-continue
Accept-Encoding: gzip, deflate
Proxy-Connection: Keep-Alive
And request body:
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"><s:Body><Login xmlns="http://tempuri.org/"><login>sdf</login><password>sdf</password></Login></s:Body></s:Envelope>