Sending HttpRequest to 2 different applications and comparing their results and performance - rest

I have a module in an application 'A' that serves a HttpRequest and computes a redirect url. I have developed another application 'B' that was written in another language that would do the same thing as that module in A plus some more amount of computation and redirect the client to another webpage.
I require ideas on the following
Q. How do I check if B redirects to the correct Url as the module in A? ( do note that the module in A would compute some url but A might do some more additional computation and redirect to another url. I am only interested in comparing the output from the module vs B)
Q. How do I performance test B against A (load, response time, etc)?
Any ideas on this would be greatly appreciated
Thanks

You can achieve this in 2 ways, server side and client side.
For server side you can use Date object in java or any other defined way in the language you are using to get the start time for http request received and the end time after redirect url call.
Print these 2 times in log for both the cases.
For client side you can also write simple code on local machine for rest calls and log the times as I mentions for server side
Please find the demo code below:
Date start1 = new Date() ;
// first http call
//process data
// redirect call http call
Date end1 = new Date()
// print time for start1 and end1
Date start2= new Date() ;
// first http call
//process data
// redirect call http call
Date end2= new Date()
// print time for start2 and end2

Related

Is there a way to set headers for GET requests in KDB?

I'm trying to make get requested with .Q.hg (HTTP get), but I need to edit the request headers to provide API keys. How can I do this?
You can try this function I wrote a few years back for a POC (similar reason - I needed to supply multiple headers). It's based on .Q.hmb which underpins .Q.hp/hg. Please note - it was never extensively tested & there are likely better alternatives out there, but it will perhaps work as a quick solution.
k)req:{[url;method;hd;bd]d:s,s:"\r\n";url:$[10=#url;url;1_$url];p:{$[#y;y;x]}/getenv`$_:\("HTTP";"NO"),\:"_PROXY";u:.Q.hap#url;t:~(~#*p)||/(*":"\:u 2)like/:{(("."=*x)#"*"),x}'","\:p 1;a:$[t;p:.Q.hap#*p;u]1;(4+*r ss d)_r:(`$":",,/($[t;p;u]0 2))($method)," ",$[t;url;u 3]," HTTP/1.1",s,(s/:("Connection: close";"Host: ",u 2),((0<#a)#,$[t;"Proxy-";""],"Authorization: Basic ",.Q.btoa a),($[#hd;(!hd),'": ",/:. hd;()])),($[#bd;(s,"Content-length: ",$#bd),d,bd;d])}
It takes 4 arguments:
Resource URL
HTTP method
Dictionary of headers
Message body as JSON object
Sending a request to a test server..
q).j.k req["https://httpbin.org/get";`GET;("Content-Type";"someOtherHeader")!(.h.ty`json;"blah");""] // no body so pass empty string
args | (`symbol$())!()
headers| `Content-Type`Host`Someotherheader`X-Amzn-Trace-Id!("application/jso..
url | "https://httpbin.org/get"

parse kdb+ HTTP response type

I am trying to communicate with kdb+ via HTTP. I succeeded to get the the response from the DB when I did it from the browser, but for some reason, the response is neither JSON nor anything that looks machine readable.
What I did:
Opened a port on q console (8080)
Defined a function getData that gives me the data I want.
The above steps can be presented in this .q file I created:
\p 8080
system["l db"]
getData: {[a;b] ?[table;((>=;`start;$[`long;a]);(<=;`end;$[`long;b]));0b;()]}
h:hopen `:localhost:8080:user:pass
I then typed http://localhost:8080/?getData[1;2] in the browser to get the results
Got the results as text with spaces inside <pre> tag.
The Problem
I have no I idea how to parse it to JSON. How can I solve this? is there a way to tell kdb+ to send response in JSON format?
The solution I found:
If you add .json before the ? in the url (i.e. http://localhost:8080/.json?getData[1;2]) the response will be in JSON.
It looks like, based on your code, the following may work to return data:
http://localhost:8080/?getData[a;b]
Where a and b are start and end times (or dates, this will depend on your data).
e.g.
http://localhost:8080/?getData[2021.07.23;2021.07.30]
In order to convert a KDB+ object to json you can make use of the .j.j operator.
For example you can change your code to something like:
getData:{[a;b] t:?[table;((>=;`start;$[`long;a]);(<=;`end;$[`long;b]));0b;()];.j.j t}
If you want to see what the incoming request looks like on your Q process you could add some logging to .z.ph (the default http message handler), e.g. something like:
.z.ph:{[zph;x]zph 0N!x}[.z.ph]
This will log any incoming requests out to your q process using 0N! and then run the usual .z.ph message handing code on it

What's the use case for gRPC where it could definitely overcome REST

I made up simple benchmarking for the simpliest case: sending string Hello world over gRPC and rest in ruby:
# REST example
require 'sinatra'
set :bind, '0.0.0.0'
set :logging, false
get '/' do
'Hello, world!'
end
gRPC example is based on official examples
// The greeting service definition.
service Greeter {
// Sends a greeting
rpc SayHello (HelloRequest) returns (HelloReply) {}
}
// The request message containing the user's name.
message HelloRequest {
string name = 1;
}
// The response message containing the greetings
message HelloReply {
string message = 1;
}
class GreeterServer < Helloworld::Greeter::Service
def say_hello(hello_req, _unused_call)
Helloworld::HelloReply.new(message: "Hello #{hello_req.name}")
end
end
deployed this code to remote server and run 1000 requests benchmark (ab for rest and looping client requests for gRPC) and get comparable results 51 sec vs 53 (REST vs gRPS)
so, I made up conclusion that in that case (with small amount of data in response) there is no benefits to gRPC. So, when would they appear? When data size would be magnitude of kilobytes or even megabytes? Or there are essentially different use cases for gRPC like streaming and duplexing data between server and client?
This blog post indicates that gRPC performs better while being slightly harder to use.
I'm think the improved performance comes from using protocol buffers for data transmission; I believe that means data is transmitted in binary format which would mean improved performance when you have more data, particularly non-string data.

Play 2.3 WS withFollowRedirects(true) does not work

The following code should post a form to an endpoint (which returns 302) and, after following the redirect, parse the url of the page and return some information from there.
val start = System.currentTimeMillis()
val requestHolder = WS.url(conf("login.url"))
.withRequestTimeout(loginRequestTimeOut)
.withFollowRedirects(true) //This appears to have no effect...
requestHolder.post(getMap(username, password))
.map(resp =>{
Logger.debug(resp.status.toString)
val loginResponse = getResponse(resp)
val end = System.currentTimeMillis()
Logger.debug("Login for the user: "+username+", request took: " + (end - start) + " milliseconds.")
loginResponse
})
The problem is that .withFollowRedirects(true) appears to have no effect on the query. The status of the response is 302 and the request does not follow the redirect.
I've gone through the process manually using httpie and following the redirects does lead to the correct page.
Any help or insight would be much appreciated.
POST redirection isn't as well supported as GET redirection. W3 specification says:
If the 301 status code is received in response to a request other than GET or HEAD, the user agent MUST NOT automatically redirect the request unless it can be confirmed by the user, since this might change the conditions under which the request was issued.
Some browsers don't do that, and just ignore. Have a look also at the 307 status:
307 Temporary Redirect (since HTTP/1.1)
In this case, the request should be repeated with another URI; however, future requests should still use the original URI. In contrast to how 302 was historically implemented, the request method is not allowed to be changed when reissuing the original request. For instance, a POST request should be repeated using another POST request.
There is also a discussion about this on Programmer Stack Exchange.
I've had a lot of trouble with withFollowRedirects and POST.
At some point, while fighting to make things work, I had .withFollowRedirects(false) in my code, then removed it during cleanups & things broke. My current guess is that if this option is not explicitly made false, the default behavior is to follow redirects (302 in my case) with some faulty mechanism. Perhaps the default mechanism uses POST again with same arguments. But in my case, interacting with Google App Script (GAS), one needs to use GET to retrieve JSON output of a POST.
Whatever the mechanism was doing, I was getting 400 with no further diagnostics.
After wasting hours, I realized that .withFollowRedirects(false) was in fact truly needed: it disabled Play's messing with redirects, I was able to see the 302 response & handle the following GET manually with success.

Play framework - retrieving the Date header in the request

I need to access the Date: header when I handle the request, but this seems to be "swallowed" by the framework; any other header (even made up FooBar ones) show up and I can get them, but this gives me None (I'm using Postman to send a simple GET request - everything else works just fine):
println("Date: " + request.headers.get("Date").getOrElse("no date!"))
returns "no date!" no matter how I try to send something sensible.
I'm wondering whether this gets processed before the request object reaches my Action.
I need the actual string value sent, as this should be part of the request's signature - so an equivalent Date object representing the same value would not be of much use (as it needs to be part of the hash, to avoid replay attacks).
Just as a test, I replaced the Date header with a Date-Auth one, and this one shows up just fine:
ArrayBuffer((Date-Auth, ArrayBuffer(Wed, 15 Nov 2014 06:25:24 GMT))
Any ideas or suggestions greatly appreciated!
Are you sure there is a Date Header in your request (tested with tools like firebug or wireshark)?
Browsers do not need to send a Date header.
RFC 2616 (HTTP 1.1) from the Date section (14.18)
Clients SHOULD only send a Date header field in messages that include an entity-body, as in the case of the PUT and POST requests, and even then it is optional. A client without a clock MUST NOT send a Date header field in a request.
I stand corrected - it turns out that Chrome blocks a whole bunch of headers:
http://www.getpostman.com/docs/requests
I wrote a Python Flask test server and, in fact, the Date header is not there.
That page has also a fix, which works just fine with Postman Version 0.10.4.3 and Interceptor(1).
sorry for wasting everyone's time!
1 Incidentally, IMO Postman is the best REST client and has now also some awesome looks, beyond incredible functionality. If you're working with REST APIs, I highly recommend it.