parse kdb+ HTTP response type - kdb

I am trying to communicate with kdb+ via HTTP. I succeeded to get the the response from the DB when I did it from the browser, but for some reason, the response is neither JSON nor anything that looks machine readable.
What I did:
Opened a port on q console (8080)
Defined a function getData that gives me the data I want.
The above steps can be presented in this .q file I created:
\p 8080
system["l db"]
getData: {[a;b] ?[table;((>=;`start;$[`long;a]);(<=;`end;$[`long;b]));0b;()]}
h:hopen `:localhost:8080:user:pass
I then typed http://localhost:8080/?getData[1;2] in the browser to get the results
Got the results as text with spaces inside <pre> tag.
The Problem
I have no I idea how to parse it to JSON. How can I solve this? is there a way to tell kdb+ to send response in JSON format?

The solution I found:
If you add .json before the ? in the url (i.e. http://localhost:8080/.json?getData[1;2]) the response will be in JSON.

It looks like, based on your code, the following may work to return data:
http://localhost:8080/?getData[a;b]
Where a and b are start and end times (or dates, this will depend on your data).
e.g.
http://localhost:8080/?getData[2021.07.23;2021.07.30]
In order to convert a KDB+ object to json you can make use of the .j.j operator.
For example you can change your code to something like:
getData:{[a;b] t:?[table;((>=;`start;$[`long;a]);(<=;`end;$[`long;b]));0b;()];.j.j t}
If you want to see what the incoming request looks like on your Q process you could add some logging to .z.ph (the default http message handler), e.g. something like:
.z.ph:{[zph;x]zph 0N!x}[.z.ph]
This will log any incoming requests out to your q process using 0N! and then run the usual .z.ph message handing code on it

Related

Is there a way to set headers for GET requests in KDB?

I'm trying to make get requested with .Q.hg (HTTP get), but I need to edit the request headers to provide API keys. How can I do this?
You can try this function I wrote a few years back for a POC (similar reason - I needed to supply multiple headers). It's based on .Q.hmb which underpins .Q.hp/hg. Please note - it was never extensively tested & there are likely better alternatives out there, but it will perhaps work as a quick solution.
k)req:{[url;method;hd;bd]d:s,s:"\r\n";url:$[10=#url;url;1_$url];p:{$[#y;y;x]}/getenv`$_:\("HTTP";"NO"),\:"_PROXY";u:.Q.hap#url;t:~(~#*p)||/(*":"\:u 2)like/:{(("."=*x)#"*"),x}'","\:p 1;a:$[t;p:.Q.hap#*p;u]1;(4+*r ss d)_r:(`$":",,/($[t;p;u]0 2))($method)," ",$[t;url;u 3]," HTTP/1.1",s,(s/:("Connection: close";"Host: ",u 2),((0<#a)#,$[t;"Proxy-";""],"Authorization: Basic ",.Q.btoa a),($[#hd;(!hd),'": ",/:. hd;()])),($[#bd;(s,"Content-length: ",$#bd),d,bd;d])}
It takes 4 arguments:
Resource URL
HTTP method
Dictionary of headers
Message body as JSON object
Sending a request to a test server..
q).j.k req["https://httpbin.org/get";`GET;("Content-Type";"someOtherHeader")!(.h.ty`json;"blah");""] // no body so pass empty string
args | (`symbol$())!()
headers| `Content-Type`Host`Someotherheader`X-Amzn-Trace-Id!("application/jso..
url | "https://httpbin.org/get"

Bittrex websockets encoding method?

It´s not a big deal to mimic the websocket connection made to bittrex from www, using chromes dev tools:
GET https://socket.bittrex.com/signalr/negotiate?clientProtocol=1.5&connectionData=%5B%7B%22name%22%3A%22c2%22%7D%5D&_=1524596108843
This return a token
"ConnectionToken":"gbLsm8C6Jck1mQTTFjmuIv5qgUuMZz/kXU1s+fAnjnW
qUFQocNBfp3VOrd/y0acxWL5Fv7MZ54heRddLYZS+EMhLnaPPQiSZblvgJPCbLKqZTIkb"
That needs to be used with:
wss://socket.bittrex.com/signalr/connect?
transport=webSockets&clientProtocol=1.5&connectionToken=
Then there are some commands sent to the websocket:
{"H":"c2","M":"SubscribeToSummaryLiteDeltas","A":[],"I":0}
{"H":"c2","M":"SubscribeToExchangeDeltas","A":["BTC-TRX"],"I":1}
{"H":"c2","M":"QueryExchangeState","A":["BTC-TRX"],"I":2}
Which makes bittrex stream data about the selected ticker, BTC-TRX. But here´s where things start to get odd:
Here´s what frames look like:
{"C":"d-C95D047D-E,0|OHjr,0|OHjs,2|CW,6116","M":[{"H":"C2","M":"uE","A":["dY+7DsIwDEX/xXOI7NiOnYwwg0TpwENd+QnUfydtBQKp9ear4yPfFxyhwr4/7PruCgFOUJO4SoA71McL+lsLAnRQMeI0xhzg3EKXQpEFzUvyMayhiWaUOJtqLCZKonkcAly+cvyXl/lCHKOyaxbDdbUuamMXzhELk5lvvKE+s0yoyWPKVIhwg81LO8OcNCZOGQn5g9IPSuyLtq1To2drNIxv"]}]}
It certainly looks like base64, but decoding it gives me something like:
u���0E��s��؎��0�D��C]� �'m����#��¾?���
NP��J�;���[tP1�4���B�B��K�1���f�8�j,&J�y\�r����B��k�u�.jc���o��>�L��c�T�p��K;Ü4&N ���J싶�S�gk4�o
Did anyone figure out how to turn this into readable data?
To decode the encoded part of the response, proceed like so (below python 3.8.2 prompt):
>>> import zlib
>>> import base64
>>> val = b'dY+7DsIwDEX/xXOI7NiOnYwwg0TpwENd+QnUfydtBQKp9ear4yPfFxyhwr4/7PruCgFOUJO4SoA71McL+lsLAnRQMeI0xhzg3EKXQpEFzUvyMayhiWaUOJtqLCZKonkcAly+cvyXl/lCHKOyaxbDdbUuamMXzhELk5lvvKE+s0yoyWPKVIhwg81LO8OcNCZOGQn5g9IPSuyLtq1To2drNIxv'
>>> zlib.decompress(base64.b64decode(val), -15)
b'{"M":"BTC-TRX","N":24854,"Z":[{"TY":2,"R":0.00000733,"Q":28491.34078928},{"TY":2,"R":0.00000721,"Q":136755.97451456}],"S":[{"TY":0,"R":0.00000739,"Q":480.53856470},{"TY":2,"R":0.00000751,"Q":738436.09317788},{"TY":2,"R":0.00000758,"Q":310528.26191108},{"TY":2,"R":0.00000763,"Q":70625.23260103},{"TY":1,"R":0.00001388,"Q":0.0}],"f":[]}'
Note the -15 as second argument to the decompress method. With that argument, the decompress function corresponds to rawInflate in equivalent javascript libraries.
The response appears to be (at least in a Python world) subject to b64decode() and decompress().
message = decompress(b64decode(message))

Using variables in SOAP request file in JMeter

In a JMeter (v2.13) test plan I have a SOAP/XML-RPC sampler. The SOAP request itself is loaded from a random file.
Sample request
<mySoapRequest>
<value>555</value>
</mySoapRequest>
This works fine.
I would now like to replace this fixed value with a variable which is defined in JMeter, i.e.
<mySoapRequest>
<value>${someValue}</value>
</mySoapRequest>
It seems as if JMeter does not resolve this variable. The actual SOAP request sent to the service does not contain 555 but ${someValue}. Is there any workaround so that I could use variables in the file?
That can be done using FileToString and eval functions.
For this XML,
<mySoapRequest>
<value>${someValue}</value>
</mySoapRequest>
In the SOAP/XML RPC request Data section, use the functions as shown below to get the value replaced at run time.
${__eval(${__FileToString(C:\users\me\desktop\soap.xml)})}
__FileToString - The FileToString function can be used to read an entire file. Each time it is called it reads the entire file.
__eval - The eval function returns the result of evaluating a string expression.

GoodData Export Reports API Call results in incomplete file

I've developed a method that does the following steps, in this order:
1) Get a report's metadata via /gdc/md//obj/
2) From that, get the report definition and use that as payload for a call to /gdc/xtab2/executor3
3) Use the result from that call as payload for a call to /gdc/exporter/executor
4) Perform a GET on the returned URI to download the generated CSV
So this all works fine, but the problem is that I often get back a blank CSV or an incomplete CSV. My workaround has been to put a sleep() in between getting the URI back and actually calling a GET on the URI. However, as our data grows, I have to keep increasing the delay on this, and even then it is no guarantee that I got complete data. Is there a way to make sure that the report has finished exporting data to the file before calling the URI?
The problem is that export runs as asynchronous task - result on the URL returned in payload of POST to /gdc/exporter/executor (in form of /gdc/exporter/result/{project-id}/{result-id}) is available after exporter task finishes its job.
If the task has not been done yet, GET to /gdc/exporter/result/{project-id}/{result-id} should return status code 202 which means "we are still exporting, please wait".
So you should periodically poll on the result URL until it returns status 200 which will contain a payload (or 40x/50x if something wrong happened).

BigCommerce API Update Order with PUT

I need to update an order which is done via PUT method passing the order id as part of the https url string and a single parameter, the status_id.
https://mystore.mybigcommerce.com/orders/12345.json
I have tried several methods to pass the status_id value but no matter what I try "status_id=12" or formatted as JSON "{"status_id": 12,}" I always get the same response:
[{"status":415,"message":"The specified input content type is not valid."}]
I have also tried as a POST request passing the JSON or XML code as raw data but that method is not supported.
How am I supposed to pass that field=value pair? can I embed it in the url string?
I also tried it but it wouldn't work for me.
Any ideas?
In case you are wondering I am doing it within FileMaker with TROIUrl plugIn, not a very popular technology, but the GET method retrieving orders works like a charm
TURL_Put( ""; $url ;"status_id=12") (I have also tried other FM plugIns to no avail)
Don't get too caught up in the Filemaker part, I don't expect many people out there to be familiar with BigCommerce and Filemaker. I just need a generic answer.
Thanks
Commandline tool curl is worth a try. It supports put and https.
Mac OS X: curl already installed, call from FileMaker via AppleScript do shell script.
Windows: must be installed, call via Powershell.
It works for me using { "status_id": "3" } which means you probably need to put quotes around the actual number.
Also, it is a PUT operation and application/json which is part of the request content.
The error message received by the OP:
[{"status":415,"message":"The specified input content type is not valid."}]
Is saying that he did not supply the 'Content-Type' header in his request or that the header supplied is for a content type that is not allowed. For the OP's case using JSON he would need to include the header:
Content-Type: application/json
in his HTTPS request. This description can be found along with those of the other status codes you may see here:
https://developer.bigcommerce.com/api/status-codes