KDB string concatenation with symbol list for dynamic query - select

In this link, there is an example on how to include a dynamic parameter. d, in a KDB select query:
h: hopen`:myhost01:8012 // open connection
d: 2016.02.15 // define date var
symList: `GBPUSD`EURUSD
h raze "select from MarketDepth where date=", string d, ", sym in `GBPUSD`EURUSD" // run query with parameter d
Here d is of type date and is easy to string concatenate in order to generate a dynamic query.
If I want to add symList as a dynamic parameter as well by converting to string:
raze "select from MarketDepth where date=", string d, ", sym in ", string symList
The concatenated string becomes: select from MarketDepth where date=2016.02.15, sym in GBPUSDEURUSD, in other words the string concatenation loses the backticks so the query does not run. How can I solve this?
p.S: I know about functional querying but after failing for 2 hours, I have given up on that.

No need for functional selects.
q)MarketDepth:([] date:9#2016.02.15; sym:9#`A`B)
q)d:2016.02.15
q)symList:`B
q)h ({[dt;sl] select from MarketDepth where date=dt,sym in sl}; d; symList)
date sym
--------------
2016.02.15 B
2016.02.15 B
2016.02.15 B
2016.02.15 B

You are right, string SYMBOL does not preserve a backtick character, so you'll have to append it yourself like this:
symList: `GBPUSD`EURUSD
strSymList: "`",'string symList / ("`GBPUSD";"`EURUSD")
I used join , with each-both adverb ' to join a backtick with each element of a list. Having your symbol list stringified your dynamic query becomes
"select from MarketDepth where date=", (string d), ", sym in ",raze"`",'string symList
You can also use parse to see how a shape of a functional form of your query will look like.
q) parse "select from MarketDepth where date=", (string d), ", sym in ",raze"`",'string symList
(?;`MarketDepth;enlist ((=;`date;2016.02.15);(in;`sym;enlist `GBPUSD`EURUSD));0b;())
Now it's easy to create a functional select:
?[`MarketDepth;enlist ((=;`date;2016.02.15);(in;`sym;enlist symList));0b;()]
Hope this helps.
Update: #Ryan Hamilton's solution is probably the best in your particular scenario. You can even make a table name an argument if you want:
h({[t;d;s]select from t where date=d,sym in s};`MarketDepth; d; symList)
But it is worth noting that you can't use this technique when you need to make a list of columns dynamic. The following will NOT work:
h({[c;d;s]select c from t where date=d,sym in s};`time`sym; d; symList)
You will have to either build a dynamic select expression like you do or use functional forms.

Others have already given good alternative approaches for your problem. But in case if you need to join string and symbols (or other data types) without losing backtick, function .Q.s1 does the task.
q) .Q.s1 `a`b
q)"`a`b"
q)"select from table where sym in ",.Q.s1 symlist
Note: Generally it is not suggested to use .Q namespace functions.

Related

How can I query the same column in a kdb table multiple times in a single statement?

I have the following table in kdb...
p:([]r:("(A|A(A|B|C|D).*)";"A(E|F|G|H|I).*";"A(J|K|L|M).*";"A(N|O|P|Q|R|S).*";"A(T|U|V|W|X|Y|Z).*";"B.*";"(C|C(A|B|C|D|E).*)";"C(F|G|H|I|J|K).*";"C(L|M|N|O|P|Q|R).*";"C(S|T|U|V|W|X|Y|Z).*";"D.*"))
r
----------------------
"(A|A(A|B|C|D).*)"
"A(E|F|G|H|I).*"
"A(J|K|L|M).*"
"A(N|O|P|Q|R|S).*"
"A(T|U|V|W|X|Y|Z).*"
"B.*"
"(C|C(A|B|C|D|E).*)"
"C(F|G|H|I|J|K).*"
"C(L|M|N|O|P|Q|R).*"
"C(S|T|U|V|W|X|Y|Z).*"
"D.*"
and the below function that parses each row of the table...
getRange:{$[x like "*(*";
[if[x like "(*"; x2:1#1_x; l:enlist x2; x:-1_(3_x)];
l,:enlist {(3#x),"-",(-3#x)} ssr[ssr[ssr[x;".";""];")";"]"];"(";"["];
if[((count l)>1)&(l[1] like "*A-*"); l[1]:ssr[l[1]; "A-";"0-9/A-"]];
:l];
:enlist ssr[x;".";""]
];
}
Which gives an output like this...
r1:raze getRange'[exec r from p]
q)r1
,"A"
"A[0-9/A-D]*"
"A[E-I]*"
"A[J-M]*"
"A[N-S]*"
"A[T-Z]*"
"B*"
,"C"
"C[0-9/A-E]*"
"C[F-K]*"
"C[L-R]*"
"C[S-Z]*"
"D*"
I'm parsing the rows so they can be inserted into a query similar to something like select from t where sym like raze getRange'[exec r from p][0]
What I'd like to be able to do is combine the first "single A" with the first "group of A" and the same with the C's (so it looks like below). But the problem I'm having is that those results can't be easily inserted into a query...
(,"A";"A[0-9/A-D]*")
,"A[E-I]*"
,"A[J-M]*"
,"A[N-S]*"
,"A[T-Z]*"
,"B*"
(,"C";"C[0-9/A-E]*")
,"C[F-K]*"
,"C[L-R]*"
,"C[S-Z]*"
,"D*"
Is there a way in q that I can do this? Essentially, select from t where sym like (enlist "A";"A[0-9/A-D]*")
Please let me know if you need any additional info. Thank you in advance.
For matching against multiple regexps we can do following
select from t where any sym like/:("A";"A[0-9/A-D]*")

Complex list of list of empty characters in a column in kdb

I have a joined table which consists of list of list of characters.
q)t:([] a:`c`d; b:("";"fill"));
q)s:([] a:`b`c`c; b:("";"";""))
q)select from t lj select b by a from s
Output:
a b
---------
c ("";"") / This is the culprit want to replace it with null character
d "fill"
The output of join consists of a list of list of empty characters.
I want to replace that with empty character.
Expected output:
a b
---------
c ""
d "fill"
Tried: Few Unsuccessful attempts
q)update b:?[null in b;raze b;b]from select from t lj select b by a from s
q)update b:?["" in b;raze b;b]from select from t lj select b by a from s
To replace a list of list of empty strings with empty string, you can try below query:
q) select from t lj select (b;"")all""~/:b by a from s
Output:
a b
--------
c ""
d "fill"
Explanation:
Basically, empty strings list is coming from group command on the right table. So during the grouping stage, we can match if all the items in a grouped list (b column values) for particular a value are an empty string. And if they are just replacing them with a single empty string.
q) select (b;"")all""~/:b by a from s
a| b
-| --
b| ""
c| ""
For a = c , b grouped values are ("";""). Lets break down the command:
q) b:("";"")
q) ""~/:b / output 11b
q) all ""~/:b / output 1b
q)(b;"") all ""~/:b / output ""
The last command is list indexing. If the return from the previous command is 1b which means all items are empty strings, then return "" else return actual b.
Edit:
Based on the discussion in the comment section of TerryLynch's answer, it looks like your requirement is:
if all values of b list after grouping are empty strings then return a single empty string.
if values of b are a mixture of empty strings and non-empty strings, then remove all empty strings.
For that, you could use the below query:
q) select from t lj select b:raze ("";b except enlist "") by a from s
But that would result in different types for different values in b column. An empty string will be 10h and all non-empty string list will be 0h.
For consistent type, can use below query which returns enlist"" instead of "" but that will not be an empty string:
q) select from t lj select b:{(c;enlist "")()~c:x except enlist ""}b by a from s
Instead of trying to fix the adverse outcome I think you need to decide what you want to do with the duplicate c rows in the s table. You're grouping by the a column but it has duplicates so how should it behave .... should it take the first value, should it take the last value? Should it append the two strings together? If you solve that then you avoid this problem, for example:
q)t lj select last b by a from s
a b
--------
c ""
d "fill"
An alternative solution would be to simply raze all the results of b together. Less where clauses in use and less match (~) operations.
q)update raze'/[b] from (t lj select b by a from s)
a b
--------
c ""
d "fill"
Here I've used over to account for more an unknown level of enlistment, as a precaustion, and then applied it to each row from the lj. For your case, an even faster solution would be
update raze each b from (t lj select b by a from s)
This will give different results than Rahuls answers
q)update raze each b from (t lj select b by a from s)
a b
--------
c "str"
d "fill"
q) select from t lj select (b;"")all""~/:b by a from s
a b
------------
c ("";"str")
d "fill"
q)update raze each b from (t lj select b by a from s)
a b
--------
c "str"
d "fill"

SQL: Split post code value to return 'outer' part of code

I've got a results set of UK postcodes. Some are formatted with spaces, and some are not e.g. S14HG and S1 4HG
I want my select query to just return the outer part of the post code value in the results, i.e. 'S1'
I can do this in Excel using the following formula:
=IF(ISERROR(LEFT(A1,LEN(A1)-3)),””,LEFT(A1,LEN(A1)-3))
Is it possible to perform the same function in SQL through a SELECT query?
UK postcode can have one of many formats for their outward code.
However, as you can see from the possible formats in that link, there is a consistent format for the remainder of the postcode. If you are confident your postcodes are correct, you can simply remove any spaces and the last 3 characters:
declare #Postcodes table (Postcode nvarchar(10));
insert into #Postcodes values
('S1 4HG')
,('S14HG')
,('S10 4HG')
,('S104HG');
select Postcode
,replace(left(Postcode,len(Postcode)-3),' ','') as OutwardCode
from #Postcodes
Output:
Postcode OutwardCode
S1 4HG S1
S14HG S1
S10 4HG S10
S104HG S10
You can use LEFT() regardless of spaces since you only want the first two, which won't have a space.
SELECT LEFT('S1 4HG',2)
Or just get rid of the spaces...
declare #t varchar(64) = 'S 1 4 H G'
SELECT LEFT(REPLACE(#t,' ',''),2)

Get substring into a new column

I have a table that contains a column that has data in the following format - lets call the column "title" and the table "s"
title
ab.123
ab.321
cde.456
cde.654
fghi.789
fghi.987
I am trying to get a unique list of the characters that come before the "." so that i end up with this:
ab
cde
fghi
I have tried selecting the initial column into a table then trying to do an update to create a new column that is the position of the dot using "ss".
something like this:
t: select title from s
update thedot: (title ss `.)[0] from t
i was then going to try and do a 3rd column that would be "N" number of characters from "title" where N is the value stored in "thedot" column.
All i get when i try the update is a "type" error.
Any ideas? I am very new to kdb so no doubt doing something simple in a very silly way.
the reason why you get the type error is because ss only works on string type, not symbol. Plus ss is not vector based function so you need to combine it with each '.
q)update thedot:string[title] ss' "." from t
title thedot
---------------
ab.123 2
ab.321 2
cde.456 3
cde.654 3
fghi.789 4
There are a few ways to solve your problem:
q)select distinct(`$"." vs' string title)[;0] from t
x
----
ab
cde
fghi
q)select distinct(` vs' title)[;0] from t
x
----
ab
cde
fghi
You can read here for more info: http://code.kx.com/q/ref/casting/#vs
An alternative is to make use of the 0: operator, to parse around the "." delimiter. This operator is especially useful if you have a fixed number of 'columns' like in a csv file. In this case where there is a fixed number of columns and we only want the first, a list of distinct characters before the "." can be returned with:
exec distinct raze("S ";".")0:string title from t
`ab`cde`fghi
OR:
distinct raze("S ";".")0:string t`title
`ab`cde`fghi
Where "S " defines the types of each column and "." is the record delimiter. For records with differing number of columns it would be better to use the vs operator.
A variation of WooiKent's answer using each-right (/:) :
q)exec distinct (` vs/:x)[;0] from t
`ab`cde`fghi

KDB: select rows based on value in one column being contained in the list of another column

very simple, silly question. Consider the following table:
tt:([]Id:`6`7`12 ;sym:`A`B`C;symlist:((`A`B`M);(`X`Y`Z);(`H`F`C)))
Id sym symlist
---------------
6 A `A`B`M
7 B `X`Y`Z
12 C `H`F`C
I would like to select all rows in tt where the element in sym is contained in the list symlist. In this case, it means just the first and third rows. However, the following query gives me a type error.
select from tt where sym in symlist
(`type)
Whats the proper way to do this? Thanks
You want to use the ' (each-both) adverb, so that they "pair up" so to speak. Recall that sym is just list, and symlist is a list of lists. You want to check each element in sym with the respective sub-list in symlist. You do this by telling it to "pair up".
q)tt:([]id:6712; sym:`A`B`C; symlist:(`A`B`M;`X`Y`Z;`H`F`C))
q)select from tt where sym in'symlist
id sym symlist
----------------
6712 A A B M
6712 C H F C
It's not entirely clear to me why your query results in a type error, so I'd be interested in hearing other people's responses.
q)select from tt where sym in symlist
'type
in
`A`B`C
(`A`B`M;`X`Y`Z;`H`F`C)
q)select from tt where {x in y}[sym;symlist]
id sym symlist
--------------
In reponse to JPCs answer (couldn't format this as a comment)....
Type error possibly caused by applying "where" to a scalar boolean
q)(`a`b`c) in (`a`g`b;`u`i`o;`g`c`t)
0b
q)where (`a`b`c) in (`a`g`b;`u`i`o;`g`c`t)
'type
Also, the reason the {x in y} lambda doesn't cause the error is because the "in" is obscured and is not visible to the parser (parser doesn't look inside lambdas)
q)0N!parse"select from tt where {x in y}[sym;symlist]";
(?;`tt;,,({x in y};`sym;`symlist);0b;())
Whereas the parser can "see" the "in" in the first case
q)0N!parse"select from tt where sym in symlist";
(?;`tt;,,(in;`sym;`symlist);0b;())
I'm guessing the parser tries to do some optimisations when it sees the "in"