How to pass a command line argument into a kdb function - sh

Question

I wrote a shell script to collect data on each character of the alphabet from a large table. As a result of the function being memory intensive, I'd like to partition the characters of the alphabet so that each character is concurrently called using a different port number. However I can't seem to successfully pass the desired command line argument into my function. Testing on a single port and a small table, what I've tried is the following...
On the server: I set up my dummy table and defined a function...
ts:([]sym:1000?`A`Ab`B`Bc`C`Ca`X`Xz`Y`Yx`Z`Zy;price:1000?100.0;num:til 1000)
collect:{[x;y]select from x where sym like y}
On the client: I open a connection handle, use the .z.X namespace to return strings of the raw, unfiltered command line arguments, index into it and store as a variable, and then attempt to pass that variable into my server side function. I've tried storing as a character and a symbol. It will run without error but neither returns any data when called. I save this file as collector.q..
/ start connection handles to remote server
h:hopen `::5000
/ index into command line arguments to get partition letter. Store as character and symbol
part:.z.X[6]
symPart:`$part
/ call server side functions
fetched:h (`collect; `ts; "symPart*")
/ close connection handle
hclose h
my shell script looks like the following...
#!/bin/sh
port=$1
partition=$2
for x in {A..Z}
do
echo running partition $x on port $port
$QHOME/l64/q collector.q -p $port -partition $x > ./stdout$port.log 2>&1 &
port=$(($port + 1))
done
After running the shell script, when calling the fetched function on the client, what's returned is an empty table...
q)fetched
symbol price ID
---------------
q)

Answer 1

I think the first issue is that you are not getting the correct value for part from the command line. .z.X returns the command line as a list of tokens while .z.x returns the same but without the q command and file name.
q test.q -p 5000 -partition a
q).z.X
"/opt/kdb/3.5/l32/q"
"test.q"
"-p"
"5000"
"-partition"
,"a"
q).z.X[6]
""
Use .Q.opt function to turn the command line parameters into a dictionary that much more reliable.
q)params:.Q.opt .z.X
q)params
p | "5000"
partition| ,"a"
q)`$first params`partition
`a
The second issue is that "symPart*" won't evaluate the variable symPart as it is inside quotes. All kdb+ sees is a string. Each loop of the bash script will send the same command (`collect; `ts; "symPart*") over the handle, with the collect checking against a "symPart" partition which I guess doesn't exist.
To pass it in as a variable you can change it to symPart,"*", but in this case symPart needs to be a string, not a symbol so you can remove the casting.
You can amend your script as follows:
/ start connection handles to remote server
h:hopen `::5000
/ convert command line arguments into dictionary and index to get partition letter
part:first .Q.opt[.z.X]`partition;
/ call server side functions
fetched:h (`collect; `ts; symPart,"*")
/ close connection handle
hclose h
Or to do this all in a single q script you could do the following:
/ start connection handles to remote server
h:hopen `::5000
/ call server side functions
fetched:.Q.a!h each (`collect; `ts),/: enlist each .Q.a,\:"*"
/ close connection handle
hclose h
.Q.a holds all the lowercase letters in a string.
q).Q.a
"abcdefghijklmnopqrstuvwxyz"
We can create all the wildcards using the each left \: with , (join) to join each of them to "*" and then create multiple commands to send over the handle using each right /: to join the unchanging left hand side to all of the different wildcards.
q).Q.a,\:"*"
"a*"
"b*"
"c*"
"d*"
..
q)(`collect; `ts),/: enlist each .Q.a,\:"*"
`collect `ts "a*"
`collect `ts "b*"
`collect `ts "c*"
..
Note the output of fetched will be a dictionary with keys as the partitions and the fetched result of each partition as the corresponding value.