How to set up kdb RDB to only subscribe to certain tables in tickerplant - kdb

I want to set up 2 RDB instances using the same script where one instance subscribes to 2 tables and the other instance subscribes to a separate table in the tickerplant. I am trying to manipulate .u.sub but with no success
if[system"p"=RDB_INSTANCE_1;
.u.rep .(hopen `$":",.u.x 0)"({.u.sub[x;`]} each `trade`quote;`.u `i`L)";
];
if[system"p"=RDB_INSTANCE_2;
.u.rep .(hopen `$":",.u.x 0)"(.u.sub[`aggTradeStats;`];`.u `i`L)";
];
or
?[system"p"=RDB_INSTANCE_1;..u.rep .(hopen `$":",.u.x 0)"({.u.sub[x;`]} each `trade`quote;`.u `i`L)"; .u.rep .(hopen `$":",.u.x 0)"(.u.sub[`aggTradeStats;`];`.u `i`L)"];
Any idea how I can achieve this?

The vanilla tick scripts aren't that flexible when it comes to subscription options, but without going too far down the rabbit hole, the few quick changes below should allow you to do what you asked (I haven't tested the below further than initiating the instances)
.u.sub changes to the below to allow subscription to all tables (`), a list of tables or single table
{$[x~`;.z.s[;y]each t;1<count x;.z.s[;y]each x;[if[not x in t;'x];del[x].z.w;add[x;y]]]};
r.q changes to the below where the required tables are passed in via a command line flag. I've removed references to .z.x and parsed the various flags. The last line builds out the list of tables in string format (not very elegant but it's a quick solution).
/q tick/r.q [host]:port[:usr:pwd] [host]:port[:usr:pwd]
/2008.09.09 .k ->.q
if[not "w"=first string .z.o;system "sleep 1"];
upd:insert;
args:.Q.opt .z.x;
/ get the ticker plant and history ports, defaults are 5010,5012
/.u.x:.z.x[0],(count .z.x 0)_(":5010";":5012");
/ end of day: save, clear, hdb reload
.u.end:{t:tables`.;t#:where `g=attr each t#\:`sym;.Q.hdpf[`$"::",first args`hdb;`:.;x;`sym];#[;`sym;`g#] each t;};
/ init schema and sync up from log file;cd to hdb(so client save can run)
.u.rep:{if[0>type first x;x:enlist x];(.[;();:;].)each x;if[null first y;:()];-11!y;system "cd ",1_-10_string first reverse y};
/ HARDCODE \cd if other than logdir/db
/ connect to ticker plant for (schema;(logcount;log))
.u.rep .(hopen `$"::",first args`tp)"(.u.sub[`",("`" sv args`tabs),";`];`.u `i`L)";
Then start up RDBs as follows
ec2-user#/home/ec2-user $ ## RDB 1 table
ec2-user#/home/ec2-user $ q tick/r.q -tp 5010 -hdb 6000 -tabs trade -q
tables[]
,`trade
ec2-user#/home/ec2-user $ # RDB 2 tables
ec2-user#/home/ec2-user $ q tick/r.q -tp 5010 -hdb 6000 -tabs trade quote -q
tables[]
`quote`trade
ec2-user#/home/ec2-user $ ## RDB all tables
ec2-user#/home/ec2-user $ q tick/r.q -tp 5010 -hdb 6000 -q
tables[]
`s#`other`quote`trade
Check subscription dict back on tp
q).u.w
other| ,(9i;`)
quote| ((8i;`);(9i;`))
trade| ((7i;`);(8i;`);(9i;`))
Hope this helps, as I said, I haven't tested beyond this point
Jason

Related

How to get a list workflows based on some filtering, like timeout/workflowType

From https://github.com/uber/cadence/issues/3820
It's command that people want to get a list of workflows based on some filtering.
For example to get a list of workflows to reset or any operation.
In WebUI, you can select by the dropdown for "timeout" workflows.
With CLI, you can use command:
./cadence --domain sample-domain wf list --status timeout
you add filtering on workflow type:
./cadence --domain sample-domain wf list --wt "MyWorkflow" --status timeout
If you have advanced visibility, you and have more customize query:
./cadence --domain sample-domain wf list --query "CloseStatus=5 AND SOME_OTHER_FILTERS"
Feel free to explore more options in using List command with ``:
./cadence --domain sample-domain wf list --help
NAME:
cadence workflow list - list open or closed workflow executions
USAGE:
cadence workflow list [command options] [arguments...]
DESCRIPTION:
list one page (default size 10 items) by default, use flag --pagesize to change page size
OPTIONS:
--print_raw_time, --prt Print raw timestamp
--print_datetime, --pdt Print full date time in '2006-01-02T15:04:05Z07:00' format
--print_memo, --pme Print memo
--print_search_attr, --psa Print search attributes
--print_full, --pf Print full message without table format
--print_json, --pjson Print in raw json format
--open, --op List for open workflow executions, default is to list for closed ones
--earliest_time value, --et value EarliestTime of start time, supported formats are '2006-01-02T15:04:05+07:00', raw UnixNano and time range (N<duration>), where 0 < N < 1000000 and duration (full-notation/short-notation) can be second/s, minute/m, hour/h, day/d, week/w, month/M or year/y. For example, '15minute' or '15m' implies last 15 minutes.
--latest_time value, --lt value LatestTime of start time, supported formats are '2006-01-02T15:04:05+07:00', raw UnixNano and time range (N<duration>), where 0 < N < 1000000 and duration (in full-notation/short-notation) can be second/s, minute/m, hour/h, day/d, week/w, month/M or year/y. For example, '15minute' or '15m' implies last 15 minutes
--workflow_id value, --wid value, -w value WorkflowID
--workflow_type value, --wt value WorkflowTypeName
--status value, -s value Closed workflow status [completed, failed, canceled, terminated, continuedasnew, timedout]
--query value, -q value Optional SQL like query for use of search attributes. NOTE: using query will ignore all other filter flags including: [open, earliest_time, latest_time, workflow_id, workflow_type]
--more, -m List more pages, default is to list one page of default page size 10
--pagesize value, --ps value Result page size (default: 10)

Redis Hashes store with new line key value

I want to store data in Redis Hashes. Data is as below (Key = Value):
30.2.25=REF_IP
30.2.24=MY_HOST_IP
30.2.32=PEER_IP
30.2.32=IM_USER_MY_HOST
30.2.2=23992
Easy way to store this info in redis is below :
hmset info 30.2.25 REF_IP 30.2.24 MY_HOST_IP 30.2.32 PEER_IP 30.2.32 IM_USER_MY_HOST 30.2.2 23992
Considering I have 1000's key value and want to change few (actually so many) values in one go so searching and editing value in above command is too painful.
i want some way to execute command in below manner, that is nice formatted command with new line after every key value :
hmset info
30.2.25 REF_IP
30.2.24 MY_HOST_IP
30.2.32 PEER_IP
30.2.32 IM_USER_MY_HOST
30.2.2 23992
Is it possible to do so ?
Currently when i copy above formatted command and paste, it ignore test after new line and giving below error which is obvious because argument is wrong due to new line.
hmset info
(error) ERR wrong number of arguments for 'hmset' command
Can anyone help please. Thanks.
Assuming you are talking about using redis-cli, there is no way to support this at the moment. There is an open issue for this. See https://github.com/antirez/redis/issues/3474
As per Redis 4.0.0, HMSET is considered deprecated. You should use HSET instead. https://redis.io/commands/hset
You can use a transaction if you want to ensure all HSETs are done at the same time, and still enter them one line at a time.
MULTI
HSET info 30.2.25 REF_IP
HSET info 30.2.24 MY_HOST_IP
...
EXEC
The commands will be sent to the server one line at a time, but they are queued and only executed at the EXEC command.
You may use another client, say in Python, and then do something fancier as well to condense your field-value hsets into one command.

How do I create and array with two heads and LOTS of members

Preface: Please tell me a better way of doing this if you know of any! Even if it changes everything.
I have a powershell script that works with Net-SNMP to store the current value of an OID, change it and then check and store the values. Right now the previous and new values are stored in their own variable. There are about 160 OIDs that I need to change on >1000 nodes in my environment.
For Example for Previous and New values:
$P_vpwrSystemTempCompensation_0 = & '.\SnmpGet.exe' -q -r:"$ip" -v:2c -c:public -o:.1.3.6.1.4.1.13858.2.2.1.0
$N_vpwrSystemTempCompensation_0 = & '.\SnmpGet.exe' -q -r:"$ip" -v:2c -c:public -o:.1.3.6.1.4.1.13858.2.2.1.0
I'm try to figure the best way to store these and output into an CSV. I would like it to be formatted liked this:
Each CSV will be saved with the name and IP of the node I'm hitting.
You can make an object like this in a loop, and that could be exported to csv:
$oid = 'vpwrSystemTempCompensation.0'
$prev = 2
$new = 4647
[pscustomobject]#{OID=$oid; Previous=$prev; New=$new}
OID Previous New
--- -------- ---
vpwrSystemTempCompensation.0 2 5647
By the way, you don't need the call operator to run smnpget, if you don't put it in quotes:
.\SnmpGet.exe -q -r:$ip -v:2c -c:public -o:.1.3.6.1.4.1.13858.2.2.1.0

Splayed table upsert leading to error: `cast

I built a data loader prototype that saves CSV into splayed tables. The workflow is as follows:
Create schema the first time e.g. volatilitysurface table:
volatilitysurface::([date:`datetime$(); ccypair:`symbol$()] atm_convention:`symbol$(); premium_included:`boolean$(); smile_type:`symbol$(); vs_type:`symbol$(); delta_ratio:`float$(); delta_setting:`float$(); wing_extrapolation:`float$(); spread_type:`symbol$());
For every file in the rawdata folder import it:
myfiles:#[system;"dir /b /o:gn ",string `$getenv[`KDBRAWDATA],"*.volatilitysurface.csv 2> nul";()];
if[myfiles~();.lg.o[`load;"no volatilitysurface files found!"];:0N];
.lg.o[`load;"loading data files ..."];
/ load each file
{
mypath:"" sv (string `$getenv[`KDBRAWDATA];x);
.lg.o[`load;"loading file name '",mypath,"' ..."];
myfile:hsym`$mypath;
tmp1:select date,ccypair,atm_convention,premium_included,smile_type,vs_type,delta_ratio,delta_setting,wing_extrapolation,spread_type from update date:x, premium_included:?[premium_included = `$"true";1b;0b] from ("ZSSSSSFFFS";enlist ",")0:myfile;
`volatilitysurface upsert tmp1;
} #/: myfiles;
delete tmp1 from `.;
.Q.gc[];
.lg.o[`done;"loading volatilitysurface data done"];
.lg.o[`save;"saving volatilitysurface schema to ",string afolder];
volatilitysurface::0!volatilitysurface;
.Q.dpft[afolder;`;`ccypair;`volatilitysurface];
.lg.o[`cleanup;"removing volatilitysurface from memory"];
delete volatilitysurface from `.;
.Q.gc[];
.lg.o[`done;"saving volatilitysurface schema done"];
This works perfectly. I use .Q.gc[]; frequently to avoid hitting the wsfull. When new CSV files are available I open the existing schema, upsert into it and save it again effectively overwriting the existing HDB file system.
Open schema:
.lg.o[`open;"tables already exists, opening the schema ..."];
#[system;"l ",(string afolder) _ 0;{.lg.e[`open;"failed to load hdb directory: ", x]; 'x}];
/ Re-create table index
volatilitysurface::`date`ccypair xkey select from volatilitysurface;
Re-run step #2 to append new CSV files into the existing volatilitysurfacetable, it upserts the first CSV perfectly but the second CSV fails with:
error: `cast
I debug to the point of the error and to double-check I see that the metadata of tmp1 and volatilitysurface are perfectly the same. Any ideas why this is happening? I get the same issue with any other table. I have tried cleaning the keys from the table after every upsert but doesn't help i.e.
volatilitysurface::0!volatilitysurface;
volatilitysurface::`date`ccypair xkey volatilitysurface;
And the metadata comparison at the point of the cast error:
meta tmp1
c | t f a
------------------| -----
date | z
ccypair | s
atm_convention | s
premium_included | b
smile_type | s
vs_type | s
delta_ratio | f
delta_setting | f
wing_extrapolation| f
spread_type | s
meta volatilitysurface
c | t f a
------------------| -----
date | z
ccypair | s p
atm_convention | s
premium_included | b
smile_type | s
vs_type | s
delta_ratio | f
delta_setting | f
wing_extrapolation| f
spread_type | s
UPDATE Using the input of the answer below I tried using Torq's .loader.loadallfiles function like this (it doesn't fail but nothing happens either, the table is not created in memory and the data is not written to the database):
.loader.loadallfiles[`headers`types`separator`tablename`dbdir`dataprocessfunc!(`x`ccypair`atm_convention`premium_included`smile_type`vs_type`delta_ratio`delta_setting`wing_extrapolation`spread_type;"ZSSSSSFFFS";enlist ",";`volatilitysurface;`:hdb; {[p;t] select date,ccypair,atm_convention,premium_included,smile_type,vs_type,delta_ratio,delta_setting,wing_extrapolation,spread_type from update date:x, premium_included:?[premium_included = `$"true";1b;0b] from t}); `:rawdata]
UDPATE2 This is the output I get from TorQ:
2017.11.20D08:46:12.550618000|wsp18497wn|dataloader|dataloader1|INF|dataloader|**** LOADING :rawdata/20171102_113420.disccurve.csv ****
2017.11.20D08:46:12.550618000|wsp18497wn|dataloader|dataloader1|INF|dataloader|reading in data chunk
2017.11.20D08:46:12.566218000|wsp18497wn|dataloader|dataloader1|INF|dataloader|Read 10000 rows
2017.11.20D08:46:12.566218000|wsp18497wn|dataloader|dataloader1|INF|dataloader|processing data
2017.11.20D08:46:12.566218000|wsp18497wn|dataloader|dataloader1|INF|dataloader|Enumerating
2017.11.20D08:46:12.566218000|wsp18497wn|dataloader|dataloader1|INF|dataloader|writing 4525 rows to :hdb/2017.09.12/volatilitysurface/
2017.11.20D08:46:12.581819000|wsp18497wn|dataloader|dataloader1|INF|dataloader|writing 4744 rows to :hdb/2017.09.13/volatilitysurface/
2017.11.20D08:46:12.659823000|wsp18497wn|dataloader|dataloader1|INF|dataloader|writing 731 rows to :hdb/2017.09.14/volatilitysurface/
2017.11.20D08:46:12.737827000|wsp18497wn|dataloader|dataloader1|INF|init|retrieving sort settings from :C:/Dev/torq//config/sort.csv
2017.11.20D08:46:12.737827000|wsp18497wn|dataloader|dataloader1|INF|sort|sorting the volatilitysurface table
2017.11.20D08:46:12.737827000|wsp18497wn|dataloader|dataloader1|INF|sorttab|No sort parameters have been specified for : volatilitysurface. Using default parameters
2017.11.20D08:46:12.737827000|wsp18497wn|dataloader|dataloader1|INF|sortfunction|sorting :hdb/2017.09.05/volatilitysurface/ by these columns : sym, time
2017.11.20D08:46:12.753428000|wsp18497wn|dataloader|dataloader1|ERR|sortfunction|failed to sort :hdb/2017.09.05/volatilitysurface/ by these columns : sym, time. The error was: hdb/2017.09.
I get the following error sorttab|No sort parameters have been specified for : volatilitysurface. Using default parameters where is this sorttab documented? does it use the table PK by default?
UPDATE3 Ok fixed UPDATE2 out by providing a non-default sort.csv under my config folder:
tabname,att,column,sort
default,p,sym,1
default,,time,1
volatilitysurface,,date,1
volatilitysurface,,ccypair,1
But now I see that if I call the function multiple times on the same files, it simply appends duplicated data instead of upserting it.
UPDATE4 Still not there yet ... assuming I can check to make sure that no duplicate file is used. When I load and then start the database I get some structure back that ressembles some sort of dictionary and not a table.
2017.10.31| (,`volatilitysurface)!,+`date`ccypair`atm_convention`premium_incl..
2017.11.01| (,`volatilitysurface)!,+`date`ccypair`atm_convention`premium_incl..
2017.11.02| (,`volatilitysurface)!,+`date`ccypair`atm_convention`premium_incl..
2017.11.03| (,`volatilitysurface)!,+`date`ccypair`atm_convention`premium_incl..
sym | `AUDNOK`AUDCNH`AUDJPY`AUDHKD`AUDCHF`AUDSGD`AUDCAD`AUDDKK`CADSGD`C..
Note that date is actually datetime Z and not just date. My full and latest version of the function invocation is:
target:hsym `$("" sv ("./";getenv[`KDBHDB];"/volatilitysurface"));
rawdatadir:hsym `$getenv[`KDBRAWDATA];
.loader.loadallfiles[`headers`types`separator`tablename`dbdir`partitioncol`dataprocessfunc!(`x`ccypair`atm_convention`premium_included`smile_type`vs_type`delta_ratio`delta_setting`wing_extrapolation`spread_type;"ZSSSSSFFFS";enlist ",";`volatilitysurface;target;`date;{[p;t] select date,ccypair,atm_convention,premium_included,smile_type,vs_type,delta_ratio,delta_setting,wing_extrapolation,spread_type from update date:x, premium_included:?[premium_included = `$"true";1b;0b] from t}); rawdatadir];
I'm going to add a second answer here to try and tackle the question about using TorQ's data loader.
I'd like to clarify what output you are getting after running this function? There should be some logging messages output, can you post these? For example when I run the function:
jmcmurray#homer ~/deploy/TorQ (master) $ q torq.q -procname loader -proctype loader -debug
<torq startup messages removed>
q).loader.loadallfiles[`headers`types`separator`tablename`dbdir`partitioncol`dataprocessfunc!(c;"TSSFJFFJJBS";enlist",";`quotes;`:testdb;`date;{[p;t] select date:.z.d,time:TIME,sym:INSTRUMENT,BID,ASK from t});`:csvtest]
2017.11.17D15:03:20.312336000|homer.aquaq.co.uk|loader|loader|INF|dataloader|**** LOADING :csvtest/tradesandquotes20140421.csv ****
2017.11.17D15:03:20.319110000|homer.aquaq.co.uk|loader|loader|INF|dataloader|reading in data chunk
2017.11.17D15:03:20.339414000|homer.aquaq.co.uk|loader|loader|INF|dataloader|Read 11000 rows
2017.11.17D15:03:20.339463000|homer.aquaq.co.uk|loader|loader|INF|dataloader|processing data
2017.11.17D15:03:20.339519000|homer.aquaq.co.uk|loader|loader|INF|dataloader|Enumerating
2017.11.17D15:03:20.340061000|homer.aquaq.co.uk|loader|loader|INF|dataloader|writing 11000 rows to :testdb/2017.11.17/quotes/
2017.11.17D15:03:20.341669000|homer.aquaq.co.uk|loader|loader|INF|dataloader|**** LOADING :csvtest/tradesandquotes20140422.csv ****
2017.11.17D15:03:20.349606000|homer.aquaq.co.uk|loader|loader|INF|dataloader|reading in data chunk
2017.11.17D15:03:20.370793000|homer.aquaq.co.uk|loader|loader|INF|dataloader|Read 11000 rows
2017.11.17D15:03:20.370858000|homer.aquaq.co.uk|loader|loader|INF|dataloader|processing data
2017.11.17D15:03:20.370911000|homer.aquaq.co.uk|loader|loader|INF|dataloader|Enumerating
2017.11.17D15:03:20.371441000|homer.aquaq.co.uk|loader|loader|INF|dataloader|writing 11000 rows to :testdb/2017.11.17/quotes/
2017.11.17D15:03:20.460118000|homer.aquaq.co.uk|loader|loader|INF|init|retrieving sort settings from :/home/jmcmurray/deploy/TorQ/config/sort.csv
2017.11.17D15:03:20.466690000|homer.aquaq.co.uk|loader|loader|INF|sort|sorting the quotes table
2017.11.17D15:03:20.466763000|homer.aquaq.co.uk|loader|loader|INF|sorttab|No sort parameters have been specified for : quotes. Using default parameters
2017.11.17D15:03:20.466820000|homer.aquaq.co.uk|loader|loader|INF|sortfunction|sorting :testdb/2017.11.17/quotes/ by these columns : sym, time
2017.11.17D15:03:20.527216000|homer.aquaq.co.uk|loader|loader|INF|applyattr|applying p attr to the sym column in :testdb/2017.11.17/quotes/
2017.11.17D15:03:20.535095000|homer.aquaq.co.uk|loader|loader|INF|sort|finished sorting the quotes table
After all this, I can run \l testdb and there is a table called "quotes" containing my loaded data
If you can post logging messages like these, it could be helpful to see what's going on.
UPDATE
"But now I see that if I call the function multiple times on the same files, it simply appends duplicated data instead of upserting it."
If I'm understanding the problem correctly, it sounds like you likely shouldn't call the function multiple times on the same files. Another process within TorQ could be useful here, the "file alerter". This process will monitor a directory for new & updated files, and can call a function on any that appear (so you can have it call the loader function with every new file automatically). It has a number of options such as moving files after processing (so you can "archive" loaded CSVs)
Note that the file alerter requires that a function take exactly two parameters - the directory & the file name. This effectively means you will need a "wrapper" function around the loader function, which takes a dictionary & a directory. I don't think TorQ includes a function similar to .loader.loadallfiles for a single file, so it might be necessary to copy the target file to a temporary directory, run loadallfiles on that directory and then delete the file from there before loading the next.
`cast error refers to a value not being enumerated
I can't see any enumeration going on here, splayed tables on disk need to have symbol columns enumerated. For example, this can be done with the following line, before calling .Q.dpft
volatilitysurface:.Q.en[afolder;volatilitysurface];
You may like to consider using an example CSV loader for loading your data. One such example is included in TorQ, the KDB framework developed by AquaQ Analytics (as a disclaimer, I work for AquaQ)
The framework is available (free of charge) here: https://github.com/AquaQAnalytics/TorQ
The specific component you will likely be interested in is dataloader.q and is documented here: http://aquaqanalytics.github.io/TorQ/utilities/#dataloaderq
This script will handle everything necessary, loading all files, enumerating, sorting on disk, applying attributes etc. as well as using .Q.fsn to prevent running out of memory

Saving all tables,variables, and functions of a KDB+ instance and then reloading

Is there a simple way to save all tables,variables and fucntions and then reload them into another kdb+ instance? For example, lets say my machine is set to restart weekly but I want reload everything that was in my kdb+ session before the restart.
Thanks.
If you want something out of the box for maintaining variable state(tables/dicts/lists/atom), then have a look at http://code.kx.com/q/cookbook/logging/
/cmdline startup
q db_main -l -p 8090
/To store state before the restart, modify .z.exit to flush to the qdb file
.z.exit:{value"\\l"}
This will not cover functions however, but ideally these shouldn't be changing anyway.
If you really need to save these you can look at saving all namespaces to disk.
.z.exit:{`:/tmp/ns set get each {x!x}`$".",/:string key`}
/on startup
{{y set x[y]}[x;]each key x} `.q`.Q _ get `:/tmp/ns
Regards,
Connor
Save the state to file using set/get. This is limited to small workspaces.
>q
KDB+ 3.4 2016.06.14 Copyright (C) 1993-2016 Kx Systems
q)a:1
q)t:([] b:til 100)
q)s:select from t where b<5
q)`:session.bin set get `.
`:session.bin
q)\\
>q
KDB+ 3.4 2016.06.14 Copyright (C) 1993-2016 Kx Systems
q)`. set get `:session.bin
`.
q)a
1
q)s
b
-
0
1
2
3
4
Logging and snapshots is an alternative:
http://www.timestored.com/kdb-guides/kdb-command-line-options#kdb-logging-replication