I would like to update my sym column in my trade table so that at the end of every sym there is a _1 appended onto the end of it.
I have tried update sym:sym _ "_1" from trade which gave me a par error so I then tried the fncol function from the dbmaint.q script which was
`fncol[`:path/to/hdb;`trade;`sym;,"_1]`
which also gave me an error on / which I'm not sure why. If anyone has any idea how to fix this or could point me in the right direction that would be great
This is probably not as trivial as it looks on paper due to the fact that it's an on-disk table (can't use update directly hence par error) and the sym column is possibly enumerated? (hence why you couldn't append string)
If the sym column is enumerated then they need to be re-enumerated after the "_1" is appended, something like:
load`:/path/to/mySymFile; /make sure sym file is loaded
fncol[`:path/to/hdb;`trade;`sym;{`:/path/to/mySymFile?`$string[x],\:"_1"}];
However personally I don't think this would be a great idea and you'd be polluting your sym file with a bunch of new symbols. Why not just append the "_1" at runtime? Does it have to be persisted?
If your sym column is actually a string column and not enumerated, then you would just need:
fncol[`:path/to/hdb;`trade;`sym;{x,\:"_1"}];
Related
I have seen this question, but I think my issue is a bit different :
kdb splayed table meta error
I am saving a table splayed in a location with the following command :
pthToSplayed upsert .Q.en[pthtohdbroot;] table
I don't have any sym so nothing gets enumerated (.Q.en is there because in the future it might have some symbols).
All works well, but when I try to do meta select from get[pathToTable] where date = .z.d I get a ..sym error.
However, the strange part is that the first time I am saving the table down ... the meta works fine. when I exit and start the proc again the problem seems to appear. what exactly happens here? I would appreciate links to a whitepaper or kx website which explains where this issue comes from.
UPDATE:
Nothing weird about the meta of the table. just a vanilla meta.
`:/home/user/hdbroot/tableName/ set .Q.en[`:/home/user/hdbroot;] table
As I am updating this table I am using upsert instead of set for subsequent operations.
Your issue is that the enumeration file named sym needs to be loaded in to the process memory.
If you load the folder using \l you will not have the issue:
q)\l hdbRoot
Or you can explicitly load the sym file:
q)tableName:get `:hdbRoot/tableName/
q)sym:get `:hdbRoot/sym
The reason it works before a restart is because .Q.en automatically loads sym in to memory for you when you call it.
If there are no symbol columns there should be no error.
Save table with no symbol column:
$q
q)`:hdbRoot/tableName/ set .Q.en[`:hdbRoot] ([] a:1 2 3)
`:hdbRoot/tableName/
q)\\
New session meta works:
$q
q)tableName:get `:hdbRoot/tableName/
q)meta tableName
c| t f a
-| -----
a| j
You can check the types of each column to confirm there are no enumerations:
q)type each flip tableName
Prior to V3.6 of kdb+, the type of an enumeration could range from 20h to 76h, with each new domain given an incremented type. In V3.6 onwards all enumerations 20h
If any column is a symbol type, even if empty then a sym file is created:
q)`:hdbRoot/tableName/ set .Q.en[`:hdbRoot] ([] a:`long$();b:`$())
`:hdbRoot/tableName/
q)get `:hdbRoot/sym
`symbol$()
I have a table in which i have a name column, The names in that column are all lower case .I want to convert the string to camel case .The table headers are a follows
Table
|Id|Name |Phone Number|
|01|bob wheeler|999999999 |
If you are trying to do this in a server job, then use Oconv(InLink.MyString,"MCT").
There is no directly-available function in a parallel Transformer stage. You could:
(a) put a server Transformer in a server Shared Container and use that in your parallel job,
(b) write your own routine in C++ and refer to it (called a "parallel routine"), or
(c) investigate whether the database in which the table resides has any kind of camel case conversion function.
I don't think there is a specific function for that.
UpCase is available.
You migh consider a workaround to convert the first and every other first character ater a blank to upper case.
Assuming name column only contains first name and Last name.
If there is middle name as well, you can simulate the formula to handle that.
The formula will convert first char after first occurrence of space to upper case.
It will concatenate first name, first character of second name with upper case and then rest of the second name. You can play with positions if getting wrong output. But this would definitely work-
left(NAME,Index(NAME,' ',1)) : Upcase(Name[(Index(NAME,' ',1))+1,1]) : Name[(Index(NAME,' ',1))+2,length(NAME)-Index(NAME,' ',1)+1]
I would like to count all tables in the same instance.
I have not used kdb for a while and I forgot how to make this work.
This is what I got:
tablelist:tables[]
{select count i from x} each tablelist
but I got a type error
Your statement doesn't contain a trailing semi colon ; at the end of the first line which will cause an error in an IDE like qpad (assuming you are running it as written).
If not running from an IDE I would check my hdb for any possible missing data and run some sanity checks (i.e can I select from each of my tables normally, do types match across partitions, i is a virtual column representing row count so issues with non-conforming types in your other columns is probably not a cause but investigating may yield the right answer)
One way to achieve what you're trying is (using dummy data):
q){flip select counts:count i,tab:1#x from x}each tablelist:tables[]
counts tab
-------------
5469 depth
3150 quotes
3005 trades
Here I select the count for each table, but also add on the name of the table, flip each result into a dictionary, which results in a list of dictionaries of conforming types and key names which is in fact a table, hence my result. In this way you have a nice way to track what you're actually counting.
Each select query you run is returning a table in the form:
x
-
3
It would be better to use exec as opposed to select to simply return the value of the count e.g:
q){exec count i from x} each tables[]
3 2
Your current method would be attempting to return a list of tables: e.g:
q){select count i from x} each tables[]
+(,`x)!,,3
+(,`x)!,,2
However, the type error makes me think there may be an issue with your tables as this should not error for in-memory tables.
Here's one way
count each `. tables[]
I am using 3.6 2018.05.17 and your expression worked for me. I then change the select to an exec to return just a list of counts.
q){exec count i from x} each tables[]
Below code helps us get the count of each table along with tablename.
q)flip (`table;`msgcount)! flip {x, count value x}#'tables[]
To get only the count and not the tablename along with it.
q){count value x}#'tables[]
A column has undergone a data type change, so the query has to be changed too:
Old query:
select from person where id = 100
New query:
select from person where id = `100
I'm new to Q and haven't been able to figure out how to do this:
Example:
I want to convert 100 to `100.
You would need to convert to string first and then cast to symbol:
q)`$string 100
`100
However, having a numerical column as symbols is a pretty bad idea. Is this table being written to disk? If so this could possibly blow up your sym file and blow up your in-memory interned symbol list (increasing your memory usage).....assuming the numerical values are not very repetitive
I am creating a new table on a KDB database as a parted splay (parted by date), the new table schema has a column called CCYY, which has a lot of repeating values. I am unsure if I should save it as char or symbols. My main goal is to use least amount of memory.
As a result which one should I use? What is the benefit/disadvantage of saving repeating values as either a char array or a symbol in a parted splayed setup?
It sounds like you should use symbol.
There's a guide to symbols/enumerations here:http://www.timestored.com/kdb-guides/strings-symbols-enumeration#when-to-use quote:
Typically you should follow the guidelines:
If the column is used in where clause equality comparisons e.g.
select from t where sym in AB -> Symbol
Short, often repeated strings -> Symbol
Else Long, Non-repeated strings -> String
When evaluating whether or not to use symbol for a column, cardinality of that column is key. Length of individual values matters less and, if anything, longer values might be better off as symbol, as they will be stored only once in the sym file, but repeated in the char vector. That consideration is pretty much moot if you compress you data on disk though.
If your values are short enough, don't forget about the possibility of using .Q.j10, .Q.x10, .Q.j12 and .Q.x12. This will use less space than a char vector. And it doesn't rely on a sym file, which in complex environments can save you from having to re-enumerate if you are, say, copying tables between hdbs who's sym files are not in sync.
If space is a concern, always compress the data on disk.