I have found a number of posts that talk about syncblk output that shows an even number for the MonitorHeld count and no owner thread. This has been explained as a quirk, the lock is in transition probably because the owner thread has failed so, there is no owner. The MonitorHeld count is incremented by 2 for each waiter and one for the owner so, an even number means you have waiters but no owner.
My syncblk shows an odd number for the MonitorHeld count yet, no owning thread. What does that indicate? Also, I have two different process dumps from two different instances that look like this so, it wasn't a case of capturing a dump at just the right time.
My syncblk output:
Index SyncBlock MonitorHeld Recursion Owning Thread Info SyncBlock Owner
164 000000000a4d2428 1 1 000000002e406d30 16ac 144 0000000012f107c0 System.Object
368 000000000a3eb378 1125 0 0000000000000000 none 00000000104b1a38 System.Object
401 000000000a4d2888 1 1 000000002e79cce0 21a0 123 00000000110b09b0 System.Object
505 000000000a516918 1 1 000000002e55ca70 95c 114 00000000110984f0 System.Object
563 0000000002ae7978 1 1 000000002e7e1910 2058 261 00000000127d0300 System.Object
678 0000000002ae7f18 3 1 000000000a462240 f44 156 0000000012a66a28 System.Object
-----------------------------
Total 814
CCW 0
RCW 1
ComClassFactory 0
Free 142
Index SyncBlock MonitorHeld Recursion Owning Thread Info SyncBlock Owner
122 000000002e561d98 3 1 000000000a621400 1410 180 0000000012cfcf28 System.Object
164 000000000a4d2428 1 1 000000002e406d30 16ac 145 0000000012f107c0 System.Object
368 000000000a3eb378 951 0 0000000000000000 none 00000000104b1a38 System.Object
401 000000000a4d2888 1 1 000000002e79cce0 21a0 124 00000000110b09b0 System.Object
505 000000000a516918 1 1 000000002e55ca70 95c 115 00000000110984f0 System.Object
563 0000000002ae7978 1 1 000000002e7e1910 2058 263 00000000127d0300 System.Object
-----------------------------
Total 814
CCW 0
RCW 20
ComClassFactory 0
Free 76
The lock is either in transition or orphaned. A SyncBlk is orphaned when the owning thread terminates without releasing it.
Related
I'm trying to perform some simple logic on a table but I'd like to verify that the columns exists prior to doing so as a validation step. My data consists of standard table names though they are not always present in each data source.
While the following seems to work (just validating AAA at present) I need to expand to ensure that PRI_AAA (and eventually many other variables) is present as well.
t: $[`AAA in cols `t; temp: update AAA_VAL: AAA*AAA_PRICE from t;()]
Two part question
This seems quite tedious for each variable (imagine AAA-ZZZ inputs and their derivatives). Is there a clever way to leverage a dictionary (or table) to see if a number of variables exists or insert a place holder column of zeros if they do not?
Similarly, can we store a formula or instructions to to apply within a dictionary (or table) to validate and return a calculation (i.e. BBB_VAL: BBB*BBB_PRICE.) Some calculations would be dependent on others (i.e. BBB_Tax_Basis = BBB_VAL - BBB_COSTS costs for example so there could be iterative issues.
Thank in advance!
A functional update may be the best way to achieve this if your intention is to update many columns of a table in a similar fashion.
func:{[t;x]
if[not x in cols t;t:![t;();0b;(enlist x)!enlist 0]];
:$[x in cols t;
![t;();0b;(enlist`$string[x],"_VAL")!enlist(*;x;`$string[x],"_PRICE")];
t;
];
};
This function will update t with *_VAL columns for any column you pass as an argument, while first also adding a zero column for any missing columns passed as an argument.
q)t:([]AAA:10?100;BBB:10?100;CCC:10?100;AAA_PRICE:10*10?10;BBB_PRICE:10*10?10;CCC_PRICE:10*10?10;DDD_PRICE:10*10?10)
q)func/[t;`AAA`BBB`CCC`DDD]
AAA BBB CCC AAA_PRICE BBB_PRICE CCC_PRICE DDD_PRICE AAA_VAL BBB_VAL CCC_VAL DDD DDD_VAL
---------------------------------------------------------------------------------------
70 28 89 10 90 0 0 700 2520 0 0 0
39 17 97 50 90 40 10 1950 1530 3880 0 0
76 11 11 0 0 50 10 0 0 550 0 0
26 55 99 20 60 80 90 520 3300 7920 0 0
91 51 3 30 20 0 60 2730 1020 0 0 0
83 81 7 70 60 40 90 5810 4860 280 0 0
76 68 98 40 80 90 70 3040 5440 8820 0 0
88 96 30 70 0 80 80 6160 0 2400 0 0
4 61 2 70 90 0 40 280 5490 0 0 0
56 70 15 0 50 30 30 0 3500 450 0 0
As you've already mentioned, to cover point 2, a dictionary of functions might be the best way to go.
q)dict:raze{(enlist`$string[x],"_VAL")!enlist(*;x;`$string[x],"_PRICE")}each`AAA`BBB`DDD
q)dict
AAA_VAL| * `AAA `AAA_PRICE
BBB_VAL| * `BBB `BBB_PRICE
DDD_VAL| * `DDD `DDD_PRICE
And then a slightly modified function...
func:{[dict;t;x]
if[not x in cols t;t:![t;();0b;(enlist x)!enlist 0]];
:$[x in cols t;
![t;();0b;(enlist`$string[x],"_VAL")!enlist(dict`$string[x],"_VAL")];
t;
];
};
yields a similar result.
q)func[dict]/[t;`AAA`BBB`DDD]
AAA BBB CCC AAA_PRICE BBB_PRICE CCC_PRICE DDD_PRICE AAA_VAL BBB_VAL DDD DDD_VAL
-------------------------------------------------------------------------------
70 28 89 10 90 0 0 700 2520 0 0
39 17 97 50 90 40 10 1950 1530 0 0
76 11 11 0 0 50 10 0 0 0 0
26 55 99 20 60 80 90 520 3300 0 0
91 51 3 30 20 0 60 2730 1020 0 0
83 81 7 70 60 40 90 5810 4860 0 0
76 68 98 40 80 90 70 3040 5440 0 0
88 96 30 70 0 80 80 6160 0 0 0
4 61 2 70 90 0 40 280 5490 0 0
56 70 15 0 50 30 30 0 3500 0 0
Here's another approach which handles dependent/cascading calculations and also figures out which calculations are possible or not depending on the available columns in the table.
q)show map:`AAA_VAL`BBB_VAL`AAA_RevenueP`AAA_RevenueM`BBB_Other!((*;`AAA;`AAA_PRICE);(*;`BBB;`BBB_PRICE);(+;`AAA_Revenue;`AAA_VAL);(%;`AAA_RevenueP;1e6);(reciprocal;`BBB_VAL));
AAA_VAL | (*;`AAA;`AAA_PRICE)
BBB_VAL | (*;`BBB;`BBB_PRICE)
AAA_RevenueP| (+;`AAA_Revenue;`AAA_VAL)
AAA_RevenueM| (%;`AAA_RevenueP;1000000f)
BBB_Other | (%:;`BBB_VAL)
func:{c:{$[0h=type y;.z.s[x]each y;-11h<>type y;y;y in key x;.z.s[x]each x y;y]}[y]''[y];
![x;();0b;where[{all in[;cols x]r where -11h=type each r:(raze/)y}[x]each c]#c]};
q)t:([] AAA:1 2 3;AAA_PRICE:1 2 3f;AAA_Revenue:10 20 30;BBB:4 5 6);
q)func[t;map]
AAA AAA_PRICE AAA_Revenue BBB AAA_VAL AAA_RevenueP AAA_RevenueM
---------------------------------------------------------------
1 1 10 4 1 11 1.1e-05
2 2 20 5 4 24 2.4e-05
3 3 30 6 9 39 3.9e-05
/if the right columns are there
q)t:([] AAA:1 2 3;AAA_PRICE:1 2 3f;AAA_Revenue:10 20 30;BBB:4 5 6;BBB_PRICE:4 5 6f);
q)func[t;map]
AAA AAA_PRICE AAA_Revenue BBB BBB_PRICE AAA_VAL BBB_VAL AAA_RevenueP AAA_RevenueM BBB_Other
--------------------------------------------------------------------------------------------
1 1 10 4 4 1 16 11 1.1e-05 0.0625
2 2 20 5 5 4 25 24 2.4e-05 0.04
3 3 30 6 6 9 36 39 3.9e-05 0.02777778
The only caveat is that your map can't have the same column name as both the key and in the value of your map, aka cannot re-use column names. And it's assumed all symbols in your map are column names (not global variables) though it could be extended to cover that
EDIT: if you have a large number of column maps then it will be easier to define it in a more vertical fashion like so:
map:(!). flip(
(`AAA_VAL; (*;`AAA;`AAA_PRICE));
(`BBB_VAL; (*;`BBB;`BBB_PRICE));
(`AAA_RevenueP;(+;`AAA_Revenue;`AAA_VAL));
(`AAA_RevenueM;(%;`AAA_RevenueP;1e6));
(`BBB_Other; (reciprocal;`BBB_VAL))
);
Ive been looking at implementing GLCM within MATLAB using graycomatrix. There are two arguments that I have discovered (NumLevels and GrayLimits) but in in my research and implementation they seem to achieve the same result.
GrayLimits specified bins between a range set [low high], causing a restricted set of gray levels.
NumLevels declares the number of gray levels in an image.
Could someone please explain the difference between these two arguments, as I don't understand why there would be two arguments that achieve the same result.
From the documentation:
'GrayLimits': Range used scaling input image into gray levels, specified as a two-element vector [low high]. If N is the number of gray levels (see parameter 'NumLevels') to use for scaling, the range [low high] is divided into N equal width bins and values in a bin get mapped to a single gray level.
'NumLevels': Number of gray levels, specified as an integer.
Thus the first parameter sets the input gray level range to be used (defaults to the min and max values in the image), and the second parameter sets the number of unique gray levels considered (and thus the size of the output matrix, defaults to 8, or 2 for binary images).
For example:
>> graycomatrix(img,'NumLevels',8,'GrayLimits',[0,255])
ans =
17687 1587 81 31 7 0 0 0
1498 7347 1566 399 105 8 0 0
62 1690 3891 1546 298 38 1 0
12 335 1645 4388 1320 145 4 0
2 76 305 1349 4894 959 18 0
0 16 40 135 965 7567 415 0
0 0 0 2 15 421 2410 0
0 0 0 0 0 0 0 0
>> graycomatrix(img,'NumLevels',8,'GrayLimits',[0,127])
ans =
1 9 0 0 0 0 0 0
7 17670 1431 156 50 31 23 15
1 1369 3765 970 350 142 84 92
0 128 1037 1575 750 324 169 167
0 46 361 836 1218 747 335 260
0 16 163 330 772 1154 741 547
0 10 74 150 370 787 1353 1208
0 4 67 136 294 539 1247 21199
>> graycomatrix(img,'NumLevels',4,'GrayLimits',[0,255])
ans =
28119 2077 120 0
2099 11470 1801 5
94 1829 14385 433
0 2 436 2410
As you can see, these parameters modify the output in different ways:
In the first case above, the range [0,255] was mapped to columns/rows 1-8, putting 32 different input grey values into each.
In the second case, the smaller range [0,127] was mapped to 8 indices, putting 16 different input grey values into each, and putting the remaining grey values 128-255 into the 8th index.
In the third case, the range [0,255] was mapped to 4 indices, putting 64 different input grey values into each.
I am joining 3 tables to get the retention rate. Here is my query:
select first_visit.first_month as first_month,
new_users.new_users as new_users,
count(distinct visit_tracker.customer__id) as retained,
cast(count(distinct visit_tracker.customer__id) / new_users.new_users as float) as retention_percent
from first_visit
left join visit_tracker
on visit_tracker.customer__id=first_visit.customer__id
left join new_users
on new_users.first_month=first_visit.first_month
group by 1,2;
I get the following output:
first_month new_users retained retention_percent
0 93 34 0
1 119 42 0
2 188 102 0
3 223 71 0
and so on
What I want is this:
first_month new_users retained retention_percent
0 93 34 0.37
1 119 42 0.35
2 188 102 0.54
3 223 71 0.32
I am not sure why it's not producing the results I want. Any inputs?
This looks like a classic case of an integer division problem.
In this case count(distinct visit_tracker.customer__id) will return an integer which is then divided by a float. It looks like the float is cast into an integer and the result of the division is therefore an integer. Because the expected answer is less than one, it truncates to zero. The as float part of your query will not help as this happens after the truncation has already occured.
Try making sure both the numerator and the denominator are floats before performing the division or multiply by 100 beforehand as this stackoverflow answer suggests.
I'm trying to create a table in KDB where the columns are the results of a query. For example , I have loaded in stock data and search for a given time window what prices the stock traded at. I created a function
getTrades[Sybmol; Date; StartTime; StopTime]
This will search through my database and return the prices that traded between the start and stop time. So my results for Apple for a 30 second window might be:
527.10, 527.45, 527.60, 526.90 etc.
What I want to do is now create a table using xbar where I have rows of every second and columns of all the prices that trade in StartTime and StopTime. I will then place an X in the column if the price traded in that 1 second. I think I can handle most of this but the main thing I'm struggling with is converting the results I got above into the name of the table. I'm also struggling with how to make it flexible so my table will have 5 columns in one scenario (5 prices traded) but 10 in another so essentially it varies depending on how many price levels traded in the window I'm searching.
Thanks.
The best and cleanest way to do programmatic selects is with the functional form of select.
from q for mortals,
?[t;c;b;a]
where t is a table, a is a dictionary of aggregates, b is a dictionary of groupbys and c is a list of constraints.
In other words, select a by b from t where c.
This will allow you to dynamically create a, which can be of arbitrary size.
You can find more information here:
http://code.kx.com/q4m3/9_Queries_q-sql/#912-functional-forms
Pivot Table
I think that pivot table will be suitable in this case. Using jgleeson example:
time price
------------------
11:27:01.600 106
11:27:02.600 102
11:27:02.600 102
11:27:03.100 100
11:27:03.100 102
11:27:03.100 102
11:27:03.100 104
11:27:03.600 104
11:27:03.600 102
11:27:04.100 106
11:27:05.100 105
11:27:06.600 106
11:27:07.100 101
11:27:07.100 104
11:27:07.600 105
11:27:07.600 105
11:27:07.600 101
not null exec (exec `$string asc distinct price from s)#(`$string price)!price by time:1 xbar time.second from s:select from t where time within 11:27:00 11:27:30
and returns:
time | 100 101 102 103 104 105 106
--------| ---------------------------
11:27:01| 0 0 0 0 0 0 1
11:27:02| 0 0 1 0 0 0 0
11:27:03| 1 0 1 0 1 0 0
11:27:04| 0 0 0 0 0 0 1
11:27:05| 0 0 0 0 0 1 0
11:27:06| 0 0 0 0 0 0 1
11:27:07| 0 1 0 0 1 1 0
It can support any numbers of unique prices.
This looks a bit convoluted... but I think this might be what you're after.
Sample table t with time and price columns:
t:`time xasc([]time:100?(.z.T+500*til 100);price:100?(100 101 102 103 104 105 106))
This table should replicate what you get from the first step of your function call - "select time,price from trade where date=x, symbol=y, starttime=t1, endtime=t2".
To return the table in the format specified:
q) flip (`time,`$string[c])!flip {x,'y}[key a;]value a:{x in y}[c:asc distinct tt`price] each group (!) . reverse value flip tt:update time:time.second from t
time 100 101 102 103 104 105 106
------------------------------------
20:34:29 0 1 0 0 0 1 0
20:34:30 0 0 0 0 0 0 1
20:34:31 0 0 1 0 0 0 0
20:34:32 0 0 1 0 1 0 0
...
This has bools instead of X as bools are probably easier to work with.
Also please excuse the one-liner... If I get a chance I'll break it up and try to make it more readable.
A more simplified version is :
q)t:`time xasc([] s:100#`s ; time:100?(.z.T+500*til 100);price:100?(100 101 102 103 104 105 106))
q)t1:update `$string price,isPrice:1b from t
q)p:(distinct asc t1`price)
q)exec p#(10b!"X ")#(price!isPrice) by time:1 xbar time.second from t1
time | 100 101 102 103 104 105 106
--------| ---------------------------
20:39:00| X X X
20:39:01| X X X X
20:39:02| X
20:39:04| X
20:39:05| X X X X
I need to know what tm->when means, but proc(5) doesn't mention anything helpful,
So, does it store the creation time of the socket? The number seems to be decreasing each time I view the file.
root#ubuntu-vm:~# cat /proc/net/tcp
sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode
0: 00000000:0CEA 00000000:0000 0A 00000000:00000000 00:00000000 00000000 104 0 17410 1 dddb6d00 100 0 0 10 -1
1: 00000000:0016 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 7959 1 dddb4500 100 0 0 10 -1
2: B238A8C0:0016 0138A8C0:9C96 01 00000000:00000000 02:00061444 00000000 0 0 8243 4 daa3c000 20 4 27 10 16
3: B238A8C0:0CEA 0138A8C0:8753 01 00000000:00000000 02:0009C787 00000000 104 0 19467 2 daa3e300 20 4 18 10 -1
From Exploring the /proc/net/ Directory
The tr field indicates whether a timer is active for this socket. A value of zero indicates the timer is not active. The tm->when field indicates the time remaining (in jiffies) before timeout occurs.