cluster the given data and use any retrieval algorithm to show output as shown below.
(any clustering algorithm)
Euclidean distance may be used for finding closest cases.
let a data file containing input vectors like
caseid f1 f2 f3 f4
1 30 45 9.5 1500
2 35 45 8 1600
3 38 47 10 1550
4 32 50 9.5 1800
..
..
..
t1 30 45 9.5 1500(target)
output should like
NO. f1 f2 f3 f4
t1 30 45 9.5 1500 (target)
21 35 45 10 1500(1st closest to target)
39 35 50 8 1500 (2nd closes)
56 35 42 9.5 1500 (3rd closes)
This looks like a classic nearest neighbor query to me, not like clustering.
Also I'd be careful with using Euclidean distance here. A difference of 1 in attribute f1 does not look like it is equal to a difference of 1 in attribute f4. The values seem to have a completely different magnitude.
Related
Using Postgres 11.6, I'm trying to analyze some event data. The goal is to find the durations for all events with a specific name, and then split each one out into evenly sized buckets. We're looking for any times that "clump" for a specific event. I'm editing my question as the specific case may be obscuring what I'm trying to ask.
Simple example
The question is "how do you group rows by a value, then split occurrences by frequency into buckets with count and average for each of those buckets." Here's a hand-done toy example with rounded averages:
Months with values, each number here represents a row.
Jan 12 24 60 150 320 488
Feb 8 16 40 100 220
Mar 4 8 20 310
Overall figures
Month Count Avg Min Max
Jan 6 176 12 488
Feb 5 77 8 220
Mar 4 86 4 310
The same original data, but with more data, including repeated values and a wider range.
Jan 12 12 12 12 24 24 60 60 150 320 488 500
Feb 8 8 8 8 8 16 40 100 220 440 1100
Mar 4 8 8 8 8 20 20 20 20 310
Overall figures
Month Count Avg Min Max
Jan 12 140 12 500
Feb 11 178 8 1100
Mar 10 43 4 310
Mock-up of one of the sets of data split out into 3 buckets
Month Count Avg Min Max Bucket
Jan 4 12 12 12 0
Jan 4 42 24 60 1
Jan 4 365 150 500 2
...and so on for Feb and Mar
I'm just guessing at how the buckets would split in the mock-up above.
That pretty much captures what I'm trying to do. Group by month name (from_to_node in my real case), split the resulting rows into buckets, and then get min, max, avg, and count for each bucket. It's starting to sound like a pivot (?)
Real Table Setup
Here's the structure of table I'm getting a feed for:
CREATE TABLE IF NOT EXISTS data.edge_event (
id uuid,
inv_id uuid,
facility_id uuid,
from_node citext,
to_node citext,
from_to_node citext,
from_node_dts timestamp without time zone,
to_node_dts timestamp without time zone,
seconds integer,
cycle_id uuid
);
The duration is pre-calculated in seconds, and the area of interest for now is only the from_to_node name. So, it's fair to think of the example as
CREATE TABLE IF NOT EXISTS data.edge_event (
from_to_node citext,
seconds integer
);
Raw Data
Within the edge_event table, there are 159 distinct from_to_node values over around 300K event rows. Some are found in only a handful of edge_event records, some are found in thousands, or tens of thousands. That's too much to provide a good sample for. But to make the problem simpler to follow, a from_to_node might be
Boxing_Assembly 1256
Meaning "it took 1256 seconds to move this part from the Boxing phase to the Assembly phase." And here we might have 10,000 other records for "Boxing_Assembly" with different durations.
Goal
We're looking for two things out of each from_to_node. For something like Boxing_Assembly, I'm trying to do this:
Sort the seconds taken into buckets, say 20 buckets. This is for a histogram.
For each bucket get the
count of edge_event rows
avg(seconds) within the bucket
min/first_value(seconds) within the bucket
max/last_value(seconds) within the bucket
So, we're looking to chart durations to look for clusters, and then get the raw seconds out of any common clusters.
What I've tried
I've tried a lot of different code, and I've not succeeded. It seems like a problem for GROUP BY and/or window functions. There's something I'm not getting, as my results are far from the mark.
I know that I haven't provided sample data, which makes it harder to help. But I'm guessing that what I'm missing is one++ concepts. Pretty much, I want to know how to split out the edge_event data by from_to_node and then by seconds. Given the huge ranges across from_to_nodes, I'm trying to bucket each individually based on their own min/max.
Thanks very much for any help.
Draft Attempt
I've developed a query that works a bit, but not entirely. This is an edit from my original post with broken code.
WITH
min_max AS
(
SELECT from_to_node,
min(seconds),
max(seconds)
FROM edge_event
GROUP BY from_to_node
)
SELECT edge_event.from_to_node,
width_bucket (seconds, min_max.min, min_max.max, 99) as bucket, -- Bucket are counted from 0, so 9 gets you 10 buckets, if you have enough data.
count(*) as frequency,
min(seconds) as seconds_min,
max(seconds) as seconds_max,
max(seconds) - min(seconds) as bucket_width,
round(avg(seconds)) as seconds_avg
FROM edge_event
JOIN min_max ON (min_max.from_to_node = edge_event.from_to_node)
WHERE min_max.min <> min_max.max AND -- Can't have a bucket with an upper and lower bound that are the same.
edge_event.from_to_node IN (
'Boxing_Assembly',
'Assembly_Waiting For QA')
GROUP BY edge_event.from_to_node,
bucket
ORDER BY from_to_node,
bucket
What I'm getting back looks pretty good:
from_to_node bucket frequency seconds_min seconds_max bucket_width seconds_avg
Boxing_Assembly 1 912 17 7052 7035 3037
Boxing_Assembly 2 226 7058 13937 6879 9472
Boxing_Assembly 3 41 14151 21058 6907 16994
Boxing_Assembly 4 16 21149 27657 6508 23487
Boxing_Assembly 5 4 28926 33896 4970 30867
Boxing_Assembly 6 1 37094 37094 0 37094
Boxing_Assembly 7 1 43228 43228 0 43228
Boxing_Assembly 10 2 63666 64431 765 64049
Boxing_Assembly 14 1 94881 94881 0 94881
Boxing_Assembly 16 1 108254 108254 0 108254
Boxing_Assembly 37 1 257226 257226 0 257226
Boxing_Assembly 40 1 275140 275140 0 275140
Boxing_Assembly 68 1 471727 471727 0 471727
Boxing_Assembly 100 1 696732 696732 0 696732
Assembly_Waiting For QA 1 41875 1 18971 18970 726
Assembly_Waiting For QA 9 1 207457 207457 0 207457
Assembly_Waiting For QA 15 1 336711 336711 0 336711
Assembly_Waiting For QA 38 1 906519 906519 0 906519
Assembly_Waiting For QA 100 1 2369669 2369669 0 2369669
One problem here is that the buckets aren't evenly sized...they seem kind of weird. I've also tried specifying 10, 20, or 100 buckets, and get similar results. I'm hoping that there is a better way to allocate the data to buckets that I'm missing, and that there's a way to have zero-entry buckets instead of gaps.
I would use the PostgreSQL optimizer for that. It collects exactly the information you want.
Create a temporary table with the values you are interested in and ANALYZE it. Then look into pg_stats for the following:
if there are "most common values", you have them and their frequency right there.
Otherwise, look for adjacent histogram boundaries that are close together. Such a bucket is an interval where values are "lumped".
I am using fminsearch to fit parameters for a system of DEs to observed data. I am not expecting to get a great fit.
fminsearch pretty quickly finds what appears to be an acceptable min for the objective function, but then does not stop. It's running for a really long time, and I cannot figure out why.
I am using the options
options = optimset('Display','iter','TolFun',1e-4,'TolX',1e-4,'MaxFunEvals',1000);
which I understood to mean that when the value of the objective function drops to below 1e-4 that would be considered sufficient. Alternatively when they could no longer change the parameters whatever is the best would be returned.
The output is
Iteration Func-count min f(x) Procedure
0 1 8.13911e+10
1 8 7.2565e+10 initial simplex
2 9 7.2565e+10 reflect
3 10 7.2565e+10 reflect
4 11 7.2565e+10 reflect
5 12 7.2565e+10 reflect
6 13 7.2565e+10 reflect
7 15 6.85149e+10 expand
8 16 6.85149e+10 reflect
9 17 6.85149e+10 reflect
10 19 6.20681e+10 expand
11 20 6.20681e+10 reflect
12 22 5.55199e+10 expand
13 23 5.55199e+10 reflect
14 25 4.86494e+10 expand
15 26 4.86494e+10 reflect
16 27 4.86494e+10 reflect
17 29 3.65616e+10 expand
18 30 3.65616e+10 reflect
19 31 3.65616e+10 reflect
20 33 2.82946e+10 expand
21 34 2.82946e+10 reflect
22 36 2.02985e+10 expand
23 37 2.02985e+10 reflect
24 39 1.20011e+10 expand
25 40 1.20011e+10 reflect
26 41 1.20011e+10 reflect
27 43 5.61651e+09 expand
28 44 5.61651e+09 reflect
29 45 5.61651e+09 reflect
30 47 2.1041e+09 expand
31 48 2.1041e+09 reflect
32 49 2.1041e+09 reflect
33 51 5.15751e+08 expand
34 52 5.15751e+08 reflect
35 53 5.15751e+08 reflect
36 55 7.99868e-05 expand
37 56 7.99868e-05 reflect
38 58 7.99835e-05 reflect
39 59 7.99835e-05 reflect
I have previously let this run for a lot longer and it's stuck with the same min f(x) for at least the next 30 print outs.
How do I set the options correctly so that when it finds a solution within an acceptable value for the objective function it stops?
Matlab requires that both TolX AND TolFun be satisfied before terminating ("Unlike other solvers, fminsearch stops when it satisfies both TolFun and TolX." See: https://www.mathworks.com/help/matlab/ref/fminsearch.html). You should check what the "x" value (your solution) is doing. I suspect that is changing more than your tolerance specification for each step. (i.e. the value of x is changing more than TolX between iterations but f(x) is not changing by more than TolFun).
For the following typical case:
n = 1000000;
r = randi(n,n,2);
(assume there are 0.05% common numbers between all rows; n could be even tens of millions)
I am looking for a CPU and Memory efficient solution to merge rows based on any common items (here integer numbers). A list of sample codes in Python is available here and a quick try to translate one into Matlab can be found here.
In my attempt they take ages (minutes to hours), so I am in favor of finding faster solution.
For the above example, the typical output should look like (cell):
{
[1 90 34 67 ... 9]
[35 89]
[45000 23 828 130 8999 45326 ... 11]
...
}
Note also that, I have tried to compile as mex but failed due to no-support for cell in Matlab-Coder.
Edit: A tiny demonstration example
%---------------------------------------
clc
n = 100;
r = randi(n,n,2); % random integers in [1,n], size(n,2)
%---------------------------------------
>> r
r =
82 17 % (1) 82 17
91 13 % (2) 91 13
13 32 % (3) 91 13 32 merged with (2), common 13
82 53 % (4) 82 17 53 merged with (1), common 82
64 17 % (5) 82 17 53 64 merged with (4), common 17
...
94 45
13 31 % (77) 91 13 32 31 merged with (3), common 13
57 51
47 52
2 13 % (80) 91 13 32 31 2 merged with (77), common 13
34 80
%---------------------------------------
c = merge(r); % cpu and memory friendly solution is searched for.
%---------------------------------------
c =
[82 17 53 64]
[91 13 32 31 2]
...
You need an index.
In Python, use a dict. In MATLAB - I'd not use MATLAB, because open-source is the future, and MATLAB is dying out.
But Python is quite slow. You can likely get a 10x speedup by using e.g. Cython to translate and optimize the code in C. Avoid using Python data types such as a list of int, because they are very memory intensive. numpy has memory-efficient arrays of integer.
If you get a new pair (a,b) you can use this dictionary to find existing items to merge. Then update the dict after the merge.
Actually for integers, you should use an array instead of a dict.
The trickiest part is handling the case when both a and b exist, but are large different groups. There are some neat optimizations possible here, if that isn't fast enough yet.
It's not clustering, but connected components.
I have read quite a lot about LibSVM library, but I would like to ask you for some advices in my particular case. The problem is that I have some 3D medical images (DCE-MRI) of a stomach. My goal is to perform a segmentation of a kidney, and find its three parts. Therefore, I need to train a classifier - I'm going to use SVM and neural network
Feature vectors:
What is available is the pixel (voxel) brightness value (I guess the value range is [0; 511]). In total, there are 71 frames, each taken every second. So the crucial feature of every voxel is how the voxel brightness/intensity is changing during the examination time. In my case, every part of a kidney has a different chart (see an example below), so the way how the voxels brightness is changing over the time will be used by the classifier.
Training sets:
Every training set is a vector of intensity value of one voxel (74 numbers). An example is presented below:
[22 29 21 7 19 12 23 25 33 28 25 5 21 18 27 21 11 11 26 12 12 31 15 15 12 29 17 34 30 11 12 24 35 28 27 26 29 22 15 23 24 14 14 37 241 313 350 349 382 402 333 344 332 366 339 383 383 379 394 398 402 357 346 379 365 376 366 365 360 363 376 383 389 385]
Summary and question to you:
I have many training sets consisting of 74 values from the range [0; 511]. I have 3 groups of voxels, which have a characteristic feature - the brightness is changing in the similar way. What I want to obtain is a classificator, which after getting one voxel vector with 74 numbers, will assess if the voxel belongs to one of these 3 groups, or to none of them.
Question: how to start with LibSVM, any advices? From what I know now is that I should transform input values to be from the range [0; 1] or [-1; 1]. I have many training sets prepared belonging to one of these 3 groups. I will be grateful for any advice, as I'm a newbie and I just need some tips just to start.
You can train and use you model like this:
model=svmtrain(train_label,train_feature,'-c 1 -g 0.07 -h 0');
% the parameters can be modified
[label, accuracy, probablity]=svmpredict(test_label,test_feaure,model);
train_label must be a vector,if there are more than two kinds of input(0/1),it will be an nSVM automatically. If you have 3 classes, you can label them using {1,2,3}.Its length is equal to the number of samples.
The feature is not restricted. It can be what ever you want.
However, you'd better preprocess them to make the results better. For example, you can change range[0:511] to range[0:1] or minus the mean of the feature.
Notice that the testset data should be preprocessed in the same way.
Hope this will help you!
I'm using Stata and trying to compute conditional means based on time/date. For each store I want to calculate mean (inventory) per year. If there are missing year gaps, then I want to take the mean from the closest two dates' inventory values.
I've used (below) to get overall means per store, but I need more granularity.
egen mean_inv = mean(inventory), by (store)
I've also tried this loop with similar results:
by id, sort: gen v1'=_n'
forvalues x = 1/'=n'{
by store: sum inventory if v1==`x'
replace mean_inv= r(mean) if v1==`x'
}
Visually, I want mean inventory per store: (store id is not sequential)
5/1/2003 2/3/2006 8/9/2006 3/5/2007 6/9/2007 2/1/2008
13 18 12 15 24 11
[mean1] [mean2] [mean3] [mean4] [mean5]
store date inventory
1 16750 17
1 18234 16
1 15844 13
1 17111 14
1 17870 13
1 16929 13.5
1 17503 13
4 15987 18
4 15896 16
4 18211 16
4 17154 18
4 17931 24
4 16776 23
12 16426 26
12 17681 17
12 16386 17
12 16603 18
12 17034 16
12 17205 16
42 15798 18
42 16022 18
42 17496 16
42 17870 18
42 16204 18
42 16778 14
33 18053 23
33 16086 13
33 16450 21
33 17374 19
33 16814 19
33 15834 16
33 16167 16
56 17686 16
56 17623 18
56 17231 20
56 15978 16
56 16811 15
56 17861 20
It is hard to relate your code to the word description of your problem.
Your egen call calculates means by store, not year.
Your longer fragment does not make complete sense given lack of definitions and at least one typo.
Note that your variable v1 contains identifiers that run 1 up within groups of store, and does not distinguish different values of store, as you (seem to) imply. It strains credibility that it produces results anywhere near those by the egen call.
n is not defined and the code evaluating it is presumably intended to be
`=n'
If you calculate
by store: sum inventory if v1 == `x'
several means will be calculated in turn but only the last to be calculated will be accessible as r(mean).
The sample data are unrelated to the problem. There is no year variable and even if the dates are Stata daily dates, they are all dates within 1960.
Setting all that aside, suppose you have variables store, inventory and year. You can try
collapse inventory, by(store year)
fillin store year
ipolate inventory year, gen(inventory2) by(store)
The collapse produces a reduced dataset of means. The ipolate interpolates across gaps, as you ask. fillin may not be adequate to give all the store and year combinations you want and you may need to add further years manually before the interpolation. If you want to put these results back with the original data, that's a merge.
In total, this is a pretty messy question.