I have a matrix (a log file) structured as follows:
Date hour sender receiver message
7/14/2014 19/54/8 C5 S a
7/14/2014 19/54/9 S C5 c
7/14/2014 19/54/10 C5 S a
7/14/2014 19/54/11 S C5 b
7/14/2014 19/54/12 C5 S d
7/14/2014 19/54/13 C5 S h
7/14/2014 19/54/14 C5 S i
7/14/2014 19/54/15 S C5 k
7/14/2014 19/54/16 C5 S p
7/14/2014 19/54/17 C3 S a
7/14/2014 19/54/18 S C3 c
7/14/2014 19/54/19 C3 S a
………………………………………………………………..
I want to create the occurrence matrix from this file which contains the numbers of occurrences of couples of messages by each day and for each client see the example:
C5 S a
S C5 c
C5 S a
S C5 b
C5 S d
C5 S h
C5 S i
C5 S k
C2 S a
S C2 c
C2 S a
S C2 b
The occurrence matrix is as follow:
C5 C2 C3 c4
ac 5 4 2 3
ca 3 2 1 2
ab 1 1 0 1
bd 2 3 4 1
dh 2 3 4 0
ik 1 1 1 1
Any help please?
Related
I have a spark dataset like this one:
key id val1 val2 val3
1 a a1 a2 a3
2 a a4 a5 a6
3 b b1 b2 b3
4 b b4 b5 b6
5 b b7 b8 b9
6 c c1 c2 c3
I would like to group all rows by id in a list or array like this:
(a, ([1 a a1 a2 a3], [2 a a4 a5 a6]) ),
(b, ([3 b b1 b2 b3], [4 b b4 b5 b6], [5 b b7 b8 b9]) ),
(c, ([6 c c1 c2 c3]) )
I have used map to output key/value pairs with the right key but I have troubles in building the final key/array.
Can anybody help with that?
how about this:
import org.apache.spark.sql.functions._
df.withColumn("combined",array("key","id","val1","val2","val3")).groupby("id").agg(collect_list($"combined"))
The Array function converts the columns into an array of column and then its a simple groupby with collect_list
import org.apache.spark.ml.feature.VectorAssembler
import org.apache.spark.sql.functions._
val assembler = new VectorAssembler()
.setInputCols(Array("key", "id", "val1", "val2", "val3","score"))
.setOutputCol("combined")
val dfRes = assembler.transform(df).groupby("id").agg(collect_list($"combined"))
File Content of my xzy.txt file
key id val1 val2 val3
1 a a1 a2 a3
2 a a4 a5 a6
3 b b1 b2 b3
4 b b4 b5 b6
5 b b7 b8 b9
6 c c1 c2 c3
Code with Required Output
Input file Content
I have a crystal report in this form:
group1
AER#
M1 A2 A3 A4 A5
M2 B2 B3 B4 B5
M2A C2 C3 C4 C5
M3 D2 D3 D4 D5
M4 E2 E3 E4 E5
These records are grouped on the basis of group1 field.
I have a requirement that these records be displayed such that
for any same AER# value different M3 and M4 are displayed as:
group1
AER#
M1 A2 A3 A4 A5
M2 B2 B3 B4 B5
M2A C2 C3 C4 C5
M3 D2a D3a D4 D5
M4 E2b E3a E4 E5
M3 D2b D3a D4 D5
M4 E2b E3b E4 E5
Instead of two separate 5 line groups.
Meaning that for same AER# there could be 3 sets of rows with only last two rows different. They should be displayed in above fashion(with last two lines repeated three times) instead of 3 separate chunks.
Any suggestions?
Thanks.
S
Note: Currently I don't have Crystal Reports hence didn't got the chance to test below solution.
Extrace last two characters of the second column and group by those characters ater the grouping `AER#'
Below is the example of the formula
Assuming your column 2 name as X.
Create a formula #group2
x[2]+x[3]
Now use #group2 to group and in group foooter place all the columns you require.
I have this matrix:
a1 a2 a3 a4 a5
b1 b2 b3 b4 b5
c1 c2 c3 c4 c5
d1 d2 d3 d4 d5
e1 e2 e3 e4 e5
and i need to transform it to this matrix:
a1 a2 a3 b1 b2 b3 c1 c2 c3
a2 a3 a4 b2 b3 b4 c2 c3 c4
a3 a4 a5 b3 b4 b5 c3 c4 c5
b1 b2 b3 c1 c2 c3 d1 d2 d3
b2 b3 b4 c2 c3 c4 d2 d3 d4
b3 b4 b5 c3 c4 c5 d3 d4 d5
c1 c2 c3 d1 d2 d3 e1 e2 e3
c2 c3 c4 d2 d3 d4 e2 e3 e4
c3 c4 c5 d3 d4 d5 e3 e4 e5
i.e. by taking a 3x3 matrix and iterating over the old matrix putting it as a new row in the new matrix.
How can i do this in matlab? I heard loops are very bad and if I can I should use matrix operations.
If you have access to image processing toolbox you can use im2col which even works for symbolic variables too.
A = sym('A%d%d', [5 5]);
A = sym(A, 'real');
Subs = mat2cell(im2col(A, [3 1])', 3*ones(1,5));
Res = im2col(Subs, [3 1]);
Res is a cell matrix containing the sub matrices.
To concatenate it into a matrix:
reshape([Res{:}]', [9 9])
or if it's not symbolic this also works: cell2mat(Res).
Take a look at this code :
clear
clc
a1 = 1; a2 = 2; a3 = 3; a4 = 4; a5 = 5;
b1 = 11; b2 = 12; b3 = 13; b4 = 14; b5 = 15;
c1 = 21; c2 = 22; c3 = 23; c4 = 24; c5 = 25;
d1 = 31; d2 = 32; d3 = 33; d4 = 34; d5 = 35;
e1 = 41; e2 = 42; e3 = 43; e4 = 44; e5 = 45;
A = [a1 a2 a3 a4 a5; b1 b2 b3 b4 b5; c1 c2 c3 c4 c5; d1 d2 d3 d4 d5; e1 e2 e3 e4 e5];
B = zeros(9,9);
sizeA = size(A);
sizeB = size(B);
jj = 1;
kk = 1;
for ii = 1 : sizeB(1)
B(ii,1:sizeB(1)) = [A(kk, jj:jj+2) A(kk + 1, jj:jj+2) A(kk + 2, jj:jj+2)]
jj = jj + 1;
if mod(ii,3) == 0
kk = kk + 1;
jj = 1;
end
end
I tested it for your case where A is 5x5 and B is 9x9. Might need to be modified to work with different size of A and B.
You can do this in a somewhat general sense, but it is not pretty:
a = [a1, a2, a3, a4, a5];
b = [b1, b2, b3, b4, b5];
% etc.
% Rearrange your vector into a matrix configured how you want it
A = flipdim(toeplitz(a),1); % 5 x 5 matrix in our case
A = A(ceil(size(A,1)/2):size(A,1), ceil(size(A,2)/2):size(A,2)); % now 3 x 3
%%%
% Repeat similar process for b --> B, c --> C, etc.
%%%
% Build the resulting matrix up block-by-block
M = [A, B, C;
B, C, D;
C, D, E];
I was unable to find a good way to generalize the construction of M, and unfortunately forming a toeplitz matrix using toeplitz([a,b,c,d,e]) fails; you do not get the right format for your sub-matrices. Still, you may be able to use this to start on something.
Let there be five matrices given as:
A= [A1 A1 A1 A1 A1; A2 A2 A2 A2 A2; A3 A3 A3 A3 A3]
B= [B1 B1 B1 B1 B1; B2 B2 B2 B2 B2;B3 B3 B3 B3 B3]
C=[ C1 C1 C1 C1 C1; C2 C2 C2 C2 C2; C3 C3 C3 C3 C3]
D= [D1 D1 D1 D1 D1 ; D2 D2 D2 D2 D2; D3 D3 D3 D3 D3]
E=[ E1 E1 E1 E1 E1; E2 E2 E2 E2 E2; E3 E3 E3 E3 E3]
I want to make a program such that ouput consists of taking each row of each given matrix and forming a new matrix. how to use looping in such cases when length of matrices increases and number of given matrices also increases. This problem seemed to me a complex one. Because I want to generalize by using loop and output for any number of matrices say 20 and having number of columns also increased to say 25, then how to get these P1 to P20 outputs. Can anyone help me regarding this complex trouble using Matlab
P1=[ A1 A1 A1 A1 A1; B1 B1 B1 B1 B1; C1 C1 C1 C1 C1 C1; D1 D1 D1 D1 D1; E1 E1 E1 E1 E1]
P2=[ A2 A2 A2 A2 A2; B2 B2 B2 B2 B2; C2 C2 C2 C2 C2 C2; D2 D2 D2 D2 D2; E2 E2 E2 E2 E2]
and similarly other matrices is obtained .
Note: That the given 5 matrices are generated with help of loop. So first I would be getting values as :
A= A1
B= B1
C=C1
D=D1
E=E1
A= A1 A1
B= B1 B1
C=C1 C1
D=D1 D1
E=E1 E1 .... AND SO ON
Get a loop and put all the matrix together to form a 3D tensor. Or just put the matrices in the 3D tensor when they are created.
M(:,:,1) = A; M(:,:,2) = B; etc
then
squeeze(M(1,:,:))' is the P1, squeeze(M(2,:,:))' is the P2
Example:
M(:,:,1) =
1 2
3 4
M(:,:,2) =
5 6
7 8
>> squeeze(M(1,:,:))'
ans =
1 2
5 6
tell me how to get every possible combination of hash
Here is an example
my %data = (
'a' => [qw(a1 a2 a3)],
'b' => [qw(b1 b2 b3)],
'c' => [qw(c1 c2 c3)]);
to get
a1
a2
a3
b1
b2
b3
c1
c2
c3
a1 b1
a1 b2
a1 b3
a1 c1
a1 c2
a1 c3
b1 c1
b1 c2
b1 c3
b2 c1
b2 c2
b2 c3
b3 c1
b3 c2
b3 c3
a1 b1 c1
a1 b1 c2
a1 b1 c3
a1 b2 c1
a1 b2 c2
a1 b2 c3
a1 b3 c1
a1 b3 c2
a1 b3 c3
a2 b1 c1
a2 b1 c2
a2 b1 c3
a2 b2 c1
a2 b2 c2
a2 b2 c3
a2 b3 c1
a2 b3 c2
a2 b3 c3
a3 b1 c1
a3 b1 c2
a3 b1 c3
a3 b2 c1
a3 b2 c2
a3 b2 c3
a3 b3 c1
a3 b3 c2
a3 b3 c3
thanks
Use brian d foy's Set::CrossProduct module. You'll need to massage your hash into array of arrays in an obvious way.
use Set::CrossProduct;
my $iterator = Set::CrossProduct->new( ARRAY_OF_ARRAYS );
my $tuples = $iterator->combinations;
My module List::Gen contains a cartesian function that can produce the results you want. This code seems to do the trick, but your example does not contain all of the permutations that this will produce, which I am assuming is just an omission in the example.
use List::Gen 'cartesian';
my %data = (
'a' => [qw(a1 a2 a3)],
'b' => [qw(b1 b2 b3)],
'c' => [qw(c1 c2 c3)],
);
my $product = cartesian {join ' ' => sort grep defined, #_}
map {[#$_, undef]}
values %data;
say for sort {length $a <=> length $b or $a cmp $b} #$product;
That is a bit dense, so to explain:
values %data returns the arrays in %data
map {[#$_, undef]} then attaches an empty value to the end of each, since you want the partial combinations
cartesian {join ' ' => sort grep defined, #_} then does the meat of the work, computing the Cartesian product of the arrays while subtracting out the undefined elements, and sorting the values as your example shows.
sort {length $a <=> length $b or $a cmp $b} #$product then prints out the product in the order specified.