Related
Is there a global vertex aggregation function in GraphX? I hope this function can calculate the number of different values of vertex attributes, like the 'collections.Counter' function in Python.
For example, I have a graph like: (Each line represents a vertex, weight of each edge = 1)
[source vertex, source vertex some-attrbute-value, [dst vertex1, dst vertex2, ...]]
{
[1, 1, [2, 3]]
[2, 1, [1, 3, 4]]
[4, 1, [2, 3]]
[3, 1, [1, 2, 4]]
[5, 2, [4, 6, 7]]
[6, 2, [5, 7]]
[7, 2, [5, 6]]
}
and the output looks like following:
{1: 4, 2: 3}
or the vertexId_list could be given (better!) like followings:
{1: [1, 2, 3, 4], 2: [5, 6, 7]}
What's more, it's perfect if this function can work together with PregelAPI. For example, Pregel control the stop point by using this function: When the number of some vertex-attr-value reach the threshold(for example, the number of value 1 = 4(There are 4 vertex which attrbute = 1)), the superStep stops.
P.S. This function should seems like "AggregatorXXX(vertex) -> Message or Sth", not the RDD-relative method, like filter/map, etc.
Sorry for my poor English. :)..
(To anyone who reads this, just to not waste your time, I wrote up this question and then came up with a solution to it right after I wrote it. I am posting this here just to help out anyone who happened to also be thinking about something like this.)
I have a vector with elements that I would like to sum up. The elements that I would like to add up are elements that share the same "triggerNumber". For example:
vector = [0, 1, 1, 1, 1]
triggerNumber = [1, 1, 1, 2, 2]
I will sum up the numbers that share a triggerNumber of 1 (so 0+1+1 =2) and share a triggerNumber of 2 (so 1+1+1 = 3). Therefore my desiredOutput is the array [2, 2].
accumarray accomplishes this task, and if I give it those two inputs:
output = accumarray(triggerNumber.',vector.').'
which returns [2, 2]. But, while my "triggerNumbers" are always increasing, they are not necessarily always increasing by one. So for example I might have the following situation:
vector = [0, 1, 1, 1, 1]
triggerNumber = [4, 4, 4, 6, 6]
output = accumarray(triggerNumber.',vector.').'
But now this returns the output:
output = [0, 0, 0, 2, 0, 2]
Which is not what I want. I want to just sum up elements with the same trigger number (in order), so the desired output is still [2, 2]. Naively I thought that just deleting the zeros would be sufficient, but then that messes up the situation with the inputs:
vector = [0, 0, 0, 1, 1]
triggerNumber = [4, 4, 4, 6, 6]
which if I deleted the zeroes would return just [2] instead of the desired [0, 2].
Any ideas for how I can accomplish this task (in a vectorized way of course)?
I just needed to turn things like [4, 4, 4, 6, 6] into [1, 1, 1, 2, 2], which can be done with a combination of cumsum and diff.
vector = [0, 0, 0, 1, 1];
triggerNumber = [4, 4, 4, 6, 6];
vec1 = cumsum(diff(triggerNumber)>0);
append1 = [0, vec1];
magic = append1+1;
output = accumarray(magic.',vector.').'
which returns [2, 2]....and hopefully my method works for all cases.
I want to learn a little more about how pyspark partitions data. I need a function such that:
a = sc.parallelize(range(10), 5)
show_partitions(a)
#output:[[0, 1], [2, 3], [4, 5], [6, 7], [8, 9]] (or however it partitions)
The glom function is what you are looking for:
glom(self): Return an RDD created by coalescing all elements within each partition into a list.
a = sc.parallelize(range(10), 5)
a.glom().collect()
#output:[[0, 1], [2, 3], [4, 5], [6, 7], [8, 9]]
val data = List((1,3),(1,2),(1,4),(2,3),(3,6),(3,8))
val rdd = sc.parallelize(data)
rdd.glom().collect()
.foreach(a => {
a.foreach(println);
println("=====")})
in this way , you can check how the data is partitioned
I wanted to make a predicate that returns a list of a number dividers.
Example: 72 = 2*2*2*3*3.
prdel(A,[],_):-
A is 1.
prdel(P,[D|L],D):-
0 is mod(P,D),
P1 is P/D,
prdel(P1,L,D).
prdel(P,L,D):-
D1 is D+1,
prdel(P,L,D1).
This works and returns the right list. The problem is that it does not stop after that but returns the same list over and over again if I press space (I am sorry I don't know the term in English when you use the same predicate to get different answer). I want it to stop after the first time.
I tried to edit the last one like that,
prdel(P,L,D):-
D1 is D+1,
D1<P,
prdel(P,L,D1).
but now it returns only false and not the list.
EDIT:
I am looking for an answer without cut.
One problem in your code is that it keeps trying to divide the number P by D even when it is clear that the division is not going to succeed because D is too high. This lets D "run away" without a limit.
Adding a check for D1 to be below or equal to P fixes this problem:
prdel(1,[],_).
prdel(P,[D|L],D):-
0 is mod(P,D),
P1 is P/D,
prdel(P1,L,D).
prdel(P,L,D):-
D1 is D+1,
D1 =< P,
prdel(P,L,D1).
This produces all combinations of divisors, including non-prime ones (demo).
[[2, 2, 2, 3, 3], [2, 2, 2, 9], [2, 2, 3, 6],
[2, 2, 18], [2, 3, 3, 4], [2, 3, 12], [2, 4, 9],
[2, 6, 6], [2, 36], [3, 3, 8], [3, 4, 6], [3, 24],
[4, 18], [6, 12], [8, 9], [72]]
If you do not want that, add the condition that mod(P,D) > 0 in the last clause:
prdel(1,[],_).
prdel(P,[D|L],D):-
0 is mod(P,D),
P1 is P/D,
prdel(P1,L,D).
prdel(P,L,D):-
mod(P,D) > 0,
D1 is D+1,
D1 =< P,
prdel(P,L,D1).
This produces only [2, 2, 2, 3, 3] (demo).
Waht i need to do is take a single dimensional array, ie:
[1, 1, 2, 2, 3, 3]
and turn it into an array of points:
[[1, 1], [2, 2], [3, 3]]
I am hoping for a simple native matlab way of doing it rather then a function. This will be going into sets of points ie:
[ [[1, 1], [2, 2], [3, 3]],
[[4, 4], [5, 5], [6, 6]],
[[7, 7], [7, 7], [8, 8]] ]
The reason this is going to happen is the points will be stored in a text file as a single stream and i need to turn them into something meaningful.
First note that a horizontal concatenation of row vectors will result in one larger row vector rather than in a row of pairs, that is [[1, 1], [2, 2], [3, 3]] is the same as [1 1 2 2 3 3]. Hence, you need to concatenate them vertically.
You can try
a = [1, 1, 2, 2, 3, 3];
b = reshape(a, 2, floor(length(a)/2))';
This will result in a matrix where each row represents the coordinates of one point.
b =
1 1
2 2
3 3
I'm just adding this answer for the sake of diversity:
Just as H.Muster said, concatenation of vectors will result in a larger vector or a matrix (depending on your operation). You can go with that.
But you can also use a cell array, which is a set of data containers called "cells". A cell can contain any type of data, regradless of what other cells contain in the same cell array.
In your case, creating a cell array can be done using a slightly different syntax (than H.Muster's answer):
a = [1, 1, 2, 2, 3, 3];
p = mat2cell(a, 1, 2 * ones(1, numel(a) / 2))
p is a cell array, each cell containing a 1-by-2 point vector. To access an element in a cell array, you'll have to use curly braces. For instance, the second point would be p{2} = [2, 2].