Related
In a typical N-Body simulation, at the end of each epoch, each locale would need to share its own portion of the world (i.e. all bodies) to the rest of the locales. I am working on this with a local-view approach (i.e. using on Loc statements). I encountered some strange behaviours that I couldn't make sense out of, so I decided to make a test program, in which things got more complicated. Here's the code to replicate the experiment.
proc log(args...?n) {
writeln("[locale = ", here.id, "] [", datetime.now(), "] => ", args);
}
const max: int = 50000;
record stuff {
var x1: int;
var x2: int;
proc init() {
this.x1 = here.id;
this.x2 = here.id;
}
}
class ctuff {
var x1: int;
var x2: int;
proc init() {
this.x1 = here.id;
this.x2 = here.id;
}
}
class wrapper {
// The point is that total size (in bytes) of data in `r`, `c` and `a` are the same here, because the record and the class hold two ints per index.
var r: [{1..max / 2}] stuff;
var c: [{1..max / 2}] owned ctuff?;
var a: [{1..max}] int;
proc init() {
this.a = here.id;
}
}
proc test() {
var wrappers: [LocaleSpace] owned wrapper?;
coforall loc in LocaleSpace {
on Locales[loc] {
wrappers[loc] = new owned wrapper();
}
}
// rest of the experiment further down.
}
Two interesting behaviours happen here.
1. Moving data
Now, each instance of wrapper in array wrappers should live in its locale. Specifically, the references (wrappers) will live in locale 0, but the internal data (r, c, a) should live in the respective locale. So we try to move some from locale 1 to locale 3, as such:
on Locales[3] {
var timer: Timer;
timer.start();
var local_stuff = wrappers[1]!.r;
timer.stop();
log("get r from 1", timer.elapsed());
log(local_stuff);
}
on Locales[3] {
var timer: Timer;
timer.start();
var local_c = wrappers[1]!.c;
timer.stop();
log("get c from 1", timer.elapsed());
}
on Locales[3] {
var timer: Timer;
timer.start();
var local_a = wrappers[1]!.a;
timer.stop();
log("get a from 1", timer.elapsed());
}
Surprisingly, my timings show that
Regardless of the size (const max), the time of sending the array and record strays constant, which doesn't make sense to me. I even checked with chplvis, and the size of GET actually increases, but the time stays the same.
The time to send the class field increases with time, which makes sense, but it is quite slow and I don't know which case to trust here.
2. Querying the locales directly.
To demystify the problem, I also query the .locale.id of some variables directly. First, we query the data, which we expect to live in locale 2, from locale 2:
on Locales[2] {
var wrappers_ref = wrappers[2]!; // This is always 1 GET from 0, okay.
log("array",
wrappers_ref.a.locale.id,
wrappers_ref.a[1].locale.id
);
log("record",
wrappers_ref.r.locale.id,
wrappers_ref.r[1].locale.id,
wrappers_ref.r[1].x1.locale.id,
);
log("class",
wrappers_ref.c.locale.id,
wrappers_ref.c[1]!.locale.id,
wrappers_ref.c[1]!.x1.locale.id
);
}
And the result is:
[locale = 2] [2020-12-26T19:36:26.834472] => (array, 2, 2)
[locale = 2] [2020-12-26T19:36:26.894779] => (record, 2, 2, 2)
[locale = 2] [2020-12-26T19:36:27.023112] => (class, 2, 2, 2)
Which is expected. Yet, if we query the locale of the same data on locale 1, then we get:
[locale = 1] [2020-12-26T19:34:28.509624] => (array, 2, 2)
[locale = 1] [2020-12-26T19:34:28.574125] => (record, 2, 2, 1)
[locale = 1] [2020-12-26T19:34:28.700481] => (class, 2, 2, 2)
Implying that wrappers_ref.r[1].x1.locale.id lives in locale 1, even though it should clearly be on locale 2. My only guess is that by the time .locale.id is executed, the data (i.e. the .x of the record) is already moved to the querying locale (1).
So all in all, the second part of the experiment lead to a secondary question, whilst not answering the first part.
NOTE: all experiment are run with -nl 4 in chapel/chapel-gasnet docker image.
Good observations, let me see if I can shed some light.
As an initial note, any timings taken with the gasnet Docker image should be taken with a grain of salt since that image simulates the execution across multiple nodes using your local system rather than running each locale on its own compute node as intended in Chapel. As a result, it is useful for developing distributed memory programs, but the performance characteristics are likely to be very different than running on an actual cluster or supercomputer. That said, it can still be useful for getting coarse timings (e.g., your "this is taking a much longer time" observation) or for counting communications using chplvis or the CommDiagnostics module.
With respect to your observations about timings, I also observe that the array-of-class case is much slower, and I believe I can explain some of the behaviors:
First, it's important to understand that any cross-node communications can be characterized using a formula like alpha + beta*length. Think of alpha as representing the basic cost of performing the communication, independent of length. This represents the cost of calling down through the software stack to get to the network, putting the data on the wire, receiving it on the other side, and getting it back up through the software stack to the application there. The precise value of alpha will depend on factors like the type of communication, choice of software stack, and physical hardware. Meanwhile, think of beta as representing the per-byte cost of the communication where, as you intuit, longer messages necessarily cost more because there's more data to put on the wire, or potentially to buffer or copy, depending on how the communication is implemented.
In my experience, the value of alpha typically dominates beta for most system configurations. That's not to say that it's free to do longer data transfers, but that the variance in execution time tends to be much smaller for longer vs. shorter transfers than it is for performing a single transfer versus many. As a result, when choosing between performing one transfer of n elements vs. n transfers of 1 element, you'll almost always want the former.
To investigate your timings, I bracketed your timed code portions with calls to the CommDiagnostics module as follows:
resetCommDiagnostics();
startCommDiagnostics();
...code to time here...
stopCommDiagnostics();
printCommDiagnosticsTable();
and found, as you did with chplvis, that the number of communications required to localize the array of records or array of ints was constant as I varied max, for example:
locale
get
execute_on
0
0
0
1
0
0
2
0
0
3
21
1
This is consistent with what I'd expect from the implementation: That for an array of value types, we perform a fixed number of communications to access array meta-data, and then communicate the array elements themselves in a single data transfer to amortize the overheads (avoid paying multiple alpha costs).
In contrast, I found that the number of communications for localizing the array of classes was proportional to the size of the array. For example, for the default value of 50,000 for max, I saw:
locale
get
put
execute_on
0
0
0
0
1
0
0
0
2
0
0
0
3
25040
25000
1
I believe the reason for this distinction relates to the fact that c is an array of owned classes, in which only a single class variable can "own" a given ctuff object at a time. As a result, when copying the elements of array c from one locale to another, you're not just copying raw data, as with the record and integer cases, but also performing an ownership transfer per element. This essentially requires setting the remote value to nil after copying its value to the local class variable. In our current implementation, this seems to be done using a remote get to copy the remote class value to the local one, followed by a remote put to set the remote value to nil, hence, we have a get and put per array element, resulting in O(n) communications rather than O(1) as in the previous cases. With additional effort, we could potentially have the compiler optimize this case, though I believe it will always be more expensive than the others due to the need to perform the ownership transfer.
I tested the hypothesis that owned classes were resulting in the additional overhead by changing your ctuff objects from being owned to unmanaged, which removes any ownership semantics from the implementation. When I do this, I see a constant number of communications, as in the value cases:
locale
get
execute_on
0
0
0
1
0
0
2
0
0
3
21
1
I believe this represents the fact that once the language has no need to manage the ownership of the class variables, it can simply transfer their pointer values in a single transfer again.
Beyond these performance notes, it's important to understand a key semantic difference between classes and records when choosing which to use. A class object is allocated on the heap, and a class variable is essentially a reference or pointer to that object. Thus, when a class variable is copied from one locale to another, only the pointer is copied, and the original object remains where it was (for better or worse). In contrast, a record variable represents the object itself, and can be thought of as being allocated "in place" (e.g., on the stack for a local variable). When a record variable is copied from one locale to the other, it's the object itself (i.e., the record's fields' values) which are copied, resulting in a new copy of the object itself. See this SO question for further details.
Moving on to your second observation, I believe that your interpretation is correct, and that this may be a bug in the implementation (I need to stew on it a bit more to be confident). Specifically, I think you're correct that what's happening is that wrappers_ref.r[1].x1 is being evaluated, with the result being stored in a local variable, and that the .locale.id query is being applied to the local variable storing the result rather than the original field. I tested this theory by taking a ref to the field and then printing locale.id of that ref, as follows:
ref x1loc = wrappers_ref.r[1].x1;
...wrappers_ref.c[1]!.x1.locale.id...
and that seemed to give the right result. I also looked at the generated code which seemed to indicate that our theories were correct. I don't believe that the implementation should behave this way, but need to think about it a bit more before being confident. If you'd like to open a bug against this on Chapel's GitHub issues page, for further discussion there, we'd appreciate that.
I am trying to run a pivot operation on a selection made within a ticker function as follows:
// Trades table
trades:(
[]time:`timespan$();
sym:`symbol$();
exch:`symbol$();
side:`symbol$();
price:`float$();
size:`float$());
// Generic pivot function
piv:{[t;k;p;v]f:{[v;P]`${raze "_" sv x} each string raze P,'/:v};v:(),v; k:(),k; p:(),p;G:group flip k!(t:.Q.v t)k;F:group flip p!t p;key[G]!flip(C:f[v]P:flip value flip key F)!raze{[i;j;k;x;y]a:count[x]#x 0N;a[y]:x y;b:count[x]#0b;b[y]:1b;c:a i;c[k]:first'[a[j]#'where'[b j]];c}[I[;0];I J;J:where 1<>count'[I:value G]]/:\:[t v;value F]};
// Ticker function that "ticks" every 5 seconds
if[not system"t";system"t 5000";
.z.ts:{
// Selection is made from the trades table with the index reset
trds:0!select first_price:first price, last_price:last price, mean_size:avg size, volume:sum size, min_price:min price, max_price:max price by time:1 xbar time.second, exch, sym from trades;
// Check if selection returns at least one row and run pivot
if[(count trds)>0; (
ptrds:piv[`trds;`time;`exch`sym;`first_price`last_price`mean_size`min_price`max_price`volume];
show ptrds;
)];
}];
However this does not function as the trds reference made in the pivot function is not picked up, presumably because the trds selection is not global, however whilst using trds (without) as a parameter the piv function somehow is not applied and returns the result as if no operation had been performed. In addition, having to create an additional variable with ptrds seems inneficcient and also throws a reference error. Could someone please advise me how I can go about implementing this logic in a canonical manner that is efficient. Thanks
I think the source of your error here is the () brackets in your final if statement. The problem is that, when multiple arguments are enclosed within brackets, the object becomes a list. In kdb, elements of a list are evaluated from right to left, e.g.
q)(a:10;a+15)
'a
[0] (a:10;a+15)
^
q)(a+15;a:10)
25 10
To solve your problem, simply remove the brackets such that it reads
if[(count trades)>0;ptrds:piv[...];show ptrds;]
I have to implement a function which takes signal of remaining char count to compute the signal of color i.e.
If there are 15 or more characters left, the color "green"
If there are between 0 and 14 characters left, included, the color "orange"
Otherwise (if the remaining count is negative), the color "red"
I have implemented this in following way:
remainingCharsCount() match {
case x if x < 0 => Signal("red")
case x if x >= 14 => Signal("green")
case _ => Signal("orange")
}
where remainingCharsCount: Signal[Int] is a signal of remaining characters in a text.
The problem with the implementation above is that it does not fire events for color change based on remaining text count. In other implementations, I have moved the logic for color change inside the Signal block.
This code works fine:
Signal {
val rc = remainingCharsCount()
if (rc >= 15) "green"
else if (rc >= 0) "orange"
else "red"
}
Can someone explain what I am doing wrong in the first implementation?
For more info you can refer this link: https://github.com/ujjkumsi/FPDesign/blob/master/README.md
In the first approach, remainingCharsCount() is called once using which the pattern matching result is evaluated.
In the second approach, everything happens inside a Signal block.
In Signal { block }, block is the code which gets evaluated when dependencies change.
The dependencies are created by the signals been used inside the callers.
For example: remainingCharsCount() adds the caller, the returned Signal in your case to it's observers.
See the apply() method in: https://github.com/omirho/scalafun2/blob/master/calculator/src/main/scala/calculator/Signal.scala
Also see the other method implementations.
Let me explain this a little more.
In approach 1:
remainingCharsCount() gets evaluated and gets assigned to a value. And the outer signal (which can't be seen in this code and can be treated as nil) gets added to its observers.
The value is then used for pattern matching to construct a constant Signal.
Hence, the required dependency is not created.
Approach 2 has been explained before.
I guess you can do this
Signal(remainingCharsCount() match {
case x if x >= 15 => "green"
case x if x >= 0 => "orange"
case _ =>"red"
})
Signal need dependencies inside it expression to observe changes.
I have no access to you link. But as I remember Signal takes a block of code that is executed each time the input is changed.
In your first approach remainingCharsCount() is called only once (when you declare the signal). In the second it is called (by a framework) while something has changed.
I've read this simple explanation in the guide:
The value of a constant doesn’t need to be known at compile time, but you must assign it a value exactly once.
But I want a little more detail than this. If the constant references an object, can I still modify its properties? If it references a collection, can I add or remove elements from it? I come from a C# background; is it similar to how readonly works (apart from being able to use it in method bodies), and if it's not, how is it different?
let is a little bit like a const pointer in C. If you reference an object with a let, you can change the object's properties or call methods on it, but you cannot assign a different object to that identifier.
let also has implications for collections and non-object types. If you reference a struct with a let, you cannot change its properties or call any of its mutating func methods.
Using let/var with collections works much like mutable/immutable Foundation collections: If you assign an array to a let, you can't change its contents. If you reference a dictionary with let, you can't add/remove key/value pairs or assign a new value for a key — it's truly immutable. If you want to assign to subscripts in, append to, or otherwise mutate an array or dictionary, you must declare it with var.
(Prior to Xcode 6 beta 3, Swift arrays had a weird mix of value and reference semantics, and were partially mutable when assigned to a let -- that's gone now.)
It's best to think of let in terms of Static Single Assignment (SSA) -- every SSA variable is assigned to exactly once. In functional languages like lisp you don't (normally) use an assignment operator -- names are bound to a value exactly once. For example, the names y and z below are bound to a value exactly once (per invocation):
func pow(x: Float, n : Int) -> Float {
if n == 0 {return 1}
if n == 1 {return x}
let y = pow(x, n/2)
let z = y*y
if n & 1 == 0 {
return z
}
return z*x
}
This lends itself to more correct code since it enforces invariance and is side-effect free.
Here is how an imperative-style programmer might compute the first 6 powers of 5:
var powersOfFive = Int[]()
for n in [1, 2, 3, 4, 5, 6] {
var n2 = n*n
powersOfFive += n2*n2*n
}
Obviously n2 is is a loop invariant so we could use let instead:
var powersOfFive = Int[]()
for n in [1, 2, 3, 4, 5, 6] {
let n2 = n*n
powersOfFive += n2*n2*n
}
But a truly functional programmer would avoid all the side-effects and mutations:
let powersOfFive = [1, 2, 3, 4, 5, 6].map(
{(num: Int) -> Int in
let num2 = num*num
return num2*num2*num})
Let
Swift uses two basic techniques to store values for a programmer to access by using a name: let and var. Use let if you're never going to change the value associated with that name. Use var if you expect for that name to refer to a changing set of values.
let a = 5 // This is now a constant. "a" can never be changed.
var b = 2 // This is now a variable. Change "b" when you like.
The value that a constant refers to can never be changed, however the thing that a constant refers to can change if it is an instance of a class.
let a = 5
let b = someClass()
a = 6 // Nope.
b = someOtherClass() // Nope.
b.setCookies( newNumberOfCookies: 5 ) // Ok, sure.
Let and Collections
When you assign an array to a constant, elements can no longer be added or removed from that array. However, the value of any of that array's elements may still be changed.
let a = [1, 2, 3]
a.append(4) // This is NOT OK. You may not add a new value.
a[0] = 0 // This is OK. You can change an existing value.
A dictionary assigned to a constant can not be changed in any way.
let a = [1: "Awesome", 2: "Not Awesome"]
a[3] = "Bogus" // This is NOT OK. You may not add new key:value pairs.
a[1] = "Totally Awesome" // This is NOT OK. You may not change a value.
That is my understanding of this topic. Please correct me where needed. Excuse me if the question is already answered, I am doing this in part to help myself learn.
First of all, "The let keyword defines a constant" is confusing for beginners who are coming from C# background (like me). After reading many Stack Overflow answers, I came to the conclusion that
Actually, in swift there is no concept of constant
A constant is an expression that is resolved at compilation time. For both C# and Java, constants must be assigned during declaration:
public const double pi = 3.1416; // C#
public static final double pi = 3.1416 // Java
Apple doc ( defining constant using "let" ):
The value of a constant doesn’t need to be known at compile time, but you must assign the value exactly once.
In C# terms, you can think of "let" as "readonly" variable
Swift "let" == C# "readonly"
F# users will feel right at home with Swift's let keyword. :)
In C# terms, you can think of "let" as "readonly var", if that construct was allowed, i.e.: an identifier that can only be bound at the point of declaration.
Swift properties:
Swift Properties official documentation
In its simplest form, a stored property is a constant or variable that is stored as part of an instance of a particular class or structure. Stored properties can be either variable stored properties (introduced by the varkeyword) or constant stored properties (introduced by the let keyword).
Example:
The example below defines a structure called FixedLengthRange, which describes a range of integers whose range length cannot be changed once it is created:
struct FixedLengthRange {
var firstValue: Int
let length: Int
}
Instances of FixedLengthRange have a variable stored property called firstValue and a constant stored property called length. In the example above, length is initialized when the new range is created and cannot be changed thereafter, because it is a constant property.
I'm using Box2D and Cocos2D for iOS.
From time to time, the game freezes and it is caused by an infinite loop on b2World::SolveTOI.
for (b2Contact* c = m_contactManager.m_contactList; c; c = c->m_next)
{
// Invalidate TOI
c->m_flags &= ~(b2Contact::e_toiFlag | b2Contact::e_islandFlag);
c->m_toiCount = 0;
c->m_toi = 1.0f;
}
I have a game where a "hero" has to collect coins. Coins are bodies, and through the contact listener I'm adding bodies in an array in order to destroy them later when the array is full (count = 2).
Here is how I add bodies to the array:
+ (void) addBodyToDestroy:(b2Body *)body {
[toDestroyArray addObject:[NSValue valueWithPointer:body]];
}
And here is how I destroy them:
+ (void) destroyAllBodies {
b2World *world = [InGame getWorld];
for (NSValue *bodyValue in toDestroyArray)
{
b2Body *body;
body = (b2Body*)[bodyValue pointerValue];
world->DestroyBody(body);
body = NULL;
}
[toDestroyArray removeAllObjects];
}
What I find very weird is that it doesn't freeze every time, just sometimes, and I can't get to know what seemes to block...
Solution :
The body was added more than once into the array to be destroyed, so when the array was full, it was destroyed more than once, which caused the infinite loop.
b2Contact of the contact manager is linked list data.
So, for (b2Contact* c = m_contactManager.m_contactList; c; c = c->m_next)
means start from m_contactList and loop until c->next meets NULL
I don't know how your contact listener works, but you have to pay attention to the timing of the world step, collision check and destroy.
Because, the contact listener is called each time when there's a contact while world is stepping. So if your contact listener is designed to store contact data, then you have to process all the contact data between world step and destroy.
(If you don't, there might be dangling pointers of the destroyed body in contact data)
If you want to destroy coins when they fill the array, you'd better check the array if there is same object.
for (b2Contact* c = m_contactManager.m_contactList; c; c = c->m_next) is quite an unusual for loop, at least from my experience. I can clearly explain what it does, and maybe this will help you.
This is how a for loop works:
for( /*code called at beginning, usually to create a counting variable*/;/*code that is checked for a boolean value every loop. True means loop again, false means stop. Usually this is checking the value of the counting variable.*/;/*code that is called at the end of each loop. Usually this is advancing the counting variable*/) {
//Code inside the loop that does stuff many times
}
So for this loop:
The loop starts. a new b2Contact pointer named c is created and is assigned to the value of the m_contactList property of m_contactManager.
The loop checks the value of c and determines a value of either true or false. In this case, since c appears to be an instance of an object, it probably is checking whether c is nil or not.
At the end of each loop, c is set to point to the m_next member of c. This is quite strange as c is no longer m_contactList but something contained by m_contactList. Then again this could be perfectly normal. You should know what your code does.