In the Couchbase API, there are store and delete operations that allow you to specify how many nodes an operation must be successfully persisted to before returning. This is expressed through two method parameters:
My question is: what is the difference between the persistTo and replicateTo parameters. For example passing in PersistTo.MASTER + ReplicateTo.THREE appears to be exactly equivalent to passing in PersistTo.FOUR. Are there actually any behavioral differences between calling these observed APIs in those two different styles?
PersistTo.MASTER + ReplicateTo.THREE means at minimum the item must be on disk on the master node and at least in memory on three replica nodes. In this case the item might not be persisted on the replicas.
PersistTo.FOUR means that the item must be persisted on the master as well as three replicas.
A good way to think about things is that just because an item is replicated to another node doesn't mean that that item has been persisted to disk.
Related
I have a Kubernetes controller written using client-go informers package. It maintains a watch on all Pods in the cluster, there are about 15k of these objects, and their YAML representation takes around 600 MB to print (I assume their in-memory representation is not that different.)
As a result, this (otherwise really small) controller watching Pods ends up with a huge memory footprint (north of 1 GiB). Even methods that you'd think offer a way of filtering, such as the one named like NewFilteredSharedInformerFactory doesn't really give you a way to specify a predicate function that chooses which objects are stored in the in-memory cache.
Instead, that method in client-go offers a TweakListOptionsFunc. It helps you control ListOptions but my predicate unfortunately cannot be satisfied with a labelSelector or fieldSelector. I need to drop the objects when they arrive to the controller through a predicate function.
Note: the predicate I have is something like "Pods that have an ownerReference by a DaemonSet" (which is not possible with fieldSelectors –also another question of mine) and there's no labelSelector that can work in my scenario.
How would I go about configuring an informer on Pods that only have DaemonSet owner references to reduce the memory footprint of my controller?
Here's an idea, you can get a list of all the DaemonSets in your cluster. Read the spec.selector.matchLabels field to retrieve the label that the DaemonSet pods are bound to have. Use those labels as part of your TweakListOptionsFunc with a logic like:
Pods with label1 OR label2 OR label3 ...
I know it's a bit of toil, but seems to be a working approach. I believe there isn't a way to specify fields in client-go.
It appears that today if you use SharedInformers, there's no way to filters which objects to keep in the shared cache and which ones to discard.
I have found an interesting code snippet in kube-state-metrics project that opts into the lower-layer of abstraction of initiating Watch calls directly (which would normally be considered as an anti-pattern) and using watch.Filter, it decides whether to return an object from a Watch() call (to a cache/reflector or not).
That said, many controller authors might choose to not go down this path as it requires you to specify your own cache/reflector/indexer around the Watch() call. Furthermore, projects like controller-runtime don't even let you get access to this low-level machinery, as far as I know.
Another aspect of reducing controllers' memory footprint can be done through field/data erasure on structs (instead of discarding objects altogether). This is possible in newer versions of client-go through cache.TransformFunc, which can let you delete fields of irrelevant objects (though, these objects would still consume some memory). This one is more of a band-aid that can make your situation better.
In my case, I mostly needed to watch for DaemonSet Pods in certain namespaces, so I refactored the code from using 1 informer (watching all namespaces) to N namespace-scoped informers running concurrently.
I am currently trying to model the data for our application. The data consists of identities and groups. One group can have multiple identities and one identity can be in multiple groups. (a typical many-to-many relationship).
So I have used the Adjacency List Design Pattern to structure my data as recommended by AWS:
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-adjacency-graphs.html
I keep all the info about identities duplicated inside the groups and reading the data works just fine - a normal query for the details and a query against the index to get the relations of my objects.
How can I ensure that all duplicated records have the same value?
Every time the group changes, I am updating all the duplicated group records in the database.
I am okay with updating multiple records at once as changes will happen rarely but I want to avoid inconsistent data.
All the tutorials and guides always just talk about reading and accessing data not about updating the data.
I know that there is a TransactWriteItem-Reuquest but it is limited to 25 items maximum. So is there another way/pattern to guarantee that all my identity records are updated when e.g. the name changes.
You have to decide for yourself how consistent is consistent enough in your application.
The CAP theorem is alive and well and it says that to get availability and partition tolerance we have to sacrifice consistency.
Since updates happen infrequently, how does your application fail if it sees inconsistent records? If you can't use the transactional API because of the 25 item limit, maybe you could roll your own "lock-out" using an attribute you would set on items that must all be updated together:
first, you identify all items that need to be updated and set the "lock_out" attribute on them (this can be a timestamp indicating when the lock_out expires)
in your application, you can add business logic to treat items with the "lock_out" in a way that makes sense (maybe show them as being updated, or not show them at all etc.)
update the items
after the update is complete, clear the "lock-out" attribute
I have a collection in my Cosmos database that I would like to observe for changes. I have many documents (official and unofficial) explaining how to do this. There is one thing though that I cannot get to work in a reliable way: how do I receive the same changes to multiple instances when I don't have any common reference for instance names?
What do I mean by this? Well, I'm running my work loads in a Kubernetes cluster (AKS). I have a variable number of instances within the cluster that should observe my collection. In order for change feeds to work properly, I have to have a unique instance name for each instance. The only candidate I have is the pod name. It's usually on the form of <deployment-name>-<random string>. E.g. pod-5f597c9c56-lxw5b.
If I use the pod name as instance name, all instances do not receive the same changes (which is my requirement), only one instance will receive the change (see https://learn.microsoft.com/en-us/azure/cosmos-db/change-feed-processor#dynamic-scaling). What I can do is to use the pod name as feed name instead, then all instances get the same changes. This is what I fear will bite me in the butt at some point; when peek into the lease container, I can see a set of documents per feed name. As pod names come and go (the random string part of the name), I fear the container will grow over time, generating a heap of garbage. I know Cosmos can handle huge work loads, but you know, I like to keep things tidy.
How can I keep this thing clean and tidy? I really don't want to invent (or reuse for that matter!) some protocol between my instances to vote for which instance gets which name out of a finite set of names.
One "simple" solution would be to build my own instance names, if AKS or Kubernetes held some "index" of some sort for my pods. I know stateful sets give me that, but I don't want to use stateful sets, as the pods themselves aren't really stateful (except for this particular aspect!).
There is a new Change Feed pull model (which is in preview at this time).
The differences are:
In your case, it looks like you don't need parallelization (you want all instances to receive everything). The important part would be to design a state storing model that can maintain the continuation tokens (or not, maybe you don't care to continue if a pod goes down and then restarts).
I would suggest that you proceed to use the pod name as unique ID. If you are concerned about sprawl of the data, you could monitor the container and devise a clean-up mechanism for the metadata.
In order to have at-least-once delivery, there is going to need to be metadata persisted somewhere to track items ACK-ed / position in a partition, etc. I suspect there could be a bit of work to get change feed processor to give you at-least-once delivery once you consider pod interruption/re-scheduling during data flow.
As another option Azure offers an implementation of checkpoint based message sharing from partitioned event hubs via EventProcessorClient. In EventProcessorClient, there is also a bit of metadata added to a storage account.
What I understand is, that GameplayAbilities do not need to replicate to update GameplayAttributes across the network, since they don't influence attributes directly. Instead, this is the task of GameplayEffects.
So what is it that updates the GameplayAbilitySystem attribute values (FGameplayAttributeData) over the network:
do only attributes replicate?
or are the GameplayEffects sent only?
or both?
To give context: I have an attribute modification system where I need several “base values”. Those “base values” change very infrequently, while the “final value” changes often. There are two possibilities to do that with GAS
use a separate attribute for each of the “base values” and “final value” or
add additional float members to the attribute struct FGameplayAttributeData, for all “base values” and the “final value”
If only GE are sent over the network (and not attributes), I would go for (2), since the size of an attribute doesn't matter for bandwidth then.
Both are replicated!
GameplayAttributes are replicated with an OnRep function, which calls FActiveGameplayEffectsContainer::SetBaseAttributeValueFromReplication(), which - despite the name - will also updates the current value using the aggregators which are existing on the local machine. For that to work, also ...
GameplayEffects replicate (see also Unreal GAS: GameplayEffect: Difference between minimal and full replication).
So for saving bandwidth in the example given, it is more meaningful to use separate attributes for the “base values” (option (1)), since they are not updated very often. So when the “final value” changes, the constant “base values” won’t have to be replicated.
Supposed I have a key-value database, and I need to build a queue on top of it. How could I achieve this without getting a bad performance?
One idea might be to store the queue inside an array, and simply store the array using a fixed key. This is a quite simple implementation, but is very slow, as for every read or write access the complete array must be loaded / saved.
I could also implement a linked list, with random keys, and there is one fixed key which acts as starting point to element 1. Depending on if I prefer a fast read or a fast write access, I could let point the fixed element to the first or the last entry in the queue (so I have to travel it forward / backward).
Or, to proceed with that - I could also have two fixed pointers: One for the first, on for the last item.
Any other suggestions on how to do this effectively?
Initially, key-value structure is extremely similar to the original memory storage where the physical address in computer memory plays as the key. So any type of data structure could be modeled upon key-value storage surely, including linked list.
Originally, a linked list is a list of nodes including the index information of previous node or following node. Then the node it self should also be viewed as a sub key-value structure. With additional prefix to the key, the information in the node could be separately stored in a flat table of key-value pairs.
To proceed with that, special suffix to the key could also make it possible to get rid of redundant pointer information. This pretend list might look something like this:
pilot-last-index: 5
pilot-0: Rei Ayanami
pilot-1: Shinji Ikari
pilot-2: Soryu Asuka Langley
pilot-3: Touji Suzuhara
pilot-5: Makinami Mari
The corresponding algrithm is also imaginable, I think. If you could have a daemon thread for manipulation these keys, pilot-5 could be renamed as pilot-4 in the above example. Even though, it is not allowed to have additional thread in some special situation, the result of the queue it self is not affected. Just some overhead would exist for the break point in sequence.
However which of the two above should be applied is the problem of balance between the cost of storage space or the overhead of CPU time.
The thread safe is exactly a problem however an ancient problem. Just like the class implementing the interface of ConcurrentMap in JDK, Atomic operation on key-value data is also provided perfectly. There are similar methods featured in some key-value middleware, like memcached, as well, which could make you update key or value separately and thread safely. However these implementation is the algrithm problem rather than the key-value structure it self.
I think it depends on the kind of queue you want to implement, and no solution will be perfect because a key-value store is not the right data structure for this kind of task. There will be always some kind of hack involved.
For a simple first in first out queue you could use a few kev-value stores like the folliwing:
{
oldestIndex:5,
newestIndex:10
}
In this example there would be 6 items in the Queue (5,6,7,8,9,10). Item 0 to 4 are already done whereas there is no Item 11 or so for now. The producer worker would increment newestIndex and save his item under the key 11. The consumer takes the item under the key 5 and increments oldestIndex.
Note that this approach can lead to problems if you have multiple consumer/producers and if the queue is never empty so you cant reset the index.
But the multithreading problem is also true for linked lists etc.