I'm reading a lot of documentation about CRD Controller
I've implemented one with my business logic, and sometimes I got this race condition:
I create a Custom Object, let's call it Foo with name bar
My business logic applies, let's says that it creates a Deployment with a generated name, and I save the name (as reference) it in the Foo object
I remove the Custom Object
I quickly recreate it with the same name, and sometimes I get this log:
error syncing 'default/bar': Operation cannot be fulfilled on Foo.k8s.io "bar":
the object has been modified; please apply your changes to the latest version
and try again, requeuing
The thing is because my Deployment has a generated name and maybe the save (Foo) has failed, I got two Deployment with two names.
I did not found a way to fix it for now, but it raises a question.
How if I've multiple controllers running ?
I've started two controllers, and I got the same race condition by just creating a new object.
So, what is the best design to scale a CRD controller and avoid this kind of race conditions?
Generally you only run one copy of a controller, or at least only one is active at any given time. As long as you were careful to write your code convergently then it shouldn't technically matter, but there really isn't much reason to run multiple.
Related
I am on Enterprise Architect 13.5, creating a deployment diagram. I am defining our servers as nodes, and using attributes on them so that I can specify their details, such as Disk Controller = RAID 5 or Disks = 4 x 80 GB.
When dragging instances of these nodes onto a diagram, I can select "Set Run State" on them and set values for all attributes I have defined - just like it is done in the deployment diagram in the EAExample project:
Since our design will have several servers using the same configuration, my plan was to use the "initial value" column in the attribute definition on the node to specify the default configuration so that all instances I create automatically come up with reasonable values, and when the default changes, I would only change the Initial Values on the original node instead of having to go to all instances:
My problem is that even though I define initial values, all instances I create do not show any values when I drag them onto the diagram. Only by setting the Run State on each instance, I can get them to show the values I want:
Is this expected behavior? Btw, I can reproduce the same using classes and instances of them, so this is not merely a deployment diagram issue.
Any ideas are greatly appreciated! I'm also thankful if you can describe a better way to achieve the same result with EA, in case I am doing it wrong.
What you could do is to either write a script to assist with it or even create an add-in to bring in more automation. Scripting is easier to implement but you need to run the script manually (which however can add the values in a batch for newly created diagram objects). Using an add-in could do this on element creation if you hook to EA_OnPostNewElement.
What you need to do is to first get the classifier of the object. Using
Repository.GetElementByID(object.ClassifierID)
will return that. You then can check the attributes of that class and make a list of those with an initial value. Finally you add the run states of the object by assigning object.RunState with a crude string. E.g. for a != 33 it would be
#VAR;Variable=a;Value=33;Op=!=;#ENDVAR;
Just join as many as you need for multiple run states.
I'm carefully making my way through the Go code in the tools/cache package for what seems like the hundredth time.
From close-reading the code, when you create a new SharedIndexInformer, the following things happen:
SharedIndexInformer creates a new Indexer.
The Indexer so created creates a new ThreadSafeStore internally for holding representations of Kubernetes resources.
SharedIndexInformer.Run() creates a new DeltaFIFO with the new Indexer as its second parameter.
The Indexer supplied to the new DeltaFIFO therefore functions as its KeyLister
and its KeyGetter. (This is used to track "prior state" for deletions; i.e. if there's an object in it but the latest sync up with Kubernetes does not contain that object, then we know it has been deleted from the cluster.)
The HandleDeltas function is the one that the underlying controller will call when Kubernetes resources are added, updated or deleted.
The HandleDeltas function will call the Add, Update and Delete methods on the Indexer (and, by extension, on its underlying ThreadSafeStore).
Let's say I write a new controller using a SharedIndexInformer. Let's further say that it is watching for Pods. Let's finally say that there are 10 Pods in the cluster.
Does this mean that after the SharedIndexInformer's Run method has been called and some requisite amount of time has passed that there will be 10 JSON representations of Pods in the ThreadSafeStore, and, further, that none will be removed from this store until an actual deletion of one of the corresponding Pods occurs in Kubernetes?
Correct, except for the JSON part.
The Store contains native Object structs, deserialized from protobuf messages.
Let's say I have a database with schema of v1, and an application which is tightly coupled to that schema of v1. i.e. SQLException is thrown if the records in the database don't match the entity classes.
How should I deploy a change which alters the database schema, and deploys the application which having a race condition. i.e. user queries the app for a field which no longer exists.
This problem actually isn't specific to kubernetes, it happens in any system with more than one server -- kubernetes just makes it more front-and-center because of how automatic the rollover is. The words "tightly coupled" in your question are a dead giveaway of the real problem here.
That said, the "answer" actually will depend on which of the following mental models are better for your team:
do not make two consecutive schemas contradictory
use a "maintenance" page that keeps traffic off of the pods until they are fully rolled out
just accept the SQLExceptions and add better retry logic to the consumers
We use the first one, because the kubernetes rollout is baked into our engineering culture and we know that pod-old and pod-new will be running simultaneously and thus schema changes need to be incremental and backward compatible for at minimum one generation of pods.
However, sometimes we just accept that the engineering effort to do that is more cost than the 500s that a specific breaking change will incur, so we cheat and scale the replicas low, then roll it out and warn our monitoring team that there will be exceptions but they'll blow over. We can do that partially because the client has retry logic built into it.
Even after deleting containers and objects directly from file system, Swift is listing the containers when executed GET command on it. However, if we try to delete the container with DELETE command then 404: Not Found error message is returned. Please explain whether there is something wrong or is there some kind of cache?
I think the problem came from deleting the containers and/or objects directly from the file system.
Swift's methods for handling write requests for object and container have to be very careful to ensure all the distributed index information remains eventually consistent. Direct modification of the file system is not sufficient. It sounds like the container databases got removed before they had a chance to update the account databases listings - perhaps manually unlinked before all of the object index information was removed?
Normally after a delete request the containers have to hang around for awhile as "tombstones" to ensure the account database gets updated correctly.
As a work around you could recreate them (with a POST) and then re-issue the DELETE; which should successfully allow the DELETE of the new empty containers and update the account database listing directly.
(Note: the container databases themselves, although empty, will still exist on disk as tombstones until the the reclaim_age passes)
Let's say I have two types of MongoDB documents: 'Projects' and 'Tasks'. A Project can have many tasks. In my case it is more suitable to link the documents rather than embed.
When a user wants to save a task I first verify that the project the task is being assigned to exists, like so:
// Create new task
var task = new Task(data);
// Make sure project exists
Project.findById(task.project, function(err, project) {
if(project) {
// If project exists, save task
task.save(function(err){
...
});
} else {
// Project not found
}
});
My concern is that if another user happens to delete the project after the Project.findById() query is run, but before the task is saved, the task will be created anyway without a referenced project.
Is this a valid concern? Is there any practice that would prevent this from happening, or is this just something that has to be faced with MongoDB?
Technically yes, this is something you need to face when using MongoDB. But it's not really a big deal as it's rarely someone to delete a project and another person is unaware of it and creating task for that project. I would not use the if statement to check the project status, rather just leave task created as a bad record. You can either manually remove those bad records or schedule a cron task to clean them.
The way you appear to be doing it, i.e, with two separate Models -- not subdocuments (hard to tell without seeing the Models), I guess you will have that race condition. The if won't help. You'd need to take advantage of the atomic modifiers to avoid this issue, and using separate Models (each being it's own MongoDB collection), the atomic modifiers are not available. In SQL world, you'd use a transaction to ensure consistentcy. Similarly, with a document store like MongoDB, you'd make each Task a subdocument of a Project, and then just .push() new tasks. But perhaps your use case necessitates separate, unrelated Models. MongoDB is great for offering that flexibility, but it enables you to retain SQL-thinking without being SQL, which can lead to design problems.
More to the point, though, the race condition you're worried about doesn't seem to be a big deal. After all, the Project could be deleted after the task is saved, too. You obviously have a method for cleaning that up. One more cleanup function isn't the end of the world -- probably a good thing to have in your back pocket anyway.