I'm carefully making my way through the Go code in the tools/cache package for what seems like the hundredth time.
From close-reading the code, when you create a new SharedIndexInformer, the following things happen:
SharedIndexInformer creates a new Indexer.
The Indexer so created creates a new ThreadSafeStore internally for holding representations of Kubernetes resources.
SharedIndexInformer.Run() creates a new DeltaFIFO with the new Indexer as its second parameter.
The Indexer supplied to the new DeltaFIFO therefore functions as its KeyLister
and its KeyGetter. (This is used to track "prior state" for deletions; i.e. if there's an object in it but the latest sync up with Kubernetes does not contain that object, then we know it has been deleted from the cluster.)
The HandleDeltas function is the one that the underlying controller will call when Kubernetes resources are added, updated or deleted.
The HandleDeltas function will call the Add, Update and Delete methods on the Indexer (and, by extension, on its underlying ThreadSafeStore).
Let's say I write a new controller using a SharedIndexInformer. Let's further say that it is watching for Pods. Let's finally say that there are 10 Pods in the cluster.
Does this mean that after the SharedIndexInformer's Run method has been called and some requisite amount of time has passed that there will be 10 JSON representations of Pods in the ThreadSafeStore, and, further, that none will be removed from this store until an actual deletion of one of the corresponding Pods occurs in Kubernetes?
Correct, except for the JSON part.
The Store contains native Object structs, deserialized from protobuf messages.
Related
I'm reading a lot of documentation about CRD Controller
I've implemented one with my business logic, and sometimes I got this race condition:
I create a Custom Object, let's call it Foo with name bar
My business logic applies, let's says that it creates a Deployment with a generated name, and I save the name (as reference) it in the Foo object
I remove the Custom Object
I quickly recreate it with the same name, and sometimes I get this log:
error syncing 'default/bar': Operation cannot be fulfilled on Foo.k8s.io "bar":
the object has been modified; please apply your changes to the latest version
and try again, requeuing
The thing is because my Deployment has a generated name and maybe the save (Foo) has failed, I got two Deployment with two names.
I did not found a way to fix it for now, but it raises a question.
How if I've multiple controllers running ?
I've started two controllers, and I got the same race condition by just creating a new object.
So, what is the best design to scale a CRD controller and avoid this kind of race conditions?
Generally you only run one copy of a controller, or at least only one is active at any given time. As long as you were careful to write your code convergently then it shouldn't technically matter, but there really isn't much reason to run multiple.
I have a cloud function which is set to be triggered by a cloud storage bucket.
Is there anyway to tell if an event is set off by a newly uploaded object or rather an overwritten object?
I tried to console.log out the event object but nothing seems to be indicative of whether or not the object has been overwritten.
I do notice that there exists an "overwroteGeneration" attribute in Cloud Pub/Sub Notifications which the trigger event here is based on, but it's not available here.
As staged by #Doug Stevenson, it looks like there is no easy way to achieve this. Moreover, I have been playing around with metageneration and, according to this example in the documentation (under "You upload a new version of the image
"), when an object is overwritten (even when versioning is enabled), it will get its own new generation and metageneration numbers, reason why, as you commented in the comment to the other answer, metageneration = 1 in such a scenario.
The only workaround I see, and which may not satisfy your specific requirements, is using two Cloud Functions (let's call them funcA and funcB), one (funcA) that identifies object creation with google.storage.object.finalize and another one (funcB) that detects object overwriting with google.storage.object.archive or google.storage.object.delete (depending if you are using versioning or not, respectively). In this case, funcA would be triggered twice, because for object overwritting it will be triggered too, but depending on your use case, you can identify the proximity in time of the create and delete events and detect that these detect to a single event:
funcA logs:
funcB logs:
I know this does not exactly solve the question you posted, but I do not think there is any way to identify, using a single Cloud Function, that an object has been either created or overwritten, it looks like you will need one Cloud Function for each of those procedures.
In order to determine the nature of the file upload (new file upload rather than existing file upload), you need to use the event and delivered to the function to figure it out.
You'll need to use an Object Finalize event type. As you can see from the documentation there:
This event is sent when a new object is created (or an existing object
is overwritten, and a new generation of that object is created) in the
bucket.
What's not so clear from that doc is that the metageneration combined with the resourceState property of the event is the indicator.
The documentation here should be clear about how to use metageneration along with resourceState to determine if a change to a bucket is a new file or a replaced file:
The resourceState attribute should be paired with the 'metageneration'
attribute if you want to know if an object was just created. The
metageneration attribute is incremented whenever there's a change to
the object's metadata. For new objects, the metageneration value is 1.
I'm trying to use Google Cloud Storage's lifecycle management features on a bucket, but I want to circumvent it for certain files (basically auto delete all files after 1 day, except for specific files that I want to keep). If I call the set metadata API endpoint will that update the age of the object and prevent the delete from occurring?
Set metadata changes the last updated time, not the creation time. TTL is keyed off of creation time, so that will not prevent TTL cleanup.
However, you could do a copy operation, and just set the destination to be the same as the source. That would update the creation time, and would be a fast operation as it can copy in the cloud.
That being said, it would probably be safer to just use a different bucket for these files. If your job to keep touching the files goes down they may get deleted.
Does anyone know what is a best practice to overwrite records under Google DNS Cloud, using API? https://cloud.google.com/dns/api/v1/changes/create does not help!
I could delete and create, but it is not nice ;) and could cause an outage.
Regards
The Cloud DNS API uses Changes objects to perform the update actions; you can create Changes but you don't ever delete them. In the Cloud DNS API, you never operate directly on the resource record sets. Instead, you create a Changes object with your desired additions and deletions and if that is created successfully, it applies those updates to the specified resource record sets in your managed DNS zone.
It's an unusual mental model, sort of like editing a file by specifying a diff to be applied, or appending to the commit history of a Git repository to change the contents of a file. Still, you can certainly achieve what you want to do using this API, and it is applied atomically at the authoritative servers (although the DNS system as a whole does not really do anything atomically, due to caching, so if you know you will be making changes, reduce your TTLs before you make the changes). The atomicity here is more about the updates themselves: if you have multiple applications making changes to your managed zones, and there are conflicts in changes to the particular record sets, the create operation will fail, and you will have retry the change with modified deletions (rather than having changes be silently overwritten).
Anyhow, what you want to do is to create a Changes object with deletions that specifies the current resource record set, and additions that specifies your desired replacement. This can be rather verbose, especially if you have a domain name with a lot of records of the same type. For example, if you have four A records for mydomain.example (1.1.1.1, 2.2.2.2, 3.3.3.3, and 4.4.4.4) and want to change the 3.3.3.3 address to 5.5.5.5, you need to list all four original A records in deletions and then the new four (1.1.1.1, 2.2.2.2, 4.4.4.4, and 5.5.5.5) in additions.
The Cloud DNS documentation provides example code boilerplate that you can adapt to do what you want: https://cloud.google.com/dns/api/v1/changes/create#examples, you just need to set the deletions and additions for the Changes object you are creating.
I have never used APIs for this purpose, but if you use command line i.e. gcloud to update DNS records, it binds the change in a single transaction and both tasks of deleting the record and adding the updated record are executed as a single transaction. Since transactions are atomic in nature, it shouldn't cause any outage.
Personally, I never witnessed any outage while using gcloud for updating DNS settings for my domain.
Even after deleting containers and objects directly from file system, Swift is listing the containers when executed GET command on it. However, if we try to delete the container with DELETE command then 404: Not Found error message is returned. Please explain whether there is something wrong or is there some kind of cache?
I think the problem came from deleting the containers and/or objects directly from the file system.
Swift's methods for handling write requests for object and container have to be very careful to ensure all the distributed index information remains eventually consistent. Direct modification of the file system is not sufficient. It sounds like the container databases got removed before they had a chance to update the account databases listings - perhaps manually unlinked before all of the object index information was removed?
Normally after a delete request the containers have to hang around for awhile as "tombstones" to ensure the account database gets updated correctly.
As a work around you could recreate them (with a POST) and then re-issue the DELETE; which should successfully allow the DELETE of the new empty containers and update the account database listing directly.
(Note: the container databases themselves, although empty, will still exist on disk as tombstones until the the reclaim_age passes)