I have been asked by security auditors to explain the underlying process of oc delete secret What they want to establish is that the secret is not recoverable by forensics tools for example, once deleted.
Thanks in advance for any input.
There are a lot of layers you would have to peel in order to do a thorough assessment/audit. Start by looking at the calls performed by the oc CLI which underneath should be sending a request to the DELETE secret endpoint on the API Server.
Ultimately, In Kubernetes, the delete operation will come down to deleting a key from the etcd datastore as seen here. You can further dig into the etcd Go API and etcd internals in order to determine how the deletion is performed across cluster nodes on commit, and determining whether a forensics tool would be able to pry into the storage blocks on disk.
Also, ensure Pods that mount the secrets are not writing the data to disk or to logs.
Related
I have a situation to which I can find no easy solution and I am looking for some common solutions to this issue.
I have a deployment with three pods doing simple CRUD operations with some HTTP resources. Now if first client POSTs a new resource to one of the pods and second clients posts a GET to some other pod shortly aftwerward, how can I make sure that second client gets the resource from another pod?
This resource is held abstract on purpose, as this problem can appear both with Restful and with database-stored resources ( when application internal caching is used )
Well, you need to share 💾 something if you want the pods to know it at the same time. Some options:
Shared volume
NFS
Shared block storage (EBS, Google Persistent Disk, etc. depends on where you are running 🏃)
Ceph
Glusterfs
...
Cache
Redis
Memcached
Another DB
Distributed K/V Stores
Zookeeper
etcd
Consul
You can run all of the above in Kubernetes or separately as long as your pods have network connectivity to these options.
✌️
We are working on provisioning our service using Kubernetes and the service needs to register/unregister some data for scaling purposes. Let's say the service handles long-held transactions so when it starts/scales out, it needs to store the starting and ending transaction ids somewhere. When it scales out further, it will need to find the next transaction id and save it with the ending transaction id that is covered. When it scales in, it needs to delete the transaction ids, etc. ETCD seems to make the cut as it is used (by Kubernetes) to store deployment data and not only that it is close to Kubernetes, it is actually inside and maintained by Kubernetes; thus we'd like to find out if that is open for our use. I'd like to ask the question for both EKS, AKS, and self-installed. Any advice welcome. Thanks.
Do not use the kubernetes etcd directly for an application.
Access to read/write data to the kubernetes etcd store is root access to every node in your cluster. Even if you are well versed in etcd v3's role based security model avoid sharing that specific etcd instance so you don't increase your clusters attack surface.
For EKS and GKE, the etcd cluster is hidden in the provided cluster service so you can't break things. I would assume AKS takes a similar approach unless they expose the instances to you that run the management nodes.
If the data is small and not heavily updated, you might be able to reuse the kubernetes etcd store via the kubernetes API. Create a ConfigMap or a custom resource definition for your data and edit it via the easily securable and namespaced functionality in the kubernetes API.
For most application uses run your own etcd cluster (or whatever service) to keep Kubernetes free to do it's workload scheduling. The coreos etcd operator will let you define and create new etcd clusters easily.
I've been working with kubernetes for the past 6 months and we've deployed a few services.
We're just about to deploy another which stores encrypted data and puts the keys in KMS. This requires two service accounts, one for the data and one for the keys.
Data access to this must be audited. Since access to this data is very sensitive we are reluctant to put both service accounts in the name namespace as if compromised in any way the attacker could gain access to the data and the keys without it being audited.
For now we have one key in a secret and the other we're going to manually post to the single pod.
This is horrible as it requires that a single person be trusted with this key, and limits scalability. Luckily this service will be very low volume.
Has anyone else came up against the same problem?
How have you gotten around it?
cheers
Requirements
No single person ever has access to both keys (datastore and KMS)
Data access to this must be audited
If you enable audit logging, every API call done via this service account will be logged. This may not help you if your service isn't ever called via the API, but considering you have a service account being used, it sounds like it would be.
For now we have one key in a secret and the other we're going to manually post to the single pod.
You might consider using Vault for this. If you store the secret in vault, you can use something like this to have the environment variable pushed down into the pod as an environment variable automatically. This is a little more involved than your process, but is considerably more secure.
You can also use Vault alongside Google Cloud KMS which is detailed in this article
What you're describing is pretty common - using a key/ service account/ identity in Kubernetes secrets to access an external secret store.
I'm a bit confused by the double key concept - what are you gaining by having a key in both secrets and in the pod? If secrets are compromised, then etcd is compromised and you have bigger problems. I would suggest you focus instead on locking down secrets, using audit logs, and making the key is easy to rotate in case of compromise.
A few items to consider:
If you're mostly using Kubernetes, consider storing (encrypted) secrets in Kubernetes secrets.
If you're storing secrets centrally outside of Kubernetes, like you're describing, consider just using a single Kubernetes secret - you will get Kubernetes audit logs for access to the secret (see the recommended audit-policy), and Cloud KMS audit logs for use of the key.
I'm looking to understand how to recreate my cluster. There's a cluster-level setting to specify the IP range for nodes created within it, which I want to use so I can set a decent firewall rule. However, it looks like that can't be changed once the cluster is created.
I have a number of namespaces, deployments, services, secrets, persistent volumes and claims. If I wanted to transfer them all to a new cluster, should I just kubectl get all --namespace=whatever --format=yaml, kubectl delete -f, and then kubectl apply -f on the new cluster?
Would something so crude work for mapping to the same load balancers / public IPs, persistent volumes, secrets, etc?
As you can see the backup and the migration of whole clusters is quite a discussed matter and still an open issue on Kubernetes github as well:
https://github.com/kubernetes/kubernetes/issues/24229
Therefore I do not believe that the command that you posted might be considered a solution or work. I think it will fail due to different resources that are cluster dependent and IPs. Moreover since this kind of use is not supported It will lead for to multiple issues.
Lets say that you change zone of the cluster, how could be possible to move the PV if the disk cannot be attached to an instance in a different zone (or possibly if you migrate to a different cloud service)?
More important I would not risk to delete my production to run a command that is not documented or indicated as best practise. You could try it on test namespace, but I would not suggest to go further.
You can check reshifter and ark since they might cover your needs. I have never tested them but they are mentioned in the thread, so they might be of your interest.
I tried this approach in one of my test cluster obtaining:
Error from server (Conflict): Operation cannot be fulfilled
Error from server (Conflict): Operation cannot be fulfilled
Error from server (Forbidden): [...]
Honestly I believe that for a limited subset of resources it might be possible (Note that some resources were created correctly) , but it cannot be considered at all a way to migrate.
I've started playing with Hashicorp's Vault to manage secrets and had some questions about the day-to-day of Vault sealing. My workflow has two auth backends; specific users access Vault with write access to add new secrets, servers have readonly access for the secrets they need.
1) Under normal circumstances, does the Vault stay in an unsealed state? I believe it would as a dynamically provisioned server should not have to coordinate an unseal.
2) Is the purpose of sealing to off-board staff to rotate keys and in case of an intrusion?
3) What's the best practice for ensuring the vault process is always running, since if it dies the Vault will seal? Also, in a highly available configuration, if one Vault node's process dies, does it seal the Vault for everyone?
I asked this question on the Vault Google Group and this was the best response:
1) Under normal circumstances, does the Vault stay in an unsealed
state? I believe it would as a dynamically provisioned server should
not have to coordinate an unseal.
Yes. Once Vault is initialized and unsealed, it 'normally' stays in an
unsealed state.
2) Is the purpose of sealing to off-board staff to rotate keys and in
case of an intrusion?
Sealing of Vault enables a turn key mechanism to stop all the services
of Vault. It would require a specific number of unseal key holders to
make Vault operational again.
3) What's the best practice for ensuring the vault process is always
running, since if it dies the Vault will seal? Also, in a highly
available configuration, if one Vault node's process dies, does it
seal the Vault for everyone?
There is no official best practice recommendation for this. But
running Vault in a dedicated instance/cluster with very
limited/no-access to its memory. Running Vault in a HA mode using a
backend which supports it is good. If any of the cluster nodes goes
down or if the Vault process is restarted, it will be in a sealed
state and would require the unseal operation to be performed to make
it operational.
Best,
Vishal
Taken from https://www.vaultproject.io/docs/concepts/seal.html:
"Under normal circumstances, does the Vault stay in an unsealed state?" -
When a Vault server is started, it starts in a sealed state. In this state, Vault is configured to know where and how to access the physical storage, but doesn't know how to decrypt any of it.
Unsealing is the process of constructing the master key necessary to
read the decryption key to decrypt the data, allowing access to the
Vault.
Prior to unsealing, almost no operations are possible with Vault. For
example authentication, managing the mount tables, etc. are all not possible. The only possible operations are to unseal the Vault and check the status of the unseal.
"Is the purpose of sealing to off-board staff to rotate keys and in case of an intrusion?" -
This way, if there is a detected intrusion, the Vault data can be locked
quickly to try to minimize damages. It can't be accessed again without access to the master key shards.
"since if it dies the Vault will seal?" - yes.