I'm using k8s on google cloud and I'm trying to use google cloud's build-in snapshotting for backups, but I would also like to use them for retrieving a db to work with locally. I've come to the conclusion that I need to first create an image of the snapshot, then export that image to a bucket before downloading. Something like this:
gcloud compute disks snapshot mydrive --snapshot-names=mydrive-snapshot
gcloud compute images create mydrive-image --source-snapshot mydrive-snapshot
gcloud compute images export --destination-uri gs://my-bucket/mydrive-image.tar.gz --image mydrive-image
gsutil cp gs://my-bucket/my-drive-image.tar.gz file://my-drive-image.tar.gz
tar xvf my-drive-image.tar.gz
This gives me a file disk.raw. Not sure how to mount this locally though?
Are there any other simple solutions to this? I would be fine to use a native k8s workflow instead as long as its on the volume level and doesn't involve actually running anything in a pod.
Why not just mount the GCS bucket locally in which you exported the disk data?
You can use gcsfuse for doing this.
Follow these instructions for installing Cloud Storage FUSE and its dependencies
Set up credentials for Cloud Storage FUSE (follow the above instructions to do this)
Create a directory (or use an already existing directory to mount the bucket)
Use Cloud Storage FUSE to mount the bucket (e.g. my-bucket).
gcsfuse my-bucket /path/to/mount
Now you can see the content inside the bucket:
ls /path/to/mount
Related
I have used Kubernetes and I deployed for example WordPress or nginx or etc. We install from yaml file. Where is it installed how can i find directory of pages(for example WordPress pages etc.) at same point at Google Cloud too. When I use Kubernetes at Google Cloud where is the path of installed files(ex. index.php).
If you are running the docker image directly without attaching anything like NFS, S3 or Disk then you will be able to get those files by default in the container file system(index.php and all).
With any K8s cluster you check files inside container either Gcloud or any :
kubectl get pods
kubectl exec -it <Wordpress pod name> -- /bin/bash
If you are attaching the File system like NFS, or object storage S3 or EFS you will be able to watch those files there unless you mount and apply config using the YAML file.
Regarding setup file (YAML),
Kubernetes uses the ETCD database as a data store. The flow is like this. Kubectl command connect to API server and sends the YAML file to API server. API parses and store the information in ETCD database so you wont be getting those file as it is in YAML format.
I have a Google Cloud Composer 1 environment (Airflow 2.1.2) where I want to run an Airflow DAG that utilizes the KubernetesPodOperator.
Cloud Composer makes available to all DAGs a shared file directory for storing application data. The files in the directory reside in a Google Cloud Storage bucket managed by Composer. Composer uses FUSE to map the directory to the path /home/airflow/gcs/data on all of its Airflow worker pods.
In my DAG I run several Kubernetes pods like so:
from airflow.contrib.operators import kubernetes_pod_operator
# ...
splitter = kubernetes_pod_operator.KubernetesPodOperator(
task_id='splitter',
name='splitter',
namespace='default',
image='europe-west1-docker.pkg.dev/redacted/splitter:2.3',
cmds=["dotnet", "splitter.dll"],
)
The application code in all the pods that I run needs to read from and write to the /home/airflow/gcs/data directory. But when I run the DAG my application code is unable to access the directory. Likely this is because Composer has mapped the directory into the worker pods but does not extend this courtesy to my pods.
What do I need to do to give my pods r/w access to the /home/airflow/gcs/data directory?
Cloud Composer uses FUSE to mount certain directories from Cloud Storage into Airflow worker pods running in Kubernetes. It mounts these with default permissions that cannot be overwritten, because that metadata is not tracked by Google Cloud Storage. A possible solution is to use a bash operator that runs at the beginning of your DAG to copy files to a new directory. Another possible solution can be to use a non-Google Cloud Storage path like a /pod path.
I am using gocd for ci/cd. Result is tar archive. I need to copy resulting tar to GCE bucket.
I have gocd-agent docker image with included google sdk.
I know how to use gcloud with service account from local machine, but not from inside pod.
How to use service account assigned to pod with gcloud auth on pod?
Final goal is to use gsutil to copy above mentioned archive to bucket in same project.
My first thought would be to create Secret based on the service account, reference it in a pod yaml definiton to mount to some file and then run gcloud auth from the pod using that file. There's more info in Google cloud docs.
Another option which is quite new is to use Workload Identitiy. Seems you'd need to configure GKE cluster to enable this option. And it's working for some versions of GKE.
I have a Google Cloud Storage Bucket which is mounted to 3 virtual machines using fstab.
When I upload a file from another machine to google bucket using gsutil command, the uploaded file is accessible from only 2 vms (Set A). The other vm (Set B) doens't show the newly uploaded file.
fstab entry used for mounting is as follows.
bucket_name mounted_path gcsfuse rw,uid=1002,gid=1003,user,allow_other,key_file=key_file_path
Content of /etc/mtab file from Set A is as follows.
bucket_name mounted_path fuse rw,nosuid,nodev,relatime,user_id=1002,group_id=1003,default_permissions 0 0
Content of /etc/mtab file from Set B is as follows.
bucket_name mounted_path fuse fuse rw,nosuid,nodev,noexec,relatime,user_id=0,group_id=0,default_permissions,allow_other 0 0
Here is how I do mounting Storage Buckets on VMs:
create a VM instance and give it Storage "full" access scope. if you
have already a VM, edit that VM with this scope. note that you need
to stop the VM to edit it's access scope.
install gcsfuse on your instance if you haven't already
create a directory on where you want to mount your bucket mkdir /path/to/bucket
go to Cloud Storage and edit your bucket permissions by adding the compute engine default service account as a Storage Admin, you can find this service account in IAM&admin --> service account, it has this structure 1213242443-compute#developer.gserviceaccount.com
use gcsfuse bucket_name /path/to/bucket to mount your bucket. here gcsfuse will use the default service account to verify access and make the connection. this is the easiest way as it conclude few steps
now any file you upload to your bucket will appear in VMs bucket mounting folder /path/to/bucket
Read more about this process here
I'm trying to create and mount a Google storage bucket on a Ubuntu Linux instance using gsutil.
sudo gsutil mb -c STANDARD -l us-central1-a gs://test-bucket
Here's what I'm getting:
Creating gs://test-bucket/...
AccessDeniedException: 403 Insufficient Permission
I've been searching around for a solution with no success. Can anyone help?
Check to see who is the account managing your VM instance from the GCloud Dashboard. Should be the compute service OR app account that is automatically created.
In the initial configuration settings you should see the Cloud API list that will state whether or not that user has Cloud Storage capability.
If not You will have to recreate your VM instance.
Create a GCP snapshot of your VM
Delete VM instance
Create a new instance using it existing snapshot. (allows you to start where you left off in new VM)
When creating the VM under API give the user full access which will allow them to write to the Cloud storage using the gsutil/gcsfuse commands.
THEN permissions from the Cloud Storage will be a concern but your root user should be able to write.