I have mounted a bucket from my Google Cloud Storage account using gcsfuse. I'm surprised to see that none of the file from that bucket are now on my machine. This is a Google "instance" Debian 8, but I suspect the problem is more my not understanding how gcsfuse works. Should I not see my files?
I also had this issue. The answer #jacobsa addressed the same issue for me.
Instead of mounting with
gcsfuse my-bucket /path/to/mount
I used
gcsfuse --implicit-dirs my-bucket /path/to/mount
I also checked that the bucket is setup with the right permissions by adding myself as the Owner in Edit bucket permissions
You are likely running into the issues with implicit directories.
Related
I want to take Oracle database backup using RMAN directly into the Google Cloud Storage
I am unable to find the plugin to use to take the RMAN backups into Cloud Storage. We have a plugin for Amazon S3 and am looking for one such related to Google Cloud Storage.
I don't believe there's an official way of doing this. Although I did file a Feature Request for the Cloud Storage engineering team to look into that you can find here.
I recommend you to star the Feature Request, for easy visibility and access, allowing you to view its status updates. The Cloud Storage team might ask questions there too.
You can use gcsfuse to mount GCS bucket as file systems on your machine and use RMAN to create backups there.
You can find more information about gcsfuse on its github page. Here are the basic steps to mount a bucket and run RMAN:
Create a bucket oracle_bucket. Check that it doesn't have a retention policy defined on it (it looks like gcsfuse has some issues with retention policies).
Please have a look at mounting.md that describes credentials for GCS. For example, I created a service account with Storage Admin role and created a JSON key for it.
Next, set up credentials for gcsfuse on your machine. In my case, I set GOOGLE_APPLICATION_CREDENTIALS to the path to JSON key from step 1. Run:
sudo su - oracle
mkdir ./mnt_bucket
gcsfuse --dir-mode 755 --file-mode 777 --implicit-dirs --debug_fuse oracle_bucket ./mnt_bucket
From gcsfuse docs:
Important: You should run gcsfuse as the user who will be using the
file system, not as root. Do not use sudo.
Configure RMAN to create a backup in mnt_bucket. For example:
configure controlfile autobackup format for device type disk to '/home/oracle/mnt_bucket/%F';
configure channel device type disk format '/home/oracle/mnt_bucket/%U';
After you run backup database you'll see a backup files created in your GCS bucket.
From document https://cloud.google.com/storage/docs/gcs-fuse, when use FUSE GCS there is only mound predefine bucket to the specified path.
In case I have many buckets, show how I can mount all buckets as root directory can access or create any bucket as the directory?
gcsfuse doesn't support mounting multiple buckets in a single process, nor does it support creating buckets for you. You'll need to create buckets in the usual way outside gcsfuse, then run one gcsfuse process per desired bucket.
I would like to know if there is a way to mount google cloud storage bucket as a folder for the first time
and each time we read the file, cache it locally (so it won't use money/bandwidth).
GCSFUSE has two type of caching available, Stat caching and type caching. You can refer to this document which provide detailed information on these types of caching with there trade-offs.
I've been using gcsfuse (FUSE) for some weeks and everything was running smoothly until my instance disk(10GB) got filled up out of nowhere.
I was trying to identify the cause and erasing some temporal files and found out that unmounting the bucket fixed the issue.
It's supposed to upload to the cloud right? So why is it taking up space as if it was counted as local instance space?
Thanks for the help guys.
Here is a reason why you would see this behaviour.
Pasting from the gcsfuse doc
https://cloud.google.com/storage/docs/gcs-fuse
Local storage: Objects that are new or modified will be stored in
their entirety in a local temporary file until they are closed or
synced. When working with large files, be sure you have enough local
storage capacity for temporary copies of the files, particularly if
you are working with Google Compute Engine instances. For more
information, see the readme documentation.
I am running some application on google compute engine where it the application reads the data from google cloud storage and writes data to persistent disk. And the bucket is mounted using gcsfuse.
But in the middle the bucket is getting unmounted and my application is going to sleep mode and getting stalled.
When I try to see the content in Mounted directory I am getting following error
cannot access /home/santhosh/MountPoint/ Transport endpoint is not connected
Is there any time limit on the bucket live? How can we see the bucket is mounted all the time?
Can someone please help me how can I resolve this? I want the program to run without any breaks in the middle.
I experience same problems, random I/O errors, unmounting. Do not use gcsfuse in production, from the doc: "Please treat gcsfuse as beta-quality software." We use it for the maintenance only.