Is it possible to enable Directory listing in Google Cloud Storage?
I was thinking on having a "domain bucket" and use it to list all the contents, similar to Nginx's autoindex on or Apache's Options +Indexes.
If I make the bucket public all contents will be listed as a XML, but not like a directory listing.
No, it is not currently possible.
You could perhaps implement such a thing by creating "index" pages in each directory that used JavaScript to query the bucket and render a list of objects, but there's no built-in support for this.
You might want to take a look at s3-bucket-listing. It's for Amazon S3 but works with GCS as well.
Related
I have a method that gets a signed url for a blob in a google bucket and then returns that to users. Ideally, I could change the name of the file shown as well. Is this possible?
An example is:
https://storage.googleapis.com/<bucket>/<path to file.mp4>?Expires=1580050133&GoogleAccessId=<access-id>&Signature=<signature>
The part that I'd like to set myself is <path to file.mp4>.
The only way I can think of is having something in the middle that will be responsible for the name "swap".
For example Google App Engine with an http trigger or Cloud Function with storage trigger that whenever you need it will fetch the object, rename it, and either provide it to the user directly or store it with the new name in another bucket.
Keep in mind that things you want to store temporarily in GAE or Cloud Functions need to be stored in "/tmp" directory.
Then for renaming, if you are using GAE possibly you can use something like:
import os
os.system([YOUR_SHELL_COMMAND])
However, the easiest but more costly approach is to set a Function with storage trigger that whenever an object is uploaded it will store a copy of it with the desired new name in a different bucket that you will use for the users.
With the Google Speech API (using the python sample code), you need to have your audio files on google cloud when longer than 1 minute. According to some sample code, you can use a path like
gs://python-docs-samples-tests/speech/audio.flac.
So I put my audio files in a bucket, and use (I believe) the correct path (i.e. gs://bucket-name/foldername/myaudiofile.wav), yet I get an error .
NotFound: 404 No such object: bucket-name/foldername/myaudiofile.wav
Even if I put the permission on public (which I rather not do), it cannot find the file. I have the feeling I am forgetting something very trivial here... But still haven't found it.
Go to the cloud console > select the project > go to Storage > Browse the buckets and make sure the file is actually there. Best way you can tell, IMO.
I am using CloverETL Designer for ETL operations and I want to load some csv files from GCS to my Clover graph. I used FlatFileReader and tried to get file using remote File URL but it is not working. Can someone please detail the entire process here??
The path for file in GCS is
https://storage.cloud.google.com/PATH/Write_to_a_file.csv
And I need to get this csv file into the FlatFileReader in CloverETL Designer
You should use the Google Cloud Storage API to GET the file; Clover's HTTPConnector component will allow you to pass in the appropriate parameters to make a GET request (you will presumably have to do an OAuth2 authentication first to get a token), and send the output to a local destination specified in "Output File URL." Then you can use a FlatFileReader to read from that local file.
GCS has several different ways to download files from your buckets. You can use the console and the Cloud Storage browser. Steps: open the storage browser, navigate to the object you want to download, right click, and save to your chosen local folder. If you use Chrome the save appears as “Save Link As…”.
To use the GS Utility, use this command:
`gsutil cp gs://[BucketName]/[ObjectName] [ObjectDestination]`.
Or you can use client libraries or the REST APIs to download files. With these last options you could work with a number of files or create a job to download them. Once they are in a location known to Clover ETL the process is straightforward.
Within Clover designer, under the navigation pane you can right click a folder and choose import. Pick the one in which you placed your GCS file. Once the file is imported then you can use data from it like any other datafile in Clover. Since this is a .csv file, remember to edit your metadata (right click the component, choose extract metadata then edit inside the Metadata Editor -- for data types, labels and such.) Assign metadata to the edges of your components so they know what is coming in/going out of that step. Depending on your file, this process may be repeated many times.
Even with an ETL tool, getting the data and data types correct can be tricky. If you have questions about how to configure data types or your edges in an ETL project, a wiki may help. The web has additional resources may help you get the end analysis you’re looking for.
I'm working on a web project but the scenario has some restrictions for a specific user case. We have been investigating a web-only solution and a dropbox-like native way to solve this.
The main restriction is that we shouldn't upload local files to a cloud. We can only track local URI's.
The use cases are:
As a developer, I should be able to link the URI of a local file to a webapp. Thus, I can click on a webapp element and the local file should be opened.
As a user, I should be able to add a directory and view the same structure on the webapp (clicking opens the file). The files are not uploaded.
Possible solutions:
We started trying the FileSystem API but when the specs. were fully defined, we figured out that a local sandbox was not enough, and we can't access to the local URI due to security issues.
We are considering a Dropbox-like native app. The Invision Sync App is closer to what we want.
The less optimal solution would be a complete native application.
The question:
Which is the more efficient way to achieve this? Any idea on some native libraries for doing this faster? Any web-only workaround?
Thanks in advance.
I am uploading objects to amazon s3 using AWS iOS SDK in Iphone, sometime error occurs and some of the objects are uploaded, remaining are not uploaded. I have created bucket and inside bucket i have created folder in which i have store my objects. I want to delete folder and all its object. Can anyone help me?
First of all, there is not such thing as "folders" in S3. Most S3 clients (including the AWS web console) show them as folders only for convenience (grouping stuff), but in fact, what you see as a "folder name" is merely a prefix.
Being that said, my suggestion to you is using the listObjectsInBucket API call, passing in your "folder name" as prefix in the S3ListObjectsRequest parameter.
When you have obtained all the keys (file names including prefix) matching that prefix, use the deleteObjects API call, passing in the keys in S3ListObjectsRequest parameter.
For more details on folder/prefix and deleting stuff, please see these related links:
Delete files, directories and buckets in amazon s3 java
Thread on AWS forum regarding this subject