I have uploaded file to google cloud storage via their php API
I am trying to use following code now to access it back.
$bucket = $storage->bucket('my-bucket');
$object = $bucket->object('filename.json');
$string = $object->downloadAsString();
echo $string;
I am just trying to retrieve a json , But problem is its getting cached and keep giving me old file which I modified 30 mins back.
How to disable Cache ? While using downloadAsString ?something like
https://storage.googleapis.com/my-bucket/filename.json?id=randomtimestamp
P.s: I know Disabling cache while uploading json will work, but I just want to disable cache in few of my php scripts.
Related
I installed latest voximal stack.
Calls are working.
I setup recognition to use Google Cloud Streaming for STT pasting the JSON credentials into browser, the credentials file created at /usr/share/voximal.
But when I try to test parrot.vxml with a call it fail to create the google_streaming resource and drop the call.
Any clues?
Please enable the logs (interpreter debug) and check the /var/log/voximal/debig.log...
You probably see in the root cause there.
I am working on a webapp for transferring files with Aspera. We are using AoC for the transfer server and an S3 bucket for storage.
When I upload a file to my s3 bucket using aspera connect everything appears to be successful, I see it in the bucket, and I see the new file in the directory when I run /files/browse on the parent folder.
I am refactoring my code to use the /files/{id}/files endpoint to list the directory because the documentation says it is faster compared to the /files/browse endpoint. After the upload is complete, when I run the /files/{id}/files GET request, the new file does not show up in the returned data right away. It only becomes available after a few minutes.
Is there some caching mechanism in place? I can't find anything about this in the documentation. When I make a transfer in the AoC dashboard everything updates right away.
Thanks,
Tim
Yes, the file-id base system uses an in-memory cache (redis).
This cache is updated when a new file is uploaded using Aspera. But for files movement directly on the storage, there is a daemon that will periodically scan and find new files.
If you want to bypass the cache, and have the API read the storage, you can add this header in the request:
X-Aspera-Cache-Control: no-cache
Another possibility is to trigger a scan by reading:
/files/{id}
for the folder id
Is there a way to serve model from Google Cloud Storage without actually downloading a copy of model? like streaming the data directly?
I'm trying to load a fasttext model that is hosted on Google Cloud Storage. everytime i run the program, it needs to get and download a copy of that model in the bucket.
language_model_filename = 'lid.176.bin' // filename in GCS
language_model_local = 'lid.176.bin' // local file name when downloaded
bucket = storage_client.get_bucket(CLOUD_STORAGE_BUCKET)
blob = bucket.blob(language_model_filename)
blob.download_to_filename(language_model_local)
language_model = FastText.load_model(language_model_local)
You can use Streaming Tranfers for that purpose. As explained in the documentation, you can use the third party boto client library plugin for Cloud Storage.
A streaming download example would look like this:
import sys
downloaded_file = 'saved_data_file'
MY_BUCKET = 'my_app_bucket'
object_name = 'data_file'
src_uri = boto.storage_uri(MY_BUCKET + '/' + object_name, 'gs')
src_uri.get_key().get_file(sys.stdout)
I am currently trying to use the FTPHook in Airflow in order to upload and download file to/from a remote ftp. But I'm not sure if I can use the gs:// path as part of the source/destination path.
I currently don't want to use local folder within the AF pod since the file size might get big, so I would rather use gcs path directly or gcs file stream.
conn = FTPHook(ftp_conn_id='ftp_default')
conn.store_file('in', 'gs://bucket_name/file_name.txt')
link to the FTPHook code:
here
Thanks for any help!
I found a simple streaming solution to upload/download from gcs to ftp server and vice versa using pysftp which I'll like to share with you.
First, I found this solution, which was working great, but the only issue with that solution was that it didn't support upload file from gcs to FTP. So I was looking for something else.
So than I was looking into different approach, so I've found this google document which basically allow you to stream to/from blob file which was exactly what I was looking for.
params = BaseHook.get_connection(self.ftp_conn_id)
cnopts = pysftp.CnOpts()
cnopts.hostkeys = None
ftp = pysftp.Connection(host=params.host, username=params.login, password=params.password,
port=params.port,
cnopts=cnopts)
#This will download file from FTP server to GCS location
with ftp.open(self.ftp_folder + '/' + file_to_load, 'r+') as remote_file:
blob = bucket.blob(self.gcs_prefix + file_to_load)
blob.upload_from_file(remote_file)
#This will upload file from GCS to FTP server
with sftp.open(self.ftp_folder + '/' +file_name,'w+') as remote_file:
blob = bucket.blob(fileObject['name'])
blob.download_to_file(remote_file)
GCS does not implement FTP support, so this won't work.
It looks like FTP hook only knows how to deal with a local file path or buffer, not one of the GCS APIs.
You might be able to find (or write) some code that reads from FTP and writes to GCS.
I am using Google App Engine PHP SDK.
Google cloud storage allows users to check a "publicly shared?" field in the storage manager that allows you to share a URL to the data directly.
I'm using google app engine and sending data to the storage, but I would like to have it publicly shared by default.
this is code which I am using to upload files
require_once 'google/appengine/api/cloud_storage/CloudStorageTools.php';
use google\appengine\api\cloud_storage\CloudStorageTools;
$options = [ 'gs_bucket_name' => 'my_bucket' ];
$upload_url = CloudStorageTools::createUploadUrl('/test.php', $options);
$gs_name = $_FILES['sample']['tmp_name'];
move_uploaded_file($gs_name, 'gs://test_sample/');
How can I do this? Their docs does not seem to mention anything about this, except manually doing it.
You can define the permissions that your new uploaded files will have by default with the command:
gsutil defacl ch -u allUsers:R gs://<bucket>
And after you upload your files with your code they should be publicly shared.
Visit the following links for more information about the command:
https://cloud.google.com/storage/docs/gsutil/commands/defacl
https://cloud.google.com/storage/docs/gsutil/commands/acl
Hope it helps.