I am trying to upload a list of files from a folder into an Amazon S3 folder.
I am able to manually upload files on the folder.But when I run the job which does the same thing ,the talend job gives an "Access Denied" error.
I have the required keys for the S3 bucket.
If you are getting the Access Denied error then it mean you do not have access to that bucke or check the access constraint again.
You can also manually copy the files to S3 by downloading the software called "CloudBerry Exlorer for Amazon S3".
Just download and provide the access key and see whether you have access to the bucket or not.
Related
I have a blob zip file in my storage account, I have a linked service and binary dataset to get the file as the source in a copy activity. There is an outside service I call in a web activity that returns a writable SAS URL to a different storage account in this format.
https://foo.blob.core.windows.net/dmf/43de9fb6-3b96-4f47-b730-eb8de040859dblah.zip?sv=2014-02-14&sr=b&sig=0mgvh25htg45b5u4ty5E%2Bf0ahMwFkHVy3iTC2nh%2FIKw%3D&st=2022-08-13T02%3A19%3A33Z&se=2022-08-13T02%3A54%3A33Z&sp=rw
I tried adding a SAS azure blob linked service, I added a parameter for the uri on the LS, then added a dataset bound to the LS and also added a parameter for the uri, I pass the SAS uri dynamically all the way down to the linked service. The copy fails each time with The remote server returned an error: (403). I have to be doing something wrong but not sure what it is. I'd appreciate any input, thanks.
I tried to reproduce the same in my environment and got same error:
To resolve the above 403 error, you need to enable it from all network option and also check whether the Storage blob data contributor was added or not. If not , Go to Azure Storage Account -> Access control (IAM) -> +Add role assignment as Storage blob data contributor.
Now, its working.
I am creating a dataset in Azure Data Factory. This dataset will be a Parquet file within a directory under a certain container in an ADLS Gen2 account. The container name is 'raw', and the directory that I want to place the file into is source/system1/FullLoad. When I click on Browse next to File path, I am able to access the container, but I cannot access the directory. When I hit folder 'source', I get the error shown below.
How can I drill to the desired directory? As the error message indicates, I suspect that it's something to do with permissions to access the data (the Parquet file doesn't exist yet, as it will be used as a sink in a copy activity that hasn't been run yet), but I don't know how to resolve.
Thanks for confirming putting the resolution for others if anyone face this issue.
The user or managed identity you are using for your data factory should have storage data blob contributor access on the storage account. You can check it from azure portal, go to your storage account, navigate to the container and then directory, click on Access Control on the left panel and check role assignment. If it is missing add the role assignment of storage data blob contributor to your managed identity.
I have created an external stage for my S3 bucket in snowflake. When I try to access the stage using
List #database.schema.stagename
am getting access denied error. We have checked the S3 bucket policy and fixed the issue.
But
I want to check log in snowflake. is it possible any of the snowflake log will give details about this issue. ? if any where I have to check ?
Open a support ticket(support#snowflake.com) to get details about a specific incident or error.
I am trying to download the exported data from my GSuite (Google Workplace) account. I ran the data export tool and it is sitting in a bucket. I want to download all of the files but it says that the only way I can download multiple files is to use the gsutil utility.
I installed it using pip instal -U gsutil.
I tried running the following command:
gsutil cp -r \
gs://takeout-export-3ba9a6a2-c080-430a-bece-6f830889cc83/20201202T070520Z/ \
gs://takeout-export-3ba9a6a2-c080-430a-bece-6f830889cc83/Status\ Report.html \
.
...but it failed with an error:
ServiceException: 401 Anonymous caller does not have storage.objects.get access to the Google Cloud Storage object.
I suppose that is because I am not authenticated. I tried going through the motions with gsutil config, but it is now asking me for a "Project ID", which I cannot find anywhere in the cloud storage web page showing the bucket with the exported files.
I tries following the top answer for this question, but the project ID does not appear to be optional anymore.
How do I download my files?
The project ID is "optional" in the sense that it's only used for certain scenarios, e.g. when you want to create a bucket (without explicitly specifying a project for it to live in), that project is specified as its parent. For most things, like your scenario of copying existing GCS objects to your local filesystem, your default project ID doesn't matter; you can just type whatever you want for the project ID in order to generate your boto file for authentication.
I'm having an issue transferring files from an S3 bucket to Google Cloud Storage.
The configuration of the job has an access key ID and secret key for a user which has AmazonS3FullAccess as policy. I verified access using aws-cli.
The path to the bucket is entered like root-bucket/folder/. There are no additional transfer options to the job.
The job runs for a while in the calculating state, and then fails with the error message
UNKNOWN: (showing 1 of 1 failures) >
Object: or
Is it possible to get a more verbose log of what failed with the job?
As Rybosome metions, only the bucket name can be written in the Amazon S3 bucket field. When wanting to include sub-folders, write it in the more section, under include Transfer files with these prefixes.