s3fs -- bucket name contains illegal character - s3fs

I am trying to mount an S3 bucket onto my ubuntu instance . But I want to only mount a "directory" or key in the s3 bucket.
bucketname : s3://my-bucket/
the directory or part of the s3 bucket I want to mount : s3://my-bucket/test/
mount point : /path/to/mount/dir/
I am using s3fs to achieve this like so:
s3fs my-bucket/test/ /path/to/mount/dir
But I get the following error all the time :
s3fs: BUCKET my-bucket/test/ -- bucket name contains an illegal character.
Any suggestions as to how I can mount a sub-directory of the s3 bucket with s3fs?
Thanks!
[CRT] s3fs_logger.cpp:LowSetLogLevel(203): change debug level from [CRT]to [INF]
[INF] s3fs.cpp:set_mountpoint_attribute(3984): PROC(uid=1002, gid=1002) - MountPoint(uid=1002, gid=1002, mode=40777)
[INF] curl.cpp:InitMimeType(432): Loaded mime information from /etc/mime.types
[INF] fdcache_stat.cpp:CheckCacheFileStatTopDir(79): The path to cache top dir is empty, thus not need to check permission.
[INF] s3fs.cpp:s3fs_init(3295): init v1.87(commit:38e1eaa) with OpenSSL
[INF] s3fs.cpp:s3fs_check_service(3411): check services.
[INF] curl.cpp:CheckBucket(3235): check a bucket.
[WAN] curl.cpp:ResetHandle(1875): The S3FS_CURLOPT_KEEP_SENDING_ON_ERROR option could not be set. For maximize performance you need to enable this option and you should use libcurl 7.51.0 or later.
[INF] curl_util.cpp:prepare_url(243): URL is https://s3.amazonaws.com/my-bucket/test/
[INF] curl_util.cpp:prepare_url(276): URL changed is https://my-bucket.s3.amazonaws.com/test/
[INF] curl.cpp:insertV4Headers(2572): computing signature [GET] [/test/] [] []
[INF] curl_util.cpp:url_to_host(320): url is https://s3.amazonaws.com
[ERR] curl.cpp:RequestPerform(2256): HTTP response code 400, returning EIO. Body Text: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'ap-south-1'</Message><Region>ap-south-1</Region><RequestId>5366A05FF406855D</RequestId><HostId>KmvHCBnlvnNZI/g3F2p+NfS0vP1Niq0+OPCZAP9u0pCjTOvpYzXfAB9QMlgQlcei2oi+lihn7F8=</HostId></Error>
[ERR] curl.cpp:CheckBucket(3261): Check bucket failed, S3 response: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'ap-south-1'</Message><Region>ap-south-1</Region><RequestId>5366A05FF406855D</RequestId><HostId>KmvHCBnlvnNZI/g3F2p+NfS0vP1Niq0+OPCZAP9u0pCjTOvpYzXfAB9QMlgQlcei2oi+lihn7F8=</HostId></Error>
[CRT] s3fs.cpp:s3fs_check_service(3444): Failed to connect region 'us-east-1'(default), so retry to connect region 'ap-south-1'.
[INF] curl_handlerpool.cpp:ReturnHandler(110): Pool full: destroy the oldest handler
[INF] curl.cpp:CheckBucket(3235): check a bucket.
[INF] curl_util.cpp:prepare_url(243): URL is https://s3-ap-south-1.amazonaws.com/my-bucket/test/
[INF] curl_util.cpp:prepare_url(276): URL changed is https://my-bucket.s3-ap-south-1.amazonaws.com/test/
[INF] curl.cpp:insertV4Headers(2572): computing signature [GET] [/test/] [] []
[INF] curl_util.cpp:url_to_host(320): url is https://s3-ap-south-1.amazonaws.com
[INF] curl.cpp:RequestPerform(2266): HTTP response code 404 was returned, returning ENOENT
[ERR] curl.cpp:CheckBucket(3261): Check bucket failed, S3 response: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>test/</Key><RequestId>FC8360185E0E5B79</RequestId><HostId>kwY8RPFw8d/6K9VL+cqc0KYDDgGVUVplbcvT+6PDBMvsdO0N53vQhfkfXRYzSQpy0sZ73f+er6o=</HostId></Error>
[CRT] s3fs.cpp:s3fs_check_service(3484): bucket not found(host=https://s3-ap-south-1.amazonaws.com) - result of checking service.
[ERR] s3fs.cpp:s3fs_exit_fuseloop(3285): Exiting FUSE event loop due to errors
[INF] s3fs.cpp:s3fs_destroy(3353): destroy

Try separating the bucket name from the bucket path with a colon:
s3fs my-bucket:/test /path/to/mount/dir
s3fs also requires that a directory object exists to mount a path. You can create this via the AWS CLI:
aws s3api put-object --bucket my-bucket --key test/

Related

Jfrog file upload failing with 403 forbidden

I have been trying to upload a file into JFrog artifactory repo from the cli as below and keep getting the 403 Forbidden error. What am I missing as env variables, etc for the upload to go through from the cli?
jf rt u "xyz" --url="http://example.com/ui/repos/tree/General/asdf/cba" --username="afad" --password="********"
Log path: /home/asdf/.jfrog/logs/jfrog-cli.ss.ss.ss.log
[Info][Thread 2]Uploading artifact: xyz
[Warn][Thread 2](Attempt 1) - Failure occurred while uploading to http://example.com/ui/repos/tree/General/asdf/cba/--username=svc#esuk.com/xyz:
server response: 403 Forbidden
[Warn][Thread 2](Attempt 2) - Failure occurred while uploading to http://example.com/ui/repos/tree/General/asdf/cba/--username=svc#esuk.com/xyz:
server response: 403 Forbidden
[Warn][Thread 2](Attempt 3) - Failure occurred while uploading to http://example.com/ui/repos/tree/General/asdf/cba/--username=svc#esuk.com/xyz:
server response: 403 Forbidden
[Warn][Thread 2](Attempt 4) - Failure occurred while uploading to http://example.com/ui/repos/tree/General/asdf/cba/--username=svc#esuk.com/xyz:
server response: 403 Forbidden
[Info] [Thread 2] executor timeout after 3 attempts with 0 milliseconds wait intervals
[Error] server response: 403 Forbidden
[Error] Failed uploading 1 artifacts.
[Error] server response: 403 Forbidden
--username is not a valid argument and the target path is also missing. Use the below command to upload a file.
jf rt u "testfile.txt" "generic-local" --url="https://myartifactory.jfrog.io/artifactory" --user="youruser" --password="yourpassword"
This will upload a file called testfile.txt to the repository generic-local. If you want to deploy to a specific path, you can pass the path after generic-local.
jf rt u "testfile.txt" "generic-local/path1/subpath/" --url="https://myartifactory.jfrog.io/artifactory" --user="youruser" --password="yourpassword"

GKE cluster (k8s 1.21) with dataplane v2 can't kubectl exec/port-forward/logs etc

We have a cluster at GKE which spontaneously stopped supporting kubectl commands exec/port-forward/logs upon cluster upgrade (seemingly to 1.21). Troubleshooting with https://cloud.google.com/kubernetes-engine/docs/troubleshooting#kubect_commands_stops did not solve the issue:
there are no egress-blocking firewall rules
control-plane-to-nodes ssh ingress rule is in place
there are four project-wide ssh keys set, this does not seem like "too many", it is unclear if it would be safe to remove some (and which?)
there is no ssh-key metadata on the node vm's
there are no egress-blocking network policies in place in k8s
Another cluster in the same GCP project (1.20, no dataplane v2) is working fine. The firewall rules are the same. We really have no clue what the problem could be, we can't find anything in the logs.
Does anybody have the same issue, or have any idea how we could troubleshoot this further?
Example output
$ kubectl -v 5 exec podname -- echo 'hi'
I0319 10:09:14.318262 8314 gcp.go:122] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.25+; use gcloud instead.
To learn more, consult https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credenkubectl -v 10 logs mainsite-test-next-web-deployment-7678d4ddc7-d57x5tial-plugins
I0319 10:09:14.396857 8314 request.go:1372] body was not decodable (unable to check for Status): couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }
I0319 10:09:14.396902 8314 cached_discovery.go:78] skipped caching discovery info due to the server is currently unable to handle the request
I0319 10:09:14.396943 8314 shortcut.go:89] Error loading discovery information: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0319 10:09:14.419116 8314 request.go:1372] body was not decodable (unable to check for Status): couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }
I0319 10:09:14.419128 8314 cached_discovery.go:78] skipped caching discovery info due to the server is currently unable to handle the request
I0319 10:09:14.454275 8314 podcmd.go:88] Defaulting container name to web
I0319 10:09:44.539168 8314 helpers.go:219] server response object: [{
"metadata": {},
"status": "Failure",
"message": "error dialing backend: dial timeout, backstop",
"code": 500
}]
Error from server: error dialing backend: dial timeout, backstop
$ kubectl -v 5 logs podname
I0319 10:12:47.021736 11845 gcp.go:122] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.25+; use gcloud instead.
To learn more, consult https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins
I0319 10:12:47.086484 11845 request.go:1372] body was not decodable (unable to check for Status): couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }
I0319 10:12:47.086501 11845 cached_discovery.go:78] skipped caching discovery info due to the server is currently unable to handle the request
I0319 10:12:47.086528 11845 shortcut.go:89] Error loading discovery information: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0319 10:12:47.110462 11845 request.go:1372] body was not decodable (unable to check for Status): couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }
I0319 10:12:47.110480 11845 cached_discovery.go:78] skipped caching discovery info due to the server is currently unable to handle the request
I0319 10:13:17.158303 11845 helpers.go:219] server response object: [{
"metadata": {},
"status": "Failure",
"message": "Get \"https://10.164.0.17:10250/containerLogs/deploy-name/pod-name/web\": dial timeout, backstop",
"code": 500
}]
Error from server: Get "https://10.164.0.17:10250/containerLogs/deploy-name/pod-name/web": dial timeout, backstop
As you have a "timeout" error it seems that kubectl can't communicate with the cluster control plane.
Can you give a try executing:
gcloud container clusters get-credentials CLUSTER_NAME
--region=COMPUTE_REGION
After this the communication with the control plane should be restored.
In the end it turned out that it was the network policies that we had set up for the kube-system namespace. This used to work just fine, until it didn't. We completely forgot that we had them in the first place :see_no_evil: Turns out we were a little overzealous there. When we removed them, everything was fine.

DataPower SSL config for "Fetch" results in "unable to get SSL Profile''

We have a javascript file hosted on GitHub that I'd like to fetch:
Error logs:
172416 cli error 320673 0x81000224 === Line 2: copy -f https://raw.githubusercontent.com/mmxxxxx/abc/master/shim_mpgw_mapping.js store:///js/shim_mpgw_mapping.js
172416 audit error 320673 0x82400020 (admin:default:web-gui:10.106.170.13): (config)# copy -f https://raw.githubusercontent.com/mm58169/datapower7/master/shim_mpgw_mapping.js store:///js/shim_mpgw_mapping.js
172416 audit error 320673 0x8240002e (admin:default:*:*): Copying file "https://raw.githubusercontent.com/mm58169/datapower7/master/shim_mpgw_mapping.js" to "store:///js/shim_mpgw_mapping.js" failed
172416 cli error 320673 0x810002e7 File copy failed - destination URL could not be opened: store:///js/shim_mpgw_mapping.js
172416 file error 320673 0x8100015c Copying file "https://raw.githubusercontent.com/mm58169/datapower7/master/shim_mpgw_mapping.js" to "store:///js/shim_mpgw_mapping.js" : failed
172416 system warning 320673 0x8040000a destination URL could not be opened: store:///js/shim_mpgw_mapping.js
172416 network error 31711 0x80e0005a Cannot establish SSL credentials (credential is NULL), URL: 'https://raw.githubusercontent.com/mm58169/datapower7/master/shim_mpgw_mapping.js'.
172416 network warning 31711 0x80e00058 SSL connection to 'https://raw.githubusercontent.com/mm58169/datapower7/master/shim_mpgw_mapping.js' failed, unable to get SSL Profile ''
172416 network warning 31711 0x80e00058 SSL connection to 'https://raw.githubusercontent.com/mm58169/datapower7/master/shim_mpgw_mapping.js' failed, unable to get SSL Profile ''
#VonC is correct that you need a SSL Client Profile. You need to add it into the User Agent of your service:
The Fetch action will download the file to your DataPower local:/// filesystem so you might as well download it manually and upload it to the DataPower instance though...
This should be related to TLS selection, like the DataPower service configuration variable called var://service/tls-info, or serviceVars.tlsInfo, which might not be properly set.
You would have then to define an SSL client profile (which secures connections between the DataPower Gateway and its targets)

Create react app service worker nginx no cache configuration

I am trying to set cache header for service worker through nginx in create react app project, in the configuration, I tried
location /service-worker.js {
add_header Cache-Control "no-cache";
proxy_cache_bypass $http_pragma;
proxy_cache_revalidate on;
expires off;
access_log off;
}
However when I load my page, sw registration fails with the message.
A bad HTTP response code (404) was received when fetching the script.
registerServiceWorker.js:71 Error during service worker registration: TypeError: Failed to register a ServiceWorker: A bad HTTP response code (404) was received when fetching the script.
Can someone please suggest a way with nginx using create-react-app?
as per your configuration service-worker.js must be in / root directory defined with root nginx directive.
Please check if the file is present there. If you are using express and express static and have placed the file in public/assets directory, it won't work. if for this file you want to to have different location. you can use alias directive.

Google cloud: Using gsutil to download data from AWS S3 to GCS

One of our collaborators has made some data available on AWS and I was trying to get it into our google cloud bucket using gsutil (only some of the files are of use to us, so I don't want to use the GUI provided on GCS). The collaborators have provided us with the AWS bucket ID, the aws access key id, and aws secret access key id.
I looked through the documentation on GCE and editied the ~/.botu file such that the access keys are incorporated. I restarted my terminal and tried to do an 'ls' but got the following error:
gsutil ls s3://cccc-ffff-03210/
AccessDeniedException: 403 AccessDenied
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied
Do I need to configure/run something else too?
thanks!
EDITS:
Thanks for the replies!
I installed the Cloud SDK and I can access and run all gsutil commands on my google cloud storage project. My problem is in trying to access (e.g. 'ls' command) the amazon S3 that is being shared with me.
I uncommented two lines in the ~/.boto file and put the access keys:
# To add HMAC aws credentials for "s3://" URIs, edit and uncomment the
# following two lines:
aws_access_key_id = my_access_key
aws_secret_access_key = my_secret_access_key
Output of 'gsutil version -l':
| => gsutil version -l
my_gc_id
gsutil version: 4.27
checksum: 5224e55e2df3a2d37eefde57 (OK)
boto version: 2.47.0
python version: 2.7.10 (default, Oct 23 2015, 19:19:21) [GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)]
OS: Darwin 15.4.0
multiprocessing available: True
using cloud sdk: True
pass cloud sdk credentials to gsutil: True
config path(s): /Users/pc/.boto, /Users/pc/.config/gcloud/legacy_credentials/pc#gmail.com/.boto
gsutil path: /Users/pc/Documents/programs/google-cloud- sdk/platform/gsutil/gsutil
compiled crcmod: True
installed via package manager: False
editable install: False
The output with the -DD option is:
=> gsutil -DD ls s3://my_amazon_bucket_id
multiprocessing available: True
using cloud sdk: True
pass cloud sdk credentials to gsutil: True
config path(s): /Users/pc/.boto, /Users/pc/.config/gcloud/legacy_credentials/pc#gmail.com/.boto
gsutil path: /Users/pc/Documents/programs/google-cloud-sdk/platform/gsutil/gsutil
compiled crcmod: True
installed via package manager: False
editable install: False
Command being run: /Users/pc/Documents/programs/google-cloud-sdk/platform/gsutil/gsutil -o GSUtil:default_project_id=my_gc_id -DD ls s3://my_amazon_bucket_id
config_file_list: ['/Users/pc/.boto', '/Users/pc/.config/gcloud/legacy_credentials/pc#gmail.com/.boto']
config: [('debug', '0'), ('working_dir', '/mnt/pyami'), ('https_validate_certificates', 'True'), ('debug', '0'), ('working_dir', '/mnt/pyami'), ('content_language', 'en'), ('default_api_version', '2'), ('default_project_id', 'my_gc_id')]
DEBUG 1103 08:42:34.664643 provider.py] Using access key found in shared credential file.
DEBUG 1103 08:42:34.664919 provider.py] Using secret key found in shared credential file.
DEBUG 1103 08:42:34.665841 connection.py] path=/
DEBUG 1103 08:42:34.665967 connection.py] auth_path=/my_amazon_bucket_id/
DEBUG 1103 08:42:34.666115 connection.py] path=/?delimiter=/
DEBUG 1103 08:42:34.666200 connection.py] auth_path=/my_amazon_bucket_id/?delimiter=/
DEBUG 1103 08:42:34.666504 connection.py] Method: GET
DEBUG 1103 08:42:34.666589 connection.py] Path: /?delimiter=/
DEBUG 1103 08:42:34.666668 connection.py] Data:
DEBUG 1103 08:42:34.666724 connection.py] Headers: {}
DEBUG 1103 08:42:34.666776 connection.py] Host: my_amazon_bucket_id.s3.amazonaws.com
DEBUG 1103 08:42:34.666831 connection.py] Port: 443
DEBUG 1103 08:42:34.666882 connection.py] Params: {}
DEBUG 1103 08:42:34.666975 connection.py] establishing HTTPS connection: host=my_amazon_bucket_id.s3.amazonaws.com, kwargs={'port': 443, 'timeout': 70}
DEBUG 1103 08:42:34.667128 connection.py] Token: None
DEBUG 1103 08:42:34.667476 auth.py] StringToSign:
GET
Fri, 03 Nov 2017 12:42:34 GMT
/my_amazon_bucket_id/
DEBUG 1103 08:42:34.667600 auth.py] Signature:
AWS RN8=
DEBUG 1103 08:42:34.667705 connection.py] Final headers: {'Date': 'Fri, 03 Nov 2017 12:42:34 GMT', 'Content-Length': '0', 'Authorization': u'AWS AK6GJQ:EFVB8F7rtGN8=', 'User-Agent': 'Boto/2.47.0 Python/2.7.10 Darwin/15.4.0 gsutil/4.27 (darwin) google-cloud-sdk/164.0.0'}
DEBUG 1103 08:42:35.179369 https_connection.py] wrapping ssl socket; CA certificate file=/Users/pc/Documents/programs/google-cloud-sdk/platform/gsutil/third_party/boto/boto/cacerts/cacerts.txt
DEBUG 1103 08:42:35.247599 https_connection.py] validating server certificate: hostname=my_amazon_bucket_id.s3.amazonaws.com, certificate hosts=['*.s3.amazonaws.com', 's3.amazonaws.com']
send: u'GET /?delimiter=/ HTTP/1.1\r\nHost: my_amazon_bucket_id.s3.amazonaws.com\r\nAccept-Encoding: identity\r\nDate: Fri, 03 Nov 2017 12:42:34 GMT\r\nContent-Length: 0\r\nAuthorization: AWS AN8=\r\nUser-Agent: Boto/2.47.0 Python/2.7.10 Darwin/15.4.0 gsutil/4.27 (darwin) google-cloud-sdk/164.0.0\r\n\r\n'
reply: 'HTTP/1.1 403 Forbidden\r\n'
header: x-amz-bucket-region: us-east-1
header: x-amz-request-id: 60A164AAB3971508
header: x-amz-id-2: +iPxKzrW8MiqDkWZ0E=
header: Content-Type: application/xml
header: Transfer-Encoding: chunked
header: Date: Fri, 03 Nov 2017 12:42:34 GMT
header: Server: AmazonS3
DEBUG 1103 08:42:35.326652 connection.py] Response headers: [('date', 'Fri, 03 Nov 2017 12:42:34 GMT'), ('x-amz-id-2', '+iPxKz1dPdgDxpnWZ0E='), ('server', 'AmazonS3'), ('transfer-encoding', 'chunked'), ('x-amz-request-id', '60A164AAB3971508'), ('x-amz-bucket-region', 'us-east-1'), ('content-type', 'application/xml')]
DEBUG 1103 08:42:35.327029 bucket.py] <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>6097164508</RequestId><HostId>+iPxKzrWWZ0E=</HostId></Error>
DEBUG: Exception stack trace:
Traceback (most recent call last):
File "/Users/pc/Documents/programs/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 577, in _RunNamedCommandAndHandleExceptions
collect_analytics=True)
File "/Users/pc/Documents/programs/google-cloud-sdk/platform/gsutil/gslib/command_runner.py", line 317, in RunNamedCommand
return_code = command_inst.RunCommand()
File "/Users/pc/Documents/programs/google-cloud-sdk/platform/gsutil/gslib/commands/ls.py", line 548, in RunCommand
exp_dirs, exp_objs, exp_bytes = ls_helper.ExpandUrlAndPrint(storage_url)
File "/Users/pc/Documents/programs/google-cloud-sdk/platform/gsutil/gslib/ls_helper.py", line 180, in ExpandUrlAndPrint
print_initial_newline=False)
File "/Users/pc/Documents/programs/google-cloud-sdk/platform/gsutil/gslib/ls_helper.py", line 252, in _RecurseExpandUrlAndPrint
bucket_listing_fields=self.bucket_listing_fields):
File "/Users/pc/Documents/programs/google-cloud-sdk/platform/gsutil/gslib/wildcard_iterator.py", line 476, in IterAll
expand_top_level_buckets=expand_top_level_buckets):
File "/Users/pc/Documents/programs/google-cloud-sdk/platform/gsutil/gslib/wildcard_iterator.py", line 157, in __iter__
fields=bucket_listing_fields):
File "/Users/pc/Documents/programs/google-cloud-sdk/platform/gsutil/gslib/boto_translation.py", line 413, in ListObjects
self._TranslateExceptionAndRaise(e, bucket_name=bucket_name)
File "/Users/pc/Documents/programs/google-cloud-sdk/platform/gsutil/gslib/boto_translation.py", line 1471, in _TranslateExceptionAndRaise
raise translated_exception
AccessDeniedException: AccessDeniedException: 403 AccessDenied
AccessDeniedException: 403 AccessDenied
I'll assume that you are able to set up gcloud credentials using gcloud init and gcloud auth login or gcloud auth activate-service-account, and can list/write objects to GCS successfully.
From there, you need two things. A properly configured AWS IAM role applied to the AWS user you're using, and a properly configured ~/.boto file.
AWS S3 IAM policy for bucket access
A policy like this must be applied, either by a role granted to your user or an inline policy attached to the user.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::some-s3-bucket/*",
"arn:aws:s3:::some-s3-bucket"
]
}
]
}
The important part is that you have ListBucket and GetObject actions, and the resource scope for these includes at least the bucket (or prefix thereof) that you wish to read from.
.boto file configuration
Interoperation between service providers is always a bit tricky. At the time of this writing, in order to support AWS Signature V4 (the only one supported universally by all AWS regions), you have to add a couple extra properties to your ~/.boto file beyond just credential, in an [s3] group.
[Credentials]
aws_access_key_id = [YOUR AKID]
aws_secret_access_key = [YOUR SECRET AK]
[s3]
use-sigv4=True
host=s3.us-east-2.amazonaws.com
The use-sigv4 property cues Boto, via gsutil, to use AWS Signature V4 for requests. Currently, this requires the host be specified in the configuration, unfortunately. It is pretty easy to figure the host name out, as it follows the pattern of s3.[BUCKET REGION].amazonaws.com.
If you have rsync/cp work from multiple S3 regions, you could handle it a few ways. You can set an environment variable like BOTO_CONFIG before running the command to change between multiple files. Or, you can override the setting on each run using a top-level argument, like:
gsutil -o s3:host=s3.us-east-2.amazonaws.com ls s3://some-s3-bucket
Edit:
Just want to add... another cool way to do this job is rclone.
1. Generate your GCS credentials
If you download the Cloud SDK, then run gcloud init and gcloud auth login, gcloud should configure the OAuth2 credentials for the account you logged in with, allowing you to access your GCS bucket (it does this by creating a boto file that gets loaded in addition to your ~/.boto file, if it exists).
If you're using standalone gsutil, run gsutil config to generate a config file at ~/.boto.
2. Add your AWS credentials to the file ~/.boto
The [Credentials] section of your ~/.boto file should have these two lines populated and uncommented:
aws_access_key_id = IDHERE
aws_secret_access_key = KEYHERE
If you've done that:
Make sure that you didn't accidentally swap the values for key and id.
Verify you're loading the correct boto file(s) - you can do this by
running gsutil version -l and looking for the "config path(s):" line.
If you still receive a 403, it's possible that they've given you either
the wrong bucket name, or a key and id corresponding to an account
that doesn't have permission to list the contents of that bucket.