I have several Google Compute Engine instances, and have set instance metadata on each, under the assumption these are available on the instance itself as an environment variable, but they don't show up. I then read here that I need to query the metadata server for this data, but that just returns a 403 unauthorized when run from the instance itself. Is there a way to access metadata as environment variables?
It may be worth studying Metadata querying a bit more, but my guess is that you are attempting to get custom metadata, which is resulting in it not being found. Make sure you are using the attributes directory to access any custom metadata.
For example, this will get the built-in tags metadata:
curl "http://metadata.google.internal/computeMetadata/v1/instance/tags" \
-H "Metadata-Flavor: Google"
while this will get your custom metadata for attribute foo:
curl "http://metadata.google.internal/computeMetadata/v1/<instance|project>/attributes/foo" \
-H "Metadata-Flavor: Google"
Related
In Vault documentation, specifically the policies page, there is this phrase:
Everything in Vault is path based, and policies are no exception
I wonder about this phrase, does it mean that in the architecture of Vault and in its internals everything really is a path, similar to the "Everything is a file" in Linux architecture, which applies on processes, files, directories, sockets, pipes, etc?
What makes me relate to this phrase is that secret engines are defined by paths, and I assume Vault infers their types and which to be used from the given paths. Also policies are relatable as you have to define exact paths for each policy, but what about other components like auth methods, audits, tokens, etc?
I just want to get what is meant by "path based" in "Everything in Vault is path based" phrase.
In Vault, everything is path based. This means that every operation that is performed in Vault is done through a path. The path is used to determine the location of the operation, as well as the permissions that are required to execute the operation.
Whether you're using the vault binary or whether you're hitting the HTTP API endpoints, secrets/configs are written to a path.
i.e. via cli:
VAULT_ADDR=https://myvault.example.com VAULT_TOKEN=xxxxxxxx-xxxxxxx-xxxxxx vault kv get mysecrets/passwords/root
would correspond to HTTP endpoint:
curl \
-H "X-Vault-Token: xxxxxxx-xxxxxx-xxxxxxx" \
-X GET \
https://myvault.example.com/v1/mysecrets/passwords/root
Here's another example:
enabling the gcp secret engine with a custom path:
vault secrets enable -path=”my-project-123” gcp
If you wanted to enable secrets engines from the HTTP API, the endpoint (path) is /sys/mounts. Details here.
creating writing a config:
vault write my-project-123/config credentials=#/path/to/creds.json ttl=3600 max_ttl=21600
Notice how the config is written to a path, and if you were to use the HTTP API endpoint to do this, then it would look something like this:
curl \
--header "X-Vault-Token: ..." \
--request POST \
--data #payload.json \
https://myvault.example.com/v1/my-project-123/config
Where the payload.json would contain your credentials in text, ttl, max_ttl
Hence why they Vault says everything is path based.
EDIT: TL;DR - path based is so that there's parity between HTTP API and CLI (or any SDKs too). Compare this to a gcloud or aws command to its HTTP API endpoint counterpart where there isn't much parity there.
I am attempting to set permissions on individual objects in a Google Cloud Storage bucket to make them publicly viewable, following the steps indicated in Google's documentation. When I try to make these requests using our application service account, it fails with HTTP status 403 and the following message:
Access denied. Provided scope(s) are not authorized.
Other requests work fine. When I try to do the same thing but by providing a token for my personal account, the PUT request to the object's ACL works... about 50% of the time (the rest of the time it is a 503 error, which may or may not be related).
Changing the IAM policy for the service account to match mine - it normally has Storage Admin and some other incidental roles - doesn't help, even if I give it the overall Owner IAM role, which is what I have.
Using neither the XML API nor the JSON version makes a difference. That the request sometimes works with my personal credentials indicates to me that the request is not incorrectly formed, but there must be something else I've thus far overlooked. Any ideas?
Check for the scope of the service account incase you are using the default compute engine service account. By default the scope is restricted and for GCS it is read only. Use rm -r ~/.gsutil to clear cache in case of clearing cache
When trying to access GCS from a GCE instance and getting this error message ...
the default scope is devstorage.read_only, which prevents all write operations.
Not sure if scope https://www.googleapis.com/auth/cloud-platform is required, when scope https://www.googleapis.com/auth/devstorage.read_only is given by default (eg. to read startup scripts). The scope should rather be: https://www.googleapis.com/auth/devstorage.read_write.
And one can use gcloud beta compute instances set-scopes to edit the scopes of an instance:
gcloud beta compute instances set-scopes $INSTANCE_NAME \
--project=$PROJECT_ID \
--zone=$COMPUTE_ZONE \
--scopes=https://www.googleapis.com/auth/devstorage.read_write \
--service-account=$SERVICE_ACCOUNT
One can also pass all known alias names for scopes, eg: --scopes=cloud-platform. The command must be run outside of the instance, because of permissions - and the instance must be shutdown, in order to change the service account.
Follow the documentation you provided, taking into account these points:
Access Control system for the bucket has to be Fine-grained (not uniform).
In order to make objects publicly available, make sure the bucket does not have the public access prevention enabled. Check this link for further information.
Grant the service account with the appropriate permissions in the bucket. The Storage Legacy Object Owner role (roles/storage.legacyObjectOwner) is needed to edit objects ACLs as indicated here. This role can be granted for individual buckets, not for projects.
Create the json file as indicated in the documentation.
Use gcloud auth application-default print-access-token to get authorization access token and use it in the API call. The API call should look like:
curl -X POST --data-binary #JSON_FILE_NAME.json \
-H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
-H "Content-Type: application/json" \
"https://storage.googleapis.com/storage/v1/b/BUCKET_NAME/o/OBJECT_NAME/acl"
You need to add OAuth scope: cloud-platform when you create the instance. Look: https://cloud.google.com/sdk/gcloud/reference/compute/instances/create#--scopes
Either select "Allow full access to all Cloud APIs" or select the fine-grained approach
So, years later, it turns out the problem is that "scope" is being used by the Google Cloud API to refer to two subtly different things. One is the available permission scopes available to the service account, which is what I (and most of the other people who answered the question) kept focusing on, but the problem turned out to be something else. The Python class google.auth.credentials.Credentials, used by various Google Cloud client classes to authenticate, also has permission scopes used for OAuth. You see where this is going - the client I was using was being created with a default OAuth scope of 'https://www.googleapis.com/auth/devstorage.read_write', but to make something public requires the scope 'https://www.googleapis.com/auth/devstorage.full_control'. Adding this scope to the OAuth credential request means the setting public permissions on objects works.
There is a method on GCP's Cloud Storage API that enables the caller to retrieve object metadata. it is documented at https://cloud.google.com/storage/docs/json_api/v1/objects/get
Is there a gsutil equivalent to this method? I've tried gsutil ls -L gs://object however it returns more information than calling the API method does.
Background to my question is that I am implementing a custom role to apply permissions on GCS buckets/objects. In order to test that custom role I am writing a script that carries out all the operations that a member of that custom role will need to be able to do. One of the permissions that the role members will require is storage.objects.get and I basically want to know what gsutil command is enabled by granting someone storage.objects.get. According to https://cloud.google.com/storage/docs/access-control/iam-json, https://cloud.google.com/storage/docs/json_api/v1/objects/get does require storage.objects.get and hence why I'm trying to find the equivalent gsutil command.
If you want to view the metadata associated with an object run:
gsutil stat gs://[BUCKET_NAME]/[OBJECT_NAME]
If you want to retrieve the object itself from the cloud and store it in a local path run:
gsutil cp gs://[BUCKET_NAME]/[OBJECT_NAME] [SAVE_TO_LOCATION]
I'd like to get the current GKE project id from within one of its clusters via the Java client or the GCloud API itself.
I'm running java containers in a GKE cluster of a specific Google Cloud project
I initialize the ClusterManagerClient with the appropriate ClusterManagerSettings
-> Is it possible to fetch this specific project id with this client?
(I'm expecting that there would be a global context within each GKE cluster where we could know the current project we're running on).
Thank you
As John Hanley mentioned in his comment above, you can use the instance metadata on the node in your cluster to determine the project that the node is a part of. The easiest way to see it is to use curl from a shell (either on the node or in a container).
If you want the project name, it can be seen at:
curl "http://metadata.google.internal/computeMetadata/v1/project/project-id" -H "Metadata-Flavor: Google"
And if you want the project number, it can be seen at:
curl "http://metadata.google.internal/computeMetadata/v1/project/numeric-project-id" -H "Metadata-Flavor: Google"
This isn't part of the container API surface, so the ClusterManagerClient isn't the right API client to use. You need to create a client to fetch the instance metadata, which I would expect might be part of the compute client libraries, or you can just make a local HTTP request if you add the right headers (as shown above) since you don't need any special client authentication / authorization to access the local metadata.
I am using the Presto REST api to query the database, but all my sessions appear to be ephemeral. For example, if I do something like
query_presto('set session columnar_processing = true')
query_presto('show session')
Despite setting the column_processing variable in the first query, the second query will show that column_processing is still the default value of false.
I read somewhere that
Presto maintains sessions for each authenticated user. This session
expires if no new requests are received within the timeout period
configured for Presto.
However, I can't find this configuration anywhere in the code nor documentation.
My question is how do I maintain a database session using the RESTful API?
After too much time digging around, I found that there is a header X-Presto-Session which you can set comma separated variables, ie
curl --data "show session" http://127.0.0.1:8889/v1/statement/ --header "X-Presto-User: myuser" --header "X-Presto-Schema: myschema" --header "X-Presto-Time-Zone: UTC" --header "X-Presto-Catalog: mycatalog" --header "User-Agent: myagent" --header "X-Presto-Session: session_variable_1=900,session_variable_2=true"
Despite what the doc says, I don't think there is a way for Presto to remember session variables set in previous executions. I have to locally cache them in my program and pass them all every execution.