Read Gcloud logs programmatically - gcloud

I am trying to understand what is the equivalent REST api for the cli command:
gcloud logging read ....
I need to execute a query on logs programmatically from a client library (Ruby if possible).
I cannot find nothing in the rest api docs.
Any suggestion?

Google's documentation for using client libraries with its services, is mostly very good.
Here's the documentation for Ruby for Cloud Logging.
https://cloud.google.com/logging/docs/reference/libraries#client-libraries-install-ruby
I encourage you to use Google's excellent APIs Explorer when developing using the client libraries too. APIs Explorer helps you craft REST requests and see the responses and this is very helpful when writing code and debugging:
Here's the APIs Explorer page for the Logging service's method to list log entries:
https://cloud.google.com/logging/docs/reference/v2/rest/v2/entries/list
Another very helpful facility is that you can append --log-http to any gcloud command it will show you the underlying REST calls that are being made. This is also helpful in determining how the gcloud commands work so that you may write your own code that replicates|extends the functionality:
gcloud logging read ${FILTER} \
--project=${PROJECT} \
--log-http
And, finally, the Console's Log Explorer is an excellent way to prototype the filters you'll probably need to use in your client code:
https://console.cloud.google.com/logs/query
Hint: when constructing filters, don't forget to escape " characters, for example FILTER="resource.type=\"gce_instance\"". In this case, the value gce_instance must be quoted (") in the filter's value.

Related

How to auth google cloud API from Java in the same way I authenticated with gcloud CLI

Using gcloud command line I can do the following operation
gcloud builds describe 74f859e9-d621-4632-b6dd-XXXXXXXX
However I wish to use the Google Cloud API from Java, now as I understand the GCloud CLI is not using a service account, it is using a user account. How can I use the same authentication from Google Cloud Java API to do this same operation to describe a build?
Google provides decent documentation that explains how to use its SDKs (Client Libraries) with all of its services.
Here's the Cloud Build client libraries documentation. Pick your preferred language and go.
If you can't use one of Google's SDKs, then you can write code directly against the underlying API. Google's APIs Explorer is an excellent tool for navigating all Google's services. Here's Cloud Build and projects.builds.get which I think (!?) maps to gcloud build describe. You can confirm that by running gcloud builds describe --log-http to see which underlying calls are made.
Code that doesn't access user data (data owned by a user account), should run as a Service Account. Code that accesses user data or operates on behalf of a user, should use the OAuth flow for the user and use an OAuth Client ID. This is what gcloud does. As a program operating on behalf of users, it authenticates you the user using a regular OAuth flow but it operates using an OAuth Client ID against a hidden backing project. Your code should probably just run as a service account.

GCP: How to implement "gcloud alpha firestore export ..." shell command through Google's Python API client

I have been running
gcloud alpha firestore export ...
to export certain collections in Firestore to GCS. Now I want to implement this with Google's Python API package, but I can't find good examples to refer to.
Or am I looking at the wrong repo?
There is no a direct function in the client libraries to do so ( Firestore GIT [1] ), but, as there is an API call that do so [2], you can call it using Python and google-api-python-client[3]. Here you can find the method details [4].
Anyway, I filed a Feature Request [5] on your behalf for you to follow it.

Example Amazon S3 Get Bucket Request using Access Key and Secret

I am struggling to understand the documentation on how to make a request to Amazon S3 API's to retrieve a list of Objects.
The documentation doesn't show how to Authorise the request using just the access key and secret. Can someone post an example? preferably something I can use in Postman to test with.
AWS supports two signature versions: Signature Version 4 and Signature Version 2.
You should use Signature Version 4. All AWS services support Signature Version 4, except Amazon SimpleDB which requires Signature Version 2.
All AWS regions support Signature Version 4.
Here is a great article by ƁUKASZ ADAMCZAK explaining how to generate and sign the S3 request using openssl and curl:
Amazon S3 REST API with curl
The real challenge will be to do the sigv4 signing. I truly urge you to use one of the established SDKs (what language are you using??). If not wrap a call to the AWS CLI.
If you really want to implement it yourself the I urge you to open source your efforts so others can benefit.
I would start by setting up the bucket with no auth (so anyone can read from it) and determine how to make a request to S3 first. You can see some raw HTTP Request examples here: https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectGET.html#RESTObjectGET-responses-examples
Then you'll be on to the fun part, sigv4 signing the request. This is a well documented process but still a good amount of effort. https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html

How to get a foundary service whitelist IPs

We have a GUI that manages Cloud Foundry, and there's a link that show an instance with IP white list external dependency (quite large) How can I easily re-create this config as JSON, and recreate to diff Foundry env ?
It's not entirely clear what is being presented in your GUI but it sounds like it might be the application security groups. You might try running cf security-groups or cf security-group <name> to see if this information matches up with what's displayed in the GUI.
If that's what you want, you can use the following API calls to obtain the JSON data & recreate it in another environment.
1.) List all the security groups: http://apidocs.cloudfoundry.org/1.40.0/security_groups/list_all_security_groups.html
2.) List security groups applied to all applications: http://apidocs.cloudfoundry.org/1.40.0/security_group_running_defaults/return_the_security_groups_used_for_running_apps.html
3.) List security groups applied to all staging containers: http://apidocs.cloudfoundry.org/1.40.0/security_group_staging_defaults/return_the_security_groups_used_for_staging.html
4.) Retrieve a particular security group: http://apidocs.cloudfoundry.org/1.40.0/security_groups/retrieve_a_particular_security_group.html
And you can find more details about the API calls here: http://apidocs.cloudfoundry.org/
You can also run the cf cli commands above with the -v flag to show the HTTP requests being made by the CLI to obtain the information that's displayed.
Hope that helps!

Reading/writing to Google Storage from Google Compute Windows 2008 VM

I have a Google Compute VM (LAMP) webserver set up to copy files to a Google Storage Bucket, which then need to be accessed (read and write) by a program on a Google Compute VM (Windows 2008). I can't seem to find any documentation about how a Google Compute Engine Windows VM can access storage buckets.
Is there a way this is possible? Thanks.
I'm doing the same thing, but not with a windows VM, but I think the principle is the same.
First you need to allow Project Access for your VM from the Google Cloud Console https://console.developers.google.com/project, see the screenshot below:
Once you've done this you need to call the metadata server to get an access token from your program. You need to make a HTTP call to the metadata server, here is an example from the docs (https://cloud.google.com/compute/docs/authentication) using curl, bear in mind when programming this you need to also provide the header "Metadata-Flavor: Google":
$ curl "http://metadata/computeMetadata/v1/instance/service-accounts/default/token" \
-H "Metadata-Flavor: Google"
{
"access_token":"ya29.AHES6ZRN3-HlhAPya30GnW_bHSb_QtAS08i85nHq39HE3C2LTrCARA",
"expires_in":3599,
"token_type":"Bearer"
}
You obviously need to code this HTTP call and the parsing of the JSON data in whichever programming language you are using for your program and extract the "access_token", based on the "expires_in" field you might also need to implement a mechanism to fetch a new token once it expires. You can then use the Google supplied cloud storage client library (https://cloud.google.com/storage/docs/json_api/v1/libraries) for your programming language and use the access token above for authenticating calls to cloud storage. I use Java and the Cloud storage class in the API library has this method that can be used:
.setOauthToken("blah")
You can mount the drive with CloudBerry. I would like to find a better way to do it though using only Google Cloud. Please let me know if you find anything better.