The IBM Analytics Engine docs have the following instructions for getting the name node hostname:
Go to Manage Cluster in IBM® Cloud and click the nodes tab to get the name node host name. It's the host name of the management-slave1 node type.
How can I programmatically retrieve the name node host name? Can I retrieve it via an API, or maybe I can get it by running a command over ssh. Failing that, can I derive it from one of the host names on vcap services?
Maybe this information should be provided to users in the vcap info?
In the end, I solved this using ambari. The solution that worked for me is captured here: https://stackoverflow.com/a/47844056/1033422
Related
This is driving me crazy, been trying to get this to work for 3 days now: I'm trying to connect a kubernetes deployment to my Cloud SQL database in GCP.
Here's what I've done so far:
Set up the cloud SQL proxy to work as a sidecar in my deployment
Created a GKE service account and attached it to my deployment
Bound the GKE service account to my GCP service account
Edited to the service account (to what I can tell) is owner permission
Yet what I run the deployment in GKE I still get:
the default Compute Engine service account is not configured with sufficient permissions to access the Cloud SQL API from this VM. Please create a new VM with Cloud SQL access (scope) enabled under "Identity and API access". Alternatively, create a new "service account key" and specify it using the -credential_file parameter
How can I fix this? I can't find any documentation on how to set up the service account to have the correct permissions with Cloud SQL or how to debug this issue. Every single tutorial I can find ends with "bind your service account" and then stops. Nothing that describes what permissions are needed, and nothing about how to actually connect to the DB from my code (how would my code talk to the proxy?).
Please help
FINALLY got it to work!
Two major pieces that the main article on this (cloud.google.com/sql/docs/mysql/connect-kubernetes-engine) glosses over:
Properly setting up workload identity, for which I found these links to be very helpful:
a) https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity
b) https://www.youtube.com/watch?v=l-nws1e4B8M
To connect to the DB you have to have your code use the DB host 127.0.0.1
I'd like to get the current GKE project id from within one of its clusters via the Java client or the GCloud API itself.
I'm running java containers in a GKE cluster of a specific Google Cloud project
I initialize the ClusterManagerClient with the appropriate ClusterManagerSettings
-> Is it possible to fetch this specific project id with this client?
(I'm expecting that there would be a global context within each GKE cluster where we could know the current project we're running on).
Thank you
As John Hanley mentioned in his comment above, you can use the instance metadata on the node in your cluster to determine the project that the node is a part of. The easiest way to see it is to use curl from a shell (either on the node or in a container).
If you want the project name, it can be seen at:
curl "http://metadata.google.internal/computeMetadata/v1/project/project-id" -H "Metadata-Flavor: Google"
And if you want the project number, it can be seen at:
curl "http://metadata.google.internal/computeMetadata/v1/project/numeric-project-id" -H "Metadata-Flavor: Google"
This isn't part of the container API surface, so the ClusterManagerClient isn't the right API client to use. You need to create a client to fetch the instance metadata, which I would expect might be part of the compute client libraries, or you can just make a local HTTP request if you add the right headers (as shown above) since you don't need any special client authentication / authorization to access the local metadata.
I am newbie at cloud servers and I've opened a google cloud storage to host image files. I've verified my domain and configured it, to view images via my domain. The problem is, same file is both accessible via my domain example.com/images/tiny.png and also via storage.googleapis.com/example.com/images/tiny.png Is there any solution to disable access via storage.googleapis.com and use only my domain?
Google Cloud Platform Support Version:
NOTE: This is the reply from Google Cloud Platform Support when contacted via email...
I understand that you have set up a domain name for one of your Cloud Storage buckets and you want to make sure only URLs starting with your domain name have access to this bucket.
I am afraid that this is not possible because of how Cloud Storage permission works.
Making a Cloud Storage bucket publicly readable also gives each of its files a public link. And currently this public link can’t be disabled.
A workaround would be implement a proxy program and running it on a Compute Engine virtual machine. This VM will need a static external IP so that you can map your domain to it. The proxy program will be in charged of returning the requested file from a predefined Cloud Storage bucket while the bucket keeps to be inaccessible to the public.
You may find these documents helpful if you are interested in this workaround:
1. Quick start to set up a Linux VM (1).
2. Python API for accessing Cloud Storage files (2).
3. How to download service account keys to grant a program access to a set of services (3).
4. Pricing calculator for getting a picture on how much a VM may cost (4).
(1) https://cloud.google.com/compute/docs/quickstart-linux
(2) https://pypi.org/project/google-cloud-storage/
(3) https://cloud.google.com/iam/docs/creating-managing-service-account-keys
(4) https://cloud.google.com/products/calculator/
My Version:
It seems the solution to this question is really a simple, just FUSE Google Cloud Storage with VM Instance.
After FUSE private files from GCS can be accessed through VM's IP address. It made Google Cloud Storage Bucket act like a directory.
The detailed documentation about how to setup FUSE in Google Cloud is here.
There is but it requires you to do more work.
Your current solution works because you've made access to the GCS bucket (example.com), public and then you're DNS aliasing from your domain.
An alternative approach would be for you to limit access to the GCS bucket to one (possibly several) accounts and then run a web-server that uses one of the accounts to access your image files. You could then also either permit access to your web-server to anyone or also limit access to it.
More work for you (and possibly cost) but more control.
Detect Google Cloud Project Id from a container in Google hosted Kubernetes cluster.
When connecting to BigTable; I need to provide the Google Project Id. Is there a way to detect this automatically from within K8s?
In Python, you can find the project id this way:
import google.auth
_, PROJECT_ID = google.auth.default()
The original question didn't mention what programming language was being used, and I had the same question for Python.
You can use the metadata service. Example:
curl -H "Metadata-Flavor: Google" -w '\n' http://metadata.google.internal/computeMetadata/v1/project/numeric-project-id
This will work from any VM running on Google Compute Engine or Container Engine.
See https://cloud.google.com/compute/docs/storing-retrieving-metadata:
Google Compute Engine defines a set of default metadata entries that provide information about your instance or project. Default metadata is always defined and set by the server.
...
numeric-project-id The numeric project ID of the instance, which is not the same as the project name visible in the Google Cloud Platform Console. This value is different from the project-id metadata entry value.
project-id The project ID.
Google has some libraries for this too: ServiceOptions.getDefaultProjectId
https://googleapis.github.io/google-cloud-java/google-cloud-clients/index.html
https://github.com/googleapis/google-cloud-java/blob/master/google-cloud-clients/google-cloud-core/src/main/java/com/google/cloud/ServiceOptions.java
https://github.com/googleapis/google-cloud-java/tree/master/google-cloud-clients/google-cloud-core
I created an instance and i connected successfully with a public external IP. I wish to know is there a way to assign authorized network with domain name instead of an IP address? I wish to use domain name with no-ip as public IP will changed after router restarted. This makes troublesome because i need to change the authorized network if there is changes in my public IP.
Currently there is no way to use a name instead of an IP. Note that Cloud SQL API [1] allows updating the list of authorized networks. The gcloud command line tool from Cloud SDK [2] has support for that.
$ gcloud sql instances patch -h
usage: gcloud sql instances patch
[optional flags] INSTANCE
Updates the settings of a Cloud SQL instance.
optional flags:
[...]
--authorized-networks AUTHORIZED_NETWORKS
The list of external networks that are allowed to
connect to the instance. Specified in CIDR notation,
also known as 'slash' notation (e.g. 192.168.100.0/24).
[...]
positional arguments:
INSTANCE Cloud SQL instance ID.
[1] https://developers.google.com/cloud-sql/docs/admin-api/v1beta3/instances
[2] https://developers.google.com/cloud-sql/docs/cloud-sdk