I am using Terraform to create a dataproc cluster that uses a GCP cloudsql instance as the hivemetastore, the terrafrm project creates the cluster and all its prerequisites (network, service account, cloudsql instance & user, etc).
cloud-sql-proxy.sh is provided to assist with this however I can't get it to work, when the cluster is created cloud-sql-proxy.sh fails with error:
nc: connect to localhost port 3306 (tcp) failed: Connection refused
I've banged my head against the wall trying to figure out why but can't get to the bottom of it so am hoping someone here can help.
I've hosted the terraform project at https://github.com/jamiekt/democratising-dataproc. Reproducing the problem is very easy, follow these steps:
Install terraform if you haven't already
Install gcloud if you haven't already
Create a new GCP project
Enable the Cloud Dataproc API for your new project
gcloud auth application-default login #creates a file containing credentials that terraform will use
git clone git#github.com:jamiekt/democratising-dataproc.git && cd democratising-dataproc
export GCP_PROJECT=name-of-project-you-just-created
make init
make apply
That should successfully spin up a network, subnetwork, cloudsql instance, a couple of storage buckets (one of them containing cloud-sql-proxy.sh), a service account, a firewall then fail when attempting to create the dataproc cluster.
if anyone could take a look and tell me why this is failing I'd be very grateful.
It seems that you are not using the latest version of cloud-sql-proxy.sh script in the cloud-sql-proxy.sh.tmpl template (diff).
You may want to try to update your template with latest script version from Dataproc Cloud SQL I/O and Hive Metastore initialization action.
There were a number of problems here that have now been solved:
service_account_scopes needed specifying and hive:hive.metastore.warehouse.dir property needed setting
Terraform removes the default root user from MySQL so it has to be recreated. I was missing (amongst other things) the host attribute (host = '%')
various other things
The state of the repo at the time of posting this message will work as intended (i.e. create, using Terraform, a dataproc cluster that uses a shared hive metastore).
Thank you #igor-dvorzhak for your responses, your link to the article on configuring Hive Metastore to use Cloud SQL put me on the right track..
Related
I have a Spring GCP service which when run locally connects fine to my Google Cloud SQL instance.
However, when I deploy and launch on my Google Cloud Kubernetes cluster, it is failing to connect with Insufficient Permissions errors.
I followed the steps https://cloud.google.com/sql/docs/mysql/connect-kubernetes-engine , but still the same connection issue.
My source code is https://github.com/christianblake/spring-boot-gcp
deployment.yml is in the root dir.
Appreciate if somebody has any pointers as I'm obviously missing a point.
Thank you.
Assuming credentials.json is installed correctly, the service account defined in credentials.json needs to have the Cloud SQL Client role. There are several ways to do this is as documented here.
From the cli, you would do something like this:
gcloud projects add-iam-policy-binding $PROJECT_NAME \
--member serviceAccount:$GOOGLE_SERIVICE_ACCOUNT.iam.gserviceaccount.com --role roles/cloudsql.client
#Mangu, I found the following error in the error logs.
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
Which led to the following similar question
Cloud SQL Proxy and Insufficient Permission
I re-created the cluster, including the sql scopes with the following.
gcloud container clusters create cloudcluster --num-nodes 2 --machine-type n1-standard-1 --zone us-central1-c --scopes https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/sqlservice.admin
And that resolved the issue.
Thank you both for the feedback, and apologies for missing the google error code in the original question.
I've created a cluster on Google Kubernetes Engine (previously Google Container Engine) and installed the Google Cloud SDK and the Kubernetes tools with it on my Windows machine.
It worked well for some time, and, out of nowhere, it stopped working. Every command I'm issuing with kubectl provokes the following:
Unable to connect to the server: net/http: TLS handshake timeout
I've searched Google, the Kubernetes Github Issues, Stack Overflow, Server Fault ... without success.
I've tried the following:
Restart my computer
Change wifi connection
Check that I'm not somehow using a proxy
Delete and re-create my cluster
Uninstall the Google Cloud SDK (and kubectl) from my machine and re-install them
Delete my .kube folder (config and cache)
Check my .kube/config
Change my cluster's version (tried 1.8.3-gke.0 and 1.7.8-gke.0)
Retry several hours later
Tried both on PowerShell and cmd.exe
Note that the cluster seem to work perfectly, since I have my application running on it and can interact with it normally through the Google Cloud Shell.
Running:
gcloud container clusters get-credentials cluster-2 --zone europe-west1-b --project ___
kubectl get pods
works on Google Cloud Shell and provokes the TLS handshake timeout on my machine.
For others seeing this issue, there is another cause to consider.
After doing:
gcloud config set project $PROJECT_NAME
gcloud config set container/cluster $CLUSTER_NAME
gcloud config set compute/zone europe-west2
gcloud beta container clusters get-credentials $CLUSTER_NAME --region europe-west2 --project $PROJECT_NAME
I was then seeing:
kubectl cluster-info
Unable to connect to the server: net/http: TLS handshake timeout
I tried everything suggested here and elsewhere. When the above worked without issue from my home desktop, I discovered that shared workspace wifi was disrupting TLS/VPNs to control the internet access!
This is what I did to solve the above problem.
I simply ran the following commands::
> gcloud container clusters get-credentials {cluster_name} --zone {zone_name} --project {project_name}
> gcloud auth application-default login
Replace the placeholders appropriately.
So this MAY NOT work for you on GKE, but Azure AKS (managed Kubernetes) has a similar problem with the same error message so who knows — this might be helpful to someone.
The solution to this for me was to scale the nodes in my Cluster from the Azure Kubernetes service blade web console.
Workaround / Solution
Log into the Azure (or GKE) Console — Kubernetes Service UI.
Scale your cluster up by 1 node.
Wait for scale to complete and attempt to connect (you should be able to).
Scale your cluster back down to the normal size to avoid cost increases.
Total time it took me ~2 mins.
More Background Info on the Issue
Added this to the full ticket description write up that I posted over here (if you want more info have a read):
'Unable to connect Net/http: TLS handshake timeout' — Why can't Kubectl connect to Azure AKS server?
Is it possible to run kubernetes from source (./hack/local-up-cluster.sh) and still properly configure the cloud provider from this type of setup? For example, if an instance is running on AWS EC2 and all prerequisites are met including proper exports, aws cli and configs but keep getting an error stating that the cloud provider was not found. KUBERNETES_PROVIDER=aws, Zone is set to us-west-2a, etc...
Failed to get AWS Cloud Provider. plugin.host.GetCloudProvider returned <nil> instead
I don't think hack/local-up-cluster.sh is designed to be run on a cloud provider. However, cluster/kube-up.sh is designed to work when building from source:
$ make release
$ export KUBERNETES_PROVIDER=aws
$ cluster/kube-up.sh # Uses the release built in step 1
There are lots of options which can be configured, and you can find more details here (just ignore the part about https://get.k8s.io).
I am using Google Cloud / Google Compute to host my application. I was on Google App Engine and I am migrating my code to Google Compute in order to use a customized VM Instance.
I am using the tutorial here, and I am deploying my app using:
$ gcloud preview app deploy
I setup a custom VM Instance using the "Create Instance" option at the top of my Google Cloud Console:
However, when I use the standard deploy gcloud command, my app is deployed to Managed VMs (managed by Google), and I have no control over those servers. I need to run the app on my custom VM because it has some custom OS-level software.
Any ideas on how to deploy the app to my custom VM Instance only? Even when I delete all the Managed VMs and try to deploy, the VMs are just re-created by Google.
The gcloud app deploy command can only be used to deploy the app to classic AppEngine sandboxed environment or to the Managed VMs. It cannot deploy your application to an instance running on GCE.
You will need to incorporate your own deployment method/script depending on the programming language you're using. Of course, since GCE is just an infrastructure-as-a-service environment (versus AppEngine being a platform-as-a-service), you will also need to take care of high-availability (what happens when your instance becomes unavailable?), scalability (what happens when one instance is not enough to sustain the load of your application?), load balancing and many more topics you'll need to address.
Finally, If you need to install packages on your application servers you may consider taking the Managed VMs route. It manages for you all the infrastructure related matters (scalability, elasticity, monitoring etc) and still allows you to have your own custom runtime. It's still beta though...
How to create a simple static Website and deploy it on Google cloud VM instance
Recommended: Docker and Google Cloud SDK should be installed
Step:1
Create a Folder “personal-website” with index.html and frontend files on your local computer
Step:2
Inside “personal-website” folder create a Dockerfile
Write two lines
FROM httpd
COPY . /usr/local/apache2/htdocs/personal-website
Step:3
Build image with docker and push it to Google cloud registry
You should have google cloud sdk and project selected and docker authorized
Select Project using these commands:
gcloud config set project [PROJECT_ID]
gcloud config set compute/zone us-central1-b
After that Run these commands
1. export PROJECT_ID="$(gcloud config get-value project -q)"
2. docker build -t gcr.io/${PROJECT_ID}/personal-website:v1 .
3. gcloud auth configure-docker
4. docker push gcr.io/${PROJECT_ID}/personal-website:v1
Step:4
Create a VM instance with command with container running into it
Run Command
1. gcloud compute instances create-with-container apache-vm2 --container-image gcr.io/test-project-220705/personal-website:v1
In Google Cloud Dataproc how can I access the Spark or Hadoop job history servers? I want to be able to look at my job history details when I run jobs.
To do this, you will need to create an SSH tunnel to the cluster and then use a SOCKS proxy with your browser. This is due to the fact that while the web interfaces are open on the cluster, firewall rules prevent anyone from connecting (for security.)
To access the Spark or Hadoop job history server, you will first need to create an SSH tunnel to the master node of your cluster:
gcloud compute ssh --zone=<master-host-zone> \
--ssh-flag="-D 1080" --ssh-flag="-N" --ssh-flag="-n" <master-host-name>
Once you have the SSH tunnel in place, you need to configure a browser to use a SOCKS proxy. Assuming you're using Chrome and know the path to Chrome on your system, you can launch Chrome with a SOCKS proxy using:
<Google Chrome executable path> \
--proxy-server="socks5://localhost:1080" \
--host-resolver-rules="MAP * 0.0.0.0 , EXCLUDE localhost" \
--user-data-dir=/tmp/
The full details on how to do this can be found here.