When trying to create a new cluster with gcloud dataproc clusters create, the following error is displayed:
ERROR: gcloud failed to load (gcloud.dataproc.clusters.create): Problem loading gcloud.dataproc.clusters.create: No module named jsonschema.
This usually indicates corruption in your gcloud installation or problems with your Python interpreter.
Please verify that the following is the path to a working Python 2.7 executable:
/usr/bin/python2
If it is not, please set the CLOUDSDK_PYTHON environment variable to point to a working Python 2.7 executable.
If you are still experiencing problems, please run the following command to reinstall:
$ gcloud components reinstall
If that command fails, please reinstall the Cloud SDK using the instructions here:
https://cloud.google.com/sdk/
Installing jsonschema does not seem to help.
This was an issue with gcloud sdk release 208.0.0. Upgrading to 208.0.1 should resolve this issue.
Related
Error on Terminal while running it locally
I am getting this error while running it locally. I have created environment using conda.Python=3.8 and Pycaret =2.3.5
Please help me to resolve it.
I created new enviornment to run it and it still didn't work.
Based on that error message, seems like you don't have pycaret properly installed. Do conda list to see what packages are installed.
I ran the gcloud components update on ubuntu. After updating I received the following warning
WARNING: There are other instances of Google Cloud Platform tools on your system PATH.
Please remove the following to avoid confusion or accidental invocation:
/usr/bin/gsutil
What will happen if I do not remove it?
How should I remove it
I am running Pycharm on my MacBook.
Client settings:
Python Interpreter -> Python 3.7 (dtabricks-connect-6.4)
Cluster settings:
Databricks Runtime Version -> 6.4 (includes Apache Spark 2.4.5, Scala 2.11)
It worked well for months but suddenly, without any updates made, I cant run my python script from Pycharm against databricks cluster anymore.
The Error is ...
Caused by: `java.lang.IllegalArgumentException: The cluster is running server version `dbr-6.4` but this client only supports Set(dbr-5.5)....`
I restarted Pycharm, I witched back and for the interpreter, I restarted the cluster and I even restarted my MacBook but it didnt helped. The Error message is simply false, because both, cluster and client are using the SAME version. I can see that when I execute my python script that the cluster is being started but fails at the end.
pyenv activate databricks-connect-6-4
pip freeze
Cython==0.29.21
databricks-connect==6.4.0
numpy==1.19.2
pandas==1.0.1
py4j==0.10.7
pyarrow==0.13.0
pycountry==20.7.3
python-dateutil==2.8.1
pytz==2020.1
six==1.15.0
It looks like that it was caused by some internal changes on the server side, and this prevents databricks-connect from working. You can always disable this check by setting environment variable DEBUG_IGNORE_VERSION_MISMATCH to value 1 (export DEBUG_IGNORE_VERSION_MISMATCH=1 in the console before executing databricks-connect test) - you can also set this environment variable in the PyCharm.
Update: it should be fixed by Databricks Connect 6.4.2 that was just released.
Thanks for raising this. The Databricks-Connect team has acknowledged this issue and we are working on a patch to address this issue. Will keep you posted. In the meantime you can use the DEBUG_IGNORE_VERSION_MISMATCH as Alex pointed out.
Update: A compatible db-connect client has been released to fix this problem: version 6.4.2 (https://pypi.org/project/databricks-connect/6.4.2/, install with: pip install databricks-connect==6.4.2).
Using node v 8.9.0 and this tutorial
When I try and debug my http google cloud function in dev tools:
C:_Users_Matt_AppData_Roaming_nvm_v8.9.0_node_modules_#google-cloud_functions-emulator_src_supervis
I get filesystem permission denied error, how can I debug my cloud functions:
I also got the filesystem permission denied error, and the issue was that you need to accept the premissions from chrome to be able to access that filesystem. initially I didn't see the premissions prompt, but then I found it on a different tab (which was kinda weird behavior). I guess just look for that permissions prompt, it should be right below your address bar.
I see that you are referring to a C directory, which means that you are trying this on Windows OS. I will put the steps below with documentation links on how to properly setup the configuration. Those steps worked for me well without giving me any issues, so I suggest you to follow them one by one and see if that helps you.
Run Google Cloud Functions Emulator on Windows OS:
Install and set up Google Cloud SDK for Windows. Link and documentation here
Install Node.js and npm for Windows. Tutorial here
Right click on Google Cloud SDK Shell and select Run as administrator.
Execute $ node --version you should get the version of Node.js without any additional errors
Execute $ npm --version you should get the version of npm without any additional errors
The tutorial that you are referring to is part of Google Cloud Functions Tutorial Series
First install and set up npm functions emulator by running $ npm install -g #google-cloud/functions-emulator as mentioned in Google Cloud Functions Tutorial : Setting up a Local Development Environment
Setup the project for the functions $ functions config set projectId PROJECT_ID as mentioned in Start and Stop the Emulator documentation.
Start the emulator by executing $ functions start. Same documentation as above.
Download the source code as mentioned in the documentation you are referring to. The GitHub repository is here.
Clone the project locally. $ git clone https://github.com/rominirani/googlecloudfunctions-training.git
Navigate to the folder $ cd googlecloudfunctions-training/helloworld-http
Follow the rest of the Google Cloud Functions Tutorial : Debugging Local Functions documentation.
NOTE: Whenever you run / execute / call the Cloud Function the Node.js
blank window will pop up. Keep it open as it is the executable that
executes your code.
I have tested the tutorial with the above set up that I described and it worked for me. You have to be the administrator of your account, since the Functions Emulator and the code is running locally, so you have to have all the permissions of the directories that are going to be used and execute all the software as administrator.
I have installed gcloud client library but GCP keeps on giving me this error what could the issue
ERROR: (gcloud.sql.connect) Mysql client not found. Please install a mysql client and make sure it is in PATH to be able to connect to the database instance
The problem is that either you have not installed the MySQL client locally, or gcloud can't find it.
If you have not installed MySQL yet, follow the instructions to do so.
If you have installed MySQL but you're still getting this error, try entering mysql at the shell prompt. If you get a command not found error, MySQL is missing from your $PATH, and you need to figure out where it has been installed.
For example, on OS X, MySQL is by default installed in the directory /usr/local/mysql/bin/. Add this to your $PATH, and it should start working:
export PATH=$PATH:/usr/local/mysql/bin/
We get this error when mysql client is not installed in your machine.
Refer https://www.youtube.com/watch?v=OM4aZJW_Ojs&t=150s&ab_channel=AmitThinks video to install my-sql in windows.
Post installation of my-sql add the path C:\Program Files\MySQL\MySQL Server 8.0\bin to PATH environment variable.
Close and then open your Google Cloud SDK shell and run the command again.
This will solve the problem.