I am trying to create a CDK Code Construct for my python scripts, in stack i have added s3 and lambda.
When I am trying to execute cdk deploy, it is exiting after 0% progress or it is giving following error.
When i tried for s3 only it is working fine but when i added the lambda it is giving me error.
file_feed_lambda = _lambda.Function(
self, id='MyLambdaHandler001',
runtime=_lambda.Runtime.PYTHON_3_7,
code=_lambda.Code.asset('lambda'),
handler='lambda_function.lambda_handler',
)
bucket = s3.Bucket(self,
"FeedBucket-01")
Note : cdk diff and cdk synth are working properly
Apparently error which is showing is wrong i have updated the version of node and cdk to latest.
After update I have received the meaningful error which was socket time out.
After setting the proxy it worked for me.
Related
My colleagues and I are facing an issue when trying to run my databricks notebook in Azure Data Factory and the error is coming from MLFlow.
The command that is failing is the following:
# Take the parent notebook path to use as path for the experiment
context = json.loads(dbutils.notebook.entry_point.getDbutils().notebook().getContext().toJson())
nb_base_path = context['extraContext']['notebook_path'][:-len("00_training_and_validation")]
experiment_path = nb_base_path + 'trainings'
mlflow.set_experiment(experiment_path)
experiment = mlflow.get_experiment_by_name(experiment_path)
experiment_id = experiment.experiment_id
run = mlflow.start_run(experiment_id=experiment_id, run_name=f"run_{datetime.now().strftime('%Y-%m-%d_%H-%M-%S')}")
And the error that is throwing is:
An exception was thrown from a UDF: 'mlflow.exceptions.RestException: INVALID_PARAMETER_VALUE: No experiment ID was specified. An experiment ID must be specified in Databricks Jobs and when logging to the MLflow server from outside the Databricks workspace. If using the Python fluent API, you can set an active experiment under which to create runs by calling mlflow.set_experiment("/path/to/experiment/in/workspace") at the start of your program.', from , line 32.
The pipeline just runs the notebook from ADF, it does not have any other step and the cluster we are using is type 7.3 ML.
Could you please help us?
Thank you in advance!
I think you need to set artifact URI and specify experiment ID (if in the artifact directory has much experiment ID
Reference: https://www.mlflow.org/docs/latest/tracking.html#how-runs-and-artifacts-are-recorded
I am working in a GCP lab (Securing Google Cloud with CFT Scorecard). All instructions for the lab are given.
First I have to run the following two commands to set environment variables
export GOOGLE_PROJECT=$DEVSHELL_PROJECT_ID
export CAI_BUCKET_NAME=cai-$GOOGLE_PROJECT
In the second command given above I don't know what to replace with my own credentials? May be that is the reason I am getting error.
Now I have to enable the "cloudasset.googleapis.com" gcloud service. For this they gave the following command.
gcloud services enable cloudasset.googleapis.com \
--project $GOOGLE_PROJECT
Error for this is given in the screeshot attached herewith:
Error in the serviec enabling command
Next step is to clone the policy: The given command for that is:
git clone https://github.com/forseti-security/policy-library.git
After that they said: "You realize Policy Library enforces policies that are located in the policy-library/policies/constraints folder, in which case you can copy a sample policy from the samples directory into the constraints directory".
and gave this command:
cp policy-library/samples/storage_blacklist_public.yaml policy-library/policies/constraints/
On running this command I received this:
error on running the directory command
Finally they said "Create the bucket that will hold the data that Cloud Asset Inventory (CAI) will export" and gave the following command:
gsutil mb -l us-central1 -p $GOOGLE_PROJECT gs://$CAI_BUCKET_NAME
I am confused in where to replace my own credentials like in the place of project_Id I wrote my own project id.
Also I don't know these errors are ocurring. Kindly help me.
I'm unable to access the tutorial.
What happens if you run the following:
echo ${DEVSHELL_PROJECT_ID}
I suspect you'll get an empty result because I think this environment variable isn't actually set.
I think it should be:
echo ${DEVSHELL_GCLOUD_CONFIG}
Does that return a result?
If so, perhaps try using that variable instead:
export GOOGLE_PROJECT=${DEVSHELL_GCLOUD_CONFIG}
export CAI_BUCKET_NAME=cai-${GOOGLE_PROJECT}
It's not entirely clear to me why this tutorial is using this approach but, if the above works, it may get you further along.
We're you asked to create a Google Cloud Platform project?
As per the shared error, this seems to be because your env variable GOOGLE_PROJECT is not set. You can verify it by using echo $GOOGLE_PROJECT and seeing whether it returns the project ID or not. You could also use echo $DEVSHELL_PROJECT_ID. If that returns the project ID and the former doesn't, it means that you didn't export the variable as stated at the beginning.
If the problem is that GOOGLE_PROJECT doesn't have any value, there are different approaches on how to solve it.
Set the env variable as you explained at the beginning. Obviously this will only work if the variable DEVSHELL_PROJECT_ID is also set.
export GOOGLE_PROJECT=$DEVSHELL_PROJECT_ID
Manually set the project ID into that variable. This is far from ideal because in Qwiklabs they create a new temporal project on every lab, so this would've only worked if you were still on that project. The project ID can be seen on both of your shared screenshots.
export GOOGLE_PROJECT=qwiklabs-gcp-03-c6e1787dc09e
Avoid using the argument --project. According to the documentation, the aforementioned argument is optional and if none is used the command will take the one by default, which will be on the configuration settings. You can get the current project by using this:
gcloud config get-value project
If the previous command matches the project ID you want to use, you can simply issue the following command:
gcloud services enable cloudasset.googleapis.com
Notice that the project ID is not being explicitly mentioned using --project.
Regarding your issue with the GitHub file, I have checked the repository and the file storage_blacklist_public.yaml doesn't seem to be in the directory policy-library/samples. There seems to be a trace that it was once there, but it isn't anymore, they should probably update the lab as it isn't anymore.
About your credentials confusion, you don't have to use your own project ID, just the one given on your lab. If I recall properly all the needed data should be on the left side of the lab. Still, you shouldn't need to authenticate in a normal situation as you are already logged in your temporal project if you are accessing it form the Cloud Shell, which is where you should be doing all this.
Adding this for the later versions
in the gcloud shell you can set a temp variable for the current project id with
PROJECT_ID="$(gcloud config get-value project)"
then use like
--project ${PROJECT_ID}
I would like to import a csv file from my Google Cloud Storage bucket into H2O running in R locally (h2o.init(ip = "localhost")).
I tried following the instructions at http://docs.h2o.ai/h2o/latest-stable/h2o-docs/cloud-integration/gcs.html?highlight=environment.
I can already upload csv files from R to GCS and vice-versa using the R package cloudml. So I am reasonably sure I have the authorizations set correctly.
I have tried using Sys.setenv(GOOGLE_APPLICATION_CREDENTIALS = "/full/path/to/auth.json"). I tried using the terminal from Rstudio to do the same thing: export GOOGLE_APPLICATION_CREDENTIALS="/full/path/to/auth.json". I also tried gcloud auth application-default login using the terminal from Rstudio.
But in every case, I could not make this work, from within Rstudio:
h2o.init()
h2o::h2o.importFile(path = "gs://[gcs_bucket]/[tbl.csv],
destination_frame = "tbl_from_gcs")
H2O throws the error:
Error in h2o.importFolder(path, pattern = "", destination_frame = destination_frame, :
all files failed to import
If I turn on logging (h2o::h2o.startLogging("logfile")), it shows:
GET http://localhost:54321/3/ImportFiles?path=gs%3A%2F%2F[gcs_bucket]%2F[tbl.csv]&pattern=
postBody:
curlError: FALSE
curlErrorMessage:
httpStatusCode: 200
httpStatusMessage: OK
millis: 182
{"__meta":{"schema_version":3,"schema_name":"ImportFilesV3","schema_type":"ImportFiles"},"_exclude_fields":"","path":"gs://[gcs_bucket]/[tbl.csv]","pattern":"","files":[],"destination_frames":[],"fails":["gs://[gcs_bucket]/[tbl.csv]"],"dels":[]}
(Obviously, I changed the bucket name and table name, but hopefully you get the idea.)
I am running h2o version 3.26.0.2 in R 3.6.1 and Rstudio 1.2.1578. (I am running Rstudio server in Docker on my local server using rocker/tidyverse:latest, FYI.)
If anyone could walk me through the steps to authenticate H2O so it can access GCS buckets directly, I would appreciate it. I know I could use cloudml or googleCloudStorageR as a workaround, but I would like to be able to use H2O directly so I can more easily switch from a local H2O cluster to a cloud H2O cluster.
I found one solution to this authentication issue: Because I am running h2o in Docker swarm, I can set an environment variable for the container in Docker Compose.
The relevant parts of the docker compose file look like this:
environment:
- GOOGLE_APPLICATION_CREDENTIALS=/run/secrets/google_auth_secret
secrets:
- google_auth_secret
...
secrets:
google_auth_secret:
file: ./gcloud_auth.json
Where gcloud_auth.json is the credentials file described here for your GCS bucket.
I am running a cloud build trigger on a cloudbuid.yaml file in which I build a docker container and then deploy it to cloud run. The error stacktrace is as follows:
API [sql-component.googleapis.com] not enabled on project
The problem is that I have enabled both SQL and SQL Admin APIs in both projects (one for the cloud build and one for the database), which was confirmed in the console and in gcloud.
Here is the yaml code for the step I am referring to:
- name: 'gcr.io/cloud-builders/gcloud'
args: [
'beta',
'run',
'deploy',
'MY_NAME',
'--image', 'gcr.io/MY_PROJECT/MY_IMAGE',
'--region', 'MY_REGION',
'--platform', 'managed',
'--set-cloudsql-instances', 'MY_CONNECTION_NAME',
'--set-env-vars', 'NODE_ENV=production,INSTANCE_CONNECTION_NAME=MY_CONNECTION_NAME,SQL_USER=MY_USER,SQL_PASSWORD=MY_PASSWORD,SQL_NAME=MY_SCHEMA,TOPIC_NAME=MY_TOPIC'
]
Any suggestions?
Thanks.
P.S.: As per Eespinola suggestion, I checked and confirmed I am running Google Cloud SDK 254.0.0.
P.S. 2: I have also tried to create a project from scratch but ended up with the same results.
Ok so as per the same thread eespinola posted (see above), the Cloud Build gcloud step will be updated according to Cloud SDK 254.0.0 update in a near future (the actual date may or may not be posted in the same thread in the future). Until then, the alternative is to use the YAML file without the --add-cloudsql-instances flag and add it manually in the UI (I still have not tried this but it should work as per Google's development team).
I am new to the Google Vision API and I would like to conduct a label detection of approx. 10 images and I would like to run the vision quickstart.py file. However when I do this with only 3 images then it is successful. With more than 3 images I am getting the error message below. I know that I would need to change something at my setup, but I do not know what I should change.
Here is my error message:
google.gax.errors.RetryError: GaxError(Exception occurred in retry method
that was not classified as transient, caused by <_Rendezvous of RPC that
terminated with (StatusCode.RESOURCE_EXHAUSTED, Insufficient tokens for
quota 'DefaultGroup' and limit 'USER-100s' of service
'vision.googleapis.com' for consumer 'project_number: XXX'.)>)
Does anybody know what I need to do?
Any help would be much appreciated
Cheers,
Andi
I ran into the same problem and fixed it with these steps:
Make sure you have the Google Cloud SDK properly installed: https://cloud.google.com/vision/docs/reference/libraries
Setup a Service Account in the Google Cloud backend: https://developers.google.com/identity/protocols/OAuth2ServiceAccount#creatinganaccount
Create a Service Account Key and download it as a JSON file to a local folder. You need to keep the key private.
Export the filepath to the key-file as an environment variable: gcloud auth activate-service-account --key-file path/to/your/keyfile/here
Log out/in of the console.
Make sure, the environment variable is properly set with printenv
Try your py-script again...
Good luck...
Edit: In addition to the mentioned steps 1.-3. you can just do vision_client = vision.Client.from_service_account_json('/path/to/your/keyfile.json') in your script. No need for the env variable then.