I have a few AWS Elastic Clusters and I am trying to import them to Pulumi. The issue is that pulumi import returned error for most of my clusters. Only a few of them were able to be detected by pulumi.
Here is the example
admin-gateway and infra-storage-thai-test-v2 are both in same aws account and same region (us-west-2). Pulumi was able to generate code for admin-gateway:
thaipham#N9GK65GXW1 quickstart % pulumi import aws:elasticache/cluster:Cluster admin-gateway admin-gateway
Previewing import (dev)
View Live: https://app.pulumi.com/thailowki/quickstart/dev/previews/2189eee2-316c-4e2d-a50f-ab03a68c4194
Type Name Plan
+ pulumi:pulumi:Stack quickstart-dev create
= └─ aws:elasticache:Cluster admin-gateway import
Resources:
+ 1 to create
= 1 to import
2 changes
Do you want to perform this import? details
Then for infra-storage-thai-test-v2
thaipham#N9GK65GXW1 quickstart % pulumi import aws:elasticache/cluster:Cluster infra-storage-thai-test-v2 infra-storage-thai-test-v2
Previewing import (dev)
View Live: https://app.pulumi.com/thailowki/quickstart/dev/previews/a5490dc5-985d-49d7-ad67-928e331dc742
Type Name Plan Info
+ pulumi:pulumi:Stack quickstart-dev create 1 error
= └─ aws:elasticache:Cluster infra-storage-thai-test-v2 import 1 error
Diagnostics:
pulumi:pulumi:Stack (quickstart-dev):
error: preview failed
aws:elasticache:Cluster (infra-storage-thai-test-v2):
error: Preview failed: resource 'infra-storage-thai-test-v2' does not exist
I tried to use --logflow enabled and still there were not good insights.
here is some of the log:
I0901 10:21:47.233425 22540 eventsink.go:59] [aws-sdk-go] DEBUG: Validate Response elasticache/DescribeCacheClusters failed, attempt 0/25, error CacheClusterNotFound: CacheCluster not found: infra-storage-thai-test-v2
I0901 10:21:47.233438 22540 eventsink.go:62] eventSink::Debug(<{%reset%}>[aws-sdk-go] DEBUG: Validate Response elasticache/DescribeCacheClusters failed, attempt 0/25, error CacheClusterNotFound: CacheCluster not found: infra-storage-thai-test-v2<{%reset%}>)
I0901 10:21:47.233675 22540 eventsink.go:59] status code: 404, request id:
Any help would be greatly appreciated.
PS: I'm using pulumi v3.39.1
Related
I try to execute the following command line:
mssparkutils.fs.ls("abfss://mycontainer#myadfs.dfs.core.windows.net/myfolder/")
I get the error:
Py4JJavaError: An error occurred while calling z:mssparkutils.fs.ls.
: java.nio.file.AccessDeniedException: Operation failed: "This request is not authorized to perform this operation.", 403, GET, https://myadfs.dfs.core.windows.net/mycontainer?upn=false&resource=filesystem&maxResults=5000&directory=myfolder&timeout=90&recursive=false, AuthorizationFailure, "This request is not authorized to perform this operation.
I followed the steps described in this link
by granting access to me and my Synapse workspace the role of "Storage Blob Data Contributor" in the container or file system level:
Even that, I still get this persistent error. Am I missing other steps?
I got the same kind of error in my environment. I just followed this official document and done the repro, now it's working fine for me. You can follow the below code it will solve your problem.
Sample code:
from pyspark.sql import SparkSession
account_name = 'your_blob_name'
container_name = 'your_container_name'
relative_path = 'your_folder path'
linked_service_name = 'Your_linked_service_name'
sas_token = mssparkutils.credentials.getConnectionStringOrCreds(linked_service_name)
Access to Blob Storage
path = 'wasbs://%s#%s.blob.core.windows.net/%s' % (container_name,account_name,relative_path)
spark.conf.set('fs.azure.sas.%s.%s.blob.core.windows.net' % (container_name,account_name),sas_token)
print('Remote blob path: ' + path)
Sample output:
Updated answer
Reference to configure Spark in pyspark notebook:
https://techcommunity.microsoft.com/t5/azure-synapse-analytics-blog/notebook-this-request-is-not-authorized-to-perform-this/ba-p/1712566
%scala
import io.sentry.{Sentry, SentryOptions}
import org.apache.spark.SparkConf
import org.apache.spark.sql.SparkSession
val sentryOptions = new SentryOptions()
sentryOptions.setDsn(sys.env.getOrElse("SENTRY_DSN", "REDACTED"))
sentryOptions.setEnvironment(sys.env.getOrElse("SENTRY_ENVIRONMENT", "dev"))
sentryOptions.setRelease(sys.env.getOrElse("SENTRY_RELEASE", "dev"))
sentryOptions.setDebug(true)
Sentry.init(sentryOptions)
Sentry.captureException(new Throwable())
Sentry.flush(2000L)
running above piece if code in the databricks notebook logs this error below. could't figure out the root cause tried. nothing much is there as part of the integration but really got bugged due to unlimited failed attempts :
INFO: Initializing SDK with DSN: 'REDACTED'
INFO: No outbox dir path is defined in options.
INFO: GlobalHubMode: 'false'
DEBUG: UncaughtExceptionHandlerIntegration removed.
INFO: Closing SentryClient.
DEBUG: Shutting down
DEBUG: UncaughtExceptionHandlerIntegration enabled: true
DEBUG: default UncaughtExceptionHandler class='com.databricks.DatabricksMain$DBUncaughtExceptionHandler'
DEBUG: UncaughtExceptionHandlerIntegration installed.
DEBUG: Capturing event: bdf91ad1b3e1451093b72970aab03bfd
INFO: Session is null on scope.withSession
ERROR: Request failed, API returned 400
ERROR: {"detail":"empty envelope"}
import io.sentry.{Sentry, SentryOptions}
import org.apache.spark.SparkConf
import org.apache.spark.sql.SparkSession
sentryOptions: io.sentry.SentryOptions = io.sentry.SentryOptions#5e0d7239
I'm trying to import a google_storage_bucket storage bucket in my Terraform state:
terraform import module.bf-nathan.google_storage_bucket.assets-bucket my-bucket
However, it fails as follows:
module.bf-nathan.google_storage_bucket.assets-bucket: Importing from ID "my-bucket"...
module.bf-nathan.google_storage_bucket.assets-bucket: Import complete! Imported google_storage_bucket (ID: next-assets-bf-nathan-botfront-cloud)
module.bf-nathan.google_storage_bucket.assets-bucket: Refreshing state... (ID: next-assets-bf-nathan-botfront-cloud)
Error: module.bf-nathan.provider.kubernetes: 1:11: unknown variable accessed: var.cluster_ip in:
https://${var.cluster_ip}
The refreshing step doesn't work. I ran the command from the project's root where a terraform.tfvars file exists.
I tried adding -var-file=terraform.tfvars but no luck.
Note that the variables are correctly interpolated with all other terraform commands.
It's worth noting that the bucket in question is defined in a module and not in the main.tf. Here is how the bucket is declared:
resource "google_storage_bucket" "assets-bucket" {
name = "${local.assets_bucket_name}"
storage_class = "MULTI_REGIONAL"
force_destroy = true
}
```
Using Terraform 11.14
My terraform file contains the following resource:
resource "google_storage_bucket" "assets-bucket" {
name = "${local.assets_bucket_name}"
storage_class = "MULTI_REGIONAL"
force_destroy = true
}
And this bucket has already been created (it exists on the infrastructure based on a previous apply)
However the state (remote on gcs) is inconsistent and doesn't seem to include this bucket.
As a result, terraform apply fails with the following error:
google_storage_bucket.assets-bucket: googleapi: Error 409: You already own this bucket. Please select another name., conflict
How can I reconcile the state? (terraform refresh doesn't help)
EDIT
Following #ydaetskcoR's response, I did:
terraform import module.bf-nathan.google_storage_bucket.assets-bucket my-bucket
The output:
module.bf-nathan.google_storage_bucket.assets-bucket: Importing from ID "my-bucket"...
module.bf-nathan.google_storage_bucket.assets-bucket: Import complete! Imported google_storage_bucket (ID: next-assets-bf-nathan-botfront-cloud)
module.bf-nathan.google_storage_bucket.assets-bucket: Refreshing state... (ID: next-assets-bf-nathan-botfront-cloud)
Error: module.bf-nathan.provider.kubernetes: 1:11: unknown variable accessed: var.cluster_ip in:
https://${var.cluster_ip}
The refreshing step doesn't work. I ran the command from the project's root where a terraform.tfvars file exists.
I tried adding -var-file=terraform.tfvars but no luck. Any idea?
You need to import it into the existing state file. You can do this with the terraform import command for any resource that supports it.
Thankfully the google_storage_bucket resource does support it:
Storage buckets can be imported using the name or project/name. If the project is not passed to the import command it will be inferred from the provider block or environment variables. If it cannot be inferred it will be queried from the Compute API (this will fail if the API is not enabled).
e.g.
$ terraform import google_storage_bucket.image-store image-store-bucket
$ terraform import google_storage_bucket.image-store tf-test-project/image-store-bucket
The Spark I connected to, is not built on my local computer but a remote one. Everytime when I connect to it http://xx.xxx.xxx.xxx:10000/, the error says:
[IPKernelApp] WARNING | Unknown error in handling PYTHONSTARTUP file /usr/local/spark/python/pyspark/shell.py:
18/03/07 08:52:53 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Anyways, I still keep trying to run it on Jupyter notebook:
from pyspark.conf import SparkConf
SparkSession.builder.config(conf=SparkConf())
dir(spark)
When I ran it yesterday, it shows directory. when I did it today, it says:
NameError: name 'spark' is not defined
Any suggestion is appreciated!
you re missing the spark variable
from pyspark.conf import SparkConf
spark=SparkSession.builder.config(conf=SparkConf())
dir(spark)