I am trying to import the azure storage blob state in to pulumi using PULUMI CLI.
tried below cmd
pulumi import --yes azure-native:storage:Blob testblob
it thrown error with below.
error: Preview failed: "resourceGroupName" not found in resource state
please let me know if any one is able to successfully import the azure storage blob resource in to pulumi.
thanks,
kumar
tried below cmd
pulumi import --yes azure-native:storage:Blob testblob
it thrown error with below.
error: Preview failed: "resourceGroupName" not found in resource state
expected result: to import successfully
actual result: import failed.
If you look at the docs for a Blob resource there's an import section (this section exists on all resources).
The actual command you'll need is:
pulumi import azure-native:storage:Blob myresource1 /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Storage/storageAccounts/{accountName}/blobServices/default/containers/{containerName}/blobs/{blobName}
Related
I am trying to access a file which is Rds extension. I am using the below code however it is not helping.
import pandas as pd
url_sas_token = 'https://<my account name>.file.core.windows.net/test/test.rds?st=2020-01-27T10%3A16%3A12Z&se=2020-01-28T10%3A16%3A12Z&sp=rl&sv=2018-03-28&sr=f&sig=XXXXXXXXXXXXXXXXX'
# Directly read the file content from its url with sas token to get a pandas dataframe
pdf = pd.read_excel(url_sas_token )
# Then, to convert the pandas dataframe to a PySpark dataframe in Azure Databricks
df = spark.createDataFrame(pdf)
I created storage account and created file share and uploaded rds file into file share.
Image for reference:
I generated SAS key in storage account.
Image for reference:
I installed azure file shares in data bricks using
pip install azure-storage-file
I installed pyreadr package to load rds file using
pip install pyreadr
I tried to load the rds extension file in databrick using
from azure.storage.file import FilePermissions, FileService
from datetime import datetime, timedelta
import pyreadr
from urllib.request import urlopen
url_sas_token="<File Service SAS URL>"
response = urlopen(url_sas_token)
content = response.read()
fhandle = open( 'counties.rds', 'wb')
fhandle.write(content)
fhandle.close()
result = pyreadr.read_r("counties.rds")
print(result)
In above code I have given File Service SAS URL at url_sas_token.
image for reference:
Above code loaded rds file data successfully.
Image for reference:
In this way I accessed rds extension file which is in azure blob file share from data bricks.
No matter i was using GCloud console or command line, when tried to import
Merchant center's data from BigQuery, it shows
Import failed: Failed to fetch service account credentials for project:
xxxx. Importing from BQ will fail.
I'm trying to connect to my Heroku DB and I'm getting the following series of errors related to SSL:
SSL connection to data store using host matching failed. Retrying without host matching.
SSL connection to data store failed. Retrying without SSL.
Check that your connection definition references your JDBC database with correct URL syntax, username, and password. org.postgresql.util.PSQLException: Connection attempt timed out.
I managed to connect to the DB with DBeaver and had similar SSL problems until I set the SSL Factory to org.postgresql.ssl.NonValidatingFactory, but Glue doesn't offer any SSL options.
The DB is actually hosted on AWS, the connection URL is:
jdbc:postgresql://ec2-52-19-160-2.eu-west-1.compute.amazonaws.com:5432/something
(p.s. the AWS Glue forums are useless! They don't seem to be answering anyones questions)
I was having the same issue and it seems that the issue is that Heroku requires a newer JDBC driver than the one that Amazon requires. See this thread:
AWS Data Pipelines with a Heroku Database
Also, it seems that you can use the jbdc directly from your python scripts. See here:
https://dzone.com/articles/extract-data-into-aws-glue-using-jdbc-drivers-and
So it seems like you need to download a new driver, upload it to s3, then manually use it in your scripts as mentioned here:
https://gist.github.com/saiteja09/2af441049f253d90e7677fb1f2db50cc
Good luck!
UPDATE: I was able to use the following code snippet in a Glue Job to connect to the data. I had to upload the Postgres driver to S3 and then add it to the path for my Glue Job. Also, make sure that either the Jars are public or you've configured the IAM user's policy such that they have access to the bucket.
%pyspark
import sys
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.dynamicframe import DynamicFrame
from awsglue.transforms import *
glueContext = GlueContext(SparkContext.getOrCreate())
source_df = spark.read.format("jdbc").option("url","jdbc:postgresql://<hostname>:<port>/<datbase>“).option("dbtable", “<table>”).option("driver", "org.postgresql.Driver").option("sslfactory", "org.postgresql.ssl.NonValidatingFactory").option("ssl", "true").option("user", “<username>”).option("password", “<password>”).load()
dynamic_dframe = DynamicFrame.fromDF(source_df, glueContext, "dynamic_df")
I would like to import mysql dump file into mysql database instance with the gcloud import tool, but I am getting an error:
ubuntu#machine:~/sql$ gcloud sql instances import sql-56-test-8ef0cb104575 gs://dbf/bt_ca_dev_tmp-2017-01-19.sql.gz
ERROR: (gcloud.sql.instances.import) HTTPError 403: Insufficient Permission
What exact permissions am I missing? I can create sql instance with registered service account but I am not possible to import data?
You have issue with permissions
create a bucket if you don't have one, run
`gsutil mb -p [PROJECT_NAME] -l [LOCATION_NAME] gs://[BUCKET_NAME]`
Describe the sql instance you are exporting from and copy the sa
`gcloud sql instances describe [INSTANCE_NAME]`
Add the service account to the bucket ACL as a writer
`gsutil acl ch -u [SERVICE_ACCOUNT_ADDRESS]:W gs://[BUCKET_NAME]`
Add the service account to the import file as a reader
`gsutil acl ch -u [SERVICE_ACCOUNT_ADDRESS]:R gs://[BUCKET_NAME]/[IMPORT_FILE_NAME]`
Import the file:
gcloud sql import csv [INSTANCE_NAME] gs://[BUCKET_NAME]/[FILE_NAME] \
--database=[DATABASE_NAME] --table=[TABLE_NAME]
According to the documentation, this should do the trick. The wierd thing is that it needs write permissions. This should do the trick:
gsutil iam ch serviceAccount:"${SERVICE_ACCOUNT}":roles/storage.legacyBucketWriter gs://${BUCKET_NAME}
gsutil iam ch serviceAccount:"${SERVICE_ACCOUNT}":roles/storage.objectViewer gs://${BUCKET_NAME}
It's a permission problem of the Cloud SQL service account in the Google Storage bucket you're trying to use. To solve it you need to grant Storage Legacy Bucket Reader, Storage Legacy Object Owner, Storage Object Viewer roles to the service account email that you get from
gcloud sql instances describe <YOUR_DB_NAME> | grep serviceAccountEmailAddress
To do it go to the Cloud Storage / your bucket in Google Cloud Console and under Permission write the serviceAccountEmailAddress in ADD. Finally, add the roles you need.
I have stored everything needed for my database in phpmyadmin , and exported the my database from it. That was saved as Cloud.sql , so now this sql file I imported it to the Google Cloud Storage with the help of this link https://developers.google.com/cloud-sql/docs/import_export.
Now after importing the Contents of .sql using the Import option present in the action of the instance, it shows the green working sign , and after a while it stops, when I check in the Logs , it shows
Failed to import gs://bucket_name/Cloud.sql: An unknown problem occurred (ERROR_RDBMS)
So ,
I am unable to find out the reason behind the error as its not clear, and how can this be solved
Google Cloud Sql probably doesn't know to which database the gs://bucket_name/Cloud.sql commands apply.
From https://groups.google.com/forum/#!topic/google-cloud-sql-discuss/pFGe7LsbUaw:
The problem is the dump doesn't contain the name of the database to use. If you add a 'USE XXX' at the top the dump where XXX is the database you want to use I would expect the import to succeed.
I had a few issues that were spitting out the ERROR_RDBMS error.
It turns out that google actually does have more precise errors now, but you have to go here
https://console.cloud.google.com/sql/instances/{DATABASE_NAME}/operations
And you will see a description of why the operation failed.