I have a storage account (StorageV2), there is a table. I would like to be able select from this table from Azure SQL database:
CREATE EXTERNAL DATA SOURCE testtable
WITH (LOCATION = 'https://<MyStorageAccount>.table.core.windows.net/test',
CREDENTIAL = testtable,
TYPE = BLOB_STORAGE
);
CREATE EXTERNAL TABLE testtable (
PartitionKey varchar(1000),
RowKey varchar(1000),
Content varchar(max)
)
WITH (
DATA_SOURCE = testtable
)
;
If it's not possible to connect as an external table, are there any other options to connect to a table in a storage account using database tools? Without datafactory, databrix, SSIS,...
Msg 46525, Level 16, State 31, Line 23
External tables are not supported with the provided data source type.
Azure SQL Database supports OPENROWSET and BULK INSERT with Azure Blob Storage files.
Examples of bulk access to data in Azure Blob Storage
So you can create a view that uses OPENROWSET over an Azure blob instead of an external table.
Eg
create or alter view my MyAzureInvoices
as
SELECT * FROM OPENROWSET(
BULK 'week3/inv-2017-01-19.csv',
DATA_SOURCE = 'MyAzureInvoices',
FORMAT = 'CSV',
FORMATFILE='invoices.fmt',
FORMATFILE_DATA_SOURCE = 'MyAzureInvoices'
) AS DataFile;
Related
I am try to create an external table on Azure Synapse by once I run select * from logdata I get the error "External table 'logdata' is not accessible because location does not exist or it is used by another proces"
below is my code
CREATE DATABASE appdb;
CREATE MASTER KEY ENCRYPTION BY PASSWORD =<>
CREATE DATABASE SCOPED CREDENTIAL SasToken
WITH IDENTITY='SHARED ACCESS SIGNATURE',
SECRET=<>
CREATE EXTERNAL DATA SOURCE log_data
WITH (
LOCATION='https://<>.dfs.core.windows.net/data',
CREDENTIAL=SasToken
)
CREATE EXTERNAL FILE FORMAT TextFileFormat WITH(
FORMAT_TYPE=DELIMITEDTEXT,
FORMAT_OPTIONS(
FIELD_TERMINATOR=',',
FIRST_ROW=2
)
)
CREATE EXTERNAL TABLE logdata(
[Id] INT,
[Correlationid] VARCHAR (200),
[Operationname] VARCHAR (200),
[Status] VARCHAR (200),
[Eventcategory] VARCHAR (200),
[Level] VARCHAR (200),
[Time] DATETIME,
[Subscription] VARCHAR (200),
[Eventinitiatedby] VARCHAR (1000),
[Resourcetype] VARCHAR (1000),
[Resourcegroup] VARCHAR (1000)
)
WITH(
LOCATION='/Log.csv',
DATA_SOURCE=log_data,
FILE_FORMAT=TextFileFormat
)
-- drop EXTERNAL table logdata ;
SELECT * from logdata;
```
I tried changing the access levels but couldn't work either.
You need the necessary access rights to the file in order to fix this problem. The simplest method is to give yourself the "Storage Blob Data Contributor" role on the storage account that you are trying to query.
And also check the details you entered while creating the external table in synapse. The location's URL is case-sensitive. and make sure your file is present at the destination.
Correct Syntax/code to create External Table:
CREATE MASTER KEY ENCRYPTION BY PASSWORD ='Welcome#Pratik123';
CREATE DATABASE SCOPED CREDENTIAL MyCred
WITH IDENTITY='SHARED ACCESS SIGNATURE',
SECRET='SAS Token';
CREATE EXTERNAL DATA SOURCE MyDs2
WITH (
LOCATION='abfss://containername#storageaccountname.dfs.core.windows.net/foldername if any',
CREDENTIAL=MyCred
)
CREATE EXTERNAL FILE FORMAT MyFile2 WITH(
FORMAT_TYPE=DELIMITEDTEXT,
FORMAT_OPTIONS(
FIELD_TERMINATOR=',',
FIRST_ROW=2
)
)
CREATE EXTERNAL TABLE MyData3(
[Id] varchar(20),
[NAME] VARCHAR (200),
[ADDRESS] VARCHAR (200)
)
WITH(
LOCATION='/dataaddress.csv',
DATA_SOURCE=MyDs2,
FILE_FORMAT=MyFile2
)
SELECT * from MyData3;
My output:
refer this similar error example by- Mike Stephenson
I have an ADF pipeline that uses copy data activity for copying data from blob storage to table storage. This pipeline runs on a trigger once every day. I have provided a table name in table storage data set as 'Table1'.
Instead of providing a hard coded table name value (Table1), is it possible to provide a dynamic value as table name in the table storage such that RUN ID of pipeline run is used as the table name in the table storage and copy data from blob to that table in table storage?
You could set a dynamic value as table name.
For example, you can add parameter to the table storage dataset:
Then you can set the pipeline parameter to specify the table name:
But we can not provide the RUN ID of pipeline run as the table name in the table storage and copy data from blob to that table in table storage.
Hope this helps.
I have a UDF function in Cosmos DB , it takes a parameter and returns the documents that meets the condition based on the parameter.
Each document returned by this UDF has 3 fields,
Customer ID
Modified Date
Customer Status
I need this information in a SQL Server SP present in another database.
I am thinking of having a powershell script to bring this data from the Cosmos DB , store it in a table local to the SQL server database , and then use this table eventually in the SP.
I wondering if my above approach to fetch data from Cosmos DB to SQL Server database is right, and if so could I know if we can execute a cosmos DB UDF from a powershell script and use the result set returned by the UDF.
Based on your description,maybe you could use Azure Data Factory.
Step1: Follow the article to create Copy activity.
Step2: Configure Cosmos db source data:
sql:
SELECT udf.adf(c.fields).CustomerID,
udf.adf(c.fields).ModifiedDate,
udf.adf(c.fields).CustomerStatus FROM c
Then,please follow the steps from this doc:
Step 3: Configure your Sink dataset:
Step 4: Configure Sink section in copy activity as follows:
Step 5: In your database, define the table type with the same name as sqlWriterTableType. Notice that the schema of the table type should be same as the schema returned by your input data.
CREATE TYPE [dbo].[CsvType] AS TABLE(
[ID] [varchar](256) NOT NULL,
[Date] [varchar](256) NOT NULL,
[Status ] [varchar](256) NOT NULL
)
Step 6: In your database, define the stored procedure with the same name as SqlWriterStoredProcedureName. It handles input data from your specified source, and merge into the output table. Notice that the parameter name of the stored procedure should be the same as the "tableName" defined in dataset.
Create PROCEDURE convertCsv #ctest [dbo].[CsvType] READONLY
AS
BEGIN
MERGE [dbo].[adf] AS target
USING #ctest AS source
ON (1=1)
WHEN NOT MATCHED THEN
INSERT (id,data,status)
VALUES (source.ID,source.Date,source.Status );
END
So, I'm taking a first look at migrating a PostgreSQL db to agensgraph db.
I'm using the manual https://bitnine.net/wp-content/uploads/2016/11/AgensGraph_Quick_Guide.pdf
first export as csv:
SET CLIENT_ENCODING TO 'utf8';
\COPY samples.samples TO
'C:\Users\garyn\Documents\graph_migration\pg_csv\samples_samples.csv'
WITH DELIMITER E'\t' CSV;
And on page 20 I follow the first steps, creating the foreign table:
CREATE EXTENSION file_fdw;
CREATE SERVER import_server FOREIGN DATA WRAPPER file_fdw;
CREATE FOREIGN TABLE vlabel_profile ( id graphid, properties text) SERVER import_server
OPTIONS( FORMAT 'csv', HEADER 'false',
FILENAME 'C:\Users\garyn\Documents\graph_migration\pg_csv\samples_samples.csv',
delimiter E'\t');
ERROR: cannot create table in graph schema
SQL state: XX000
Now, I haven't set any column names (as header=false) and I haven't changed the id graphid, properties text since the manual says it is setting up the table, but it states the file directory, any ideas how to get past this error? I'm back to being a noob.
The next steps will be:
CREATE FOREIGN TABLE elabel_profile ( id graphid, start graphid, "end" graphid, properties text) SERVER import_server OPTIONS( FORMAT 'csv', HEADER 'false', FILENAME '/path/file.csv', delimiter E'\t');
Then execute the import
CREATE VLABEL test_vlabel; LOAD FROM vlabel_profile AS profile_name CREATE (a:test_vlabel =row_to_json(profile_name)::jsonb);
CREATE ELABEL test_elabel; LOAD FROM elabel_profile AS profile_name MATCH (a:test_vlabel), (b:test_vlabel) WHERE (a).id::graphid = (profile_name).start AND (b).id::graphid = (profile_name).end CREATE (a)-[:test_elabel]->(b);
------------ UPDATE ------------
I'm now trying with the northwind dataset, again following the agens tutorial: https://bitnine.net/tutorial/english-tutorial.html
DROP GRAPH northwind CASCADE;
CREATE GRAPH northwind;
SET graph_path = northwind;
DROP SERVER northwind;
CREATE SERVER northwind FOREIGN DATA WRAPPER file_fdw;
CREATE FOREIGN TABLE categories (
CategoryID int,
CategoryName varchar(15),
Description text,
Picture bytea
)
SERVER northwind
OPTIONS (FORMAT 'csv', HEADER 'true', FILENAME 'D:\northwind\categories.csv', delimiter ',', quote '"', null '');
Same error
I have tried to create a foreign table with northwind dataset you mentioned but it works just fine for me as you see the below screen shot.
I installed the agensgraph and tried the sample with its latest version which is 2.1.0 since I didn't have agensgraph on my window OS.
If you let me know the version of agensgraph you are currently using and how you are accessing to agensgraph, I would be able to help you out more.
re: cannot create table in graph schema
This is an error you will get when your schema is the same as the name of a graph - or there is some other problem related to the default schema.
The default schema is called public. To check your current schema enter
select current_schema();
If it's not public you can set it with
set schema public;
then try to create a table
create table mytable(id int);
I have a U-SQL managed table that contains schematized structured data.
CREATE TABLE [AdlaDb].[dbo].[User]
(
UserGuid Guid,
Postcode string,
Age int?
DateOfBirth DateTime?,
)
And a Azure SQL Database table.
CREATE TABLE [SqlDb].[dbo].[User]
(
UserGuid uniqueidentifier NOT NULL,
Postcode varchar(15) NULL,
Age int NULL,
DateOfBirth Date NULL,
)
I would like to transfer data from U-SQL managed table to Azure SQLDB table without losing the data types.
I'm using azure data factory, seems like I cannot
directly query the U-SQL managed table as an input dataset for data factory
do a federated write query to Azure SQLDB
Hence I'm having an intermediate step where I copy from U-SQL managed table to Azure Blob and then move to Azure SQLDB table. Doing this, I'm losing the data type and having to have type conversion/transformations later again before inserting.
Is there any better way to transfer data from U-SQL managed table to Azure SQL Database table without losing data type? Or am I missing something?
At this point you have the following option:
Export the U-SQL table into an intermediate format (e.g., CSV) in ADLS or blob storage.
Use ADF to move the file into Azure SQL DB.
I know that the ADF team has a work item to do this for you. I will ask them to reply to this thread as well.
Directly writing into a table from a U-SQL script has a lot of challenges due to the fault-tolerant retry and scale-out processing in U-SQL. This makes atomic writing in parallel into a transacted store a bit more complex (see for example http://www.vldb.org/conf/1996/P460.PDF).
There is now another option to transfer data from USQL managed table to Azure SQL Database table.
Write out the data from USQL Managed table or from a USQL script to Azure Blob Storage as a text file (.csv, .txt etc..)
And then make use of the public preview feature in Azure SQL Database - BULK INSERT - wrap this into a stored procedure
Add an Stored procedure activity in Azure Data Factory to schedule
Note: There is one thing to be aware of when creating DATABASE SCOPED CREDENTIAL, refer this Stack Overflow question