How to query all DBs in Azure ElasticDB Pool? - tsql

Question

How do I perform SELECT on a table on all databases available in an ElasticDB Pool. All of them have same DB schema and they're created dynamically. I've explored Elastic Database Query but getting lost in the middle.
Reporting across scaled-out cloud databases
It asks to download a sample console application first, create a shard and then run the query which is a bit confusing. Is there anyway I can run T-SQL queries from SQL Server Management Studio to query all the databases.
PS: The DBs are not sharded. They're on DB per customer.
Thanks in Advance!

Answer 1

I'm thinking you need to add the databases as external sources so you can do a cross database query, you will be able to query the tables as if they were local.
I found a guide that can help you set it up:
Link to guide:
https://www.mssqltips.com/sqlservertip/4550/sql-azure-cross-database-querying/
the guide:
DB1 has Db1Table table:
CREATE TABLE DB1.dbo.Db1Table (
ID int IDENTITY(1, 1) NOT NULL PRIMARY KEY,
CustomerId INT,
CustomerName NVARCHAR(50));
INSERT INTO DB1.dbo.Db1Table(CustomerId, CustomerName) VALUES
( 1, 'aaaaaaa' ),
( 2, 'bbbbbbb' ),
( 3, 'ccccccc' ),
( 4, 'ddddddd' ),
( 5, 'eeeeeee' );
CREATE TABLE DB1.dbo.Db1Table (
ID int IDENTITY(1, 1) NOT NULL PRIMARY KEY,
CustomerId INT,
CustomerName NVARCHAR(50));
INSERT INTO DB1.dbo.Db1Table(CustomerId, CustomerName) VALUES
( 1, 'aaaaaaa' ),
( 2, 'bbbbbbb' ),
( 3, 'ccccccc' ),
( 4, 'ddddddd' ),
( 5, 'eeeeeee' );
DB2 has Db2Table table:
CREATE TABLE DB2.dbo.Db2Table (
ID int IDENTITY(1, 1) NOT NULL PRIMARY KEY,
CustomerId INT,
Country NVARCHAR(50));
INSERT INTO DB2.dbo.Db2Table(CustomerId, Country) VALUES
( 1, 'United States' ),
( 3, 'Greece' ),
( 4, 'France' ),
( 5, 'Germany' ),
( 6, 'Ireland' );
CREATE TABLE DB2.dbo.Db2Table (
ID int IDENTITY(1, 1) NOT NULL PRIMARY KEY,
CustomerId INT,
Country NVARCHAR(50));
INSERT INTO DB2.dbo.Db2Table(CustomerId, Country) VALUES
( 1, 'United States' ),
( 3, 'Greece' ),
( 4, 'France' ),
( 5, 'Germany' ),
( 6, 'Ireland' );
If we want to fetch customers whose country is Greece then we could do the following query:
SELECT
db1.CustomerId,
db1.CustomerName
FROM DB1.dbo.Db1Table db1
INNER JOIN DB2.dbo.Db2Table db2 ON db1.CustomerId = db2.CustomerId
WHERE db2.Country = 'Greece';
SELECT
db1.CustomerId,
db1.CustomerName
FROM DB1.dbo.Db1Table db1
INNER JOIN DB2.dbo.Db2Table db2 ON db1.CustomerId = db2.CustomerId
WHERE db2.Country = 'Greece';
but instead of returning customerId 3 we get the following error:
Reference to database and/or server name in 'DB2.dbo.Db2Table' is not supported in this version of SQL Server.
Reference to database and/or server name in 'DB2.dbo.Db2Table' is not supported in this version of SQL Server.
In order to be able to perform a cross database query we need to perform the following steps:
Step1: Create Master Key
The database master key is a symmetric key used to protect the private keys of certificates and asymmetric keys that are present in the database. More info here.
CREATE MASTER KEY ENCRYPTION BY PASSWORD = '<password>';
-- Example --
CREATE MASTER KEY ENCRYPTION BY PASSWORD = '3wbASg68un#q'
CREATE MASTER KEY ENCRYPTION BY PASSWORD = '<password>';
-- Example --
CREATE MASTER KEY ENCRYPTION BY PASSWORD = '3wbASg68un#q'
Step 2: Create Database Scoped Credential “my_credential”
A database credential is not mapped to a server login or database user. The credential is used by the database to access the external location anytime the database is performing an operation that requires access.
CREATE DATABASE SCOPED CREDENTIAL <credential_name>
WITH IDENTITY = '<user>',
SECRET = '<secret>';
-- Example --
CREATE DATABASE SCOPED CREDENTIAL my_credential
WITH IDENTITY = 'dbuser',
SECRET = '9Pfwbg68un#q';
CREATE DATABASE SCOPED CREDENTIAL <credential_name>
WITH IDENTITY = '<user>',
SECRET = '<secret>';
-- Example --
CREATE DATABASE SCOPED CREDENTIAL my_credential
WITH IDENTITY = 'dbuser',
SECRET = '9Pfwbg68un#q';
credential_name
Specifies the name of the database scoped credential being created. credential_name cannot start with the number (#) sign. System credentials start with ##.
IDENTITY =’identity_name‘
Specifies the name of the account to be used when connecting outside the server.
SECRET =’secret‘
Specifies the secret required for outgoing authentication.
Step 3: Create External Data Source “my_datasource” of type RDBMS
This instruction creates an external data source for use in Elastic Database queries. For RDBMS, it specifies the logical server name of the remote database in Azure SQL Database.
-- (only on Azure SQL Database v12 or later)
CREATE EXTERNAL DATA SOURCE <data_source_name>
WITH (
TYPE=RDBMS,
LOCATION='<server_name>.database.secure.windows.net',
DATABASE_NAME='<remote_database_name>',
CREDENTIAL = <sql_credential>);
-- Example --
CREATE EXTERNAL DATA SOURCE my_datasource
WITH (
TYPE=RDBMS,
LOCATION='ppolsql.database.secure.windows.net',
DATABASE_NAME='DB2',
CREDENTIAL = my_credential);
-- (only on Azure SQL Database v12 or later)
CREATE EXTERNAL DATA SOURCE <data_source_name>
WITH (
TYPE=RDBMS,
LOCATION='<server_name>.database.secure.windows.net',
DATABASE_NAME='<remote_database_name>',
CREDENTIAL = <sql_credential>);
-- Example --
CREATE EXTERNAL DATA SOURCE my_datasource
WITH (
TYPE=RDBMS,
LOCATION='ppolsql.database.secure.windows.net',
DATABASE_NAME='DB2',
CREDENTIAL = my_credential);
data_source_name
Specifies the user-defined name for the data source. The name must be unique within the database in Azure SQL Database.
TYPE = [ HADOOP | SHARD_MAP_MANAGER | RDBMS ]
Use RDBMS with external data sources for cross-database queries with Elastic Database query on Azure SQL Database.
LOCATION =
specifies the logical server name of the remote database in Azure SQL Database.
DATABASE_NAME = ‘remote_database_name’
The name of the remote database (for RDBMS).
CREDENTIAL = credential_name
Specifies a database-scoped credential for authenticating to the external data source.
Step 4: Create External Table “mytable”
This instruction creates an external table for Elastic Database query.
CREATE EXTERNAL TABLE [ database_name . [ schema_name ] . | schema_name. ] table_name
( <column_definition> [ ,...n ] )
WITH (
DATA_SOURCE = <data_source_name>);
-- Example --
CREATE EXTERNAL TABLE [dbo].[Db2Table] (
[ID] int NOT NULL,
[CustomerId] INT,
[Country] NVARCHAR(50)
) WITH ( DATA_SOURCE = my_datasource )
CREATE EXTERNAL TABLE [ database_name . [ schema_name ] . | schema_name. ] table_name
( <column_definition> [ ,...n ] )
WITH (
DATA_SOURCE = <data_source_name>
);
-- Example --
CREATE EXTERNAL TABLE [dbo].[Db2Table] (
[ID] int NOT NULL,
[CustomerId] INT,
[Country] NVARCHAR(50)
) WITH ( DATA_SOURCE = my_datasource )
database_name . [ schema_name ] . | schema_name. ] table_name
The one to three-part name of the table to create. For an external table, only the table metadata is stored in SQL along with basic statistics about the file and/or folder referenced in Hadoop or Azure blob storage. No actual data is moved or stored in SQL Server.
[ ,…n ]
The column definitions, including the data types and number of columns, must match the data in the external files.
DATA_SOURCE = external_data_source_name
Specifies the name of the external data source that contains the location of the external data.
After running the DDL statements, you can access the remote table Db2Table as though it were a local table.
So, now if we want to fetch customers whose country is Greece the query would be executed successfully:
SELECT
db1.CustomerId,
db1.CustomerName
FROM DB1.dbo.Db1Table db1
INNER JOIN DB1.dbo.Db2Table db2 ON db1.CustomerId = db2.CustomerId
WHERE db2.Country = 'Greece';
-- Result --
CustomerId | CustomerName
-------------------------
3 ccccccc
SELECT
db1.CustomerId,
db1.CustomerName
FROM DB1.dbo.Db1Table db1
INNER JOIN DB1.dbo.Db2Table db2 ON db1.CustomerId = db2.CustomerId
WHERE db2.Country = 'Greece';
-- Result --
CustomerId | CustomerName
-------------------------
3 ccccccc