SQL Server BDC Pools and Performance - tsql

Question

Against an AKS based SQL Server 2019 BDC, I loaded the Flight_delay dataset that is available at www.kaggle.com. I wanted to test the performance of the various data stores, ie, master instance, data pool, HDFS-storage pool and ADLS-storage pool (via HDFS Tiering).
Against the pools - data pool, HDFS-storage pool and ADLS-storage pool - I created External tables to access as shown in the script below.
Select Count(*)
From [dbo].[Master_Flights] F
Inner join [dbo].[Master_Airports] A on F.ORIGIN_AIRPORT = A.IATA_CODE
Inner join [dbo].[Master_Airlines] L on F.AIRLINE = L.IATA_CODE
GO
Select Count(*)
From [dbo].[DataPool_Flights] F
Inner join [dbo].[DataPool_Airports] A on F.ORIGIN_AIRPORT = A.IATA_CODE
Inner join [dbo].[DataPool_Airlines] L on F.AIRLINE = L.IATA_CODE
GO
Select Count(*)
From [dbo].[HDFS_StoragePool_Flights] F
Inner join [dbo].[HDFS_StoragePool_Airports] A on F.ORIGIN_AIRPORT = A.IATA_CODE
Inner join [dbo].[HDFS_StoragePool_Airlines] L on F.AIRLINE = L.IATA_CODE
GO
Select Count(*)
From [dbo].[adls_StoragePool_Flights] F
Inner join [dbo].[adls_StoragePool_Airports] A on F.ORIGIN_AIRPORT = A.IATA_CODE
Inner join [dbo].[adls_StoragePool_Airlines] L on F.AIRLINE = L.IATA_CODE
GO
The performance from "best to worst":
Master Instance - 2 seconds
Data Pool - 16 seconds
HDFS-Storage Pool - 90 seconds
ADLS-Storage Pool - 220 seconds
Are my test results on expected lines? I was hoping to get pretty good performance results for the data pool option at least.
If the performance is this bad for the tables in the pools why would anyone want to use them?
Thanks,
grajee

Answer 1

let's ignore the ADLS for now - there is too much other stuff (network latency between regions etc.) involved.
This might be an indexing or IO issue.
How big is you AKS cluster (machine family and node count)?
How do the numbers compare when running the count JUST against the fact table (flights) without joining?
How does the HDFS perform when you convert the CSV to parquet?
Ben