Import Azure Monitor log data into Azure Data Factory - azure-data-factory

Question

Is it possible to connect Azure Data Factory to the Azure Monitor logs to extract the data?
You can connect from Power BI as described here:
https://learn.microsoft.com/en-us/azure/azure-monitor/platform/powerbi
But I want to be able to connect from Data Factory to the log.

Answer 1

Per my knowledge,there is no direct way which is similar to the PB way in the link you mentioned in your question in the ADF. Based on this document and ADF portal UI, we could store the log in three residences:
Azure Storage Account.
Event Hub.
Log Analytics.
For storage account ,you could access them in copy activity.
For Event hub, maybe you could use Event Hub REST API in REST dataset and ADF Web Activity or you could get an idea of Azure Stream Analytics.
For Log Analytics, you could use Log Analytics REST API in REST dataset and ADF Web Activity：
POST https://api.loganalytics.io/v1/workspaces/DEMO_WORKSPACE/query
X-Api-Key: DEMO_KEY
Content-Type: application/json
{
"query": "AzureActivity | summarize count() by Category"
}

Answer 2

Got is all working. I have done the following:
Create a Pipeline which contains 2 Web Activities, 1 For Each Loop & Call to stored procedure to insert the data
First Web Activity gets the bearer token
Second Web Activity calls the REST API GET and has a Header name Authorization which brings in the access_token for the first web activity Bearer {access_token}
Then A For Each Loop which I pass the output for the second Web Activity
Stored procedure Activity which passes in all my fields into an insert stored procedure
Finally that all worked. I had a lot of trouble using the Copy Activity so resorted to the For Each Loop and stored procedure call to insert each record from the output of the REST API call in the web activity.
I will post more detailed info once I get some sleep!

Answer 3

Partial Answer:
I have been able to use 2 Web Activities in a pipeline. 1 which gets the bearer token and 2 which then uses the bearer token to carry out the GET command. BUT now the question is how can I use the Output from the Web Activity in a subsequent Copy Activity so I can load the data into SQL ????

Answer 4

There are two methods witch depend on the method of authentication to the API.
The first is with a service principle, High level steps described above. This Blog on the topic is also useful: https://datasavvy.me/2020/12/24/retrieving-log-analytics-data-with-data-factory/comment-page-1/#comment-28467
Second is with Managed Identity:
first give ADF access to Log Analytics using IAM How can I use this API in Azure Data Factory
Then connect to Log Analytic API with Web activity or a copy activity (these are the two i got working).
Web Activity
URL: https://api.loganalytics.io/v1/workspaces/[Workspace ID]/query
Body: {"query":"search '*'| where TimeGenerated >= datetime(#{pipeline().parameters.it_startDate}) and TimeGenerated < datetime(#{pipeline().parameters.it_endDate}) | distinct $table "}
Copy Activity
First the linked service.
ADF Datasets:
Base URL: URL: https://api.loganalytics.io/v1/workspaces/[Workspace ID]/
Copy Source:
Body: {
"query": "#{item()[0]} | where TimeGenerated >= datetime(#{pipeline().parameters.it_startDate}) and TimeGenerated < datetime(#{pipeline().parameters.it_endDate})"
}
Additional:
The body code above, gets a list of the table names in log analytics using the web activity. Which I then pass to the Copy Activity to exports copy of the data for each table.