How do I trigger a notebook in my Azure Machine Learning notebook workspace from Azure Data Factory
I want to run a notebook in my Azure ML workspace when there are changes to my Azure storage account.
My understanding is that your use case is 100% valid and it is currently possible with the azureml-sdk. It requires that you create the following:
Create an Azure ML Pipeline. Here's a great introduction.
Add a NotebookRunnerStep to your pipeline. Here is a notebook demoing the feature. I'm not confident that this feature is still being maintained/supported, but IMHO it's a valid and valuable feature. I've opened this issue to learn more
Create a trigger using Logic apps to run your pipeline anytime a change in the datastore is detected.
There's certainly a learning curve to Azure ML Pipelines, but I'd argue the payoff is in the flexibility you get in composing steps together and easily scheduling and orchestrating the result.
This feature is currently supported by Azure ML Notebooks. You can also use Logic apps to trigger a run of your Machine Learning pipeline when there are changes to your Azure storage account.
Related
We are in the process of setting up the CI / CD process for AWS Glue ETL Process. The existing ETL process contains the following AWS Glue Components - Crawlers, Registered tables in catalog, Jobs, Triggers and workflows.
Obviously the first step is to set up a code repository and link the existing artifacts from different components mentioned above to the repository, which will ideally need to facilitate the developers in performing the check-ins and pull request from the tool (Something similar to ADF and Databricks). However as far as we have explored, AWS glue does not have integration to any of the source code repository which can directly provide this feature unless we are missing something.
Hence what is the method to setup the environment for CI (I'm still not talking about CD), the below link gives a reference for CI/CD:
https://aws.amazon.com/blogs/big-data/implement-continuous-integration-and-delivery-of-serverless-aws-glue-etl-applications-using-aws-developer-tools/
However it mentions at the beginning that, AWS CloudFormation template file for deploying the ETL jobs are both committed to version control - so not clear on how this is done for the on-going regular commits from the developers.
However as far as we have explored, AWS glue does not have integration
to any of the source code repository which can directly provide this
feature unless we are missing something.
Correct, Glue does not have VC integration.
I develop (python and cloudformation) locally on vscode and use it's git integration plugin. And I use a container if I want to test something locally, but Glue also has a Dev Endpoint for similar tasks.
To promote CI/CD for Analysis Services tabular models (SSAS, Azure AS or Power BI datasets), I always recommend people to use Tabular Editor with Azure DevOps. One popular feature of Tabular Editor's CLI, is that it can perform a schema check, in which Tabular Editor connects to the data source defined within the tabular model, in order to validate the partition queries against the actual columns specified in the model metadata.
Microsoft recommends the use of the MSOLEDBSQL provider (1,2) for SQL Server-based data sources. Unfortunately, this provider is not available on Microsoft-hosted build agents in Azure Pipelines (neither vs2017-win2016 nor windows-2019).
Unfortunately, the installer for MSOLEDBSQL requires admin permission, so I don't think that we can install the driver as part of our pipeline.
One workaround is to use Tabular Editor's scripting functionality to temporarily change the data source to use for example the SQLNCLI provider when performing the schema check. However, it feels like the missing MSOLEDBSQL driver on the build agents is an oversight on Microsoft's part, especially considering that they're recommending the use of this driver for production purposes.
Is there any way we can have the MSOLEDBSQL driver available on a Microsoft-hosted Windows-based build agent?
Is there any way we can have the MSOLEDBSQL driver available on a
Microsoft-hosted Windows-based build agent?
Virtual Environments repo contains the source used to create the virtual environments for GitHub Actions hosted runners, as well as the VM images of Microsoft-hosted agents used for Azure Pipelines.To file bug reports, or request that tools be added/updated, please open an issue using the appropriate template.
So I think you can open a Tool Request here with the given template, then the team there would consider and check your feedback.
In addition: As temporary workaround you can consider installing one self-hosted agent in your local machine, so that you can run the pipeline with local environment. (With more control to install dependent software needed for your build and deployment)
I have a logic app that connects to an sftp server (virtual machine that I created on azure) and does actions when a file is added to that sftp:
When a file is added I create a new blob on the blob storage.
Delete the file from the SFTP server
I have also created a blob trigger-based azure function that, every time a blob is created, processes some actions (like blob content decryption and parsing).
Next steps will be chaining some other azure functions executions in my logic app (like sending e-mail after executing and azure function etc... )…
Now, I have two main questions:
In order to have the best CI/CD pipeline suited for this workflow, do I create the logic app from the portal or from visual studio and why please?
Do I put azure function and logic app in the same solution/Repo? Same project?
Then, how can I create the CI/CD pipeline (type template and steps please)?
Ps: I want to add unit tests to test if my logic app and azure function are working correctly so I want to integrate test step in my build definition.
For more details about the logic app please see this Stack overflow question in which i detailed the process
and here is the logic app
Please find the below points:
I would recommend using Visual Studio. The main advantage is it gives you the same
designer experience, and you can make use of ARM Template and parameters to
deploy your logic app robustly to multiple environments to dev, Staging , proc etc.,
making a robust CI/CD pipeline. It also gives you an advantage of using Azure key vaults using ARM template and the parameter syntax to store any sensitive data.
Also Visual studio provides you to connect to the cloud using cloud
explorer where you can mimic resubmit , run history etc..
If you are using Azure function only for one process then you can
put it under the same solutions, but keeping Azure functions as a
separate Repo gives you more flexibility of re-usability, so that
other applications can also make use of it.
You can utilise Speck flow for automate logic app testing Automated tesing logic app with speckflow this link explained it in detail.
I am doing some research for Powerapps integration with Azure DevOps.
However there is limitated information for it.
It is possible to integrate powerapps inside a Task for AzureDevops?
Based on, that we have a .zip file with the Powerapp, and we want to create a Build and Release/Deploy for several environments.
Thank You.
It is possible to integrate powerapps inside a Task for AzureDevops?
Yes it is.
You can leverage the Solution concept of the Microsoft Power Platform and the Power Apps BuildTools (preview) extension for Azure DevOps.
Update 11/2020: This is now GA and called Power Platform Build Tools
I've written a complete step-by-step guide on this topic:
A Continuous Delivery Approach for No-Code Solutions in Microsoft’s Power Platform
Bottom line:
With this build tool, you can automatically check-in a Solution into source control and deploy it using a continuous delivery approach with the help of Azure DevOps. See the screenshot for a sample configuration of the Export and Import Solution Task.
It works for everything you can organize inside a Solution, e.g.:
Power Apps
Power Automate Flows
AI Builder Models
Common Data Service Entities
It is possible to integrate powerapps inside a Task for AzureDevops?
I am afraid there is no such Task integrate powerapps for AzureDevops at this moment.
If you want to integrate powerapps with azure devops, you can follow the guide step by step:
Microsoft Teams – Integration with Visual Studio Team Services using PowerApps.
Besides, AFAIK, PowerApps should not be "Build/Deployed" through Azure Devops.
When you are developing with PowerApps, there is no way to do Source
Control. There are no source files. The only artifact you can version
control is the .zip file that you can export.
And
In PowerApps, you don’t have to build your code. Any change you make
to the application is live for you to test it. In that way it is very
productive. To publish the application you just click on the publish
button and it is live.
Check this great blog: PowerApps From A DevOps Perspective for some more details.
Hope this helps.
Solutions are a way to package your components in a single zip file and use Powerapps build tools to import your solution on to a different environment or tenant.
It is still a an improvement from manually importing each app or environment variable and then import it on to target system, but it lacks what we call as automation of deployment.
To provide an eg, I will explain what I have done, and what still constitutes of a manual task:
I created an enterprise level app using powerapp canvas model. My app consumes data from around 20 APIs. These API calls are implemented in power automate.
We have 4 environments, dev, sit, uat and prod. Now I cant keep on importing flows in each environment and change their api URLs to point to the deployed environment. So I used environment variables for each environment which stores api URLs for each environment. This can be done under solution.
Under the same solution, I added my app. So now my solution has 2 things, my app and the environment variable which consists of api URLs.
I then use powerapps build tools to move this solution from dev to sit.
Steps: use build tools tasks to perform the following
Export solution
Unpack it in git
Pack it
Import the solution.
This successfully moves my solution to sit.
But the solution environment variable still points to the dev url.
So I have to override environment variables to store sit URLs.
This manual intervention to edit environment variable is as good as doing all the tasks manually.
This was the case when PowerApps was first announced; however, this is no longer the case.
While it is technically true that there is no actual code that would be managed and deployed with a PowerApp or Flow but that doesn't mean that you can not use the power of Azure DevOps. Additionally, when creating a PowerApp / flow you would also be creating entities and even Model Driven apps - and these uses solutions - which naturally work well to deplooy within Azure DevOps.
Microsoft is building out this whole construct to enable all these to deploy...
While the whole incorporation of PowerApps and flows into Solutions is not fully baked yet - they are targeting to have this ready around the October time frame this year.
We have been talking to Microsoft about also enabling PowerApps and flows to follow the same expansion that solutions do so that they can take advantage of the full branching strategy.
So even though you would be simply exporting out zip files into your repo - you can still take advantage of the full devops pipeline which is highly recommended.
Use this component, it still on preview mode but is working fine on my side
https://marketplace.visualstudio.com/items?itemName=microsoft-IsvExpTools.PowerApps-BuildTools
Are there any built-in options to perform Regression testing and Performance testing of APIs in Azure API Management? If not, are there any options to do that within the Azure ecosystem (like VSTS Performance tests tool etc.)? Thanks!
I think there are no built-in options to perform Regression testing that available in Azure. But for performance testing, there are some.
Regression Testing
For regression.I'd recommend going with manual testing like creating Test Plan, Test Cases execute/test them manually using VSTS
Creating a Test Plan in VSTS
Creating a Test Case in VSTS
Run Manual Test in VSTS
Load/Performance Testing
There are 3 ways you can easily do this
With Visual Studio
With VSTS
Azure Portal
I prefer going with VSTS