The Situation
I have an Azure Devops build pipeline that is building and deploying to an existing AzureSql Database instance via the outputted .dacpac.
I would like to have the ability to run a script or execute API calls to create new AzureSql database instances based on that project. I have found the New-AzSqlDatabaseImport powershell cmdlet that ALMOST lets me do that, requiring a .bacpac rather than a .dacpac. I attempted to use the .dacpac and naturally the process failed.
The Question
Can I output a .bacpac from my SqlProj build process?
Alternatively is there a way to create a new database and have that database schema imported from the dacpac in a relatively smooth elegant fashion?
What we have gone with is the following:
Host a "template" database alongside the other databases.
Update the "template" database during each update cycle with the dacpac changes.
On new user/organization creation, execute single call powershell script that performs a quick copy of the "template" database. New-AzSqlDatabaseCopy
This appears to go faster than separate provision and dacpac deploy, and is a single call to execute. In the future the powershell execution is likely to be changed to an Azure API call.
Related
I am trying to run the pipeline in azure data factory without using trigger.
So Is there any service which we can use instead of trigger in Azure?
In ADF, every pipeline will be executed by manual or by trigger.
The alternative way to ADF triggers can be done by following.
By Powershell.
Logic app.
Using REST API.
By Powershell:
Use Invoke-AzureRmDataFactoryV2Pipeline or Invoke-AzDataFactoryV2Pipeline in Powershell.
Command:
Invoke-AzureRmDataFactoryV2Pipeline -ResourceGroupName "RG name" -DataFactoryName "rakeshdfactory" -PipelineName "MyADFpipeline"
You will get the Pipeline run id in Powershell after executing and you can see the pipeline run in Monitor pipelines-> Trigger runs of ADF.
Schedule this Powershell script using Azure Automation to execute the pipeline daily.
Reference:
Third party tutorial to learn about powershell automation from sharepointdairy by Salaudeen Rajack
Using Logic apps:
ADF event triggers are only permitted to storage accounts, but logic apps can give other options like SQL tables as well.
Here I have created a blob trigger for the logic app. You can create Recurrence trigger for the logic app instead of blob trigger, by which you can schedule the logic app to invoke the ADF pipeline.
Reference:
Article by MITCHELL PEARSON
Assuming you mean without using the Trigger definitions inside Data Factory, then yes there are other options. The two that I use almost exclusively are 1) Logic Apps and 2) the Azure Data Factory SDK (this document may also be useful). I use a Function App to house my SDK code, and in most cases I have Logic Apps call the AF instead of executing the pipeline directly.
NOTE: purely for pedantic purposes, every pipeline run has a Trigger. When you execute by using the Trigger Now feature inside the workspace or execute a pipeline externally using one of the above methods, the Trigger Type will show as "Manual" in the Monitor section.
We have a database project that uses a dacpac to deploy schema changes and also allows a pre-deployment and post-deployment script.
However, we frequently have to run one-off scripts and security would prefer that developers not have write access in prod (we do not have DBA role at this time). I'm trying to find a solution that would work with azure devops to store one-time run scripts in git, run the script if it has not been run before, and not run the script the next time the pipeline runs. We'd like this done through devops so the SP has access to run the queries and not the dev, and anything flowing through the pipe has been through our peer review process, plus we have record of what was executed.
I'm looking for suggestions from anyone who has done this or is aware of any product which can do this.
Use liquibase. Though I would have it as part of my code base you can also use it from the CLI and run your scripts using that tool.
Liquibase keeps track of what SQL files you have published across deployments so you can have multiple stages say DIT, UAT, STAGING, PROD and it can apply the remaining one off SQL changes over time.
Generally unless you really need support, I doubt you'd need the commercial version. The opensource version is more than sufficient for my system needs and I have a relatively complex system already.
The main reason I like liquibase over other technologies is it allows for SQL based change sets. So the learning curve is a lot lower.
Two tips:
don't rely on the automatic computation of the logicalFilePath, explicitly set it even if it is repeating yourself. This allows you to refactor your scripts so instead of lumping everything into a single folder you may group them later on.
Name your scripts with the date first. That way you can leverage the natural sorting order.
I've faced a similar problem in the past:
Option 1
If you can afford to have an additional table in your database to keep track of what was executed or not, your problem can be easily solved, there is a tool which helps you: https://github.com/DbUp/DbUp
Then you would have a new repository let's call it OneOffSqlScriptsRepository and your pipeline would consume this repository:
resources:
repositories:
- repository: OneOffSqlScriptsRepository
endpoint: OneOffSqlScriptsEndpoint
type: git
Thus you'd create a pipeline to run this DbUp application consuming the scripts from the OneOffSqlScripts repository, the DB would take care of executing the scripts only once (it's configurable).
The username/password for the database can be stored safely in the library combined with azure keyvaults, so only people with the right access rights could access them (apart from the pipeline).
Option 2
This option assumes that you wanna do everything by using only the native resources that azure pipelines can provide.
Create a OneOffSqlScripts as in option1
Create a ScriptsRunner repository
In the ScriptRunner repository, you'd create a folder containing a .json file with the name of the scripts and the amount of times (or a boolean) you've had run them.
eg.:
[{
"id": 1
"scriptName" : "myscript1.sql"
"runs": 0 //or hasRun : false
}]
Then write a python script that reads and writes a json file by updating the amount of runs, thus you'd need to update your repository after each pipeline run. It would mean that your pipeline will perform a git commit / push operation after each run in case there new scripts to be run.
The algorithm is like these, the implementation can be tuned.
I am trying to run a runbook on azure that contains the following command:
Remove-AzureRmDataLakeStoreItem
When the Runbook is run, the following error comes out:
"Remove-AzureRmDataLakeStoreItem : The term 'Remove-AzureRmDataLakeStoreItem' is not recognized as the name of a cmdlet,..."
What should I do?
This issue typically happens when there is a version mismatch between PS modules in your runbook and the Azure Automation account. To resolve, you will need to update your Azure PS Modules within the Azure Automation Account. "update" steps are published HERE.
Important note:
"Because modules are updated regularly by the product group, changes can occur with the included cmdlets, which may negatively impact your runbooks depending on the type of change, such as renaming a parameter or deprecating a cmdlet entirely. To avoid impacting your runbooks and the processes they automate, it is recommended that you test and validate before proceeding. If you do not have a dedicated Automation account intended for this purpose, consider creating one so that you can test many different scenarios and permutations during the development of your runbooks, in addition to iterative changes such as updating the PowerShell modules. After the results are validated and you have applied any changes required, proceed with coordinating the migration of any runbooks that required modification and perform the following update as described in production."
I have an application with an Azure SQL Server. I have a test environment where I deploy the app for end to end testing. I want to reset the database to a certain state before the tests. I have a bacpac file with that state. My goal is to do this with a ci tool (appveyor). I am aware that I can drop the entire database with azure powershell api. Should I drop the database or should I empty it? Or am I thinking this totally wrong?
You could use Powershell to restore the BACPAC before the tests and drop the database after the tests. An example is available on Azure docs: https://learn.microsoft.com/en-us/azure/sql-database/scripts/sql-database-import-from-bacpac-powershell
An option that will probably execute faster, is to directlry run T-SQL against the database to setup your initial state before the tests and empty the database after the tests.
However, doing a full BACPAC restore and drop out is much easier to setup in such a way that you can run multiple testruns in parallel as it is not relying on a pre-existent database that it is claiming for the duration of the run.
Edit: Changed link to en-us instead of nl-nl
Another option is you run Azure database as docker service, this may simplify your initial databse setup.
I have found this great script which backs up SQL Azure database to BLOB.
I want to run many different variations of this script - e.g. DB1 goes to Customer1Blob, DB2 goes to Customer2Blob.
I have looked at Scheduler Job Collections. However I can only see options (Action settings) for HTTP(S)/ Storage Queue / Service Bus.
Is it possible to run a specific .ps1 script (with commands) scheduled?
You can definitely run a Powershell script as a WebJob. If you want to run a script on a schedule, you can add a settings.job file containing a chron expression with your webjob. The docs for doing so are here.
For this type of automation tasks, I prefer to use the Azure Automation service. You can create runbooks using powershell and then schedule this with the use of the Azure scheduler. You can have it run "on azure" so you do not need to use compute power that you pay for (rather you pay by the minute the job runs) or you can configure it to run with a hybrid worker.
For more information, please see the documentation
When exporting from SQL DB or from SQL Server, make sure you are exporting from a quiescent database. Exporting from a database with active transactions can result in data integrity issues - data being added to various tables while they are also being exported.