Consider project with different language modules that use same database. Database is controlled by liquibase. Liquibase changeLog files placed in special github repository. So every developer could download it and run locally to keep local instance of database in actual state.
Often developers use inmemory database like H2 for testing and liquibase rebuilds database structure every test run. To keep testing database in actual state every developer need to download changeLogs and put in their modules manually. But sometimes he could miss that somebody else changed database structure, so his tests should fail, but will pass. This is more actual for testing in CI.
Could liquibase download changelogs automatically from internet source? Could it use private github repositories? Could it use some property like changeLogFile for this?
May I confirm some assumptions? It sounds you have a team of developers all commiting liquibase database changes and you want to be able to automatically deploy in a CI/CD process?
Thanks,
Ronak
Liquibase Developer Evangelist
Liquibase can't run changelogs placed into some remote git repo. But in your CI you can setup such thing: if there is a pull-request from branch 'feature' to branch 'master' automatically:
Before tests are executed checkout branch 'feature'
and merge 'master' there
Apply all changelogs
Run tests after it
So if another developer change DB structure and commit it into master tests in 'feature' branch will be executed with latest DB structure.
Related
Is there any way we can connect snowflake with GIT for version control. With the help of that, we can maintain version of our merge statement and any other sql script in GIT.
DBeaver has git integration and is the best solution my team has found for version control with Snowflake. It's not perfect but it allows you to run your scripts against Snowflake and then push your SQL code to a git repository through the app UI or command line.
Yes! One way to do this is to store your Snowflake SQL code in a file/files with the sql extension (i.e. filename.sql). You can add those files to a GIT repo and track them in the repo accordingly.
This is an age old question when dealing with databases and how one should go about versioning them. Unfortunately, no database really integrates directly into any VCS that I'm aware of.
My team has settled on using dbt. This essentially turns the database into a series of text files that are easily integrated with git. The short of it is that you edit your models as local text files, and then use dbt run to put these models into Snowflake itself. This is kind of nice as you can configure separate environments such as dev and prod.
Other answers help with using an IDE as a go-between for git and Snowflake. These projects could be useful also:
https://medium.com/snowflake/snowflake-vs-code-sql-tools-and-github-7eab915e10cb
use VSCode as the IDE with a useful snowflake extension
https://github.com/Snowflake-Labs/schemachange
manage schema changes as script in git, deploy them with CI/CD
https://github.com/Snowflake-Labs/sfsnowsightextensions#get-sfworksheets
the missing feature of SnowSight -- export worksheets
There is now a VSCode extension for Snowflake. I'm able to connect vscode to our repo (Azure DevOps in my case) and Snowflake. It's got some nice features too like being able to easily cycle through past queries (including query results) and gives the same level of detail (or more) than the Snowflake UI.
When we deploy to our Testing environment with Sqitch, the migrations run and all is well. However, when we have to test a different code branch, the Sqitch migrations aren't rolled back. What I'm looking for is an idea of how others are solving this problem. For example, if I have a migration that creates a table my_table, and then deploy a migration from a different branch, Sqitch gets too confused to move forward.
Reason for branch change: we have features using our develop branch, and hotfixes being applied to master (I know; we're going to change that). Swapping over to test a hotfix caused all of this mess.
We considered having a Jenkins job to run a sqitch revert and have the user supply the migration to roll back to, but we want to eliminate the manual step.
How do you manage migrations when having to switch code branches?
I believe this is the purpose of sqitch-checkout
I want to create automated deployment pipeline for azure datafactory.
For one stream of development we can configure it using doc
https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment
But when it comes to deploying to two diff test datafactories for parrallel features development (in two different branches), it is not working because the adb_publish which gets generated is only specific to the one datafactory.
Currently we are doing deployement using powershell scripts and passing object list which needs to be deployed.
Our repo is in Azure devops.
I tried
linking the repo to multiple df but then it is causing issue, perhaps when finding deltas to publish.
Creating forks of repo instead of branches so that adb_publish can be seperate for the every datafactory - but this approach will not work when there is a conflict, which needs manual merge, so the testing will be required again instead of moving to prod.
Adf_publish get generated whenever you publish. Publishing takes whatever you have in your repo and updates data factory with it.
To develop multiple features in parallel, you need to just use "Save". Save will commit your changes to the branch you are actually working on. Other branches will do the same. Whenever you want to publish, you need to first make a pull request from your branch to master, then publish. Any merge conflict should be solved when merging everything in the master branch. Then just publish and there shouldn't be any conflicts, and adf_publish will get generated after that.
Hope this helped!
Since a GitHub repository can be associated with only one data factory. And you are only allowed to publish to the Data Factory service from your collaboration branch. Check this
It seems there is not a direct and easy way to accomplish this. If forking repo as workaround, you may have to solve the conflicts before merging as #Martin suggested.
I'm working on a web project(built with the .Net framework) on a remote windows server, and this project is connected to a database my SQL server management studio, now on multiple other remote windows servers exist the same web project linked to the same database, now I change a page's code in my project or add/remove a table or stored procedure in my database, is there a way(or an already existing software) which will my to deploy the changes that I made to all the others(or to choose multiple servers if I don't want to deploy the changes to all of them)?
If it were me, I would stand up a git server somewhere (cloud or local vm), make a branch called something like Prod or Stable, and create a script (powershell if the servers are windows, bash on anything else) on a nightly or hourly job to pull from that branch. Only push to that branch after testing thoroughly. If your code requires compilation, you have the choice to compile once before committing (in which case you're probably going to commit to releases), or on each endpoint after the pull. I would have the script that does the pull also compile and restart the service (only if there was something new in the pull).
You can probably achieve this by following two things :
Create a separate publishing profile for each server.
Use git/vsts branches to keep the code separate. (as suggested by #memtha).
Let's say you have total 6 servers and two branches A and B. So, you'll have to create 6 publishing profiles. Then, you can choose which branch to deploy where. e.g. you can deploy branch B on server 1,3 and 4.
For the codebase you could use Git Hooks.
https://gist.github.com/noelboss/3fe13927025b89757f8fb12e9066f2fa
And for the database, maybe you could use migrations or something similar. You will need to provide more info about your database, do you store your database across multiple servers etc.
If the same web project is connecting to the same database and the database changes, I suspect you would need to update all the web apps to ensure the database changes don't break any of the apps and to keep all the apps updated to prevent any being left behind.
You should look at using Azure Devops to build and deploy your apps and update the database.
If you use Entity Framework, you can run the migrations on startup and have the application update the database when deployed manually or automatically using devops.
To maintain the software updated in multiple server you could use Git with hooks, post-receive hook is what you need.
The idea is to use one server as your Remote Repository and here configure the post-receive hook to update the codebase in the same server and the others.
we are using Redshift as our EDW and we have quite a bit of tables and view there. at the moment we keeping all DDL's in our organisation's knowledge centre, but this is basically copy and paste and not very smart. is there any other option that is quicker better to do so?
thanks
Not very sure what you meant be "copy and paste" but you can try to put all the scripts in a github/ SVN repository and make sure that all the DDLs actually get fired using the scripts from the repo.
We did this using git and Jenkins (and little bit of Shell programs to do the code checkins and checkouts). We blocked all the users from running DDL statements and the Jenkins job would just pull the latest scripts from the repo and deploy it automatically from the RC (release candidate) branch of the repo.
If you need to export the DDL scripts out from the system you can use the script provided by the AWS folks,
https://github.com/awslabs/amazon-redshift-utils/blob/master/src/AdminViews/v_generate_tbl_ddl.sql
If you want to automate the checkin process to some code repository, you can build a wrapper python code using this code.