There is a need to deploy project to kubernetes with multiple environments "development,staging,production" and to multiple countries "Germany, France, Spain" (examples).
Until now I have been using pipeline in original project that deploys to germany and it is done with gitlab ci "environment" for "development,staging,production". That part is ok.
For other countries, I mirror the project and add different variables in the mirror project. Then both countries are deployed.
Problem is that each country builds a separate image even though the code base is the same. So for 5 countries, 5 images are built instead of 1 for each environment.
Current solution:
I have determined for now that I will be adding extra_{{country_id}}_{{environment}} environment for each country in the same project.
So then for example, I will build image for development environment and deploy it to extra_GER_development, extra_FRA_development and extra_SPA_development.
For now I have found a lot of info on how to deploy for "development, staging, production", but none how to split it even further.
Question: Is there a better way on how to deploy single image for multiple environments?
Any info about possible solution articles is also appreciated. thank you.
If by image you mean Docker Image, you can build a single image and set 5 or more tags for countries and push different tags to the repository or deploy every tag to their environment.
Since it's a repetitive process, you can also write a shell script and automate the process. You can pass the environments or countries list, iterate over the list and set appropriate tags or deploy them one by one.
Consider having an environment variable that includes all countries; you can write a shell script like the following:
#!/bin/bash
for country in "$#"
do
# set taga
docker tag SOURCE_IMAGE:tag TARGET_IMAGE:tag
# or maybe deploy
...
done
and call this script in the Gitlab CI script:
...
deploy:
stage: deploy
script:
- ./set-country-tags ${COUNTRIES}
...
Related
We are planning to implement CI/CD for our project. In summary, our setup is the following:
Code Repository (contains solution and project files)
Config Repository (contains .config files for different countries)
Aside from having multiple environments, we also have instances in multiple countries. Sample below:
Country 1
Dev
QA
Prod
Country 2
Dev
QA
Prod
I understand that Azure Release Pipelines can have multiple Stages (Dev, QA, Prod) and Variables that can be used for deployment. Then we can have a 1 Release Pipeline for each country.
What I am having difficulty is the Config Repository. Sometimes, developers may need to update a specific config file for a Country for a specific environment.
The only solution I can think of is trigger a release pipeline with this again, but just to replace the Web.config on IIS in the VM.
However, this is not good, because if developers only update Code Repository, any changes from Config Repository will be overwritten.
Any suggestions on how to automate multiple countries / environments with pipelines?
Thank you
Hi you can use Azure DevOps Environments to target environment strategy (in your case country and Dev/QA/Prod)
Normal practice shouldn't using repository for .config and any settings files.
you can also consider using Secure files for Azure Pipeline or inline script (within your environment deplyonce)
you can start exploring that Azure Pipeline Environment to understand it and YAML and how can it apply to you Multi-Stage YAML.
I'successfully set up a CICD pipeline following this tutorial.
It shows clearly how to make Google Cloud Build and Kubernetes work with one environment: production.
For simplicity, this tutorial uses a single environment —production—
in the env repository, but you can extend it to deploy to multiple
environments if needed.
Right, but some details are missing: is there one kubernetes.yaml file by environment? What about kubernetes namespaces?...
More precisely, what would be the way to handle multiple environments (staging...)?
There could be a bizillion ways of doing environments , but what I understand from this line:
env repository: contains the manifests for the Kubernetes Deployment
That the default master/production branch maps to the production environment , then you can create for example testing , and staging branches , where you test and stage your things , and later on port the change to master branch.
Infact if you keep reading that document , it will tell you something:
The env repository can have several branches that each map to a
specific environment (you only use production in this tutorial) and
reference a specific container image, whereas the app repository does
not.
One more thing , if you have access to gitlab and kubernetes , you can implement it without google GKE and clud build.
I'm using Gitlab CI for my project. When I push on develop branch, it runs tests and update the code on my test environment (a remote server).
But the gitlab runner is already using the same build folder : builds/a3ac64e9/0/myproject/myproject
But I would like to create a now folder every time :
builds/a3ac64e9/1/yproject/myproject
builds/a3ac64e9/2/yproject/myproject
builds/a3ac64e9/3/yproject/myproject
and so on
Using this, I could just update my website by changing a symbolic link pointing to the last runner directory.
Is there a way to configure Gitlab Runner this way ?
While it doesn't make sense to use your build directory as your deployment directory, you can setup a custom build directory
Open config.toml in a text editor: (more info on where to find it here)
Set enabled = true under [runners.custom_build_dir] (more info here)
[runners.custom_build_dir]
enabled = true
In your .gitlab-ci.yml file, under variables set GIT_CLONE_PATH. It must start with $CI_BUILDS_DIR/, e.g. $CI_BUILDS_DIR/$CI_JOB_ID/$CI_PROJECT_NAME, which will probably give you what you're looking for, although if you have multiple stages, they will have different job IDs. Alternatively, you could try $CI_BUILDS_DIR/$CI_COMMIT_SHA, which would give you a unique folder for each commit. (More info here)
variables:
GIT_CLONE_PATH: '$CI_BUILDS_DIR/$CI_JOB_ID/$CI_PROJECT_NAME'
Unfortunately there is currently an issue with using GIT_BUILDS_DIR in GIT_CLONE_PATH, if you're using Windows and Powershell, so you may have to do something like this as a work-around, if all your runners have the same build directory: GIT_CLONE_PATH: 'C:\GitLab-Runner/builds/$CI_JOB_ID/$CI_PROJECT_NAME'
You may want to take a look at the variables available to you (predefined variables) to find the most suitable variables for your path.
You might want to read the following answer Changing the build intermediate paths for gitlab-runner
I'll repost my answer here:
Conceptually, this approach is not the way to go; the build directory is not a deployment directory, it's a temporary directory, to build or to deploy from, whereas on a shell executor this could be fixed.
So what you need is to deploy from that directory with a script as per gitlab-ci.yml below, to the correct directory of deployment.
stages:
- deploy
variables:
TARGET_DIR: /home/ab12/public_html/$CI_PROJECT_NAME
deploy:
stage: deploy
script:
mkdir -pv $TARGET_DIR
rsync -r --delete ./ $TARGET_DIR
tags:
- myrunner
This will move your projectfiles in /home/ab12/public_html/
naming your projects as project1 .. projectn, all your projects could use this same .gitlab-ci.yml file.
You can not achieve this only with Gitlab CI runner configuration, but you can create 2 runners, and assign them exclusively to each branch by using a combination of only and tags keywords.
Assuming your two branches are named master and develop and two runners have been tagged with master_runner and develop_runner tags, your .gitlab-ci.yml can look like this:
master_job:
<<: *your_job
only:
- master
tags:
- master_runner
develop_job:
<<: *your_job
only:
- develop
tags:
- develop_runner
(<<: *your_job is your actual job that you can factorize)
I have an ASP .Net MVC application with 4 different publishing profiles: dev, test, demo and prod.
These publising profiles are build using the same two steps: NuGet restore followed by an MSBuild. Then, they are deployed to lots of different servers: a few dev servers (one server dev per team), one test server, one demo server and several production servers.
msbuild /p:Configuration="$(Configuration)"
/p:PlatformTarget="any cpu"
/p:DeployOnBuild="True"
/p:DeployTarget="MsDeployPublish"
/p:MSDeployServiceURL="$(MSDeployServiceURL)"
/p:DeployIISAppPath="portal"
/p:CreatePackageOnPublish="False"
/p:MsDeployPublishMethod="WMSVC"
/p:AllowUntrustedCertificate="True"
/p:UserName="Deploy_User_For_TFS"
/p:Password="P#ssw0rd"
/p:AutoParameterizationWebConfigConnectionStrings=False
/p:ExcludeFilesFromDeployment="Cache"
Currently I have 4 TFS Build configurations (one for every publishing profile) and a file where I have all the possible values for MSDeployServiceURL parameter.
There are two issues with this approach:
When we had to add a new parameter AutoParameterizationWebConfigConnectionStrings we had to change it in 4 places instead of one.
We have to have a shared file as the source for the parameters. it is not easy to understand where which value should be copied and people often make mistakes.
So I have two questions:
Is there any way to have a one universal template where I can specify only my parameters Configuration and list of possible MSDeployServiceURLs and have everything else stay the same? Having such a template should fix problem #1.
Is there a way to define a drop-down like variable, where the value could not be typed in by the user, but should be selected from a pre-defined list of values?
Is there any way to have a one universal template where I can specify
only my parameters Configuration and list of possible
MSDeployServiceURLs and have everything else stay the same? Having
such a template should fix problem #1.
You could simplify such that the release configuration creates a templated publish profile. Then using TFS's release management, you could update the publish profile with the appropriate values. Based on your description, it seems like you are trying to combine both the compilation and the release.
For example, in TFS you could have one build (for example, MyApp-Release) that builds the code in the release configuration. As part of that process, it passes in placeholders for things like the deploy URL. For example, /p:MSDeployServiceURL="$(MSDeployServiceURL)" would be /p:MSDeployServiceURL="__MSDeployServiceURL__".
In the TFS release, you'd have a step that the replaces tokens (if you need one, you can use Colin's ALM Corner Custom Build Tasks) in the publish profile. The replace token task would then update the __MSDeployServiceURL__ with the value from an release environment variable with the same name (minus the underscores). So your release would have a dev, test, demo, and prod environment and for each environment, there would be a variable named MSDeployServerURL in each with a different value and a replace tokens step.
My question is relatively simple,
I have gitlab set-up, gitlab CI too and two separate server which each have their own runner.
Both runners are working and can execute a build successfully.
What I'd like to achieve now is to have one project be build by both runners, perferrably even with seperate commands. This last thing doesn't seem possible however if I add both runners to a project it just seems to build on one of them and not the other.
Is it possible to get it to build on both and maybe even vary the scripts?
Someone brought this up on the GitLab CI issue tracker (https://gitlab.com/gitlab-org/gitlab-ci/issues/237). The workaround proposed there is as follows:
Create multiple jobs with different tags and assign different tags to these runners:
job1:
script: echo 1
tags:
- runner1
job2:
script: echo 2
tags:
- runner2
Not a great solution, especially if you want to run the exact same job on a bunch of runners (as I do), but it can be made to work.