My question is relatively simple,
I have gitlab set-up, gitlab CI too and two separate server which each have their own runner.
Both runners are working and can execute a build successfully.
What I'd like to achieve now is to have one project be build by both runners, perferrably even with seperate commands. This last thing doesn't seem possible however if I add both runners to a project it just seems to build on one of them and not the other.
Is it possible to get it to build on both and maybe even vary the scripts?
Someone brought this up on the GitLab CI issue tracker (https://gitlab.com/gitlab-org/gitlab-ci/issues/237). The workaround proposed there is as follows:
Create multiple jobs with different tags and assign different tags to these runners:
job1:
script: echo 1
tags:
- runner1
job2:
script: echo 2
tags:
- runner2
Not a great solution, especially if you want to run the exact same job on a bunch of runners (as I do), but it can be made to work.
Related
I'm trying to create CI that does the following:
Run terraform plan -out=plan.out to generate a Terraform plan.
After looking at the Terraform plan output in Github actions, I can manually run another job or workflow that calls terraform apply plan.out with the previously generated plan. I want to manually run this automation after the other automation has successfully run, dependent on the previous automation's success, using an artifact from the previous automation.
I've looked online for some examples of this but all the examples of this I can find just run terraform apply without actually allowing someone to verify the plan output.
Is this something that's possible to do in Github Actions?
This can be done using protected environments' required reviewers: https://docs.github.com/en/actions/deployment/targeting-different-environments/using-environments-for-deployment#required-reviewers
What you would do is setup an environment e.g. production and add yourself as reviewer.
In your workflow, you would then add the environments like so:
jobs:
plan:
steps:
- run: terraform plan
apply:
environment: production
steps:
- run: terraform apply
This means that as soon as the workflow reaches the job apply, it is going to stop and you'll need to manually click a button to approve.
My solution ended up being the following:
When the PR is approved and merged, a Terraform plan is created and pushed to an S3 bucket with the commit hash in the path. Then when the apply workflow is triggered via workflow dispatch it looks for a plan for the commit hash of the code it's running and applies it.
Using pull requests as suggested wasn't the right solution for me because of the following:
How do you know that the plan that was run for the pull request was run with the latest changes on the base branch? The plan could be invalid in this case. The way I solved this was by having the plan workflow run on push of a specific branch that corresponds to the environment being Terraformed. This way the plan is always generated for the state the Terraform says the specific environment should be in.
How do you know that an apply is applying the exact plan that was generated for the pull request? All the examples I saw actually ended up re-running the plan in the apply workflow, which breaks the intended use of Terraform plans. The way I solved this was by having the apply workflow look for a specific commit hash in cloud storage.
There is a need to deploy project to kubernetes with multiple environments "development,staging,production" and to multiple countries "Germany, France, Spain" (examples).
Until now I have been using pipeline in original project that deploys to germany and it is done with gitlab ci "environment" for "development,staging,production". That part is ok.
For other countries, I mirror the project and add different variables in the mirror project. Then both countries are deployed.
Problem is that each country builds a separate image even though the code base is the same. So for 5 countries, 5 images are built instead of 1 for each environment.
Current solution:
I have determined for now that I will be adding extra_{{country_id}}_{{environment}} environment for each country in the same project.
So then for example, I will build image for development environment and deploy it to extra_GER_development, extra_FRA_development and extra_SPA_development.
For now I have found a lot of info on how to deploy for "development, staging, production", but none how to split it even further.
Question: Is there a better way on how to deploy single image for multiple environments?
Any info about possible solution articles is also appreciated. thank you.
If by image you mean Docker Image, you can build a single image and set 5 or more tags for countries and push different tags to the repository or deploy every tag to their environment.
Since it's a repetitive process, you can also write a shell script and automate the process. You can pass the environments or countries list, iterate over the list and set appropriate tags or deploy them one by one.
Consider having an environment variable that includes all countries; you can write a shell script like the following:
#!/bin/bash
for country in "$#"
do
# set taga
docker tag SOURCE_IMAGE:tag TARGET_IMAGE:tag
# or maybe deploy
...
done
and call this script in the Gitlab CI script:
...
deploy:
stage: deploy
script:
- ./set-country-tags ${COUNTRIES}
...
I'm trying to use GitHub to trigger on PR a GitLab pipeline.
Practically when a developer creates a PR in GitHub, his/her code get tested against a GitLab pipeline.
I'm trying to follow this user guide: https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/github_integration.html
and we have a silver account, but it won't work. When creating the PR, the GitLab pipeline is not triggered.
Anyone with this kind of experience who can help?
Thanks
Joe
I've found the cause of the issue.
In order for GitHub to trigger GitLab as CD/CI mostly in PR request, you need to have a Silver/Premium account AND, very important, being the root owner.
Any other case, you won't be able to see github in the integration list on GitLab. People from gitlab had the brilliant idea to hide it instead of showing it disabled (which would had been a tip to understand that you needed an upgraded license)
In the video above it's not explained.
Firstly, you need to give us the content of your .gitlab-ci.yaml file. In your question you asked about GitHub but you're following Gitlab documentation which is completely different. Both are using git commands to commit and push repos but Github & Gitlab are different.
For Github pipelines, you need to create a repository, then you go to Actions. Github will propose you to configure a .github/workflows directory which contain a file.yaml. In this .yaml file you can code your pipelines. According to your project, Github will propose you several linux machines with the adequate configuration to run your files (If it's a Java Project --> you'll be proposed maven machines, Python --> Python Machines, React/Angular -> machines with npm installed, Docker, Kubernetes for deployments...) and you're limited to 4 private project as far as I know (check this last information).
For Gitlab you have two options, you can use preconfigured machines like github, and you call them by adding for example atag: npm in your .gitlab-ci.yaml file, to call a machine with npm installed, but you need to pay an amount of money. Or you can configure your own runners by following the Gitlab documentation with gitlab commands (which is the best option), but you'll need good machines and servers to run npm - mvn - python3 - ... commands
Of course, in your Gitlab repository, and finally to answer your question this an example, of .gitlab-ci.yaml file with two simple stages: build & test, the only statement specifies that these pipelines will run if there is a merge request ( I use the preconfigured machines of Gitlab as a sample here) More details on my python github project https://github.com/mehdimaaref7/Scrapping-Sentiment-Analysis and for gitlab https://docs.gitlab.com/runner/
stages:
- build
- test
build:
tags:
- shell
- linux
stage: build
script:
- echo "Building"
- mkdir build
- touch build/info.txt
artifacts:
paths:
- build/
only:
- merge_requests
test:
tags:
- shell
- linux
stage: test
script:
- echo "Testing"
- test -f "build/info.txt"
only:
- merge_requests
I'm using Gitlab CI for my project. When I push on develop branch, it runs tests and update the code on my test environment (a remote server).
But the gitlab runner is already using the same build folder : builds/a3ac64e9/0/myproject/myproject
But I would like to create a now folder every time :
builds/a3ac64e9/1/yproject/myproject
builds/a3ac64e9/2/yproject/myproject
builds/a3ac64e9/3/yproject/myproject
and so on
Using this, I could just update my website by changing a symbolic link pointing to the last runner directory.
Is there a way to configure Gitlab Runner this way ?
While it doesn't make sense to use your build directory as your deployment directory, you can setup a custom build directory
Open config.toml in a text editor: (more info on where to find it here)
Set enabled = true under [runners.custom_build_dir] (more info here)
[runners.custom_build_dir]
enabled = true
In your .gitlab-ci.yml file, under variables set GIT_CLONE_PATH. It must start with $CI_BUILDS_DIR/, e.g. $CI_BUILDS_DIR/$CI_JOB_ID/$CI_PROJECT_NAME, which will probably give you what you're looking for, although if you have multiple stages, they will have different job IDs. Alternatively, you could try $CI_BUILDS_DIR/$CI_COMMIT_SHA, which would give you a unique folder for each commit. (More info here)
variables:
GIT_CLONE_PATH: '$CI_BUILDS_DIR/$CI_JOB_ID/$CI_PROJECT_NAME'
Unfortunately there is currently an issue with using GIT_BUILDS_DIR in GIT_CLONE_PATH, if you're using Windows and Powershell, so you may have to do something like this as a work-around, if all your runners have the same build directory: GIT_CLONE_PATH: 'C:\GitLab-Runner/builds/$CI_JOB_ID/$CI_PROJECT_NAME'
You may want to take a look at the variables available to you (predefined variables) to find the most suitable variables for your path.
You might want to read the following answer Changing the build intermediate paths for gitlab-runner
I'll repost my answer here:
Conceptually, this approach is not the way to go; the build directory is not a deployment directory, it's a temporary directory, to build or to deploy from, whereas on a shell executor this could be fixed.
So what you need is to deploy from that directory with a script as per gitlab-ci.yml below, to the correct directory of deployment.
stages:
- deploy
variables:
TARGET_DIR: /home/ab12/public_html/$CI_PROJECT_NAME
deploy:
stage: deploy
script:
mkdir -pv $TARGET_DIR
rsync -r --delete ./ $TARGET_DIR
tags:
- myrunner
This will move your projectfiles in /home/ab12/public_html/
naming your projects as project1 .. projectn, all your projects could use this same .gitlab-ci.yml file.
You can not achieve this only with Gitlab CI runner configuration, but you can create 2 runners, and assign them exclusively to each branch by using a combination of only and tags keywords.
Assuming your two branches are named master and develop and two runners have been tagged with master_runner and develop_runner tags, your .gitlab-ci.yml can look like this:
master_job:
<<: *your_job
only:
- master
tags:
- master_runner
develop_job:
<<: *your_job
only:
- develop
tags:
- develop_runner
(<<: *your_job is your actual job that you can factorize)
I am using TravisCI's Build Stages to separate my job to two stages:
Build and test on multiple environments.
Build and deploy if stage 1 pass.
And I want to Travis to run the jobs on commits to two GitHub branches, master and dev, and pull requests to master and dev. However, I only want to run the stage two when a commit to master happens.
Is there a way to completely ignore stage 2 in commits to branches that are not master, and on pull requests?
This is what my .travis.yml looks like at the time of writing this:
https://github.com/SammyIsra/photostream-react/blob/c354a62c3cc963b345a5c2fb95658c90ddc39d21/.travis.yml
Update:
this seems to not be possible as of yet. However, the TravisCI team may be working on something like that, as of this comment on the Build Stages feedback board. Whenever I learn that it was added as a feature, or that it will definitely not be possible ever, I will change this question.
There doesn't seem to be a way to ignore specific branches when triggering jobs in the build stages section, however it looks like there's a way to only fire your deployments on pushes to master.
In your build stage script section, you could wrap the npm run build command in a quick Bash if statement to test the environment variable which shows what branch you're on:
if [ TRAVIS_BRANCH == "master" ]; then npm run build; fi
Then, in your Surge deploy section, you can restrict the deployment to a particular branch with:
deploy:
...
on: master
And basically the same for the NPM deployment:
deploy:
...
on:
branch: master
Note: I've not tried any of this myself, so it may or may not work in combination with the new build stages functionality.