I compile my website into static HTML using Harp. I then upload the static HTML to my webserver's data directory to deploy the build. I store the markdown source in a private GitLab repository (hosted by me).
All the uploading and mucking about with Harp is a little tedious and I'd like to automate it. I would like to set things up so that as soon as I push a new commit to the GitLab repo, Harp automatically compiles and copies over the HTML files.
How can I accomplish this?
The webserver and GitLab hosts are actually both virtual hosts on the same Ubuntu machine, and I use the machine itself to run harp compile.
We do this on the harpjs.com website repo: when a commit is pushed or a pull request merged, harp compile is run on Travis CI and then we publish the static files to Surge.sh.
.travis.yml
language: node_js
sudo: false
node_js:
- "4.0"
after_success:
- npm run deploy
package.json run scripts
"scripts": {
"compile": "harp compile",
"deploy": "npm run compile && surge ./www harpjs.com"
}
Full disclosure, I am on the Harp team and we also make Surge, but I think that example in the .travis.yml file of the harpjs.com repo should still be applicable, even if you’re using GitLab and something other than Surge.
Related
CI Runner Context
Gitlab version : 13.12.2 (private server)
Gitlab Runner version : 14.9.1
Executor : shell executor (PowerShell)
Exploitation system : Windows 10
Project in Python (may be unrelated)
(using Poetry for dependency management)
The Problem
I am setting up an automated integration system for a project that has several internal dependencies that are hosted on the same server as the project being integrated. If I run the CI with a poetry update in the yml file, the Job console sends an exit with error code 128 upon calling a git clone on my internal dependency.
To isolate the problem, I tried simply calling a git clone on that same repo. The response is that the runner cannot authenticate itself to the Gitlab server.
What I Have Tried
Reading through the Gitlab docs, I found that the runners need authorization to pull any private dependencies. For that, Gitlab has created deploy keys.
So I followed the instructions to create the deploy key for the dependency and added it to the sub-project's deploy key list. I then ran into the exact same permissions problem.
What am I missing?
(For anyone looking for this case for a Winodws PowerShell, the user that the runner uses is nt authority/system, a system only user that I have not found a way to access as a human. I had to make the CI runner do the ssh key creation steps.)
Example .gitlab-ci.yml file:
#Commands in PowerShell
but_first:
#The initial stage, always happens first
stage: .pre
script:
# Start ssh agent for deploy keys
- Start-Service ssh-agent
# Check if ssh-agent is running
- Get-Service ssh-agent
- git clone ssh://git#PRIVATE_REPO/software/dependency-project.git
I solved my problem of pulling internal dependencies via completely bypassing the ssh pull of the source code and by switching from poetry to hatch for dependency management (I'll explain why further down).
Hosting the compiled dependencies
For this, I compiled my dependency project's source code into a distribution-ready package (in this context it was a python wheel).
Then used Gitlab's Packages and Registries offering to host my package. Instead of having packages in each source code project, I pushed the packages of all my dependencies to a project I created for this single purpose.
My .gitlab-ci.yaml file looks like this when publishing to that project:
deploy:
# Could be used to build the code into an installer
stage: Deploy
script:
- echo "deploying"
- hatch version micro
# only wheel is built (without target, both wheel and sdist are built)
- hatch build -t wheel
- echo "Build done ..."
- hatch publish --repo http://<private gitlab repo>/api/v4/projects/<project number>/packages/pypi --user gitlab-ci-token --auth $CI_JOB_TOKEN
- echo "Publishing done!"
Pulling those hosted dependencies (& why I ditched poetry)
My first problem was having pip find the extra pypi repository with all my packages. But pip already has a solution for that!
In it's pip.ini file(to find where it is, you can do pip config -v list), 2 entries need to be added:
[global]
extra-index-url = http://__token__:<your api token>#<private gitlab repo>/api/v4/projects/<project number>/packages/pypi/simple
[install]
trusted-host = <private gitlab repo>
This makes it functionally the same as adding the --extra-index-url and --trusted-host tags while calling pip install.
Since I was using a dependency manager, I was not directly using pip, but the manager's wrapper for pip. And here comes the main reason why I decided to change dependency managers: poetry does not read or recognize pip.ini. So any changes done in any of those files will be ignored.
With the configuration of the pip.ini file, any dependencies I have in the private package repo will also be searched for the installation of projects. So the line:
- git clone ssh://git#PRIVATE_REPO/software/dependency-project.git
changes to a simple line:
- pip install dependency-project
Or a line in pyproject.toml:
dependencies = [
"dependency-project",
"second_project",
]
I'm trying to use GitHub to trigger on PR a GitLab pipeline.
Practically when a developer creates a PR in GitHub, his/her code get tested against a GitLab pipeline.
I'm trying to follow this user guide: https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/github_integration.html
and we have a silver account, but it won't work. When creating the PR, the GitLab pipeline is not triggered.
Anyone with this kind of experience who can help?
Thanks
Joe
I've found the cause of the issue.
In order for GitHub to trigger GitLab as CD/CI mostly in PR request, you need to have a Silver/Premium account AND, very important, being the root owner.
Any other case, you won't be able to see github in the integration list on GitLab. People from gitlab had the brilliant idea to hide it instead of showing it disabled (which would had been a tip to understand that you needed an upgraded license)
In the video above it's not explained.
Firstly, you need to give us the content of your .gitlab-ci.yaml file. In your question you asked about GitHub but you're following Gitlab documentation which is completely different. Both are using git commands to commit and push repos but Github & Gitlab are different.
For Github pipelines, you need to create a repository, then you go to Actions. Github will propose you to configure a .github/workflows directory which contain a file.yaml. In this .yaml file you can code your pipelines. According to your project, Github will propose you several linux machines with the adequate configuration to run your files (If it's a Java Project --> you'll be proposed maven machines, Python --> Python Machines, React/Angular -> machines with npm installed, Docker, Kubernetes for deployments...) and you're limited to 4 private project as far as I know (check this last information).
For Gitlab you have two options, you can use preconfigured machines like github, and you call them by adding for example atag: npm in your .gitlab-ci.yaml file, to call a machine with npm installed, but you need to pay an amount of money. Or you can configure your own runners by following the Gitlab documentation with gitlab commands (which is the best option), but you'll need good machines and servers to run npm - mvn - python3 - ... commands
Of course, in your Gitlab repository, and finally to answer your question this an example, of .gitlab-ci.yaml file with two simple stages: build & test, the only statement specifies that these pipelines will run if there is a merge request ( I use the preconfigured machines of Gitlab as a sample here) More details on my python github project https://github.com/mehdimaaref7/Scrapping-Sentiment-Analysis and for gitlab https://docs.gitlab.com/runner/
stages:
- build
- test
build:
tags:
- shell
- linux
stage: build
script:
- echo "Building"
- mkdir build
- touch build/info.txt
artifacts:
paths:
- build/
only:
- merge_requests
test:
tags:
- shell
- linux
stage: test
script:
- echo "Testing"
- test -f "build/info.txt"
only:
- merge_requests
I currently have TravisCI building on PRs in a public GitHub repo.
The instructions for Coveralls say to put this in a .coveralls.yml file:
service_name: travis-pro
repo_token: <my_token>
That doesn't work for me because the .coveralls.yml file would be public--checked into GitHub. My TravisCI is integrated into my GitHub repo wired to a branch and fires on PR.
So I tried this:
In TravisCI's site I set an environment var:
COVERALLS_REPO_TOKEN to my token's value.
Then modded my .travis.yml to look like this:
language: scala
scala:
- 2.11.7
notifications:
email:
recipients:
- me#my_email.com
jdk:
- oraclejdk8
script: "sbt clean coverage test"
after_success: "sbt coverageReport coveralls"
script:
- sbt clean coverage test coverageReport &&
sbt coverageAggregate
after_success:
- sbt coveralls
Now when I create a PR on the branch this runs ok--no errors and I see output in Travis' console that the coverage test ran and generated files. But when I go to Coveralls I see nothing--"There have been no builds for this repo."
How can I set this up?
EDIT: I also tried creating a .coveralls.yml with just service_name: travis-ci
No dice, sadly.
How can I set this up?
Step 1 - Enable Coveralls
The first thing to do is to enable Coveralls for your repository.
You can do that on their website http://coveralls.io:
go to http://coveralls.io
sign in with your GitHub credentials
click on "Repositories", then "Add Repo"
if the repo isn't listed, yet, then "Sync GitHub Repos"
finally, flip the "enable coveralls" switch to "On"
Step 2 - Setup Travis-CI to push the coverage infos to Coveralls
You .travis.yml file contains multiple entries of the script and after_success sections. So, let's clean that up a bit:
language: scala
scala: 2.11.7
jdk: oraclejdk8
script: "sbt clean coverage test"
after_success: "sbt coveralls"
notifications:
email:
recipients:
- me#my_email.com
Now, when you push, the commands in the script sections are executed.
This is were your coverage data is generated.
When the commands finish successfully the after_success section is executed.
This is were the coverage data is pushed to coveralls.
The .coveralls config file
The .coveralls file is only needed to:
public Travis-CI repos do not need this config file since Coveralls can get the information via their API (via access token exchange)
the repo_token (found on the repo page on Coveralls) is only needed for private repos and should be kept secret. If you publish it, then anyone could submit some coverage data for your repo.
Boils down to: you need the file only in two cases:
to specify a custom location to the files containing the coverage data
or when you are using Travis-Pro and private repositories. Then you have to configure "travis-pro" and add the token:
service_name: travis-pro
repo_token: ...
I thought it might be helpful to explain how to set this up for PHP, given that the question applies essentially to any language that Coveralls supports (and not just Lua).
The process is particularly elusive for PHP because the PHP link on Travis-CI's website points to a password-protected page on Coveralls' site that provides no means by which to login using GitHub, unlike the main Coveralls site.
Equally confusing is that the primary PHP page on Coveralls' site seems to contain overly-complicated instructions that require yet another library called atoum/atoum (which looks to be defunct) and are anything but complete.
What ended-up working perfectly for me is https://github.com/php-coveralls/php-coveralls/ . The documentation is very thorough, but it boils-down to this:
Enable Coveralls for your repository (see Step 1 in the Accepted Answer).
Ensure that xdebug is installed and enabled in PHP within your Travis-CI build environment (it should be by default), which is required for code-coverage support in PHPUnit.
Add phpunit and the php-coveralls libraries to the project with Composer:
composer require phpunit/phpunit php-coveralls/php-coveralls
Update travis.yml at the root of the project to include the following directives:
script:
- mkdir -p build/logs
- vendor/bin/phpunit tests --coverage-clover build/logs/clover.xml
after_success:
- travis_retry php vendor/bin/php-coveralls
Create .coveralls.yml at the root of the project and populate it with:
service_name: travis-ci
I'm not positive that this step is necessary for public repositories (the Accepted Answer implies that it's not), but the php-coveralls documentation says of this directive (emphasis mine):
service_name: Allows you to specify where Coveralls should look to find additional information about your builds. This can be any string, but using travis-ci or travis-pro will allow Coveralls to fetch branch data, comment on pull requests, and more.
Push the above changes to the remote repository on GitHub and trigger a Travis-CI build (if you don't already have hooks to make it happen automatically).
Slap a Coveralls code-coverage badge in your README (or wherever else you'd like). The required markup may be found on the Coveralls page for the repository in question, in the Badge column.
I have generated documentation for my project with cargo doc, and it is made in the target/doc directory. I want to allow users to view this documentation without a local copy, but I cannot figure out how to push this documentation to the gh-pages branch of the repository. Travis CI would help me automatically do this, but I cannot get it to work either. I followed this guide, and set up a .travis.yml file and a deploy.sh script. According to the build logs, everything goes fine but the gh-pages branch never gets updated. My operating system is Windows 7.
It is better to use travis-cargo, which is intended to simplify deploying docs and which also has other features. Its readme provides an example of .travis.yml file, although in the simplest form it could look like this:
language: rust
sudo: false
rust:
- nightly
- beta
- stable
before_script:
- pip install 'travis-cargo<0.2' --user && export PATH=$HOME/.local/bin:$PATH
script:
- |
travis-cargo build &&
travis-cargo test &&
travis-cargo --only beta doc
after_success:
- travis-cargo --only beta doc-upload
# needed to forbid travis-cargo to pass `--feature nightly` when building with nightly compiler
env:
global:
- TRAVIS_CARGO_NIGHTLY_FEATURE=""
It is very self-descriptive, so it is obvious, for example, what to do if you want to use another Rust release train for building docs.
In order for the above .travis.yml to work, you need to set your GH_TOKEN somehow. There are basically two ways to do it: inside .travis.yml via an encrypted string, or by configuring it in the Travis itself, in project options. I prefer the latter way, so I don't need to install travis command line tool or pollute my .travis.yml (and so the above config file does not contain secure option), but you may choose otherwise.
I'm wondering if there are any convenient ways to automate deployment of code to a live server in GO, either standard built-in methods, or otherwise.
I want something google app engine like, I just run the command and it uploads to the server and triggers a restart.
(Ultimately I want a git commit to trigger a rebuild and redeploy, but thats for down the track in the future)
I recommend Travis CI + Heroku.
You can deploy to heroku directly with just a git push, but I like to use Travis to build and run the tests before that.
There are some guides online but I'll try to go directly to the point:
What you will need?
Github account
Travis account (linked with github, free if open source)
Empty Heroku app (Free dyno works great)
Setup
In your github repo, create the following files:
.travis.yml (more info on the Travis CI documentation)
Procfile
.go-dir
After that go to your Travis account, add your repository and enabled the build for it.
Here is a sample minimal config file content (based on my app that I deploy to heroku):
.travis.yml
language: go
go:
- tip
deploy:
provider: heroku
buildpack: https://github.com/kr/heroku-buildpack-go.git
api_key:
secure: <your heroku api key encripted with travis encrypt>
on: master
Procfile
worker: your-app-binary
.go-dir
your-app-binary
Procfile and .go-dir are heroku configs so it can vary if you are deploying a web app, you can read more at the heroku documentation
One important and easily missed point is the build pack, without it the deploy will not work.
Read the Travis docs to see how to encrypt the heroku key
How it works?
Basically, every push to your repository will trigger the Travis CI build, if it passes it will deploy the app to heroku, so you set this up once and build + deploy is just a push away ;)
Also Travis will build and updated the status of all Pull Requests to your repository automagically.
To see my config and build, please take a look at my Travis build and my repository with my working configs