I am trying to use AWS CodeBuild for building my code from github. These are the steps I followed so far,
1) Created a windows docker image with all the pre-req software
needed (git, npm, node.js etc) and pushed to Amazon ECS.
2)Created a project in AWS CodeBuild using
a) github as the source (What to build)
b) docker image created in Step 1 (How to build)
I setup buildspec.yml as below:
env:
#variables:
#parameter-store:
phases:
#install:
#pre_build:
build:
commands:
- git clone https://github.com/OrgName/RepName.git "c:\www\localfolder"
#post_build:
#artifacts:
#files:
But this is always failing during DOWNLOAD_SOURCE STEP saying "CodeBuild is experiencing Issues"
Please suggest how to setup buildspec.yml for github clone\fetch\checkout purpose.
Thanks.
The issue you encountered may not be related to git clone\fetch\checkout failure. The build could also fail at "DOWNLOAD_SOURCE" step if CodeBuild failed/timed out when pulling the Windows Docker image; especially when the image is large.
Workarounds you can try:
1) use the windows image provided by CodeBuild and install the pre-req software during the install phase. (you will need to update your buildspec.yml)
OR
2) use a BUILD_GENERAL1_LARGE instance. maybe you will need to increase the timeout too.
Related
I'm trying to use GitHub to trigger on PR a GitLab pipeline.
Practically when a developer creates a PR in GitHub, his/her code get tested against a GitLab pipeline.
I'm trying to follow this user guide: https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/github_integration.html
and we have a silver account, but it won't work. When creating the PR, the GitLab pipeline is not triggered.
Anyone with this kind of experience who can help?
Thanks
Joe
I've found the cause of the issue.
In order for GitHub to trigger GitLab as CD/CI mostly in PR request, you need to have a Silver/Premium account AND, very important, being the root owner.
Any other case, you won't be able to see github in the integration list on GitLab. People from gitlab had the brilliant idea to hide it instead of showing it disabled (which would had been a tip to understand that you needed an upgraded license)
In the video above it's not explained.
Firstly, you need to give us the content of your .gitlab-ci.yaml file. In your question you asked about GitHub but you're following Gitlab documentation which is completely different. Both are using git commands to commit and push repos but Github & Gitlab are different.
For Github pipelines, you need to create a repository, then you go to Actions. Github will propose you to configure a .github/workflows directory which contain a file.yaml. In this .yaml file you can code your pipelines. According to your project, Github will propose you several linux machines with the adequate configuration to run your files (If it's a Java Project --> you'll be proposed maven machines, Python --> Python Machines, React/Angular -> machines with npm installed, Docker, Kubernetes for deployments...) and you're limited to 4 private project as far as I know (check this last information).
For Gitlab you have two options, you can use preconfigured machines like github, and you call them by adding for example atag: npm in your .gitlab-ci.yaml file, to call a machine with npm installed, but you need to pay an amount of money. Or you can configure your own runners by following the Gitlab documentation with gitlab commands (which is the best option), but you'll need good machines and servers to run npm - mvn - python3 - ... commands
Of course, in your Gitlab repository, and finally to answer your question this an example, of .gitlab-ci.yaml file with two simple stages: build & test, the only statement specifies that these pipelines will run if there is a merge request ( I use the preconfigured machines of Gitlab as a sample here) More details on my python github project https://github.com/mehdimaaref7/Scrapping-Sentiment-Analysis and for gitlab https://docs.gitlab.com/runner/
stages:
- build
- test
build:
tags:
- shell
- linux
stage: build
script:
- echo "Building"
- mkdir build
- touch build/info.txt
artifacts:
paths:
- build/
only:
- merge_requests
test:
tags:
- shell
- linux
stage: test
script:
- echo "Testing"
- test -f "build/info.txt"
only:
- merge_requests
I have a dotnet core web app built on windows using GitHub Actions workflow steps. The last step is to build and push the container to GitHub packages (using docker build and docker push commands).
docker push of windows container image to GitHub packages always fails with message below:
denied: No matching package_file with sha256 "b9e6fec25718aef5ed18d499b27e43adb524f9ee4f2eb3f0fffaea018e7e86b0" found in repository "myrepo/dotnet-ci".
Is windows container not supported in GitHub packages?
I am successful if I use linux for GitHub Actions to build the dotnet core app for linux, build and push linux container to GitHub packages.
Sadly it appears to be the case that Windows images are not supported by the GitHub registry: https://docs.github.com/en/packages/using-github-packages-with-your-projects-ecosystem/configuring-docker-for-use-with-github-packages
Note: When installing or publishing a docker image, GitHub Packages does not currently support foreign layers, such as Windows images.
dockerhub provides automated build option on every new git push.
With automated build option,
we know, which image corresponds to which docker filethis is a important.
On git push of Dockerfile in GitHub repo or Bitbucket, build starts immediately.
1)
Does AWS ECR provide such automated build option? because I don't prefer the approach of running docker push through some automation
2)
if yes, can the automated build support docker-compose build?
ECR does not contain a build tool. But you can use AWS Codebuild AWS's continuous integration tool. I am pretty sure it supports both of your requirements.
I have several Scala applications that I want to deploy in a Docker multi-container environment on Amazon's Elastic Beanstalk.
It seems like the whole process is a bit more complicated that I was expecting. So I'm really looking forward to hear some feedback for best practices and other ways to improve my entire process and be able to "automate" some steps (if possible).
This is my current process:
To generate my projects' artifacts I'm using the sbt-docker plugin. This
plugin generates the projects artifacts (jars and Dockerfile) under
[app-route]/target/docker.
I upload these artifacts (jars and Dockerfile) into a git
repository (currently doing this "manually").
As Amazon's Elastic Beanstalk requires for Docker
multi-containers, I need an online repository to "host" the
images: Could be Docker-Hub or Quay.io. Either require me
to have a git repository in which it can find the artifacts to be
able to generate the project's image.
Having created the multi-container environment in Elastic Beanstalk,
I proceed to upload the Dockerrun.aws.json file as detailed in
Amazon's documentation and also the
.ebextensions/elb-listeners.config file with the settings of the
ports (Since I'm running multiple apps)
Magic! Amazon generates my environment. Same url, different ports
for all my apps (as specified in the configuration files in step
4.
I would love to find a way to automate step 2. Since this requires me to have an extra repo per each app. I have my apps hosted in a git repo, and I have an "extra" repo per each where I host the artifacts generated in step 1 to be able to do step 3.
If you're willing to use a different SBT plugin for step 1, then you can automate step 2.
Although quay.io supports building your image from GitHub, they do not require it. (You can publish a local Docker image directly to your quay.io repository.)
Use the sbt-native-packager plugin in project/plugins.sbt.
Setup the plugin settings in build.sbt, like: dockerRespository := Some("quay.io/myaccount")
Your step 1 becomes: sbt docker:stage
Followed by: sbt docker:publishLocal
Check your image names and tags with docker images. The new image should have a name like quay.io/myaccount/app
Before you can publish to quay.io, you must docker login quay.io. Read their tutorial.
Your step 2 becomes sbt docker:publish. Now your quay.io account should contain the same IMAGE ID as your local Docker daemon.
Proceed with steps 3+ on the AWS side...
I am not really familiar with Scala however I believe the artifacts could be generated by Jenkins/CircleCI inside of your container that is built on Jenkins/CircleCI then the appropriate image tags referenced within your Dockerrun.aws.json.
Hope that helps.
I'm wondering if there are any convenient ways to automate deployment of code to a live server in GO, either standard built-in methods, or otherwise.
I want something google app engine like, I just run the command and it uploads to the server and triggers a restart.
(Ultimately I want a git commit to trigger a rebuild and redeploy, but thats for down the track in the future)
I recommend Travis CI + Heroku.
You can deploy to heroku directly with just a git push, but I like to use Travis to build and run the tests before that.
There are some guides online but I'll try to go directly to the point:
What you will need?
Github account
Travis account (linked with github, free if open source)
Empty Heroku app (Free dyno works great)
Setup
In your github repo, create the following files:
.travis.yml (more info on the Travis CI documentation)
Procfile
.go-dir
After that go to your Travis account, add your repository and enabled the build for it.
Here is a sample minimal config file content (based on my app that I deploy to heroku):
.travis.yml
language: go
go:
- tip
deploy:
provider: heroku
buildpack: https://github.com/kr/heroku-buildpack-go.git
api_key:
secure: <your heroku api key encripted with travis encrypt>
on: master
Procfile
worker: your-app-binary
.go-dir
your-app-binary
Procfile and .go-dir are heroku configs so it can vary if you are deploying a web app, you can read more at the heroku documentation
One important and easily missed point is the build pack, without it the deploy will not work.
Read the Travis docs to see how to encrypt the heroku key
How it works?
Basically, every push to your repository will trigger the Travis CI build, if it passes it will deploy the app to heroku, so you set this up once and build + deploy is just a push away ;)
Also Travis will build and updated the status of all Pull Requests to your repository automagically.
To see my config and build, please take a look at my Travis build and my repository with my working configs