I'm trying to achieve a dynamic deployment with a gitlab runner.
I already added a runner in my config.toml:
[[runners]]
name = "Deploy-Runner"
url = "https://my-gitlab.server.com/ci"
token = "my_super_secret_token"
executor = "ssh"
[runners.ssh]
host = "$HOSTNAME"
user = "deployuser"
identity_file = "/home/deployuser/.ssh/id_rsa.pub"
In my .gitlab-ci.yml I would then write maybe:
variables:
HOSTNAME: preprod.server.com
deploy to preprod:
stage: deploy
script: cd to_my_repo && git pull
environment:
name: preprod
Obviously I added the public key of my deployuser to the preprod.
But what I get is that he can't find the ssh key, which is funny because it is there!
Using SSH executor...
ERROR: Preparation failed: ssh: no key found
Did anyone try to achieve the same. Maybe that isn't even possible?
The error is in identity_file. You have configured the path to de public key, but you must set your rsa private key file.
You have to set:
identity_file = "/home/deployuser/.ssh/id_rsa"
Related
Im publishing a project via docker compose to AWS ECR but it fails on the last couple of steps. Its based on the new "docker compose" integration with an AWS context
The error i receive is:
MicroservicedocumentGeneratorService TaskFailedToStart: ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve ecr registry auth: service call has been retried 3 time(s): RequestError: send request failed caused by: Post https://api.ecr....
The image is in an ECR private repository along with the others from the compose file.
I have authenticated with:
aws ecr get-login-password
The docker compose is:
microservice_documentGenerator:
image: xxx.dkr.ecr.xxx.amazonaws.com/microservice_documentgenerator:latest
networks:
- publicnet
The original dockerfile is
FROM openjdk:11-jdk-slim
COPY /Microservice.DocumentGenerator/Microservice.DocumentGenerator.jar app.jar
ENTRYPOINT ["java","-jar","/app.jar"]
The output for before the error was:
[+] Running 54/54
- projext DeleteComplete 355.3s
- PublicnetNetwork DeleteComplete 310.5s
- LogGroup DeleteComplete 306.1s
- MicroservicedocumentGeneratorTaskExecutionRole DeleteComplete 272.2s
- MicroservicedocumentGeneratorTaskDefinition Del... 251.2s
- MicroservicedocumentGeneratorServiceDiscoveryEntry DeleteComplete 220.1s
- MicroservicedocumentGeneratorService DeleteComp... 211.9s
try authentication with:
aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com
Plus can you mention from where you are making the call and if the server has the permission to make the call to ECR?
I have a java maven project that I am building with an azure pipeline with as host "windows-latest" as it contains the correct java 13 version. However, for the integration tests, I need a postgres db and the "windows-latest" agent contains a postgres db, see: link. But how can I use this? I tried to use it by including it's serviceName in the Maven task as service:
services:
postgres: postgresql-x64-13
But then I get the error it can not find a service by that name.
I tried defining the db properties through env settings (see yml below), and then it shows the error:
Caused by: java.net.ConnectException: Connection refused
I also tried running it through a script task through the docker-compose.yml in the root of the project that I use during development, but docker-compose throws an error saying it can't find the compose file, I also doubt this the correct way.
So can I use the postgres db on the windows agent? and how?
My azure pipeline snippet:
variables:
MAVEN_CACHE_FOLDER: $(Pipeline.Workspace)/.m2/repository
MAVEN_OPTS: "-Dmaven.repo.local=$(MAVEN_CACHE_FOLDER)"
application_name: clearsky
service_name: backend
mygetUsername: myserUsername
mygetPassword: mytoken
SPRING_DATASOURCE_URL: jdbc:postgresql://localhost:5432/postgres
SPRING_DATASOURCE_USER: postgres
SPRING_DATASOURCE_PASSWORD: root
stages:
- stage: create_artifact
displayName: Create artifact
jobs:
- job: build
displayName: Build, test and publish artifact
steps:
- task: Maven#3
name: maven_package
displayName: Maven package
inputs:
goals: "package"
mavenPomFile: "backend/pom.xml"
options: '--settings backend/.mvn/settings.xml -DmygetUsername=$(mygetUsername) -DmygetPassword=$(mygetPassword)'
mavenOptions: "-Xmx3072m $(MAVEN_OPTS)"
javaHomeOption: "JDKVersion"
jdkVersionOption: "1.13"
mavenAuthenticateFeed: true
In Azure Devops Windows agen, the postgresql is disabled/stop by default.
Here is the configuration doc.
Property Value
ServiceName postgresql-x64-13
Version 13.2
ServiceStatus Stopped
ServiceStartType Disabled
You could try the following command to start the postgresql.
"C:\Program Files\PostgreSQL\13\bin\pg_ctl.exe" start -D "C:\Program Files\PostgreSQL\13\data" -w
I'm trying to install packages from a private repository I've deployed using chart-releases, but I'm not being able to do it.
Here is what I've done:
I've created a new private repository, added a sample chart to it and ran the following commands:
helm package charts/* --destination .deploy
cr upload -o odelucca -r helm-charts -p .deploy -t $MY_TOKEN
I've created the index.yaml with the following command:
cr index --config .cr.yaml -t $MY_TOKEN
# My .cr.yaml file:
# owner: odelucca
# git-repo: helm-charts
# package-path: .deploy
# index-path: index.yaml
# charts-repo: https://github.com/odelucca/helm-charts/
I've commit the index.yaml to the repo
I've added the remote helm repo with the following command:
helm repo add helm-charts https://raw.githubusercontent.com/odelucca/helm-charts/master --username $MY_EMAIL --password $MY_TOKEN
The repo was added, then I've added the following dependency to a local chart:
dependencies:
- name: serverless-common
version: 1.0.0
repository: "#helm-charts"
Now, I've tried to run the following:
helm dep update
I get the following errors:
Hang tight while we grab the latest from your chart repositories...
...Unable to get an update from the "local" chart repository (http://127.0.0.1:8879/charts):
Get http://127.0.0.1:8879/charts/index.yaml: dial tcp 127.0.0.1:8879: connect: connection refused
...Successfully got an update from the "helm-charts" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete.
Saving 1 charts
Downloading serverless-common from repo https://raw.githubusercontent.com/odelucca/helm-charts/master
Save error occurred: could not download https://github.com/odelucca/helm-charts/releases/download/serverless-common-1.0.0/serverless-common-1.0.0.tgz: Failed to fetch https://github.com/odelucca/helm-charts/releases/download/serverless-common-1.0.0/serverless-common-1.0.0.tgz : 404 Not Found
Deleting newly downloaded charts, restoring pre-update state
Error: could not download https://github.com/odelucca/helm-charts/releases/download/serverless-common-1.0.0/serverless-common-1.0.0.tgz: Failed to fetch https://github.com/odelucca/helm-charts/releases/download/serverless-common-1.0.0/serverless-common-1.0.0.tgz : 404 Not Found
Anyone can help me? I've tried a lot of different approaches, and none of them fix it
Have you tried editing the repository config file?
You should have something like this:
vim ${HOEM}/.config/helm/repositories.yaml
apiVersion: ""
generated: "0001-01-01T00:00:00Z"
repositories:
- caFile: ""
certFile: ""
insecure_skip_tls_verify: false
keyFile: ""
name: helm-charts
password: ""
url: "https://raw.githubusercontent.com/odelucca/helm-charts/master"
username: ""
Edit it and put your username and password to connect to your private registry.
For me, this works fine.
Maybe you also need to check your repo path.
You should be able to download with an URL like this:
${repository}/${name}-${version}.tgz
This post contins some information about how we integrated SonarQube in our workflow using Docker and Saltslack as Docker Container Configuration Management.
It also contains the setup used with Gradle in Travis-CI in order to execute analysis of code and analysis of Pull Requests on Github.
Also, if you see any improvements to this setup, please comment!
(If using Docker Compose, see https://github.com/SonarSource/docker-sonarqube. Feel free to maintain this answer here or copy it to a SCM.)
Requires Docker Engine 1.9
Setting up a SonarQube Server using Salt
Create this pillar file applicable for your SonarQube server:
sonar-qube:
name: sonar-qube
port: 9000
version: <ENTER SOME VERSION>
version_postgresql: <ENTER SOME VERSION>
# Using a shared disk allows you to move the SonarQube container between different servers and still keep the data.
host_storage_path: /some/shared/disk
Create this sonarqube.sls as your Docker State file.
(It requires you to have a network created named sonarnet configured in a configuration named sonarnet-config)
{% set name = salt['pillar.get']('sonar-qube:name') %}
{% set port = salt['pillar.get']('sonar-qube:port') %}
{% set tag = salt['pillar.get']('sonar-qube:version') %}
{% set pg_tag = salt['pillar.get']('sonar-qube:version_postgresql') %}
{% set host_storage_path = salt['pillar.get']('sonar-qube:host_storage_path') %}
include:
- <state file of the sonarnet-config network definition>
sonar-qube-image:
dockerng.image_present:
- name: sonarqube:{{tag}}
sonar-qube:
dockerng.running:
- name: {{name}}
- image: sonarqube:{{tag}}
- network_mode: sonarnet
- port_bindings:
- {{port}}:{{port}}
- environment:
- SONARQUBE_JDBC_URL: jdbc:postgresql://sonar-db:5432/sonar
- binds:
- {{host_storage_path}}/sonarqube/conf:/opt/sonarqube/conf
- {{host_storage_path}}/sonarqube/data:/opt/sonarqube/data
- {{host_storage_path}}/sonarqube/extensions:/opt/sonarqube/extensions
- {{host_storage_path}}/sonarqube/lib/bundled-plugins:/opt/sonarqube/lib/bundled-plugins
- require:
- dockerng: sonarnet-config
sonar-db:
dockerng.running:
- image: postgres:{{pg_tag}}
- network_mode: sonarnet
- port_bindings:
- 5432:5432
- environment:
- POSTGRES_USER: sonar
- POSTGRES_PASSWORD: sonar
- binds:
- {{host_storage_path}}/postgresql:/var/lib/postgresql
# This needs explicit mapping due to https://github.com/docker-library/postgres/blob/4e48e3228a30763913ece952c611e5e9b95c8759/Dockerfile.template#L52
- {{host_storage_path}}/postgresql/data:/var/lib/postgresql/data
- require:
- dockerng: sonarnet-config
Use regular salt to start your containers.
Once this SonarQube server is started, you should be able to reach the web gui of SonarQube.
Execute automated analysis (with Gradle in Travis CI)
These bullests will be described one by one
Enable Gradle plugin
Create users at SonarQube and Github
Write a bash script that executes analysis
Invoke bash script from Travis CI.
1) Enable the Gradle plugin
Enable the plugin according to documentation at https://plugins.gradle.org/plugin/org.sonarqube
plugins {
id "org.sonarqube" version "2.0.1"
}
2) Setup users in Github and Sonar
Github requires a user with write access (soon only read access?) to the repo. Create a sonar-ci user to a team, and provide write access to the repo for the team. See this post: https://github.com/janinko/ghprb/issues/232#issuecomment-149649126 Then create an access token for that user, the access token must grant "Full control of private repositories".
Sonar requires a user that has permission to "Execute Analysis" and "Create Projects" under Global Permissions. It also needs permissions to "BROWSE", "SEE SOURCE CODE" and "EXECUTE ANALYSIS" under Project Permissions. Generate an access token for this user.
3) Write bash script
This script will do a full analysis and publish the result at the SonarQube web GUI when merged to git branch master. This keeps track of the code evolvement over time. It will also analyze pull requests in github and write its findings directly as review comments.
These env variables needs to be set:
TRAVIS_*- set by Travis: see https://docs.travis-ci.com/user/environment-variables/
SONAR_TOKEN is the access token for the sonar server
GITHUB_SONAR_TOKEN is the access token for the sonar alaysis user on Github
sonarqube.sh:
SONAR_URL="https://sonar.example.com"
if [ -z "$SONAR_TOKEN" ] || [ -z "$GITHUB_SONAR_TOKEN" ]; then
echo "Missing environemnt variable(s) for SonarQube. Make sure all environment variables are set."
exit 1
fi
if [ "$TRAVIS_PULL_REQUEST" != "false" ]; then
echo "Running SonarQube analysis for pull request nr $TRAVIS_PULL_REQUEST..."
./gradlew sonarqube \
-Dsonar.host.url=$SONAR_URL \
-Dsonar.login=$SONAR_TOKEN \
-Dsonar.github.pullRequest=$TRAVIS_PULL_REQUEST \
-Dsonar.github.repository=$TRAVIS_REPO_SLUG \
-Dsonar.github.oauth=$GITHUB_SONAR_TOKEN \
-Dsonar.analysis.mode=issues
elif [ "$TRAVIS_BRANCH" == "master" ]; then
echo "Starting publish SonarQube analyzis results to $SONAR_URL"
./gradlew sonarqube \
-Dsonar.host.url=$SONAR_URL \
-Dsonar.login=$SONAR_TOKEN \
-Dsonar.analysis.mode=publish
fi
4) Integrate from Travis CI
In the .travis.yml add:
after_success:
- ./sonarqube.sh
before_cache:
- rm -rf $HOME/.gradle/caches/*/gradle-sonarqube-plugin
cache:
directories:
- $HOME/.sonar
i'm try to use wercker,
but i don't know my testing can't connect into my mongodb.
i'm using sails + sails mongo, and when npm test...i'm always get error can connect into mongo db, this is my wercker.yml :
box: nodesource/trusty:0.12.7
services:
- id: mongo:2.6
# Build definition
build:
# The steps that will be executed on build
steps:
- script:
name: set NODE_ENV
code: export NODE_ENV=development
# A step that executes `npm install` command
- npm-install
# A step that executes `npm test` command
- npm-test
# A custom script step, name value is used in the UI
# and the code value contains the command that get executed
- script:
name: echo nodejs information
code: |
echo "node version $(node -v) running"
echo "npm version $(npm -v) running"
this is my error message :
warn: `sails.config.express` is deprecated; use `sails.config.http` instead.
Express midleware for passport
error: A hook (`orm`) failed to load!
1) "before all" hook
2) "after all" hook
0 passing (2s)
2 failing
1) "before all" hook:
Uncaught Error: Failed to connect to MongoDB. Are you sure your configured Mongo instance is running?
Error details:
{ [MongoError: connect ECONNREFUSED] name: 'MongoError', message: 'connect ECONNREFUSED' }
at net.js:459:14
2) "after all" hook:
Uncaught Error: Failed to connect to MongoDB. Are you sure your configured Mongo instance is running?
Error details:
{ [MongoError: connect ECONNREFUSED] name: 'MongoError', message: 'connect ECONNREFUSED' }
at net.js:459:14
While out of the box, MongoDB has no authentication so you just have to provide to sails the right host and port.
Define a new connection in your sails app in config/connection.js:
mongodbTestingServer: {
adapter: 'sails-mongo',
host: process.env.MONGO_PORT_27017_TCP_ADDR,
port: process.env.MONGO_PORT_27017_TCP_PORT
},
Concerning MONGO_PORT_27017_TCP_ADDR and MONGO_PORT_27017_TCP_PORT, these 2 environment variable are created by Wercker when you declared a mongo service. Like That, you will be able to connected your application to your database with the right host and port.
Add a new environment in your sails sails app in config/env/testing.js. It will be used by Wercker :
module.exports = {
models: {
connection: 'mongodbTestingServer'
}
};
In your wercker file wercker.yml. I recommend you to use the ewok stack (based on Docker), you can active it in the settings of your application. Here is some useful informations concerning migration to Ewok stack. My example use a box based on a Docker image.
# use the latest official stable node image hosted on DockerHub
box: node
# use the mongo (v2.6) image hosted on DockerHub
services:
- id: mongo:2.6
# Build definition
build:
steps:
# Print node and npm version
- script:
name: echo nodejs information
code: |
echo "node version $(node -v) running"
echo "npm version $(npm -v) running"
- script:
name: set NODE_ENV
code: |
export NODE_ENV=testing
# install npm dependencies of your project
- npm-install
# run tests
- npm-test
To see all environment variables in your Wercker build, add this line :
- script:
name: show all environment variables
code: |
env
It should work.