I have a bash script for deploying my app to my production server, how can I use GitHub to automate when someone pushes to master? - github

The script is in ./bin/deploy.sh and uses an SSH key to connect to the server.
I have a script in package.json called npm run deploy:prod which runs this bash script.
What do I need to get GitHub to run this script automatically when someone pushes or merges a PR to master?
Here is my deploy.sh script:
#!/usr/bin/env bash
. $HOME/.bashrc
. .env
. .env.local
args=(-azvP --delete --exclude=node_modules --exclude=.idea --exclude=.git)
hosts=($HOST_DOMAIN) # tornado lightning thunder tundra jefferson
dry=() #add --dry-run to enable testing
user=$HOST_USER
name=$HOST_PATH
project=$HOST_PROJECT
for host in "${hosts[#]}"
do
echo ""
date
echo "---------------------"
echo "syncing ${host}"
echo "---------------------"
rsync ${dry[#]} ${args[#]} ./ ${user}#${host}:www/${name}/${project}
ssh -t ${user}#${host} \$HOME/www/${name}/${project}/bin/post-deploy.sh
done
version=$(jq -r .version package.json)
say "$HOST_PROJECT is live!"
exit
Here is my post-deploy.sh script that gets executed on the server:
#!/usr/bin/env bash
cd "$(dirname "$0")/.."
. $HOME/.bashrc
. .env
. .env.local
host=$HOST_DOMAIN
name=$HOST_PATH
project=$HOST_PROJECT
echo "current name: $name"
cd $HOME/www/${name}/${project}
nvm install v18
node -v
npm -v
npm i
sudo /etc/init.d/nginx reload
sudo systemctl daemon-reload
sudo systemctl restart ${META_SERVICE}

Related

gitlab ci does not update code at remote server

I want to deploy ma test app from local repo to gitlab repo and with gitlab ci push it to my remote server. SSH connection is working, gitlab CI shows that job is passed, but code on remote server is not updated.
I made bare repo in: /home/repos/testDeploy.git
And folder for files is in: /home/example.com/web/testDeploy
I added
My .gitlab-ci.yml file
stages:
- deploy
deployment:
stage: deploy
environment:
name: production
url: http://www.example.com/testDeploy
only:
- master
before_script:
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- chmod 600 ~/.ssh/id_rsa_gitlab && chmod 700 ~/.ssh
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
script:
- git remote add production ssh://user#server:port/home/repos/testDeploy.git
- git push -f production master
- echo "Deployed to production!"
Also, i have post-receive hook:
#!/bin/sh
git --git-dir=/home/repos/testDeploy.git --work-tree=/home/example.com/web/testDeploy checkout -f
I make changes in my local repo, commit and push to origin master to gitlab. Job is passed, but as I mention above, file on remote server is not update.
Output from gitlab job is:
Fetching changes...
HEAD is now at 595db67 as
Checking out 595db67b as master...
Skipping Git submodules setup
$ which ssh-agent || ( apt-get update -y && apt-get install openssh- client -y )
/usr/bin/ssh-agent
$ eval $(ssh-agent -s)
Agent pid 40589
$ chmod 600 ~/.ssh/id_rsa_gitlab && chmod 700 ~/.ssh
$ [[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
$ git branch
* (HEAD detached at 595db67)
master
production
$ git push -f production master
Everything up-to-date
$ echo "Deployed to production!"
Deployed to production!
Job succeeded
What I am doing wrong? Can you someone help me please to figure it out? Thank you for all your answers.

Install MongoDB on Debian Buster

How to install the latest MongoDB 3.4 or even 3.6?
They support with Ubuntu, but my server is Debian Buster and I am stuck with MongoDB 3.2.
I don't know if this is a good idea yet, but I just installed it by adding the sid repos and installing using the mongodb-server package. For me this installs version 3.4.18.
I created /etc/apt/sources.list.d/sid.list with:
deb http://deb.debian.org/debian/ sid main
deb-src http://deb.debian.org/debian/ sid main
then did
apt update
apt install mongodb-server
and verified that it's working by connecting with mongo.
I have found the solution for a build script, the description is found here:
https://github.com/patrikx3/docker-debian-testing-mongodb-stable
The description:
Debian Stretch / Buster / Bullseye / Testing MongoDB and MongoDB Tools build stable builder script, what it does as exactly:
It is basically a built for the latest MongoDB for Debian.
The current varsion is the r4.0.x build (release).
Warning It will remove all mongodb* apt packages in ./scripts/build-server.sh and /etc/systemd/system/mongodb-server.service is replaced.
It install the required apt dependencies and generates the SystemD service and makes it enabled.
Check if the build works (building is below). It runs all tests, so if it works, then it really does, actually. If there is an error, of course, you will not deploy on your server. So, if building and testing works, then it puts the binaries as it follow and you are sure and done.
The build as follows build-server.sh:
#!/usr/bin/env bash
# based on https://github.com/mongodb/mongo/wiki/Build-Mongodb-From-Source
# the current directory
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
# if an error exit right away, don't continue the build
set -e
# some info
echo
#echo "Works like command, use a tag: sudo ./scripts/build-server.sh r4.2.0"
echo "Works like command, use a tag: sudo ./scripts/build-server.sh r4.0.12"
echo
# check if we are root
if [[ $EUID -ne 0 ]]; then
echo "This script must be ran via root 'sudo' command or using in 'sudo -i'."
exit 1
fi
# require mongo branch
#if [ -z "${1}" ]; then
# echo "First argument must be the MONGODB_BRANCH for example 'v4.1'"
# exit 1
#fi
#MONGODB_BRANCH="${1}"
# require mongo release
#if [ -z "${2}" ]; then
# echo "The second argument must be the MONGODB_RELEASE for example 'r4.1.0'"
# exit 1
#fi
#MONGODB_RELEASE="${2}"
# require mongo release
if [ -z "${1}" ]; then
echo "The first argument must be the MONGODB_RELEASE for example 'r4.0.12'"
exit 1
fi
MONGODB_RELEASE="${1}"
# delete all mongo other programs, we self compile
apt remove --purge mongo*
# the required packages for debian
apt -y install libboost-filesystem-dev libboost-program-options-dev libboost-system-dev libboost-thread-dev build-essential gcc python scons git glibc-source libssl-dev python-pip libffi-dev python-dev libcurl4-openssl-dev #libcurl-dev
pip install -U pip pyyaml typing
# generate build directory variable
BUILD=$DIR/../build
# delete previous build directory
rm -rf $BUILD/mongo
# generate new build directory
mkdir -p $BUILD
# the mongodb.conf and systemd services files in a directory variable
ROOT_FS=$DIR/../artifacts/root-filesystem
# find out how many cores we have and we use that many
if [ -z "$CORES" ]; then
CORES=$(grep -c ^processor /proc/cpuinfo)
fi
echo Using $CORES cores
# go to the build directory
pushd $BUILD
# clone the mongo by branch
#git clone -b ${MONGODB_BRANCH} https://github.com/mongodb/mongo.git
# clone the mongo by branch
git clone https://github.com/mongodb/mongo.git
# the mongo directory is a variables
MONGO=$BUILD/mongo
# go to the mongo directory
pushd $MONGO
# checkout the mongo release
git checkout tags/${MONGODB_RELEASE}
# hack to old version python pip cryptography from 1.7.2 to use the latest
sed -i 's#cryptography == 1.7.2#\#cryptography == 1.7.2#g' buildscripts/requirements.txt
# this is only because 4.0.12 uses 1.7.2 and
# https://github.com/pyca/cryptography/issues/4193#issuecomment-381236459
# support minimum latest (2.2)
pip install cryptography
# install the python requirements
#pip install -r etc/pip/dev-requirements.txt
pip install -r buildscripts/requirements.txt
# somewhere in the build it says if we install this, it is faster to build
pip2 install --user regex
# build everything
scons all --disable-warnings-as-errors -j $CORES --ssl
# install the mongo programs all
scons install --disable-warnings-as-errors -j $CORES --prefix /usr
# create a copy of the old config
#TIMESTAMP=$(($(date +%s%N)/1000000))
#cp /etc/mongodb.conf /etc/mongodb.conf.$TIMESTAMP.save
# copy the mongodb.conf configured and the systemd service file
# dangerous!!! removed
# cp -avr $ROOT_FS/. /
MONGODB_SERVICE=etc/systemd/system/mongodb-server.service
cp $ROOT_FS/$MONGODB_SERVICE /$MONGODB_SERVICE
chown root:root /$MONGODB_SERVICE
chmod o-rwx /$MONGODB_SERVICE
# generate mongodb user and group
useradd mongodb -d /var/lib/mongodb -s /bin/false || true
# create the required mongodb database directory and add safety
mkdir -p /var/lib/mongodb
chmod o-rwx -R /var/lib/mongodb
chown -R mongodb:mongodb /var/lib/mongodb
# create the required mongodb log directory and add safety
mkdir -p /var/log/mongodb
chmod o-rwx -R /var/log/mongodb
chown -R mongodb:mongodb /var/log/mongodb
# create the required run socket directory and add safety
mkdir -p /run/mongodb
chmod o-rwx -R /run/mongodb
chown -R mongodb:mongodb /run/mongodb
# add safety to the mongodb config file
chmod o-rwx /etc/mongodb.conf || true
chown mongodb:mongodb /etc/mongodb.conf || true
# reload systemd services
systemctl daemon-reload
# enable the mongodb-server
systemctl enable mongodb-server
# start the mongodb-server
#service mongodb-server start
# exit of the mongo directory
popd
# exit the build directory
popd
# delete current build directory
rm -rf $BUILD/mongo
The build as follows build-tools.sh:
#!/usr/bin/env bash
# based on https://github.com/mongodb/mongo-tools
# the current directory
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
# if an error exit right away, don't continue the build
set -e
# some info
echo
echo "Works like command: sudo ./scripts/build-tools.sh r4.0.12"
echo
# check if we are root
if [[ $EUID -ne 0 ]]; then
echo "This script must be ran via root 'sudo' command or using in 'sudo -i'."
exit 1
fi
# require mongo release
if [ -z "${1}" ]; then
echo "The first argument must be the MONGODB_RELEASE for example 'r4.0.12'"
exit 1
fi
MONGODB_RELEASE="${1}"
## delete all mongo other programs, we self compile
##apt remove --purge mongo*
## the required packages for debian
##apt -y install gcc python scons git glibc-source libssl-dev python-pip
apt -y install golang libpcap-dev
export GOROOT=$(go env GOROOT)
# generate build directory variable
BUILD=$DIR/../build/src/github.com/mongodb/
# delete previous build directory
rm -rf $BUILD/mongo-tools
# generate new build directory
mkdir -p $BUILD
# find out how many cores we have and we use that many
CORES=$(grep -c ^processor /proc/cpuinfo)
# go to the build directory
pushd $BUILD
# clone the mongo by branch
git clone https://github.com/mongodb/mongo-tools
# the mongo directory is a variables
MONGO_TOOLS=$BUILD/mongo-tools
# go to the mongo directory
pushd $MONGO_TOOLS
# checkout the mongo release
git checkout tags/${MONGODB_RELEASE}
bash ./build.sh
chown root:adm -R ./bin
chmod o-rwx -R ./bin
chmod ug+rx ./bin/*
cp -r ./bin/. /usr/bin
# for PROGRAM in bsondump mongodump mongoexport mongofiles mongoimport mongoreplay mongorestore mongostat mongotop
# do
# go build -o bin/${PROGRAM} -tags "ssl sasl" ${PROGRAM}/main/${PROGRAM}.go
# done
# exit of the mongo directory
popd
# exit the build directory
popd
# delete current build directory
rm -rf $BUILD/mongo-tools
popd
# delete current build directory
rm -rf $BUILD/mongo-tools

Starting services at container startup

I'm trying to run 3 services at my container startup (snmpd, sshd and centengine)
As runlevel is unknown in the container, services won't start.
I built an image with this Dockerfile :
FROM centos:6.7
MAINTAINER nael <me#mail>
# Update CentOS
RUN yum -y update
# Install wget
RUN yum install -y wget
# Get Centreon Repo
RUN wget http://yum.centreon.com/standard/3.0/stable/ces-standard.repo -O /etc/yum.repos.d/ces-standard.repo
# Install Packages (SSH, sudo, Centreon Poller & Engine, SNMP)
RUN yum install -y --nogpgcheck openssh-clients openssh-server centreon-poller-centreon-engine sudo net-snmp net-snmp-utils
# Install supervisord
RUN rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
RUN yum --enablerepo=epel install -y supervisor
RUN mv -f /etc/supervisord.conf /etc/supervisord.conf.org
ADD supervisord.conf /etc/
# For sshd & centengine
EXPOSE 22 5669
# Change user password
RUN echo -e "password" | (passwd --stdin user)
# Disable PAM (causing issues while ssh login)
RUN sed -ri 's/UsePAM yes/#UsePAM yes/g' /etc/ssh/sshd_config
RUN sed -ri 's/#UsePAM no/UsePAM no/g' /etc/ssh/sshd_config
# Start supervisord
CMD ["/usr/bin/supervisord"]
Here is the supervisord.conf file
[supervisord]
nodaemon=true
pidfile=/var/run/supervisord.pid
logfile=/var/log/supervisor/supervisord.log
[program:centengine]
command=service centengine start
[program:snmpd]
command=service snmpd start
[program:sshd]
command=service sshd start
But with this Dockerfile and supervisord.conf, when I start my container theses services aren't running.
What could be the problem ?
Not anymore using supervisord.
I just include a script with all the services ... start commands in the Dockerfile. When I create my container with docker run ... I just specify that I want to start it with my script.
& that's working very well.
Thanks #warmoverflow for trying to solve this.
You may find my dockerfy utility useful starting services, pre-running initialization commands before the primary command starts. See https://github.com/markriggins/dockerfy
For example:
RUN wget https://github.com/markriggins/dockerfy/releases/download/0.2.4/dockerfy-linux-amd64-0.2.4.tar.gz; \
tar -C /usr/local/bin -xvzf dockerfy-linux-amd64-*tar.gz; \
rm dockerfy-linux-amd64-*tar.gz;
ENTRYPOINT dockerfy
COMMAND --start bash -c "while false; do echo 'Ima Service'; sleep 1; done" -- \
--reap -- \
nginx
Would run a bash script as a service, echoing "Ima Service" every second, while the primary command nginx runs. If nginx exits, then the "Ima Service" script will automatically be stopped.
As an added benefit, any zombie processes left over by nginx will be automatically cleaned up.
You can also tail log files such as /var/log/nginx/error.log to stderr, edit nginx's configuration prior to startup and much more

How to start railo service in background on the Docker

My name Trang,
I have created Docker image on https://registry.hub.docker.com/u/trangunghoa/railo-mysql/
It is run ok.
Now I created Dockerfile but I can't start automatic Railo service. Please help me.
I have start by some commands at shell script:
exec /opt/railo/railo_ctl start
exec /opt/railo/railo_ctl start -D FOREGROUND
service railo_ctl restart
exec service railo_ctl restart
No command it work.
I looked inside your Dockerfile and identified the problem.
You can only use one CMD inside a Dockerfile. (if you use multiple CMD the old one will override) More info : https://docs.docker.com/reference/builder/#cmd
You need to know that Docker isn't made for running multiple process without a little bit of help. I suggest using supervisord : https://docs.docker.com/articles/using_supervisord/
You can't use RUN service inside a Dockerfile, the reason is simple the command service will be executed and start a daemon, then notify the execution was successful. The temporary container will be killed (and the daemon too) and after that the change will be committed.
What your Dockerfile should look like :
FROM ubuntu:trusty
MAINTAINER Trang Lee <trangunghoa#gmail.com>, Seta International Vietnam(info#setacinq.vn)
#Install base packages
RUN apt-get -y update
RUN apt-get install -y openjdk-7-jre-headless
RUN apt-get install -y tomcat7 tomcat7-admin apache2 libapache2-mod-jk
RUN apt-get purge -y openjdk-6-jre-headless icedtea-6-jre-cacao openjdk-6-jre-lib icedtea-6-jre-jamvm
RUN apt-get install -y supervisor
# config to enable .htaccess
ADD apache_default /etc/apache2/sites-available/000-default.conf
RUN a2enmod rewrite
ENV APACHE_RUN_USER www-data
ENV APACHE_RUN_GROUP www-data
ENV APACHE_LOG_DIR /var/log/apache2
# start service
ADD start-apache2.sh /start-apache2.sh
ADD railo.sh /railo.sh
ADD run.sh /run.sh
RUN chmod +x /*.sh
#RUN sudo service apache2 start
# install railo
RUN apt-get install -y wget
RUN wget http://www.getrailo.org/railo/remote/download42/4.2.1.000/tomcat/linux/railo-4.2.1.000-pl2-linux-x64-installer.run
RUN chmod -R 744 railo-4.2.1.000-pl2-linux-x64-installer.run
RUN ./railo-4.2.1.000-pl2-linux-x64-installer.run --mode unattended --railopass “123456”
# remove railo setup
#RUN rm -rf railo-4.2.1.000-pl2-linux-x64-installer.run
#RUN sudo service railo_ctl start
RUN mkdir -p /etc/service/railo
ADD start-railo.sh /etc/service/railo/run
RUN chmod 755 /etc/service/railo/run
# EXPOSE <port>
EXPOSE 80 8888
#CMD ["/railo.sh"]
#CMD ["/start-apache2.sh"]
# Supervisord configuration
RUN mkdir /var/log/supervisor
ADD ./supervisord.conf /etc/supervisor/conf.d/supervisord.conf
CMD ["/usr/bin/supervisord"]
With your supervisord.conf file looking something like that :
[supervisord]
nodaemon=true
[program:apache2]
command=/bin/bash -c "source /etc/apache2/envvars && exec /usr/sbin/apache2 -DFOREGROUND"
[program:railo]
command=/bin/bash -c "exec /opt/railo/railo_ctl start -D FOREGROUND"

Run Python 2.7 by default in a Dotcloud custom service

I need to make Python 2.7 the default version of Python for running a Jenkins build server. I'm trying to use python_version to do this, but Python 2.6 remains the default version. I'm probably missing something really simple. Any suggestions?
dotcloud.yml
jenkins:
type: custom
buildscript: jenkins/builder
ports:
www: http
config:
python_version: v2.7
processes:
sshagent: ssh-agent /bin/bash
jenkins: ~/run
db:
type: postgresql
builder
#!/bin/bash
if [ -f ~/jenkins.war ]
then
echo 'Found jenkins installation.'
else
echo 'Installing jenkins.'
wget -O ~/jenkins.war http://mirrors.jenkins-ci.org/war/latest/jenkins.war
fi
echo 'Installing dotCloud scaffolding.'
cp -a jenkins/. ~
echo 'Setting up SSH.'
mkdir -p ~/.ssh
cp jenkins_id ~/.ssh/id_rsa
chmod 0600 ~/.ssh/id_rsa
ssh-keygen -R bitbucket.org
ssh-keyscan -H bitbucket.org >> ~/.ssh/known_hosts
I'm still not sure why my build file didn't solve the problem, but I was able to work around it by using the --python=/usr/bin/python2.7 option for virtualenv in my Jenkins build script.