I have a .gitlab-ci.yml file in which I want to do the following:
Build a Docker image and push it to AWS ECR
Restart a specific deployment in my EKS cluster that uses this Docker image
Building and pushing the Docker image works fine, however I'm failing to connect to my EKS cluster.
My idea is to use aws eks to update my kubeconfig file, and kubectl to restart my deployment, but I don't know how to use the AWS CLI and Kubectl in my .gitlab-ci.yml file.
I have AWS_ACCESS_KEY_ID, AWS_ACCOUNT_ID, and AWS_DEFAULT_REGION defined in my CI/CD variables. I've got the following .gitlab-ci.yml file:
stages:
- build
- deploy staging
<build stage omitted for brevity>
staging:
stage: deploy staging
image: bitnami/kubectl:latest
only:
- staging
script: |
# install AWS CLI
apk add --no-cache python3 py3-pip \
&& pip3 install --upgrade pip \
&& pip3 install awscli \
&& rm -rf /var/cache/apk/*
aws eks update-kubeconfig --region eu-west-1 --name my-cluster-name
kubectl rollout restart deployment my-deployment
This pipeline fails with the error:
error: unknown command "sh" for "kubectl"
Did you mean this?
set
cp
I've found this issue and solution, but changing the .gitlab-ci.yml file accordingly prevents me from using apk and installing the AWS cli:
stages:
- build
- deploy staging
<build stage omitted for brevity>
staging:
stage: deploy staging
image:
name: bitnami/kubectl:latest
entrypoint: [""]
only:
- staging
script: |
# install AWS CLI
apk add --no-cache python3 py3-pip \
&& pip3 install --upgrade pip \
&& pip3 install awscli \
&& rm -rf /var/cache/apk/*
aws eks update-kubeconfig --region eu-west-1 --name my-cluster-name
kubectl rollout restart deployment my-deployment
Results in the error:
$ # install AWS CLI # collapsed multi-line command
/bin/bash: line 140: apk: command not found
/bin/bash: line 144: aws: command not found
So that leads me to the following question: how do I use both the AWS CLI and Kubectl in my .gitlab-ci.yml file? Or is there another easier way that allows me to restart a deployment in my EKS cluster?
I solved it myself. For future readers: using the alpine/k8s image solves my problem. It has both Kubectl and AWScli installed.
I run a python script to download file via sftp using vpn snx vpn and sftp paramiko. I invoke the script via cronjobs,
Here are my cronjobs script:
apiVersion: batch/v1
kind: CronJob
metadata:
name: file-uploader-a
labels:
app: file-uploader
spec:
schedule: "*/1 0-10 * * *"
jobTemplate:
spec:
parallelism: 1 # How many pods will be instantiated at once.
completions: 1 # How many containers of the job are instantiated one after the other (sequentially) inside the pod.
backoffLimit: 5 # Maximum pod restarts in case of failure
template:
spec:
containers:
- name: file-uploader-a
image: image-a
imagePullPolicy: IfNotPresent
envFrom:
- configMapRef:
name: file-env
- secretRef:
name: file-secret
securityContext:
capabilities:
add:
- CAP_NET_ADMIN
- CAP_SYS_MODULE
command:
- sh
- "-c"
- ". /root/.venv/bin/activate && python -m python.module.a"
restartPolicy: OnFailure
terminationGracePeriodSeconds: 8
My Docker file
FROM ubuntu:18.04
ADD scripts/snx_install_800010013.sh /root
ADD scripts/SINAR33-exp-13May2022.pfx /root
ADD scripts/post_install.sh /root
ADD scripts/init_snx.sh /root
ADD requirements.txt /root
RUN cd root && mkdir bss_uploader
RUN cd root/bss_uploader && mkdir temp
ADD bss_uploader /root/bss_uploader
ARG SNX_SERVER
ARG FTP_HOST
ARG DEBIAN_FRONTEND=noninteractive
RUN dpkg --add-architecture i386 && apt-get update && \
apt-get install bzip2 kmod libstdc++5:i386 \
libpam0g:i386 libx11-6:i386 expect iptables \
net-tools iputils-ping iproute2 python3-venv \
linux-modules-5.4.0-1063-aws python3-pip \
software-properties-common tmux openssh-client -y
RUN cd /usr/bin && ln -s python3 python
WORKDIR /root
RUN bash -x snx_install_800010013.sh
RUN bash -x post_install.sh $SNX_SERVER $FTP_HOST
post_install.sh script
#!/bin/bash
SNX_SERVER=$1
FTP_HOST=$2
mkdir ~/.ssh && touch ~/.ssh/config
echo -e "Host $FTP_HOST\n\tStrictHostKeyChecking no\n\nHost $SNX_SERVER\n\tStrictHostKeyChecking no" >> ~/.ssh/config
chmod 644 ~/.ssh/config
uname=$(uname -r)
mkdir /lib/modules/$uname
# move kernel modules installed to current
cp -a /lib/modules/5.4.0-1063-aws/. /lib/modules/$uname/
modprobe tun
python -m venv .venv
. .venv/bin/activate && pip install --upgrade pip && pip install -r requirements.txt
init_snx.sh script to init on first run
#!/bin/bash
iptables -t nat -A POSTROUTING -o tunsnx -j MASQUERADE
iptables -A FORWARD -i eth0 -j ACCEPT
SNX_SERVER=$1
SNX_PASSWORD=$2
SNX_COMMAND="snx -s $SNX_SERVER -c /root/SINAR33-exp-13May2022.pfx -g"
/usr/bin/expect <<EOF
spawn $SNX_COMMAND
expect "*?assword:"
send "$SNX_PASSWORD\r"
expect "*Do you accept*"
send "y\r"
expect "SNX - connected."
spawn sleep 4
expect "Waiting up to*"
spawn snx -d
expect "SNX - Disconnecting*"
spawn sleep 2
expect "Waiting up to*"
EOF
When I try to run the script via CronJobs, I got connection refused error while connecting to SFTP.
But when i try to run manualy from docker-container (via cli docker container) i got succeed
docker run --name xt_up --cap-add=ALL -t -d image:latest
I already tried to add networkPolicies.egress but still got no luck
could you please help me regarding this ?
thank you and sorry for my bad english
I am facing an error while deploying deployment in CircleCI. Please find the configuration file below.
When running the kubectl CLI, we got an error between kubectl and the EKS tool of the aws-cli.
version: 2.1
orbs:
aws-ecr: circleci/aws-ecr#6.3.0
docker: circleci/docker#0.5.18
rollbar: rollbar/deploy#1.0.1
kubernetes: circleci/kubernetes#1.3.0
deploy:
version: 2.1
orbs:
aws-eks: circleci/aws-eks#1.0.0
kubernetes: circleci/kubernetes#1.3.0
executors:
default:
description: |
The version of the circleci/buildpack-deps Docker container to use
when running commands.
parameters:
buildpack-tag:
type: string
default: buster
docker:
- image: circleci/buildpack-deps:<<parameters.buildpack-tag>>
description: |
A collection of tools to deploy changes to AWS EKS in a declarative
manner where all changes to templates are checked into version control
before applying them to an EKS cluster.
commands:
setup:
description: |
Install the gettext-base package into the executor to be able to run
envsubst for replacing values in template files.
This command is a prerequisite for all other commands and should not
have to be run manually.
parameters:
cluster-name:
default: ''
description: Name of the EKS Cluster.
type: string
aws-region:
default: 'eu-central-1'
description: Region where the EKS Cluster is located.
type: string
git-user-email:
default: "deploy#mail.com"
description: Email of the git user to use for making commits
type: string
git-user-name:
default: "CircleCI Deploy Orb"
description: Name of the git user to use for making commits
type: string
steps:
- run:
name: install gettext-base
command: |
if which envsubst > /dev/null; then
echo "envsubst is already installed"
exit 0
fi
sudo apt-get update
sudo apt-get install -y gettext-base
- run:
name: Setup GitHub access
command: |
mkdir -p ~/.ssh
echo 'github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==' >> ~/.ssh/known_hosts
git config --global user.email "<< parameters.git-user-email >>"
git config --global user.name "<< parameters.git-user-name >>"
- aws-eks/update-kubeconfig-with-authenticator:
aws-region: << parameters.aws-region >>
cluster-name: << parameters.cluster-name >>
install-kubectl: true
authenticator-release-tag: v0.5.1
update-image:
description: |
Generates template files with the specified version tag for the image
to be updated and subsequently applies that template after checking it
back into version control.
parameters:
cluster-name:
default: ''
description: Name of the EKS Cluster.
type: string
aws-region:
default: 'eu-central-1'
description: Region where the EKS Cluster is located.
type: string
image-tag:
default: ''
description: |
The tag of the image, defaults to the value of `CIRCLE_SHA1`
if not provided.
type: string
replicas:
default: 3
description: |
The replica count for the deployment.
type: integer
environment:
default: 'production'
description: |
The environment/stage where the template will be applied. Defaults
to `production`.
type: string
template-file-path:
default: ''
description: |
The path to the source template which contains the placeholders
for the image-tag.
type: string
resource-name:
default: ''
description: |
Resource name in the format TYPE/NAME e.g. deployment/nginx.
type: string
template-repository:
default: ''
description: |
The fullpath to the repository where templates reside. Write
access is required to commit generated templates.
type: string
template-folder:
default: 'templates'
description: |
The name of the folder where the template-repository is cloned to.
type: string
placeholder-name:
default: IMAGE_TAG
description: |
The name of the placeholder environment variable that is to be
substituted with the image-tag parameter.
type: string
cluster-namespace:
default: sayway
description: |
Namespace within the EKS Cluster.
type: string
steps:
- setup:
aws-region: << parameters.aws-region >>
cluster-name: << parameters.cluster-name >>
git-user-email: dev#sayway.com
git-user-name: deploy
- run:
name: pull template repository
command: |
[ "$(ls -A << parameters.template-folder >>)" ] && \
cd << parameters.template-folder >> && git pull --force && cd ..
[ "$(ls -A << parameters.template-folder >>)" ] || \
git clone << parameters.template-repository >> << parameters.template-folder >>
- run:
name: generate and commit template files
command: |
cd << parameters.template-folder >>
IMAGE_TAG="<< parameters.image-tag >>"
./bin/generate.sh --file << parameters.template-file-path >> \
--stage << parameters.environment >> \
--commit-message "Update << parameters.template-file-path >> for << parameters.environment >> with tag ${IMAGE_TAG:-$CIRCLE_SHA1}" \
<< parameters.placeholder-name >>="${IMAGE_TAG:-$CIRCLE_SHA1}" \
REPLICAS=<< parameters.replicas >>
- kubernetes/create-or-update-resource:
get-rollout-status: true
namespace: << parameters.cluster-namespace >>
resource-file-path: << parameters.template-folder >>/<< parameters.environment >>/<< parameters.template-file-path >>
resource-name: << parameters.resource-name >>
jobs:
test:
working_directory: ~/say-way/core
parallelism: 1
shell: /bin/bash --login
environment:
CIRCLE_ARTIFACTS: /tmp/circleci-artifacts
CIRCLE_TEST_REPORTS: /tmp/circleci-test-results
KONFIG_CITUS__HOST: localhost
KONFIG_CITUS__USER: postgres
KONFIG_CITUS__DATABASE: sayway_test
KONFIG_CITUS__PASSWORD: ""
KONFIG_SPEC_REPORTER: true
docker:
- image: 567567013174.dkr.ecr.eu-central-1.amazonaws.com/core-ci:test-latest
aws_auth:
aws_access_key_id: $AWS_ACCESS_KEY_ID_STAGING
aws_secret_access_key: $AWS_SECRET_ACCESS_KEY_STAGING
- image: circleci/redis
- image: rabbitmq:3.7.7
- image: circleci/mongo:4.2
- image: circleci/postgres:10.5-alpine
steps:
- checkout
- run: mkdir -p $CIRCLE_ARTIFACTS $CIRCLE_TEST_REPORTS
# This is based on your 1.0 configuration file or project settings
- restore_cache:
keys:
- v1-dep-{{ checksum "Gemfile.lock" }}-
# any recent Gemfile.lock
- v1-dep-
- run:
name: install correct bundler version
command: |
export BUNDLER_VERSION="$(grep -A1 'BUNDLED WITH' Gemfile.lock | tail -n1 | tr -d ' ')"
echo "export BUNDLER_VERSION=$BUNDLER_VERSION" >> $BASH_ENV
gem install bundler --version $BUNDLER_VERSION
- run: 'bundle check --path=vendor/bundle || bundle install --path=vendor/bundle --jobs=4 --retry=3'
- run:
name: copy test.yml.sample to test.yml
command: cp config/test.yml.sample config/test.yml
- run:
name: Precompile and clean assets
command: bundle exec rake assets:precompile assets:clean
# Save dependency cache
- save_cache:
key: v1-dep-{{ checksum "Gemfile.lock" }}-{{ epoch }}
paths:
- vendor/bundle
- public/assets
- run:
name: Audit bundle for known security vulnerabilities
command: bundle exec bundle-audit check --update
- run:
name: Setup Database
command: bundle exec ruby ~/sayway/setup_test_db.rb
- run:
name: Migrate Database
command: bundle exec rake db:citus:migrate
- run:
name: Run tests
command: bundle exec rails test -f
# By default, running "rails test" won't run system tests.
- run:
name: Run system tests
command: bundle exec rails test:system
# Save test results
- store_test_results:
path: /tmp/circleci-test-results
# Save artifacts
- store_artifacts:
path: /tmp/circleci-artifacts
- store_artifacts:
path: /tmp/circleci-test-results
build-and-push-image:
working_directory: ~/say-way/
parallelism: 1
shell: /bin/bash --login
executor: aws-ecr/default
steps:
- checkout
- run:
name: Pull latest core images for cache
command: |
$(aws ecr get-login --no-include-email --region $AWS_REGION)
docker pull "${AWS_ECR_ACCOUNT_URL}/core:latest"
- docker/build:
image: core
registry: "${AWS_ECR_ACCOUNT_URL}"
tag: "latest,${CIRCLE_SHA1}"
cache_from: "${AWS_ECR_ACCOUNT_URL}/core:latest"
- aws-ecr/push-image:
repo: core
tag: "latest,${CIRCLE_SHA1}"
deploy-production:
working_directory: ~/say-way/
parallelism: 1
shell: /bin/bash --login
executor: deploy/default
steps:
- kubernetes/install-kubectl:
kubectl-version: v1.22.0
- rollbar/notify_deploy_started:
environment: report
- deploy/update-image:
resource-name: deployment/core-web
template-file-path: core-web-pod.yml
cluster-name: report
environment: report
template-repository: git#github.com:say-way/sw-k8s.git
replicas: 3
- deploy/update-image:
resource-name: deployment/core-worker
template-file-path: core-worker-pod.yml
cluster-name: report
environment: report
template-repository: git#github.com:say-way/sw-k8s.git
replicas: 4
- deploy/update-image:
resource-name: deployment/core-worker-batch
template-file-path: core-worker-batch-pod.yml
cluster-name: report
environment: report
template-repository: git#github.com:say-way/sw-k8s.git
replicas: 1
- rollbar/notify_deploy_finished:
deploy_id: "${ROLLBAR_DEPLOY_ID}"
status: succeeded
deploy-demo:
working_directory: ~/say-way/
parallelism: 1
shell: /bin/bash --login
executor: deploy/default
steps:
- kubernetes/install-kubectl:
kubectl-version: v1.22.0
- rollbar/notify_deploy_started:
environment: demo
- deploy/update-image:
resource-name: deployment/core-web
template-file-path: core-web-pod.yml
cluster-name: demo
environment: demo
template-repository: git#github.com:say-way/sw-k8s.git
replicas: 2
- deploy/update-image:
resource-name: deployment/core-worker
template-file-path: core-worker-pod.yml
cluster-name: demo
environment: demo
template-repository: git#github.com:say-way/sw-k8s.git
replicas: 1
- deploy/update-image:
resource-name: deployment/core-worker-batch
template-file-path: core-worker-batch-pod.yml
cluster-name: demo
environment: demo
template-repository: git#github.com:say-way/sw-k8s.git
replicas: 1
- rollbar/notify_deploy_finished:
deploy_id: "${ROLLBAR_DEPLOY_ID}"
status: succeeded
workflows:
version: 2.1
build-n-test:
jobs:
- test:
filters:
branches:
ignore: master
build-approve-deploy:
jobs:
- build-and-push-image:
context: Core
filters:
branches:
only: master
- approve-report-deploy:
type: approval
requires:
- build-and-push-image
- approve-demo-deploy:
type: approval
requires:
- build-and-push-image
- deploy-production:
context: Core
requires:
- approve-report-deploy
- deploy-demo:
context: Core
requires:
- approve-demo-deploy
There is an issue in aws-cli. It is already fixed.
Option 1:
In my case, updating aws-cli + updating the ~/.kube/config helped.
Update aws-cli (following the documentation)
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install --update
Update the kube configuration
mv ~/.kube/config ~/.kube/config.bk
aws eks update-kubeconfig --region ${AWS_REGION} --name ${EKS_CLUSTER_NAME}
Option 2:
Change v1alpha1 to v1beta1:
diff ~/.kube/config ~/.kube/config-backup
691c691
< apiVersion: client.authentication.k8s.io/v1beta1
---
> apiVersion: client.authentication.k8s.io/v1alpha1
We HAVE a fix here: https://github.com/aws/aws-cli/issues/6920#issuecomment-1119926885
Update the aws-cli (aws cli v1) to the version with the fix:
pip3 install awscli --upgrade --user
For aws cli v2 see this.
After that, don't forget to rewrite the kube-config with:
aws eks update-kubeconfig --name ${EKS_CLUSTER_NAME} --region ${REGION}
This command should update the kube apiVersion to v1beta1
In my case, changing apiVersion to v1beta1 in the kube configuration file helped:
apiVersion: client.authentication.k8s.io/v1beta1
There is a glitch with the very latest version of kubectl.
For now, you can follow these steps to get rid of the issue:
curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.23.6/bin/linux/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
sudo kubectl version
There is a problem with the latest kubectl and the aws-cli:
https://github.com/aws/aws-cli/issues/6920
An alternative is to update the AWS cli. It worked for me.
The rest of the instructions are from the answer provided by bigLucas.
Update the aws-cli (aws cli v2) to the latest version:
winget install Amazon.AWSCLI
After that, don't forget to rewrite the kube-config with:
aws eks update-kubeconfig --name ${EKS_CLUSTER_NAME} --region ${REGION}
This command should update the kube apiVersion to v1beta1.
I changed the alpha1 value to the beta1 value, and it’s working for me under the configuration file.
The simplest solution: (it appears here but in complicated words..)
Open your kube config file and replace all alpha instances with beta.
(Editors with find&replace are recommended: Atom, Sublime, etc..).
Example with Nano:
nano ~/.kube/config
Or with Atom:
atom ~/.kube/config
Then you should search for the alpha instances and replace them with beta and save the file.
Open ~/.kube/config
Search for the user within the cluster you have a problem with and replace the client.authentication.k8s.io/v1alpha1 with client.authentication.k8s.io/v1beta1
I was facing the same issue for solution, please follow the below setups:
take backup existing config file mv ~/.kube/config ~/.kube/config.bk
run below command:
aws eks update-kubeconfig --name ${EKS_CLUSTER_NAME} --region ${REGION}
then open the config ~/.kube/config file in any text editor, update v1apiVersion1 to v1beta1 and then try again.
Using kubectl 1.21.9 fixed it for me, with asdf:
asdf plugin-add kubectl https://github.com/asdf-community/asdf-kubectl.git
asdf install kubectl 1.21.9
And I would recommend having a .tools-versions file with:
kubectl 1.21.9
I was able to fix this by running on a MacBook Pro M1 chip (Homebrew):
brew upgrade awscli
Try upgrading the AWS Command Line Interface:
Steps
curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"
sudo installer -pkg ./AWSCLIV2.pkg -target
You can use other ways from the AWS documentation: Installing or updating the latest version of the AWS CLI
Try updating your awscli (AWS Command Line Interface) version.
For Mac, it's brew upgrade awscli (Homebrew).
I got the same problem:
EKS version 1.22
kubectl works, and its version: v1.22.15-eks-fb459a0
helm version is 3.9+, when I execute helm ls -n $namespace I got the error
Error: Kubernetes cluster unreachable: exec plugin: invalid apiVersion "client.authentication.k8s.io/v1alpha1"
from here: it is helm version issue.
so I use the command
curl -L https://git.io/get_helm.sh | bash -s -- --version v3.8.2
downgraded the helm version. helm works
fixed for me only change in kubeconfig
-- >v1alpha1 to v1beta1
In case of Windows, first delete the configuration file in $HOME/.kube folder.
Then run the aws eks update-kubeconfig --name command as suggested by bigLucas.
I just simplified the workaround by updating awscli to awscli-v2, but that also requires Python and pip to be upgraded. It requires minimum Python 3.6 and pip3.
apt install python3-pip -y && pip3 install awscli --upgrade --user
And then update the cluster configuration with awscli
aws eks update-kubeconfig --region <regionname> --name <ClusterName>
Output
Added new context arn:aws:eks:us-east-1:XXXXXXXXXXX:cluster/mycluster to /home/dev/.kube/config
Then check the connectivity with cluster
dev#ip-10-100-100-6:~$ kubectl get node
NAME STATUS ROLES AGE VERSION
ip-X-XX-XX-XXX.ec2.internal Ready <none> 148m v1.21.5-eks-9017834
You can run the below command on your host machine where kubectl and aws-cli exist:
export KUBERNETES_EXEC_INFO='{"apiVersion":"client.authentication.k8s.io/v1beta1"}'
If using ‘sudo’ while running kubectl commands, then export this as root user.
apt install python3-pip -y
pip3 install awscli --upgrade --user
try diffrent version of kubectl ,
if kubernetes version is a 1.23 then we can use (one near) kubectl version 1.23,1.24,1.22
I deployed a mysql monitor application image in kubernetes cluster which run as non root user. When I tried to mount a path to make the data persistent,its overriding the directory(creates a new directory by deleting everything inside that path) in which my application configuration files has to be present.Even I tried using init container still,i am not able to mount it.
my docker file:
FROM centos:7
ENV DIR /binaries
ENV PASS admin
WORKDIR ${DIR}
COPY libstdc++-4.8.5-39.el7.x86_64.rpm ${DIR}
COPY numactl-libs-2.0.12-3.el7.x86_64.rpm ${DIR}
COPY mysqlmonitor-8.0.18.1217-linux-x86_64-installer.bin ${DIR}
RUN yum install -y libaio && yum -y install gcc && yum -y install gcc-c++ && yum -y install compat-libstdc++-33 && yum -y install libstdc++-devel && yum -y install elfutils-libelf-devel && yum -y install glibc-devel && yum -y install libaio-devel && yum -y install sysstat
RUN yum install -y gcc && yum install -y make && yum install -y apr-devel && yum install -y openssl-devel && yum install -y java
RUN rpm -ivh numactl-libs-2.0.12-3.el7.x86_64.rpm
RUN useradd sql
RUN chown sql ${DIR}
RUN chmod 777 ${DIR}
RUN chmod 755 /home/sql
USER sql
WORKDIR ${DIR}
RUN ./mysqlmonitor-8.0.18.1217-linux-x86_64-installer.bin --installdir /home/sql/mysql/enterprise/monitor --mode unattended --tomcatport 18080 --tomcatsslport 18443 --adminpassword ### --dbport 13306
RUN rm -rf /binaries/*
VOLUME /home/mysql/mysql/enterprise/monitor/mysql/data
ENTRYPOINT ["/bin/bash", "-c", "/home/sql/mysql/enterprise/monitor/mysqlmonitorctl.sh start && tail -f /home/sql/mysql/enterprise/monitor/apache-tomcat/logs/mysql-monitor.log"]
my deployment file
apiVersion: apps/v1
kind: Deployment
metadata:
name: mypod
spec:
replicas: 1
selector:
matchLabels:
app: mem
template:
metadata:
labels:
app: mem
spec:
containers:
- name: mem
image: 22071997/mem
command:
volumeMounts:
- mountPath: /home/sql/mysql/enterprise/monitor/mysql/data
name: volume
volumes:
- name: volume
persistentVolumeClaim:
claimName: mem-pvc1
initContainers:
- name: permissionsfix
image: alpine:latest
command: ["/bin/sh", "-c"]
args:
- chown -R 1000:1000 /home/sql/mysql/enterprise/monitor/ && chmod -R 777 /home/sql/mysql/enterprise/monitor/ ;
volumeMounts:
- name: volume
mountPath: /home/sql/mysql/enterprise/monitor
output:
[sql#mypod-775764db45-bzs8n enterprise]$ cd monitor/mysql
[sql#mypod-775764db45-bzs8n mysql]$ ls
LICENSE LICENSE.router README.meb bin docs lib my-large.cnf my-small.cnf new runtime support-files var
LICENSE.meb README README.router data include man my-medium.cnf my.cnf run share tmp
[sql#mypod-775764db45-bzs8n mysql]$ cd data
[sql#mypod-775764db45-bzs8n data]$ ls
mypod-775764db45-bzs8n.err
This doesn't seem related to mounting as a non-root user, but more so that mounting a volume over an existing directory will result in that directory looking as if it is empty (or containing whatever happens to be on the volume already). If you have configuration stored on a non-volume that you would like to be on the volume, then you will need to mount the volume to a different location (so it doesn't overwrite your local configuration) and copy that configuration to the mounted volume location. You can do this in an init container, but be careful not to overwrite the volume contents on every startup of the container.
I am trying to mount a persistent disk on my container which runs a Postgres custom image. I am using Kubernetes and following this tutorial.
This is my db_pod.yaml file:
apiVersion: v1
kind: Pod
metadata:
name: lp-db
labels:
name: lp-db
spec:
containers:
- image: my_username/my-db
name: my-db
ports:
- containerPort: 5432
name: my-db
volumeMounts:
- name: pg-data
mountPath: /var/lib/postgresql/data
volumes:
- name: pg-data
gcePersistentDisk:
pdName: my-db-disk
fsType: ext4
I create the disk using the command gcloud compute disks create --size 200GB my-db-disk.
However, when I run the pod, delete it, and then run it again (like in the tutorial) my data is not persisted.
I tried multiple versions of this file, including with PersistentVolumes and PersistentVolumeClaims, I tried changing the mountPath, but to no success.
Edit
Dockerfile for creating the Postgres image:
FROM ubuntu:trusty
RUN rm /bin/sh && \
ln -s /bin/bash /bin/sh
# Get Postgres
RUN echo "deb http://apt.postgresql.org/pub/repos/apt/ trusty-pgdg main" >> /etc/apt/sources.list.d/pgdg.list
RUN apt-get update && \
apt-get install -y wget
RUN wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
# Install virtualenv (will be needed later)
RUN apt-get update && \
apt-get install -y \
libjpeg-dev \
libpq-dev \
postgresql-9.4 \
python-dev \
python-pip \
python-virtualenv \
strace \
supervisor
# Grab gosu for easy step-down from root
RUN gpg --keyserver pool.sks-keyservers.net --recv-keys B42F6819007F00F88E364FD4036A9C25BF357DD4
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates wget && rm -rf /var/lib/apt/lists/* \
&& wget -O /usr/local/bin/gosu "https://github.com/tianon/gosu/releases/download/1.2/gosu-$(dpkg --print-architecture)" \
&& wget -O /usr/local/bin/gosu.asc "https://github.com/tianon/gosu/releases/download/1.2/gosu-$(dpkg --print-architecture).asc" \
&& gpg --verify /usr/local/bin/gosu.asc \
&& rm /usr/local/bin/gosu.asc \
&& chmod +x /usr/local/bin/gosu \
&& apt-get purge -y --auto-remove ca-certificates wget
# make the "en_US.UTF-8" locale so postgres will be utf-8 enabled by default
RUN apt-get update && apt-get install -y locales && rm -rf /var/lib/apt/lists/* \
&& localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8
ENV LANG en_US.utf8
# Adjust PostgreSQL configuration so that remote connections to the database are possible.
RUN echo "host all all 0.0.0.0/0 md5" >> /etc/postgresql/9.4/main/pg_hba.conf
# And add ``listen_addresses`` to ``/etc/postgresql/9.4/main/postgresql.conf``
RUN echo "listen_addresses='*'" >> /etc/postgresql/9.4/main/postgresql.conf
RUN echo "log_directory='/var/log/postgresql'" >> /etc/postgresql/9.4/main/postgresql.conf
# Add all code from the project and all config files
WORKDIR /home/projects/my-project
COPY . .
# Add VOLUMEs to allow backup of config, logs and databases
ENV PGDATA /var/lib/postgresql/data
VOLUME /var/lib/postgresql/data
# Expose an entrypoint and a port
RUN chmod +x scripts/sh/*
EXPOSE 5432
ENTRYPOINT ["scripts/sh/entrypoint-postgres.sh"]
And entrypoint script:
echo " I am " && gosu postgres whoami
gosu postgres /etc/init.d/postgresql start && echo 'Started postgres'
gosu postgres psql --command "CREATE USER myuser WITH SUPERUSER PASSWORD 'mypassword';" && echo 'Created user'
gosu postgres createdb -O myuser mydb && echo 'Created db'
# This just keeps the container alive.
tail -F /var/log/postgresql/postgresql-9.4-main.log
In the end, it seems that the real problem was the fact that I was trying to create the database from my entrypoint script.
Things such as creating a db or a user should be done at container creation time so I ended up using the standard Postgres image, which actually provides a simple and easy way to create an user and a db.
This is the fully functional configuration file for Postgres.
apiVersion: v1
kind: Pod
metadata:
name: postgres
labels:
name: postgres
spec:
containers:
- name: postgres
image: postgres
env:
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
- name: POSTGRES_USER
value: myuser
- name: POSTGRES_PASSWORD
value: mypassword
- name: POSTGRES_DB
value: mydb
ports:
- containerPort: 5432
volumeMounts:
- mountPath: /var/lib/postgresql/data
name: pg-data
volumes:
- name: pg-data
persistentVolumeClaim:
claimName: pg-data-claim
Thanks to all those who helped me :)
does your custom postgresql persist data at /var/lib/postgresql/data?
are you able to get logs from your postgresql container and spot anything interesting?
when your pod is running, can you see the mountpoints inside your container and check the persistent disk is there?
I followed this scenario and I was able to persist my data by changing the mountPath to /var/lib/postgresql and also reproduced using cassandra (i.e. /var/lib/cassandra for mountPath)
I was able to delete/restart pods from different nodes/hosts and still see my "users" table and the data I previously entered. However, I was not using a custom image, I just used standard docker images.