Push images into docker hub and deploy in aws cluster? - kubernetes

I am trying to deploy my application into aws cluster as follows
Steps
Build image and push into docker hub (it is working)
Deploy the image into aws cluster (I couldn't make it work)
I searched in google, but couldn't find any solution.
Here is my GitHub workflow file
deploy.yml. Any help is appreciated to make it work.
# This is a basic workflow that is manually triggered
name: Deploy Manual
# Controls when the action will run. Workflow runs when manually triggered using the UI
# or API.
on:
workflow_dispatch:
# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
# This workflow contains a single job called "deploy"
deploy:
# The type of runner that the job will run on
runs-on: ubuntu-latest
env:
IMAGE_TAG: ${{ github.sha }}
KUBE_CONFIG_DATA: ${{ secrets.KUBE_CONFIG_DATA }}
KUBE_NAMESPACE: production
DOCKER_USER: ${{secrets.DOCKER_HUB_USERNAME}}
DOCKER_PASSWORD: ${{secrets.DOCKER_HUB_ACCESS_TOKEN}}
RELEASE_IMAGE: ucars/ucars-ui3:${{ github.sha }}
steps:
# This step instructs Github to cancel any current run for this job on this very repository.
- name: Cancel Previous Runs
uses: styfle/cancel-workflow-action#0.4.1
with:
access_token: ${{ github.token }}
- uses: actions/checkout#v2
- name: docker login
run: |
docker login -u $DOCKER_USER -p $DOCKER_PASSWORD
- name: Build the Docker image
run: docker build . --file Dockerfile --tag $RELEASE_IMAGE
- name: Docker Push
run: docker push $RELEASE_IMAGE
- name: Deploy to Kubernetes cluster
uses: kodermax/kubectl-aws-eks#master
with:
args: set image deployment/ucars-ui3-pod app=${{ env.RELEASE_IMAGE }} --record -n $KUBE_NAMESPACE
It is failing at the step Deploy to Kubernetes cluster
2022-01-14T18:22:14.4557590Z ##[group]Run kodermax/kubectl-aws-eks#master
2022-01-14T18:22:14.4558128Z with:
2022-01-14T18:22:14.4559002Z *** set image deployment/***-ui3-pod app=***/***-ui3:3d23d9fb07a2ce43b3a27502359c1a0685705200 --record -n $KUBE_NAMESPACE
2022-01-14T18:22:14.4559708Z ***
2022-01-14T18:22:14.4560253Z IMAGE_TAG: 3d23d9fb07a2ce43b3a27502359c1a0685705200
2022-01-14T18:22:14.4608584Z KUBE_CONFIG_DATA: ***
2022-01-14T18:22:14.4609135Z KUBE_NAMESPACE: production
2022-01-14T18:22:14.4609639Z DOCKER_USER: ***
2022-01-14T18:22:14.4610253Z DOCKER_PASSWORD: ***
2022-01-14T18:22:14.4610915Z RELEASE_IMAGE: ***/***-ui3:3d23d9fb07a2ce43b3a27502359c1a0685705200
2022-01-14T18:22:14.4611509Z ##[endgroup]
2022-01-14T18:22:14.4809817Z ##[command]/usr/bin/docker run --name a74655ce21da3d4675874b9544657797b0_b31db8 --label 9916a7 --workdir /github/workspace --rm -e IMAGE_TAG -e KUBE_CONFIG_DATA -e KUBE_NAMESPACE -e DOCKER_USER -e DOCKER_PASSWORD -e RELEASE_IMAGE -e INPUT_ARGS -e HOME -e GITHUB_JOB -e GITHUB_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_REPOSITORY_OWNER -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RETENTION_DAYS -e GITHUB_RUN_ATTEMPT -e GITHUB_ACTOR -e GITHUB_WORKFLOW -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e GITHUB_EVENT_NAME -e GITHUB_SERVER_URL -e GITHUB_API_URL -e GITHUB_GRAPHQL_URL -e GITHUB_REF_NAME -e GITHUB_REF_PROTECTED -e GITHUB_REF_TYPE -e GITHUB_WORKSPACE -e GITHUB_ACTION -e GITHUB_EVENT_PATH -e GITHUB_ACTION_REPOSITORY -e GITHUB_ACTION_REF -e GITHUB_PATH -e GITHUB_ENV -e RUNNER_OS -e RUNNER_ARCH -e RUNNER_NAME -e RUNNER_TOOL_CACHE -e RUNNER_TEMP -e RUNNER_WORKSPACE -e ACTIONS_RUNTIME_URL -e ACTIONS_RUNTIME_TOKEN -e ACTIONS_CACHE_URL -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/***-ui3/***-ui3":"/github/workspace" 9916a7:4655ce21da3d4675874b9544657797b0 set image deployment/***-ui3-pod app=***/***-ui3:3d23d9fb07a2ce43b3a27502359c1a0685705200 --record -n $KUBE_NAMESPACE
2022-01-14T18:22:14.7791749Z base64: invalid input

I think I have found the issue, apparently, KUBE_CONFIG_DATA is invalid. Your entrypoint.sh in kodermax/kubectl-aws-eks#master image is trying to decode it, but can't and throwing the error.
#!/bin/sh
set -e
# Extract the base64 encoded config data and write this to the KUBECONFIG
echo "$KUBE_CONFIG_DATA" | base64 -d > /tmp/config
export KUBECONFIG=/tmp/config
sh -c "kubectl $*"
Please fix the KUBE_CONFIG_DATA, it must be in a valid base64 format. if you put raw kubeconfig file there, you may have to convert it to base64 format first.
KUBE_CONFIG_DATA: ${{ secrets.KUBE_CONFIG_DATA }}

Related

How to run multiple lines command in docker compose?

I am trying to make this run but it retrieves this error:
fscrawler | sed: -e expression #2, char 31: unknown option to `s'
I'm trying to run this command:
command: >
sh -c "sed -i -e "s/{ELASTIC_PASSWORD}/${ELASTIC_PASSWORD}/g"
-e "s/{ELASTICSEARCH_HOST}/${ELASTICSEARCH_HOST}/g"
-e "s/{FSCRAWLER_HOST}/${FSCRAWLER_HOST}/g" /root/.fscrawler/job1/_settings.yaml
&& fscrawler job1 --restart --rest"
I've tried with simple quotes and many other options (backslashes at the end as well) but couldn't make it work.
SOLUTION:
docker-compose.yml
entrypoint: /path/to/entrypoint.sh
environment:
- ELASTIC_HOST=${ELASTIC_HOST}
- ELASTIC_USER=${ELASTIC_USER}
- ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
- FSCRAWLER_HOST=${FSCRAWLER_HOST}
Dockerfile FSCrawler
...
COPY /host/path/to/entrypoint.sh /docker/path/to/entrypoint.sh
RUN chmod u+x /entrypoint.sh
Entrypoint
#!/bin/bash
sed -i -e "s|{ELASTIC_USER}|${ELASTIC_USER}|g" \
-e "s|{ELASTIC_PASSWORD}|${ELASTIC_PASSWORD}|g" \
-e "s|{ELASTIC_HOST}|${ELASTIC_HOST}|g" \
-e "s|{FSCRAWLER_HOST}|${FSCRAWLER_HOST}|g" /root/.fscrawler/job1/_settings.yaml
fscrawler job1 --restart --rest

Trying to run postgres on docker. It gives error

I am trying to run postgres on docker with this cmd and It gives error
PS D:\Data Engineering with Zoomcamp> docker run -it -e POSTGRES_USER="root" -e POSTGRES_PASSWORD="root" -e POSTGRES_DB="ny_taxi"D:\Data Engineering with Zoomcamp\ny_taxi_postgres:/var/lib/postgresql/data -p 5432:5432 postgres:13
docker: invalid reference format: repository name must be lowercase.
See 'docker run --help'.
you forgot to add the volume tag. Use below command
docker run -it -e POSTGRES_USER="root" -e POSTGRES_PASSWORD="root" -e POSTGRES_DB="ny_taxi" -v "D:\Data Engineering with Zoomcamp\ny_taxi_postgres":/var/lib/postgresql/data -p 5432:5432 postgres:13

Why does Snyk GitHub Actions fail due to ".. Your package.json and undefined are probably out of sync ..."

Hello StackOverflowians.
I'm currently trying to set up Snyk in my GitHub Actions workflow, in a Node project.
The idea is to run two jobs:
A Snyk security gate as per their documentation (found here), such as the first example for keeping it simple.
A build and push job (that works as intended on its own)
However, when attempting to run the first job, it fails with the following log during the "Run Snyk to check for vulnerabilities" step:
Run snyk/actions/node#master
with:
command: test
json: false
env:
REGISTRY: ghcr.io
IMAGE_NAME: <IMAGENAME>
SNYK_TOKEN: ***
/usr/bin/docker run --name snyksnyknode_3aa871 --label 6a6825 --workdir /github/workspace --rm -e REGISTRY -e IMAGE_NAME -e SNYK_TOKEN -e INPUT_ARGS -e INPUT_COMMAND -e INPUT_JSON -e SNYK_INTEGRATION_NAME -e SNYK_INTEGRATION_VERSION -e HOME -e GITHUB_JOB -e GITHUB_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_REPOSITORY_OWNER -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RETENTION_DAYS -e GITHUB_RUN_ATTEMPT -e GITHUB_ACTOR -e GITHUB_WORKFLOW -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e GITHUB_EVENT_NAME -e GITHUB_SERVER_URL -e GITHUB_API_URL -e GITHUB_GRAPHQL_URL -e GITHUB_REF_NAME -e GITHUB_REF_PROTECTED -e GITHUB_REF_TYPE -e GITHUB_WORKSPACE -e GITHUB_ACTION -e GITHUB_EVENT_PATH -e GITHUB_ACTION_REPOSITORY -e GITHUB_ACTION_REF -e GITHUB_PATH -e GITHUB_ENV -e RUNNER_OS -e RUNNER_ARCH -e RUNNER_NAME -e RUNNER_TOOL_CACHE -e RUNNER_TEMP -e RUNNER_WORKSPACE -e ACTIONS_RUNTIME_URL -e ACTIONS_RUNTIME_TOKEN -e ACTIONS_CACHE_URL -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/SOMEPROJECT/SOMEPROJECT":"/github/workspace" snyk/snyk:node "snyk" "test" "--severity-threshold=high --fail-on=upgradable"
Dependency bindings was not found in undefined. Your package.json and undefined are probably out of sync. Please run "undefined" and try again.
The last part Dependency bindings was not found in undefined. Your package.json and undefined are probably out of sync. Please run "undefined" and try again. is that which I do not understand how it helps me debug.
Is this a known problem with a known solution? If not, how can I go about finding what undefined is referring to?
Thank you in advance,
Raoul
Currently, it seems as though deleting node_modules/ as well as package-lock.json and regenerating them with npm install remedies this issue.
<in root>
rm -rf node_modules/
npm install

Why does this GitHub Actions script not build my Hugo page?

I have a GitHub project with a Hugo-based web site in it. Whenever someone pushes something to the prod branch, I want to build the Hugo page (transform Markdown files to HTML) and upload it to my hosting provider. I have problems building the page.
I have this script in GitHub Actions:
name: Publish prod branch
on:
push:
branches:
- prod
jobs:
build:
name: Greeting
runs-on: ubuntu-latest
steps:
- name: Hello world
uses: actions/hello-world-javascript-action#v1
with:
who-to-greet: Dmitrii
id: hello
- name: Echo the greeting's time
run: echo 'The time was ${{ steps.hello.outputs.time }}.'
- name: Build Hugo
uses: srt32/hugo-action#master
It fails because it does not find the configuration file config.toml, even though it is there:
/usr/bin/docker run --name e87b520e21a5125f094485b4e030650bd57153_f8bc76 --label e87b52 --workdir /github/workspace --rm -e HOME -e GITHUB_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_ACTOR -e GITHUB_WORKFLOW -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e GITHUB_EVENT_NAME -e GITHUB_WORKSPACE -e GITHUB_ACTION -e GITHUB_EVENT_PATH -e RUNNER_OS -e RUNNER_TOOL_CACHE -e RUNNER_TEMP -e RUNNER_WORKSPACE -e ACTIONS_RUNTIME_URL -e ACTIONS_RUNTIME_TOKEN -e ACTIONS_CACHE_URL -e GITHUB_ACTIONS=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/hugo-wp-site/hugo-wp-site":"/github/workspace" e87b52:0e21a5125f094485b4e030650bd57153
Error: Unable to locate config file or config directory. Perhaps you need to create a new site.
#################################################
Run `hugo help new` for details.
Starting the Hugo Action
Total in 0 ms
How can I fix it, i. e. make hugo see my config.toml file?
Update 1: I tried to find out the version of Hugo being used by modifying the script as follows:
name: Publish prod branch
on:
push:
branches:
- prod
jobs:
build:
name: Build and publish web site to hosting provider
runs-on: ubuntu-latest
steps:
- name: Hello world
uses: actions/hello-world-javascript-action#v1
with:
who-to-greet: Dmitrii
id: hello
- name: Echo the greeting's time
run: echo 'The time was ${{ steps.hello.outputs.time }}.'
- name: Output the version of Hugo
run: hugo version
- name: Build Hugo
uses: srt32/hugo-action#master
But when I run it, I get the following error:
hugo version
shell: /bin/bash -e {0}
/home/runner/work/_temp/9e57960c-2f2c-4f2a-870c-c1cbc41d820f.sh: line 1: hugo: command not found
##[error]Process completed with exit code 127.
Update 2: Found out the version of Hugo in the output:
(7/7) Installing hugo (0.61.0-r0)
Update 3: The earliest Hugo version that may have the issue 6794 fixed is v0.64.0 because that issue was merged on January 31st and v0.64.0 is the first version that came out after that day.
Update 4: It seems that in order to fix this error, I need to make sure that the Hugo action uses a more recent version of Hugo. To achieve this, I changed the Dockerfile so that version 0.65.3-r0 is installed (according to this answer):
RUN apk add --no-cache hugo=0.65.3-r0 bash
But when I run the script, Alpine Linux fails to install Hugo:
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/community/x86_64/APKINDEX.tar.gz
ERROR: unsatisfiable constraints:
hugo-0.61.0-r0:
breaks: world[hugo=0.65.3-r0]
The command '/bin/sh -c apk add --no-cache hugo=0.65.3-r0 bash' returned a non-zero code: 1
##[warning]Docker build failed with exit code 1, back off 9.558 seconds before retry.
/usr/bin/docker build -t e87b52:dfe904e1240c4dbea120e452e5568b51 "/home/runner/work/_actions/dpisarenko/hugo-action/master"
Sending build context to Docker daemon 7.168kB
Any help on how to fix this is highly appreciated.
Update 5: After changing the section for installation of Hugo to
RUN echo "http://dl-cdn.alpinelinux.org/alpine/edge/community" >> /etc/apk/repositories
RUN apk update
RUN apk add --no-cache hugo=0.65.3-r0 bash
the action installs a more recent version of Hugo:
Step 10/13 : RUN apk add --no-cache hugo=0.65.3-r0 bash
---> Running in 633b06ba9a65
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/community/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
(1/7) Installing ncurses-terminfo-base (6.1_p20200118-r2)
(2/7) Installing ncurses-libs (6.1_p20200118-r2)
(3/7) Installing readline (8.0.1-r0)
(4/7) Installing bash (5.0.11-r1)
Executing bash-5.0.11-r1.post-install
(5/7) Installing libgcc (9.2.0-r3)
(6/7) Installing libstdc++ (9.2.0-r3)
(7/7) Installing hugo (0.65.3-r0)
But I still get the same error:
Run dpisarenko/hugo-action#master
/usr/bin/docker run --name e87b52fba2a6bbd65d4e86b03264ae4ae92e94_cbeaf6 --label e87b52 --workdir /github/workspace --rm -e HOME -e GITHUB_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_ACTOR -e GITHUB_WORKFLOW -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e GITHUB_EVENT_NAME -e GITHUB_WORKSPACE -e GITHUB_ACTION -e GITHUB_EVENT_PATH -e RUNNER_OS -e RUNNER_TOOL_CACHE -e RUNNER_TEMP -e RUNNER_WORKSPACE -e ACTIONS_RUNTIME_URL -e ACTIONS_RUNTIME_TOKEN -e ACTIONS_CACHE_URL -e GITHUB_ACTIONS=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/hugo-wp-site/hugo-wp-site":"/github/workspace" e87b52:fba2a6bbd65d4e86b03264ae4ae92e94
#################################################
Starting the Hugo Action
Error: Unable to locate config file or config directory. Perhaps you need to create a new site.
Run `hugo help new` for details.
Update 6: I added the commands pwd and ls -al into the file entrypoint.sh in which hugo is being called:
echo "pwd:"
pwd
echo "ls -al:"
ls -al
hugo "$#"
Here is its output:
Starting the Hugo Action
pwd:
/github/workspace
ls -al:
total 8
drwxr-xr-x 2 1001 115 4096 Mar 15 17:39 .
drwxr-xr-x 5 root root 4096 Mar 15 17:39 ..
Error: Unable to locate config file or config directory. Perhaps you need to create a new site.
Run `hugo help new` for details.
It seems that the action tries to run hugo inside the directory /github/workspace which is empty.
My next step is to find out in which directory the contens of my git branch is located.
Update 7: I tried to output the contents of the directories
/home/runner/work/_temp/_github_home,
/github/home,
/home/runner/work/_temp/_github_workflow,
/github/workflow,
/home/runner/work/hugo-wp-site/hugo-wp-site, and
/github/workspace
in entrypoint.sh, but none of them contains my Hugo code.
Update 8: I added the following line to entrypoint.sh to find the directory with Hugo sources:
find / -name "*archetypes*"
All Hugo projects contain that directory.
But find did not find anything. It looks like Docker of the GitHub action is running in the wrong directory.
This is based on srt32/hugo-action which uses possibly an older version of Hugo.
Check first the hugo version, to see if issue 6794 applies (it was fixed in january 2020 with PR 6834.
It seems that the Hugo code was not checked out at all. Therefore the solution is to modify the GitHub action so that
git is installed in the Dockerized Linux and
the Hugo source code is checked out.
To do the former, the Dockerfile needs to be modified like shown below (see RUN apk add --no-cache git):
FROM alpine:latest
LABEL "com.github.actions.name"="Hugo Actions"
LABEL "com.github.actions.description"="Commands to help with building Hugo based static sites"
LABEL "com.github.actions.icon"="mic"
LABEL "com.github.actions.color"="yellow"
LABEL "repository"="http://github.com/dpisarenko/hugo-action"
LABEL "homepage"="http://github.com/dpisarenko/hugo-action"
RUN echo "http://dl-cdn.alpinelinux.org/alpine/edge/community" >> /etc/apk/repositories
RUN apk update
RUN apk add --no-cache hugo=0.65.3-r0 bash
RUN apk add --no-cache bash
RUN apk add --no-cache git
ADD entrypoint.sh /
RUN chmod +x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
Then we need to call git clone in entrypoint.sh:
#!/bin/bash
set -e
echo "#################################################"
echo "Starting the Hugo Action"
git clone --branch prod https://github.com/dpisarenko/hugo-wp-site.git /hugo
cd /hugo
hugo "$#"
echo "#################################################"
echo "Completed the Hugo Action"

Call Perl Script using Ansible

I have the below .sh code which need to get converted to Ansible tasks.
#!/bin/sh
echo "Installing Sonar"
SONAR_HOME=/tui/hybris/sonar
if [ ! -d "$SONAR_HOME" ]; then
mkdir -p $SONAR_HOME
fi
cd $SONAR_HOME
wget https://s3-eu-west-1.amazonaws.com/tuiuk/source/sonarqube/sonarqube-5.4.zip
unzip sonarqube-5.4.zip
echo "Modifying Sonar config file"
cd sonarqube-5.4/conf
perl -p -i -e 's/#sonar.jdbc.username=/sonar.jdbc.username=sonar/g' sonar.properties
perl -p -i -e 's/#sonar.jdbc.password=/sonar.jdbc.password=sonar/g' sonar.properties
perl -p -i -e 's/#sonar.jdbc.url=jdbc:mysql/sonar.jdbc.url=jdbc:mysql/g' sonar.properties
cd $SONAR_HOME
echo "downloading and copying plugins"
wget https://s3-eu-west-1.amazonaws.com/tuiuk/source/sonarqube/sonarqube5.4_plugins.zip
unzip sonarqube5.4_plugins.zip
cp plugins/* sonarqube-5.4/extensions/plugins/
cd sonarqube-5.4/bin/linux-x86-64
echo "Starting Sonar"
./sonar.sh start
Below is my task.I got stuck where I need to execute perl script. Could any of you help me in proceeding further.
- hosts: docker_test
tasks:
- name: Creates directory
file: path=/tui/hybris/sonar state=directory mode=0777
sudo: yes
- name: Installing Sonar
get_url:
url: "https://s3-eu-west-1.amazonaws.com/tuiuk/source/sonarqube/sonarqube-5.4.zip"
dest: "/tui/hybris/sonar/sonarqube-5.4.zip"
register: get_solr
- debug:
msg: "solr was downloaded"
when: get_solr|changed
- name: Unzip SonarQube
unarchive: src=/tui/hybris/sonar/sonarqube-5.4.zip dest=/tui/hybris/sonar copy=no
I bet you don't need perl here, use lineinfile with regex option (if you need to modify a single line in the file) or replace module (if you need to modify all occurrences).
Just call perl with command or shell-module:
- task: Modifying Sonar config file
shell: cd sonarqube-5.4/conf && perl -p -i -e ...