Send arguments to a Job - kubernetes

I have a docker Image that basically runs a one time script. That scripts takes 3 arguments. My docker file is
FROM <some image>
ARG URL
ARG USER
ARG PASSWORD
RUN apt update && apt install curl -y
COPY register.sh .
RUN chmod u+x register.sh
CMD ["sh", "-c", "./register.sh $URL $USER $PASSWORD"]
When I spin up the contianer using docker run -e URL=someUrl -e USER=someUser -e PASSWORD=somePassword -itd <IMAGE_ID> it works perfectly fine.
Now I want to deploy this as a job.
My basic Job looks like:
apiVersion: batch/v1
kind: Job
metadata:
name: register
spec:
template:
spec:
containers:
- name: register
image: registeration:1.0
args: ["someUrl", "someUser", "somePassword"]
restartPolicy: Never
backoffLimit: 4
But this the pod errors out on
Error: failed to start container "register": Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \"someUrl\": executable file not found in $PATH"
Looks like it is taking my args as commands and trying to execute them. Is that correct ? What can I do to fix this ?

In the Dockerfile as you've written it, two things happen:
The URL, username, and password are fixed in the image. Anyone who can get the image can run docker history and see them in plain text.
The container startup doesn't take any arguments; it just runs the single command with its fixed set of arguments.
Especially since you're planning to pass these arguments in at execution time, I wouldn't bother trying to include them in the image. I'd reduce the Dockerfile to:
FROM ubuntu:18.04
RUN apt update \
&& DEBIAN_FRONTEND=noninteractive \
apt install --assume-yes --no-install-recommends \
curl
COPY register.sh /usr/bin
RUN chmod u+x /usr/bin/register.sh
ENTRYPOINT ["register.sh"]
When you launch it, the Kubernetes args: get passed as command-line parameters to the entrypoint. (It is the same thing as the Docker Compose command: and the free-form command at the end of a plain docker run command.) Making the script be the container entrypoint will make your Kubernetes YAML work the way you expect.
In general I prefer using CMD to ENTRYPOINT. (Among other things, it makes it easier to docker run --rm -it ... /bin/sh to debug your image build.) If you do that, then the Kubernetes args: need to include the name of the script it's running:
args: ["./register.sh", "someUrl", "someUser", "somePassword"]

Use:
args: ["sh", "-c", "./register.sh someUrl someUser somePassword"]

Related

Docker 'backup' process container not seeing Database container postgres

I have a simple docker-compose.yml & associated Dockerfiles that give me a simple dev and prod environment for a nginx-uvicorn-django-postgres stack. I want to add an optional 'backup' container that just runs cron to periodically connect to the 'postgres' container.
# backup container - derived from [this blog][1]
ARG DOCKER_REPO
ARG ALPINE_DOCKER_IMAGE # ALPINE
ARG ALPINE_DOCKER_TAG # LATEST
FROM ${DOCKER_REPO}${ALPINE_DOCKER_IMAGE}:${ALPINE_DOCKER_TAG}
ARG DB_PASSWORD
ARG DB_HOST # "db"
ARG DB_PORT # "5432"
ARG DB_NAME # "ken"
ARG DB_USERNAME # "postgres"
ENV PGPASSWORD=${DB_PASSWORD} HOST=${DB_HOST} PORT=${DB_PORT} PSQL_DB_NAME=${DB_NAME} \
USERNAME=${DB_USERNAME}
RUN printenv
RUN mkdir /output && \
mkdir /output/backups && \
mkdir /scripts && \
chmod a+x /scripts
COPY ./scripts/ /scripts/
COPY ./scripts/in_docker/pg_dump.sh /etc/periodic/15min/${DB_NAME}_15
COPY ./scripts/in_docker/pg_dump.sh /etc/periodic/daily/${DB_NAME}_day
COPY ./scripts/in_docker/pg_dump.sh /etc/periodic/weekly/${DB_NAME}_week
COPY ./scripts/in_docker/pg_dump.sh /etc/periodic/monthly/${DB_NAME}_month
RUN apk update && \
apk upgrade && \
apk add --no-cache postgresql-client && \
chmod a+x /etc/periodic/15min/${DB_NAME}_15 && \
chmod a+x /etc/periodic/daily/${DB_NAME}_day && \
chmod a+x /etc/periodic/weekly/${DB_NAME}_week && \
chmod a+x /etc/periodic/monthly/${DB_NAME}_month
The django container is derived from the official Python image and connects (through psycopg2) with values (as ENV value) for host, dbname, username, password and port. The 'backup' container has these same values, but I get this error from the command line:
> pg_dump --host="$HOST" --port="$PORT" --username="$USERNAME" --dbname="$PSQL_DB_NAME"
pg_dump: error: could not translate host name "db" to address: Name does not resolve
Is Alpine missing something relevant that is present in the official Python?
Edit:
I am running with a system of shell scripts that take care of housekeeping for different configurations. so
> ./ken.sh dev_server
will set up the environment variables and then run docker-compose for the project and the containers
docker-compose.yml doesn't explicitly create a network.
I don't know what "db" should resolve to beyond just 'db://'? - its what the django container gets and it is able to resolve a connection to the 'db' service.
service:
db:
image: ${DOCKER_REPO}${DB_DOCKER_IMAGE}:${DB_DOCKER_TAG} #postgres: 14
container_name: ${PROJECT_NAME}_db
volumes:
- pgdata:/var/lib/postgresql/data
environment:
- PGPASSWORD
- POSTGRES_DB=${DB_NAME}
- POSTGRES_USER=${DB_USERNAME}
- POSTGRES_PASSWORD=${DB_PASSWORD}
command: ["postgres", "-c", "log_statement=all"]
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres -h db"]
interval: 2s
timeout: 5s
retries: 25
This is the 'dev_server' script run by the parent ken.sh script
function dev_server() {
trap cleanup EXIT
wait_and_launch_browser &
docker-compose -p "${PROJECT_NAME}" up -d --build db nginx web pgadmin backup
echo "Generate static files and copy them into static and file volumes."
source ./scripts/generate_static_files.sh
docker-compose -p "${PROJECT_NAME}" logs -f web nginx backup
}
Update: Worked through "Reasons why docker containers can't talk to each other" and found that all the containers are on a ken_default network, from 170.20.0.2 to 170.20.0.6.
I can docker exec ken_backup backup ken_db -c2, but not from db to backup, because the db container doesn't include ping.
From a shell on backup I cannot ping ken_db - ken_db doesn't resolve, nor does 'db'.
I can't make much of that and I'm not sure what to try next.
You are running the backup container as a separate service.
Docker-compose creates a unique network for each service (docker-compose.yml file).
You need to get the DB and your backup container on the same docker network.
See this post

docker-compose entrypoint with mutiple commands

How to run docker-compose entrypoint configuration option with multiple bash commands
commands:
yarn install
yarn build
sleep infinity
In docker-compose.yml, for service gvhservice
gvhservice:
entrypoint:
- "/bin/sh"
- -ecx
- |
yarn install
yarn build
sleep infinity
OR
optionally, add all these commands to a file say - entrypoint.sh
and in docker-compose.yml,
gvhservice:
entrypoint: entrypoint.sh
OR,
Using the option of entrypoint.sh and command configuration option in docker-compose.yml (suitable for a variable number of commands to be passed during runtime)
entrypoint.sh
#!/bin/sh
set -ex
exec "$#"
docker-compose.yml
command:
- /bin/sh
- -ecx
- |
yarn install
yarn build
sleep infinity

Is it possible to install curl into busybox in kubernetes pod

I am using busybox to detect my network problem in kubernetes v1.18 pods. I created the busybox like this:
apiVersion: v1
kind: Pod
metadata:
name: busybox
namespace: default
spec:
containers:
- name: busybox
image: busybox:1.28
command:
- sleep
- "3600"
imagePullPolicy: IfNotPresent
restartPolicy: Always
and login to find the kubernetes clusters network situation:
kubectl exec -it busybox /bin/bash
What surprises me is that the busybox does not contain curl. Why does the busybox package not include the curl command? I am searching the internet and find the docs do not talk about how to add curl into busybox. I tried to install curl, but found no way to do this. Is there anyway to add curl package into busybox?
The short answer, is you cannot.
Why?
Because busybox does not have package manager like: yum, apk, or apt-get ..
Acutally you have two solutions:
1. Either use a modified busybox
You can use other busybox images like progrium/busybox which provides opkg-install as a package manager.
image: progrium/busybox
Then:
kubectl exec -it busybox -- opkg-install curl
2. Or if your concern to use a minimal image, you can use alpine
image: alpine:3.12
then:
kubectl exec -it alpine -- apk --update add curl
No. Consider alpine as a base image instead that includes BusyBox plus a package manager, or building (or finding) a custom image that has the tools you need pre-installed.
BusyBox is built as a single binary that contains implementations of many common Linux tools. The BusyBox documentation includes a listing of the included commands. You cannot "install" more commands into it without writing C code.
BusyBox does contain an implementation of wget, which might work for your purposes (wget -O- http://other-service).
BusyBox has a subset of wget. The usage patterns of curl are significantly more complex in your OS than the one that comes with Busybox.
To clarify what I mean, run the following in your OS:
$ wget --help | wc -l
207
while running wget's help inside Busybox container should give you a minimal subset package:
$ docker run --rm busybox wget --help 2>&1 | wc -l
20
In K8s, you could run the following:
$ kubectl run -i --tty --rm busybox --image=busybox -- sh
If you don't see a command prompt, try pressing enter.
/ # wget
BusyBox v1.33.1 (2021-06-07 17:33:50 UTC) multi-call binary.
Usage: wget [-cqS] [--spider] [-O FILE] [-o LOGFILE] [--header 'HEADER: VALUE'] [-Y on/off]
[--no-check-certificate] [-P DIR] [-U AGENT] [-T SEC] URL...
Retrieve files via HTTP or FTP
--spider Only check URL existence: $? is 0 if exists
--no-check-certificate Don't validate the server's certificate
-c Continue retrieval of aborted transfer
-q Quiet
-P DIR Save to DIR (default .)
-S Show server response
-T SEC Network read timeout is SEC seconds
-O FILE Save to FILE ('-' for stdout)
-o LOGFILE Log messages to FILE
-U STR Use STR for User-Agent header
-Y on/off
If curl is something required for your use case, I wouldsuggest to use Alpine which is busybox + a minimal package manager and libc implementation such that you can trivially do apk add --no-cache curl and get real curl (or even apk add --no-cache wget to get the "real" wget instead of BusyBox's wget).
As others said, the answer is no and you need to use another image.
There is:
Official curl alpine based image: https://hub.docker.com/r/curlimages/curl with curlimages/curl
Busyboxplus Images: https://hub.docker.com/r/radial/busyboxplus with radial/busyboxplus:curl
Nixery with nixery.dev/curl
Image sizes:
$ docker images -f "reference=*/*curl"
REPOSITORY TAG IMAGE ID CREATED SIZE
curlimages/curl latest ab35d809acc4 9 days ago 11MB
radial/busyboxplus curl 71fa7369f437 8 years ago 4.23MB
nixery.dev/curl latest aa552b5bd167 N/A 56MB
As #abdennour is suggesting, I'm no longer sticking with busybox anymore. Alpine is a very lightweight Linux container image as others suggest here in which you can literally install any UNIX-like tool handy to accomplish your troubleshooting task. In fact, I use this function within my dotfiles at .bashrc to spin a handy ephemeral ready-to-rock Alpine pod:
## This function takes an optional argument to run a pod within a Kubernetes NS, if it's not provided it fallsback to `default` NS.
function kalpinepod () { kubectl run -it --rm --restart=Never --image=alpine handytools -n ${1:-default} -- /bin/ash }
❯ kalpinepod kube-system
If you don't see a command prompt, try pressing enter.
/ # cat /etc/resolv.conf
search kube-system.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.245.0.10
options ndots:5
/ # apk --update add curl openssl
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/community/x86_64/APKINDEX.tar.gz
(1/6) Installing ca-certificates (20191127-r5)
(2/6) Installing brotli-libs (1.0.9-r3)
(3/6) Installing nghttp2-libs (1.42.0-r1)
(4/6) Installing libcurl (7.74.0-r1)
(5/6) Installing curl (7.74.0-r1)
(6/6) Installing openssl (1.1.1j-r0)
Executing busybox-1.32.1-r3.trigger
Executing ca-certificates-20191127-r5.trigger
OK: 9 MiB in 20 packages
Or just copy a statically built curl into Busybox:
https://github.com/moparisthebest/static-curl/releases
Radial has an overlay of busybox images adding cURL. docker pull radial/busyboxplus:curl
They also have a second images having cURL + Git. docker pull radial/busyboxplus:git
Install the curl binary from the source website
Replace binary-url with the URL of the binary file found from curl.se
export BINARY_URL="<binary-url>"
wget $BINARY_URL -O curl && install curl /bin; rm -f curl
Worked with busybox:latest image

Docker container for app tests with postgres database

I'm new to Docker.
I'm trying to run my node app tests in a Docker container.
I want to run the tests with a real postgres db.
I'm creating this container with the following Dockerfile:
# Set image
FROM postgres:alpine
# Install node latest
RUN apk add --update nodejs nodejs-npm
# Set working dir
WORKDIR .
# Copy the current directory contents into the container at .
ADD src src
ADD .env.testing .env
ADD package.json .
ADD package-lock.json .
# Run tests
CMD npm install && npm run coverage
From the image docs, when I run the container with:
$ docker run build-name -d postgres
I see that the container takes time to start postgresql service.
When I run the container without the "-d postgres" param:
$ docker run build-name
The service does not start and the tests fail due to "could not connect to server".
Questions:
A. How can I run the tests AFTER the postgresql service starts?
B. I saw some examples using docker-composer but can I do this without composer?
Thanks
Thanks to #Bogdan I found the complete solution:
Dockerfile should be:
# Set image
FROM postgres:alpine
# Install node latest
RUN apk add --update nodejs nodejs-npm
# Set working dir
WORKDIR .
# Copy the current directory contents into the container at .
ADD src src
ADD .env.testing .env
ADD package.json .
ADD package-lock.json .
# Install
RUN npm install
# Init container
CMD psql -U postgres -c "SELECT 1;" postgres
Build container:
$ docker build -t test .
Run container:
$ docker run --name startedtest -d test -d postgres
Run tests after conatiner is running:
$ docker exec startedtest some_create_schema_script && npm run coverage
If the goal is just to run the tests in the Postgres container, one solution could be to install NodeJs in your postgres:alpine derived image and run the container normally. Once the database is up, you can run npm using docker exec like this:
docker exec <container_id> npm run coverage

Executing wait-for-it.sh in python Docker container

I have a Python docker container that needs to wait until another container (postgres server) finishes setup. I tried the standard wait-for-it.sh but several commands weren't included. I tried a basic sleep (again in an sh file) but now it's reporting exec: 300: not found when trying to finally execute the command I'm waiting on.
How do I get around this (preferably without changing the image, or having to extend an image.)
I know I could also just run a Python script, but ideally I'd like to use wait-for-it.sh to wait for the server to turn up rather than just sleep.
Dockerfile (for stuffer):
FROM python:2.7.13
ADD ./stuff/bin /usr/local/bin/
ADD ./stuff /usr/local/stuff
WORKDIR /usr/local/bin
COPY requirements.txt /opt/updater/requirements.txt
COPY internal_requirements.txt /opt/stuff/internal_requirements.txt
RUN pip install -r /opt/stuff/requirements.txt
RUN pip install -r /opt/stuff/other_requirements.txt
docker-compose.yml:
version: '3'
services:
local_db:
build: ./local_db
ports:
- "localhost:5432:5432"
stuffer:
build: ./
depends_on:
- local_db
command: ["./wait-for-postgres.sh", "-t", "300", "localhost:5432", "--", "python", "./stuffing.py", "--file", "./afile"]
Script I want to use (but can't because no psql or exec):
#!/bin/bash
# wait-for-postgres.sh
set -e
host="$1"
shift
cmd="$#"
until psql -h "$host" -U "postgres" -c '\l'; do >&2 echo "Postgres is unavailable - sleeping"
sleep 1
done
>&2 echo "Postgres is up - executing command"
exec $cmd
Sergey's comment. I had wrong argument order. This issue had nothing to do with docker and everything to do with my inability to read.
I made an example so you can see it working:
https://github.com/nitzap/wait-for-postgres
On the other hand also you can have errors inside the execution of the script to validate that the service is working. You should not refer as localhost .... because that is within the contexts of containers, if you want to point to another container has to be through the name of the service.