I am trying to run a docker image in Google Container Engine. The instance comes up with no running docker images. I can ssh and run the docker commands and the service comes up. But nothing happens when I just launch the instance from the terminal. Can some one take a look at what I am doing wrong.
My docker file looks like
FROM golang
RUN mkdir -p /app
COPY . /app
RUN go get golang.org/x/tools/cmd/present
ENTRYPOINT cd /app && /go/bin/present -http=":8080"
EXPOSE 8080
containers.yaml looks like
version: v1beta3
containers:
- name: talks
image: sheki/talks
ports:
- name: http-port
containerPort: 8080
hostPort: 80'
The command to launch the instance is
gcloud compute instances create zoop \
--image container-vm \
--metadata-from-file google-container-manifest=containers.yaml \
--zone us-central1-a \
--machine-type f1-micro
You mentioned in your question that you are using google container engine, but in fact you are using the container vm (which is a bit different). If you want to use container engine, please check out the documentation to create a container cluster.
I ran your example, and in /var/log/kubelet.log saw the following error:
E0519 17:05:41.285556 2414 http.go:54] Failed to read URL: http://metadata.google.internal/computeMetadata/v1beta1/instance/attributes/google-cont
ainer-manifest: received 'version: v1beta3
containers:
- name: talks
image: sheki/talks
ports:
- name: http-port
containerPort: 8080
hostPort: 80'
', but couldn't parse as neither single (error unmarshaling JSON: json: cannot unmarshal string into Go value of type int: {Version:v1beta3 ID: UUID:
Volumes:[] Containers:[{Name:talks Image:sheki/talks Entrypoint:[] Command:[] WorkingDir: Ports:[{Name:http-port HostPort:0 ContainerPort:8080 Proto
col: HostIP:}] Env:[] Resources:{Limits:map[] Requests:map[]} CPU:0 Memory:0 VolumeMounts:[] LivenessProbe:<nil> ReadinessProbe:<nil> Lifecycle:<nil>
TerminationMessagePath: Privileged:false ImagePullPolicy: Capabilities:{Add:[] Drop:[]}}] RestartPolicy:{Always:<nil> OnFailure:<nil> Never:<nil>} D
NSPolicy: HostNetwork:false}) or multiple manifests (error unmarshaling JSON: json: cannot unmarshal object into Go value of type []v1beta1.Container
Manifest: []) nor single (kind not set in '{"containers":[{"image":"sheki/talks","name":"talks","ports":[{"containerPort":8080,"hostPort":"80'","name
":"http-port"}]}],"version":"v1beta3"}') or multiple pods (kind not set in '{"containers":[{"image":"sheki/talks","name":"talks","ports":[{"container
Port":8080,"hostPort":"80'","name":"http-port"}]}],"version":"v1beta3"}').
It looks like the documentation for container vms is out of date.
Related
A new version of MLFlow (1.23) provided a --serve-artifacts option (via this pull request) along with some example code. This should allow me to simplify the rollout of a server for data scientists by only needing to give them one URL for the tracking server, rather than a URI for the tracking server, URI for the artifacts server, and a username/password for the artifacts server. At least, that's how I understand it.
A complication that I have is that I need to use podman instead of docker for my containers (and without relying on podman-compose). I ask that you keep those requirements in mind; I'm aware that this is an odd situation.
What I did before this update (for MLFlow 1.22) was to create a kubernetes play yaml config, and I was successfully able to issue a podman play kube ... command to start a pod and from a different machine successfully run an experiment and save artifacts after setting the appropriate four env variables. I've been struggling with getting things working with the newest version.
I am following the docker-compose example provided here. I am trying a (hopefully) simpler approach. The following is my kubernetes play file defining a pod.
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: "2022-01-14T19:07:15Z"
labels:
app: mlflowpod
name: mlflowpod
spec:
containers:
- name: minio
image: quay.io/minio/minio:latest
ports:
- containerPort: 9001
hostPort: 9001
- containerPort: 9000
hostPort: 9000
resources: {}
tty: true
volumeMounts:
- mountPath: /data
name: minio-data
args:
- server
- /data
- --console-address
- :9001
- name: mlflow-tracking
image: localhost/mlflow:latest
ports:
- containerPort: 80
hostPort: 8090
resources: {}
tty: true
env:
- name: MLFLOW_S3_ENDPOINT_URL
value: http://127.0.0.1:9000
- name: AWS_ACCESS_KEY_ID
value: minioadmin
- name: AWS_SECRET_ACCESS_KEY
value: minioadmin
command: ["mlflow"]
args:
- server
- -p
- 80
- --host
- 0.0.0.0
- --backend-store-uri
- sqlite:///root/store.db
- --serve-artifacts
- --artifacts-destination
- s3://mlflow
- --default-artifact-root
- mlflow-artifacts:/
# - http://127.0.0.1:80/api/2.0/mlflow-artifacts/artifacts/experiments
- --gunicorn-opts
- "--log-level debug"
volumeMounts:
- mountPath: /root
name: mlflow-data
volumes:
- hostPath:
path: ./minio
type: Directory
name: minio-data
- hostPath:
path: ./mlflow
type: Directory
name: mlflow-data
status: {}
I start this with podman play kube mlflowpod.yaml. On the same machine (or a different one, it doesn't matter), I have cloned and installed mlflow into a virtual environment. From that virtual environment, I set an environmental variable MLFLOW_TRACKING_URI to <name-of-server>:8090. I then run the example.py file in the mlflow_artifacts example directory. I get the following response:
....
botocore.exceptions.NoCredentialsError: Unable to locate credentials
Which seems like the client needs the server credentials to minIO, which I thought the proxy was supposed to take care of.
If I also provide the env variables
$env:MLFLOW_S3_ENDPOINT_URL="http://<name-of-server>:9000/"
$env:AWS_ACCESS_KEY_ID="minioadmin"
$env:AWS_SECRET_ACCESS_KEY="minioadmin"
Then things work. But that kind of defeats the purpose of the proxy...
What is it about the proxy setup via kubernates play yaml and podman that is going wrong?
Just in case anyone stumbles upon this, I had same issue based on your description. However the problem on my side was that I was that I tried to test this with a preexisting experiment (default), and I did not create new one, so the old setting carried over, thus resulting in MLFlow trying to use s3 trough credentials and not https.
Hope this helps at least some of you out there.
I'm trying to deploy Mongo onto Azure Container Instances as part of a container group. To do this, I use a Storage Account with a file share to persist the mongo data. It's impossible to mount the volume in the /data/db default location, so I mount it elsewhere and start mongod using the --db-path flag. This all works fine using the CLI, full command below.
However, when I want to translate all of these commands into my YAML config file it doesn't work. Mongo crashes out with an unknown file or directory error. If I start the container without the --db-path flag but still mount the volume, I can exec into the running container and see that the volume is there and is attached. I can even manually create folders in the share via the Azure Portal and see them appear in the container.
Documentation and examples are a little thin on the ground, especially YAML based examples. The biggest difference with the container group is having to define a named volume separate from the container which is used by the volumeMounts property. Is it just a syntax error? Are the CLI command and the YAML not equivalent in some way?
CLI Command
az container create
--resource-group rsenu-hassPilots-group
--name mongo
--image mongo
--azure-file-volume-account-name <account>
--azure-file-volume-account-key "<key>" --azure-file-volume-share-name mongodata
--azure-file-volume-mount-path "/data/mongoaz"
--ports 27017
--cpu 1
--ip-address public
--memory 1.5
--os-type Linux
--protocol TCP
--command-line "mongod --dbpath=/data/mongoaz"
YAML Config
apiVersion: 2018-10-01
location: uksouth
name: trustNewArtGroup
properties:
containers:
- name: mongo
properties:
image: mongo:4.2.3
resources:
requests:
cpu: 1
memoryInGb: 1.5
ports:
- port: 27017
volumeMounts:
- name: database
mountPath: /data/azstorage
environmentVariables:
- name: 'MONGO_INITDB_DATABASE'
value: 'trust-new-art-db'
command:
- "mongod --dbpath=/data/azstorage"
osType: Linux
ipAddress:
type: Public
dnsNameLabel: trustnewart
ports:
- protocol: tcp
port: '27017'
volumes:
- name: database
azureFile:
sharename: mongodata
storageAccountName: <account>
storageAccountKey: <key>
tags: null
type: Microsoft.ContainerInstance/containerGroups
With a bit of help from this page in the documentation, I've discovered it was a syntax issue. The correct way to override the entrypoint in a YAML config file is as follows:
command: ['mongod', '--dbpath', '/data/azstorage']
I just deployed a docker with Postgres on it on AWS EKS.
Below is the description details.
How do i access or test if postgres is working. I tried accessing both IP with post within VPC from worker node.
psql -h #IP -U #defaultuser -p 55432
Below is the deployment.
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: postgres
spec:
replicas: 1
template:
metadata:
labels:
app: postgres
spec:
containers:
- name: postgres
image: postgres:10.4
imagePullPolicy: "IfNotPresent"
ports:
- containerPort: 55432
# envFrom:
# - configMapRef:
# name: postgres-config
volumeMounts:
- mountPath: /var/lib/postgresql/data
name: postgredb
volumes:
- name: postgredb
persistentVolumeClaim:
claimName: efs
Surprisingly I am able to connect to psql but on 5432. :( Not sure what I am doing wrong. I passed containerPort as 55432
In short, you need to run the following command to expose your database on 55432 port.
kubectl expose deployment postgres --port=55432 --target-port=5432 --name internal-postgresql-svc
From now on, you can connect to it via port 55432 from inside your cluster by using the service name as a hostname, or via its ClusterIP address:
kubectl get internal-postgresql-svc
What you did in your deployment manifest file, you just attached additional information about the network connections a container uses, between misleadingly, because your container exposes 5432 port only (you can verify it by your self here). You should use a Kubernetes Service - abstraction which enables access to your PODs, and does the necessary port mapping behind the scene.
Please check also different port Types, if you want to expose your postgresql database outside of the Kubernetes cluster.
To test if progress is running fine inside POD`s container:
kubectl run postgresql-postgresql-client --rm --tty -i --restart='Never' --namespace default --image bitnami/postgresql --env="PGPASSWORD=<HERE_YOUR_PASSWORD>" --command -- psql --host <HERE_HOSTNAME=SVC_OR_IP> -U <HERE_USERNAME>
I want to set an environment variable (I'll just name it ENV_VAR_VALUE) to a container during deployment through Kubernetes.
I have the following in the pod yaml configuration:
...
...
spec:
containers:
- name: appname-service
image: path/to/registry/image-name
ports:
- containerPort: 1234
env:
- name: "ENV_VAR_VALUE"
value: "some.important.value"
...
...
The container needs to use the ENV_VAR_VALUE's value.
But in the container's application logs, it's value always comes out empty.
So, I tried checking it's value from inside the container:
$ kubectl exec -it appname-service bash
root#appname-service:/# echo $ENV_VAR_VALUE
some.important.value
root#appname-service:/#
So, the value was successfully set.
I imagine it's because the environment variables defined from Kubernetes are set after the container is already initialized.
So, I tried overriding the container's CMD from the pod yaml configuration:
...
...
spec:
containers:
- name: appname-service
image: path/to/registry/image-name
ports:
- containerPort: 1234
env:
- name: "ENV_VAR_VALUE"
value: "some.important.value"
command: ["/bin/bash"]
args: ["-c", "application-command"]
...
...
Even still, the value of ENV_VAR_VALUE is still empty during the execution of the command.
Thankfully, the application has a restart function
-- because when I restart the app, ENV_VAR_VALUE get used successfully.
-- I can at least do some other tests in the mean time.
So, the question is...
How should I configure this in Kubernetes so it isn't a tad too late in setting the environment variables?
As requested, here is the Dockerfile.
I apologize for the abstraction...
FROM ubuntu:18.04
RUN apt-get update && apt-get install -y some-dependencies
COPY application-script.sh application-script.sh
RUN ./application-script.sh
# ENV_VAR_VALUE is set in this file which is populated when application-command is executed
COPY app-config.conf /etc/app/app-config.conf
CMD ["/bin/bash", "-c", "application-command"]
You can try also running two commands in Kubernetes POD spec:
(read in env vars): "source /env/required_envs.env" (would come via secret mount in volume)
(main command): "application-command"
Like this:
containers:
- name: appname-service
image: path/to/registry/image-name
ports:
- containerPort: 1234
command: ["/bin/sh", "-c"]
args:
- source /env/db_cred.env;
application-command;
Why don't you move the
RUN ./application-script.sh
below
COPY app-config.conf /etc/app/app-config.conf
Looks like the app is running before the env conf is available for it.
Using fleet I can specify a command to be run inside the container when it is started. It seems like this should be easily possible with Kubernetes as well, but I can't seem to find anything that says how. It seems like you have to create the container specifically to launch with a certain command.
Having a general purpose container and launching it with different arguments is far simpler than creating many different containers for specific cases, or setting and getting environment variables.
Is it possible to specify the command a kubernetes pod runs within the Docker image at startup?
I spend 45 minutes looking for this. Then I post a question about it and find the solution 9 minutes later.
There is an hint at what I wanted inside the Cassandra example. The command line below the image:
id: cassandra
kind: Pod
apiVersion: v1beta1
desiredState:
manifest:
version: v1beta1
id: cassandra
containers:
- name: cassandra
image: kubernetes/cassandra
command:
- /run.sh
cpu: 1000
ports:
- name: cql
containerPort: 9042
- name: thrift
containerPort: 9160
env:
- key: MAX_HEAP_SIZE
value: 512M
- key: HEAP_NEWSIZE
value: 100M
labels:
name: cassandra
Despite finding the solution, it would be nice if there was somewhere obvious in the Kubernetes project where I could see all of the possible options for the various configuration files (pod, service, replication controller).
for those looking to use a command with parameters, you need to provide an array
for example
command: [ "bin/bash", "-c", "mycommand" ]
or also
command:
- "bin/bash"
- "-c"
- "mycommand"
To answer Derek Mahar's question in the comments above:
What is the purpose of args if one could specify all arguments using command?
Dockerfiles can have an Entrypoint only or a CMD only or both of them together.
If used together then whatever is in CMD is passed to the command in ENTRYPOINT as arguments i.e.
ENTRYPOINT ["print"]
CMD ["hello", "world"]
So in Kubernetes when you specify a command i.e.
command: ["print"]
It will override the value of Entrypoint in the container's Dockerfile.
If you only specify arguments then those arguments will be passed to whatever command is in the container's Entrypoint.
In order to specify the command a kubernetes pod runs within the Docker image at startup we need to include the command and args fields inside the yaml file for command and arguments to be passed. For example,
apiVersion: v1
kind: Pod
metadata:
name: command-demo
labels:
purpose: demo-command
spec:
containers:
- name: command-demo-container
image: ubuntu
command: ["/bin/sh"]
args: ["-c", "while true; do echo hello; sleep 10;done"]
Additionally to the accepted answer, you can use variables with values from secrets in the commands as follows:
command: ["/some_command","-instances=$(<VARIABLE_NAME>)"]
env:
- name: <VARIABLE_NAME>
valueFrom:
secretKeyRef:
name: <secret_name>
key: <secret_key>