I installed the Dask Helm Chart with a revised values.yaml to have 10 workers, however instead of Dask Workers I want to create Dash CUDA Workers to take advantage of the NVIDIA GPUs on my multi-node cluster.
I tried to modify the values.yaml as follows to get Dask CUDA workers instead of Dask Workers, but the worker pods are able to start. I did already install the NVIDIA GPUs on all my nodes on the Kubernetes per the official instructions so I'm not sure what DASK needs to see in order to create the Dask-Cuda-Workers.
worker:
name: worker
image:
repository: "daskdev/dask"
tag: 2.19.0
pullPolicy: IfNotPresent
dask_worker: "dask-cuda-worker"
#dask_worker: "dask-worker"
pullSecrets:
# - name: regcred
replicas: 15
default_resources: # overwritten by resource limits if they exist
cpu: 1
memory: "4GiB"
env:
# - name: EXTRA_APT_PACKAGES
# value: build-essential openssl
# - name: EXTRA_CONDA_PACKAGES
# value: numba xarray -c conda-forge
# - name: EXTRA_PIP_PACKAGES
# value: s3fs dask-ml --upgrade
resources: {}
# limits:
# cpu: 1
# memory: 3G
# nvidia.com/gpu: 1
# requests:
# cpu: 1
# memory: 3G
# nvidia.com/gpu: 1
As dask-cuda-worker is not yet officially in the dask image you will need to pull the image a different image: rapidsai/rapidsai:latest
Related
In Kubernetes I would like to deploy Apache Nifi Cluster in StatefulSet with 3 nodes.
Problem is I would like to modify node adresses recursively in an init container in my yaml file.
I have to modify these parameters for each nodes in Kubernetes:
'nifi.remote.input.host'
'nifi.cluster.node.address'
I need to have these FQDN added recursively in Nifi properties:
nifi-0.nifi.NAMESPACENAME.svc.cluster.local
nifi-1.nifi.NAMESPACENAME.svc.cluster.local
nifi-2.nifi.NAMESPACENAME.svc.cluster.local
I have to modify the properties before deploying so I tried the following init container but doesn't work :
initContainers:
- name: modify-nifi-properties
image: busybox:v01
command:
- sh
- -c
- |
# Modify nifi.properties to use the correct hostname for each node
for i in {1..3}; do
sed -i "s/nifi-$((i-1))/nifi-$((i-1)).nifinamespace.nifi.svc.cluster.local/g" /opt/nifi/conf/nifi.properties
done
resources:
requests:
cpu: 100m
memory: 100Mi
limits:
cpu: 100m
memory: 100Mi
How can I do it ?
There is such structure of the project, I'm trying to understand:
charts/
spark-0.0.1-100.tgz
templates/
Chart.yaml
values.yaml
Chart.yaml
appVersion: 0.1.0
dependencies:
- name: spark
version: "0.0.1-100"
repository: https://helm.<corporation>.com/<project>
condition: spark.enabled
values.yaml (some values are omitted for simplicity)
spark:
enabled: true
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/spark-service-account
image:
tag: "3.3.0-dev-28"
extraEnv:
- name: AWS_STS_REGIONAL_ENDPOINTS
value: regional
Master:
Requests:
Cpu: ...
Memory: ...
Disk: ...
Limits:
Cpu: ...
Memory: ...
Disk: ...
Worker:
Replicas: 3
Requests:
Cpu: ...
Memory: ...
Disk: ...
Limits:
Cpu: ...
Memory: ...
Disk: ...
zookeeper:
host: "project-zookeeper"
port: 2181
Then, I have unzipped charts/spark-0.0.1-100.tgz into folder charts/spark/:
charts/
spark/
templates/
Chart.yaml
values.yaml
charts/spark/values.yaml:
global:
aci:
sdrAppname: spark
image:
repository: "docker.<corporation>.com/<project>/spark"
tag: "1.0.1"
spark:
path: "/opt/spark"
user: 1000
group: 1000
config: |
SPARK_WORKER_OPTS="-Dspark.worker.cleanup.enabled=true -Dspark.worker.cleanup.appDataTtl=21600"
Master:
Name: master
Replicas: 1
Component: "spark-core"
Requests:
Cpu: ...
Memory: ...
Disk: ...
Limits:
Cpu: ...
Memory: ...
Disk: ...
ServicePort: <port>
ContainerPort: <port>
RestEnabled: "true"
RestPort: <port>
ServiceType: LoadBalancer
WebUi:
Name: webui
ServicePort: <port>
ContainerPort: <port>
The question is, how values from values.yaml and charts/spark/values.yaml are being corresponded?
Are values from root values.yaml are replaced with values from charts/spark/values.yaml?
Thank you in advance.
The question is, how values from values.yaml and charts/spark/values.yaml are being corresponded?
The short version is they have almost nothing to do with one another, they're just the defaults used by their respective charts. The medium version is that the outer chart can supersede values in the subordinate charts if it chooses to, but ultimately the user has the final word in that discussion because helm --values win out over the defaults (same for usage of --set, but --values are far easier to discuss due to not having to delve into that --set DSL)
The subcharts are not aware of the parent chart's defaults. The parent chart doesn't have to be aware of the child chart's values, and cannot -- itself -- refer to the subchart's defaults
Are values from root values.yaml are replaced with values from charts/spark/values.yaml?
For the most part, no, they're completely separate namespaces. However, as we saw above, for every - name: key in the dependencies: list, those top-level keys in the parent chart's values.yaml becomes special in that it overlays the values on top of the subordinate chart's defaults (you see in your example of { spark: { path: "/opt/spark" } }; that top level spark: key matches up with - name: in the dependencies: list)
It's kind of like "duck typing" though, because the top level key is free to use any random structure and the child chart will use ones that it is aware of. For example:
# values.yaml
spark:
path: /opt/spark
value-of-pi: 3.1415
is perfectly legal in the top chart's values.yaml even though the child spark chart will only notice the { spark: { path: "" } } because it does not have any {{ .Values.value-of-pi }} in its templates
But, again, for clarity: the top level values.yaml, even if it includes spark: { path: /alpha } can be superseded by the user with --values <(echo '{spark: {path: /beta } }') and the resulting spark install will have path: /beta when that chart is templated out
What is the best way to enable BBR on default for my clusters?
In this link, I didn't see an option for controlling the congestion control.
Google BBR can only be enabled in Linux operating systems. By default the Linux servers uses Reno and CUBIC but the latest version kernels also includes the google BBR algorithms and can be enabled manually.
To enable it on CentOS 8 add below lines in /etc/sysctl.conf and issue command sysctl -p
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
For more Linux distributions you can refer to this link.
Maybe set it as part of your Deployment spec:
spec:
initContainers:
- name: sysctl-buddy
image: busybox:1.29
securityContext:
privileged: true
command: ["/bin/sh"]
args:
- -c
- sysctl -w net.core.default_qdisc=fq net.ipv4.tcp_congestion_control=bbr
resources:
requests:
cpu: 1m
memory: 1Mi
I’m having some trouble limiting my Pods access to the GPUs available on my cluster.
Here is my .yaml:
kind: Pod
metadata:
name: train-gpu
spec:
containers:
- name: train-gpu
image: index.docker.io/myprivaterepository/train:latest
command: ["sleep"]
args: ["100000"]
resources:
limits:
nvidia.com/gpu: 1 # requesting 1 GPU
When I run the nvidia-smi command inside of this pod all of the GPUs show up, rather than just 1.
Any advice would be much appreciated.
Some information that may be useful:
Kubernetes version:
Client Version: version.Info{Major:“1”, Minor:“16”, GitVersion:“v1.16.1”, GitCommit:“d647ddbd755faf07169599a625faf302ffc34458”, GitTreeState:“clean”, BuildDate:“2019-10-07T14:30:40Z”, GoVersion:“go1.12.10”, Compiler:“gc”, Platform:“linux/amd64”}
Server Version: version.Info{Major:“1”, Minor:“15”, GitVersion:“v1.15.3”, GitCommit:“2d3c76f9091b6bec110a5e63777c332469e0cba2”, GitTreeState:“clean”, BuildDate:“2019-08-19T11:05:50Z”, GoVersion:“go1.12.9”, Compiler:“gc”, Platform:“linux/amd64”}
Docker base image:
FROM nvidia/cuda:10.1-base-ubuntu18.04
I am trying to run DPDK L2FWD application on a container managed by Kubernetes.
To achieve this I have done the below steps -
I have created single node K8s setup where both master and client are running on host machine. As network plug-in, I have used Calico Network.
To create customized DPDK docker image, I have used the below Dockerfile
FROM ubuntu:16.04 RUN apt-get update
RUN apt-get install -y net-tools
RUN apt-get install -y python
RUN apt-get install -y kmod
RUN apt-get install -y iproute2
RUN apt-get install -y net-tools
ADD ./dpdk/ /home/sdn/dpdk/
WORKDIR /home/sdn/dpdk/
To run DPDK application inside POD, below host's directories are mounted to POD with privileged access:
/mnt/huge
/usr
/lib
/etc
Below is k8s deployment yaml used to create the POD
apiVersion: v1
kind: Pod
metadata:
name: dpdk-pod126
spec:
containers:
- name: dpdk126
image: dpdk-test126
imagePullPolicy: IfNotPresent
command: ["/bin/sh"]
args: ["-c", "while true; do echo hello; sleep 10;done"]
resources:
requests:
memory: "2Gi"
cpu: "100m"
volumeMounts:
- name: hostvol1
mountPath: /mnt/huge
- name: hostvol2
mountPath: /usr
- name: hostvol3
mountPath: /lib
- name: hostvol4
mountPath: /etc
securityContext:
privileged: true
volumes:
- name: hostvol1
hostPath:
path: /mnt/huge
- name: hostvol2
hostPath:
path: /usr
- name: hostvol3
hostPath:
path: /home/sdn/kubernetes-test/libtest
- name: hostvol4
hostPath:
path: /etc
Below configurations are already done in host -
Huge page mounting.
Interface binding in user space.
After successful creation of POD, when trying to run a DPDK L2FWD application inside POD, I am getting the below error -
root#dpdk-pod126:/home/sdn/dpdk# ./examples/l2fwd/build/l2fwd -c 0x0f -- -p 0x03 -q 1
EAL: Detected 16 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: No free hugepages reported in hugepages-1048576kB
EAL: 1007 hugepages of size 2097152 reserved, but no mounted hugetlbfs found for that size
EAL: FATAL: Cannot get hugepage information.
EAL: Cannot get hugepage information.
EAL: Error - exiting with code: 1
Cause: Invalid EAL arguments
According to this, you might be missing
medium: HugePages from your hugepage volume.
Also, hugepages can be a bit finnicky. Can you provide the output of:
cat /proc/meminfo | grep -i huge
and check if there's any files in /mnt/huge?
Also maybe this can be helpful. Can you somehow check if the hugepages are being mounted as mount -t hugetlbfs nodev /mnt/huge?
First of all, you have to verify, that you have enough hugepages in your system. Check it with kubectl command:
kubectl describe nodes
where you could see something like this:
Capacity:
cpu: 12
ephemeral-storage: 129719908Ki
hugepages-1Gi: 0
hugepages-2Mi: 8Gi
memory: 65863024Ki
pods: 110
If your hugepages-2Mi is empty, then your k8s don't see mounted hugepages
After mounting hugepages into your host, you can prepare your pod to work with hugepages. You don't need to mount hugepages folder as you shown. You can simply add emptyDir volume like this:
volumes:
- name: hugepage-2mi
emptyDir:
medium: HugePages-2Mi
HugePages-2Mi is a specific resource name that corresponds with hugepages of 2Mb size. If you want to use 1Gb size hugepages then there is another resource for it - hugepages-1Gi
After defining the volume, you can use it in volumeMounts like this:
volumeMounts:
- mountPath: /hugepages-2Mi
name: hugepage-2mi
And there is one additional step. You have to define resource limitations for hugepages usage:
resources:
limits:
hugepages-2Mi: 128Mi
memory: 128Mi
requests:
memory: 128Mi
After all this steps, you can run your container with hugepages inside container
As #AdamTL mentioned, you can find additional info here