Kubernetes giving each pod access to GPU - kubernetes

Question

Im new to Kubernetes, my goal is to create a serverless like architecture on GPUs (i.e fan out to 1000+ pods)
I understand a node may be a virtual or physical machine. I am using GKE to help manage k8s. My node machine config is n1-standard-4 with 1 x NVIDIA Tesla T4.
With that setup it seems I could only have 4 pods, if I wanted lets say 16 pods per node, I could use n1-standard-16.
Lets say we are using n1-standard-4 and ran 4 pods on that node, how can we give each node access to the GPU? Currently I am only able to run one pod, while the other pods stay on pending. This seems to only happen when I add the gpu resource in my YAML file.
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/name: load-balancer-example
name: hello-world
spec:
replicas: 4
selector:
matchLabels:
app.kubernetes.io/name: load-balancer-example
template:
metadata:
labels:
app.kubernetes.io/name: load-balancer-example
spec:
containers:
- image: CUSTOM_IMAGE_WITH_NVIDIA/CUDA/UBUNTU
name: test
ports:
- containerPort: 3000
resources:
limits:
nvidia.com/gpu: 1
Without the GPU resource and with a basic node container it seems to fan out fine. With the GPU resource I can only get one POD to run.

Answer 1

What you are creating is not a Pod but a Deployment with a replica count of 4, which is essentially 4 pods. All 4 of these pods are using your n1-standard-4 type of node.
There are certain limitations when it comes to using GPUs with pods. This is very different from CPU sharing. In short, GPUs are only supposed to be specified in the limits section, which means:
You can specify GPU limits without specifying requests because Kubernetes will use the limit as the request value by default.
You can specify GPU in both limits and requests but these two values must be equal.
You cannot specify GPU requests without specifying limits.
Containers (and Pods) do not share GPUs. There's no overcommitting of GPUs.
Each container can request one or more GPUs. It is not possible to request a fraction of a GPU.
You can read more about these limitations here.
Your best option, will be to create a node pool with your desired GPU type. This node pool will have # nodes = # pods in your deployment and each node will host only 1 pod, and will have 1 GPU of your choice. I suggest this instead of multiple GPUs/node because you want to have a fan-out/scale-out architecture, so more smaller nodes will be better than less larger nodes.
You can read more about how to do this on the GKE docs here.
Note that having n1-standard-4 doesn't mean you can have 4 pods on the node. It simply means the node has 4 vCPUs which you can share across as many pods as needed. But since you want to run GPU workloads, this node type should not matter much, as long as you attach the right amount of GPU resources.