GPU-based workloads

Cloudfleet Kubernetes Engine (CFKE) supports NVIDIA GPUs for workloads that require GPU acceleration. This guide explains how to provision nodes with NVIDIA GPUs and how to schedule workloads on them.

Adding nodes with NVIDIA GPUs

Depending on the node type, you can add nodes with NVIDIA GPUs to your cluster in two ways:

• Self-managed nodes: To enable GPU support when you add a self-managed node, add the --install-nvidia-drivers flag to the cloudfleet clusters add-self-managed-node CLI command. For more information, refer to the Self-managed nodes document.

• Auto-provisioned nodes: Fleets support automatic provisioning of cloud instances with NVIDIA GPUs. If a workload requires a GPU and no existing node can satisfy this requirement, CFKE automatically provisions a new GPU-equipped node. The next section describes the labels for automated GPU node provisioning.

In both cases, the NVIDIA drivers and the NVIDIA Container Toolkit are installed and configured automatically for GPU workloads. Use of the NVIDIA drivers is subject to the NVIDIA Driver License Agreement. By using the NVIDIA drivers, you agree to the terms of the NVIDIA Driver License Agreement.

Scheduling GPU-based workloads with CFKE

The node object shows the number of GPUs in the node as the capacity field nvidia.com/gpu. Use this field to schedule workloads on GPU-enabled nodes. For example:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-test
spec:
  restartPolicy: OnFailure
  containers:
  - name: cuda-vector-add
    image: "nvidia/samples:vectoradd-cuda11.6.0"
    resources:
      limits:
        nvidia.com/gpu: 1

In this example, the nvidia.com/gpu field requests one GPU. The Kubernetes scheduler assigns this workload to a node with at least one available GPU. If no GPU-enabled node is available and a Fleet is configured, CFKE provisions a new GPU-equipped node.

Requesting specific GPU models

The nvidia.com/gpu field sets the number of GPUs, but not the GPU model. To schedule workloads on specific GPU models, use the following labels:

• cfke.io/accelerator-manufacturer: Manufacturer (for example, NVIDIA).
• cfke.io/accelerator-name: Model name (for example, V100).
• cfke.io/accelerator-memory: Memory size in GiB (for example, 24).

To schedule a workload on a node with an NVIDIA V100 GPU:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-test
spec:
  nodeSelector:
    cfke.io/accelerator-name: V100
  restartPolicy: OnFailure
  containers:
  - name: cuda-vector-add
    image: "nvidia/samples:vectoradd-cuda11.6.0"
    resources:
      limits:
        nvidia.com/gpu: 1

If multiple clouds support the specified GPU (for example, AWS and GCP for V100), CFKE provisions the most cost-effective option. To target a specific cloud provider, use the cfke.io/provider label:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-test
spec:
  nodeSelector:
    cfke.io/accelerator-name: V100
    cfke.io/provider: GCP
  restartPolicy: OnFailure
  containers:
  - name: cuda-vector-add
    image: "nvidia/samples:vectoradd-cuda11.6.0"
    resources:
      limits:
        nvidia.com/gpu: 1

Available GPU models and providers

After CFKE provisions a suitable node, it updates the nvidia.com/gpu field on the node object. The scheduler can then schedule the workload.

Your cloud account must have the necessary quota to provision GPU-equipped instances.

For more details on GPU scheduling, refer to the Kubernetes Documentation.

GPU sharing with time slicing and MPS

CFKE supports GPU sharing technologies that let multiple containers access a single physical GPU. GPU sharing improves GPU utilization and reduces costs. Two primary methods are available: time slicing and Multi-Process Service (MPS).

GPU time slicing uses NVIDIA’s built-in time-sharing capability. It switches context between processes with instruction-level preemption, which is available in GPUs from the Pascal architecture onward. Time slicing provides software-level isolation between containers: address space, performance, and error isolation. When time slicing is enabled, each container requests one full GPU (nvidia.com/gpu: 1), and all containers share the physical GPU.

Time slicing is optimal for bursty or interactive workloads with idle periods, testing and prototyping environments, and workloads that don’t need dedicated GPU resources.

However, time slicing does not enforce memory limits between shared workloads. The rapid context switching can also add performance overhead. Multi-Process Service (MPS) is an alternative sharing strategy. It lets multiple CUDA processes from different containers share a single GPU context. Time slicing relies on context switching and time-multiplexing. MPS instead submits work streams to the GPU concurrently, through a control daemon that coordinates access.

MPS reduces context switching overhead and has lower latency than time slicing. It is particularly useful for inference workloads with small batch sizes, multiple small CUDA processes, and applications that are CUDA-aware. MPS is generally better suited for inference workloads than for training workloads.

As with time slicing, MPS requires each container to request one full GPU (nvidia.com/gpu: 1). MPS needs proper configuration, and it may not suit all workload types.

GPU time slicing and MPS are cluster-level features, and both are disabled by default. You configure GPU sharing directly on the cluster:

1. Choose a sharing strategy (time_slicing or mps).
2. Set the maximum number of Pods that can share a single GPU.

You can set these when you create or edit a cluster, in the cluster’s feature settings. Use the Cloudfleet Console, the CLI, or the Terraform provider. GPU sharing is available on the Pro and Enterprise tiers.