How do you run GPU jobs in CI/CD pipelines?
GPU jobs in CI/CD run on runners that have a GPU attached: either a GPU runner from the CI provider or a self-hosted runner on GPU hardware, often a Pod on a Kubernetes GPU node pool that scales to zero between jobs. On GitHub Actions, the hosted option is a larger runner with one NVIDIA Tesla T4 and 16 GB of video memory at $0.052 per minute on Linux as of November 2026, available on the Team and Enterprise Cloud plans. Whichever option you use, keep CPU-only work off the GPU runner and pin driver and CUDA versions so results are reproducible.
When a CI job needs a GPU
Many GPU-related steps do not need a GPU. Compiling CUDA code with nvcc, building container images with CUDA libraries and running most unit tests work on ordinary CPU runners. A GPU is needed when the job executes GPU code:
| Job | Needs a GPU | Notes |
|---|---|---|
| Building CUDA code and images | No | Runs on CPU runners |
| Unit tests for model code | Usually not | Run on CPU with small inputs; mark GPU-only tests separately |
| GPU kernel and library tests | Yes | Correctness depends on the real driver and hardware |
| Model smoke tests | Yes | Load the model, run a few steps or a few inferences |
| Inference regression tests | Yes | Compare outputs or latency with a baseline |
| Performance benchmarks | Yes, dedicated | Shared or different GPUs make results incomparable |
Splitting the pipeline so only the last group runs on GPU runners is the largest single cost saving.
Option 1: GitHub-hosted GPU runners
GitHub offers GPU-powered larger runners with 4 vCPUs, 28 GB of RAM, 176 GB of SSD storage and one Tesla T4 with 16 GB of video memory, on Ubuntu or Windows. As of November 2026, the runner pricing reference lists $0.052 per minute for the Linux GPU runner and $0.102 for the Windows one. Larger runners are billed in public repositories too, and GitHub’s limits allow 100 concurrent GPU jobs on Team and Enterprise plans.
At that rate, a 15-minute GPU test job costs $0.78, and 20 such runs a day cost about $15.60. The main limits are the hardware: one T4 with 16 GB cannot load larger models, and it cannot test features of newer GPU generations.
Option 2: self-hosted GPU runners on Kubernetes
Self-hosted runners let you choose the GPU model, keep large model weights cached near the runner and reach private data. On Kubernetes, the usual setup combines:
- A GPU node pool with the vendor’s drivers and device plugin installed. The device plugin exposes GPUs as a schedulable resource such as
nvidia.com/gpu, which Pods request in theirlimits(Kubernetes GPU scheduling). - A dedicated runner scale set in Actions Runner Controller whose Pod template requests one GPU and tolerates the GPU nodes’ taint. If jobs run in a separate job container, make sure the GPU request reaches the Pod that runs the tests.
- Scale to zero. With no minimum runner count and a node autoscaler, GPU nodes exist only while GPU jobs run. The first job after a quiet period waits for a new node, its drivers and a large image pull.
For short tests that use a fraction of a GPU, time slicing or Multi-Instance GPU (MIG) lets several jobs share one card. The comparison of GPU sharing methods explains the isolation each one gives. Do not share GPUs for benchmark jobs.
Pin drivers and CUDA versions
A GPU test can pass on one runner and fail on another because of the software stack, not the code. The NVIDIA driver runs on the node, while the CUDA toolkit and libraries come from the container image, and a CUDA version in the image needs a driver that supports it. To keep runs reproducible:
- Pin the CUDA base image by tag or digest instead of using a floating tag.
- Pin the driver version on self-hosted GPU nodes and upgrade it deliberately.
- Run
nvidia-smias the first step of every GPU job, so the log records the GPU model and driver version. - Pin framework versions, such as PyTorch builds for a specific CUDA release, in the lock file.
Keep the cost per run down
GPU minutes cost several times more than CPU minutes, so the usual CI levers matter more:
- Trigger GPU jobs only for changes to model, kernel or inference code, using path filters, or run them nightly and before releases.
- Cache model weights and datasets on the runner or in a nearby object store instead of downloading them every run.
- Use small test models or reduced inputs for smoke tests.
- Set
timeout-minutesso a hung job does not hold a GPU for hours. - On self-hosted pools, use spot or preemptible GPU instances for jobs that can be retried.
The broader article on reducing GPU costs covers the same levers for training and inference.
GPU runners on Cloudfleet
On Cloudfleet Kubernetes Engine (CFKE), a Pod that requests nvidia.com/gpu makes node auto-provisioning create a GPU node when a Fleet is configured, and CFKE installs the NVIDIA drivers and Container Toolkit automatically (GPU-based workloads). Fleet taints keep a Fleet dedicated to GPU jobs, so other workloads do not land on those nodes.