Skip to content
Christopher Shaun Godwin cgodwin.io
Blog

kubernetes

1 post with the tag “kubernetes”

Kubernetes 1.37 Garhwal: Gang Scheduling and Device Taints Fix the GPU Deadlock

Christopher Shaun Godwin
Author

Kubernetes v1.37 “Garhwal” landed on August 26, 2026. This is the release where Kubernetes stops scheduling pods one at a time and starts scheduling AI workloads as groups. The release carries 67 enhancements: 16 to Stable, 23 to Beta, 27 into Alpha, and 1 deprecation (Kubernetes Blog, 2026).

If you run GPUs, this release changes your day job. Dynamic Resource Allocation (DRA) moved four features to Stable in one shot. Gang scheduling, the fix for the distributed-training deadlock, hit Beta. And the removal list put kube-dns and IPVS on the clock.

The release by the numbers

Section titled “The release by the numbers”
  • 67 enhancements: 16 Stable, 23 Beta, 27 Alpha, 1 deprecation (Kubernetes Blog, 2026).
  • A 15-week cycle with contributions from over 1,700 individuals across 212 companies (The Register, 2026).
  • 118 API reviews, the most in project history, up from 88 in v1.36 and 41 in v1.21 back in April 2021 (Cloud Native Now, 2026).
  • Release lead Dipesh Rawat, and a theme named Garhwal, the Himalayan region of Uttarakhand in northern India (Network World, 2026).

The 118-review number is the hard one to absorb. The team handled it because the project formalized declarative validation in the previous release. That system generates API validation code from the type definitions instead of hand-writing every check (Cloud Native Now, 2026).

DRA grows up

Section titled “DRA grows up”

DRA is how Kubernetes assigns GPUs and other accelerators. The DRA core APIs reached general availability in v1.34 (Cloud Native Now, 2026). v1.37 graduates four more pieces to Stable (Kubermatic, 2026).

  1. ResourceClaim device status (KEP-4817) — drivers report per-device data on .status.devices. Before this field, a Pod could request a DRA-managed network device and no other component could learn the IP assigned to it. DRA is now fully usable for attaching secondary network interfaces (Kubernetes Blog, 2026).

  2. Extended resources through a DRA driver (KEP-5004) — assign an extended resource name such as abc.example/gpu: 3 to a DeviceClass. DRA allocates the device with no device plugin in the path. Existing manifests keep working, which makes this the migration route off device plugins (Kubermatic, 2026).

  3. Device taints and tolerations (KEP-5055) — taint one overheating GPU instead of draining the node it sits in. A driver can taint a single device, or an administrator can write a DeviceTaintRule by selection criteria, such as every device one driver manages. Claims that still need the degraded device declare tolerations (Kubernetes Blog, 2026). The old model was worse. A driver’s only option for a failing device was to pull it out of the ResourceSlice entirely. That stopped everyone from using it, including healthy running workloads (MetalBear, 2026).

  4. A standard numaNode device attribute — resource.kubernetes.io/numaNode becomes the shared attribute name. Devices managed by different DRA drivers now compare on the same NUMA placement (Kubernetes Blog, 2026).

Gang scheduling kills the partial-deadlock

Section titled “Gang scheduling kills the partial-deadlock”

The default scheduler places pods one by one. A 32-replica training job with only 20 free GPUs used to take the 20 and sit. Those 20 pods hold resources that can never finish the job, and other workloads starve behind them (Kubernetes Blog, 2026).

Gang scheduling flips the rule. The scheduler commits the group only when cluster capacity covers the whole group. The beta runs through the Workload API and the PodGroup concept as an all-or-nothing strategy (KEP-4671) (Kubernetes Blog, 2026).

The beta also adds two supporting pieces. Workload-aware preemption stops the scheduler from evicting a job for a preemptor that would not make progress anyway. PodGroup queueing coordinates competing workloads and stops the livelock where two large jobs keep bouncing each other without progress (Kubernetes Blog, 2026).

One caveat before you upgrade. Gang scheduling does not activate on its own. You opt in through the Workload API and PodGroup objects (TechTimes, 2026).

HPA can scale to zero

Section titled “HPA can scale to zero”

Scale to zero for the HorizontalPodAutoscaler reached Beta and is enabled by default. Set spec.minReplicas: 0 on a queue consumer, batch job, or GPU workload. The workload drops to zero pods when idle and comes back when demand returns. It works with object or external metrics only, because CPU and memory scaling needs live pods to produce numbers (Kubernetes Blog, 2026).

The control plane gets calmer under load

Section titled “The control plane gets calmer under load”
  • Resilient watchcache initialization completed its graduation. When the API server restarts, the watchcache warm-up no longer fires a list-and-watch storm at etcd. The API server bounds the burst and rejects the rest with HTTP 429 plus a Retry-After header (Kubernetes Blog, 2026).
  • Pod-level checkpoint and restore entered Alpha as KEP-5823. The CRI gains CheckpointPod and RestorePod RPCs. Your container runtimes must implement them before this is useful (Kubernetes Blog, 2026, and The Register, 2026).

The theme of the release, according to the trade press, is control-plane hardening for production AI workloads rather than shiny additions (ITOps Times, 2026).

The removal list is your upgrade checklist

Section titled “The removal list is your upgrade checklist”

Garhwal prunes as hard as it adds. The Register called it “Kubernetes cleans house” (The Register, 2026).

  • kube-dns is deprecated. CoreDNS has been the default cluster DNS since v1.13, and kube-dns never gained EndpointSlices or dual-stack Services. No new kube-dns packages ship after v1.40 (Kubernetes Blog, 2026).
  • kube-proxy IPVS mode is deprecated. IPVS arrived in v1.8 to fix iptables bottlenecks, but it still leans on iptables underneath. The project is consolidating on nftables, and clusters running IPVS now log a startup deprecation warning (Kubernetes Blog, 2026, and Network World, 2026).
  • cgroup v1 is on the way out. Since v1.35 the kubelet failCgroupV1 setting defaults to true. Nodes still on cgroup v1 fail kubelet init unless you apply a temporary override. Memory QoS and in-place memory-volume scaling work only on cgroup v2 (Kubernetes Blog, 2026).
  • Smaller cuts: kubectl run --filename/-f is deprecated, and static Pods can no longer reference Secrets or ConfigMaps (Kubernetes Blog, 2026).

What to do this week

Section titled “What to do this week”
  1. Verify cgroup v2 on every node before you schedule the 1.37 upgrade.
  2. Inventory kube-dns. Start the CoreDNS migration before v1.40 ends the packages.
  3. If kube-proxy runs IPVS, test nftables mode on a canary node.
  4. Start moving device plugins to DRA. KEP-5004 keeps your old extended-resource manifests working during the move.
  5. Opt in to gang scheduling for training jobs through the Workload API. It does not auto-enable.
  6. Teach your controllers to handle HTTP 429 with exponential backoff. The API server now expects it.

The bottom line

Section titled “The bottom line”

Kubernetes 1.37 treats an AI training run like a unit of work, not a pile of pods. DRA finished growing into a real device scheduler, and gang scheduling ends the deadlock that has haunted distributed AI training for years (TechTimes, 2026). The release logo shows terraced Himalayan fields, each shelf cut one step at a time and each shelf holding the next one up. That image fits the work: no single leap, four DRA graduations, one beta that rewrites the scheduling contract, and a removal list that tells you where the next three upgrades bite. The release is available for download on GitHub (GitHub, 2026).