How to Run Isolated Tenant Kubernetes Clusters on Shared GPU Infrastructure

Running a dedicated Kubernetes cluster per team often results in more isolation than an organization requires. While one cluster can be successfully shared across many teams, the coordination costs increase as the number of teams grows. Challenges include conflicting CRD versions, overlapping RBAC, and no clean way to carve GPU capacity into team-level budgets. At a certain scale, teams might start asking for their own clusters just to regain autonomy.

This post provides a pattern that preserves team autonomy without splitting the hardware. This solution involves a single control plane cluster with a GPU pool, GPU sharing with per-team quotas, and isolated Kubernetes control plane per team including an API server, controller, data store, syncer, and scheduler. This can be achieved using two open source tools: KAI Scheduler and vCluster. 

Follow along with the tutorial steps and you’ll have three teams running real GPU Kubernetes pods in their own tenant clusters, all sharing a single physical GPU. You’ll also be able to verify that each team only sees their own workloads.

Tutorial prerequisites and notes

To keep the process reproducible for users that have limited resources, this tutorial uses a cluster with one NVIDIA L40S GPU and three teams sharing fractions of it. This makes the moving parts easy to see and try. The process works the same way on a larger cluster with hundreds of GPU nodes and dozens of teams. You can scale the node pool, the queue hierarchy, and the number of tenant clusters. 

KAI Scheduler is a robust, efficient, and scalable topology-aware Kubernetes scheduler that was purpose-built for optimizing GPU resource allocation for AI workloads. It’s designed to manage large-scale GPU clusters, including thousands of nodes, and a high throughput of workloads. With KAI Scheduler, you can dynamically allocate GPU resources to workloads. It can run alongside the default kube-scheduler. Any pod with schedulerName: kai-scheduler is handled by KAI Scheduler. Everything else goes through the usual kube-scheduler process. 

The vCluster Kubernetes platform provisions fully isolated tenant clusters on your infrastructure or directly on bare metal. Each tenant cluster gets its own API server, custom resource definitions (CRDs), and role-based access control (RBAC), indistinguishable from a dedicated Kubernetes cluster, while sharing the underlying nodes and hardware. The virtualized control plane is invisible to tenants: no shared control plane nodes, no in-cluster agent pods, and no lateral path between environments. That makes vCluster a natural fit for GPU infrastructure where teams need their own clean cluster experience without splitting the hardware.

The vCluster shared-nodes model is used for this tutorial, so teams share the GPU node while each gets its own isolated control plane, the right fit for trusted internal teams. For untrusted tenants needing node-, network-, and storage-level separation, the same pattern extends to vCluster private nodes.

The example in this post uses three teams: NLP Team, Vision Team, and Recommender System Team. The NLP Team wants to install their own CRDs. The Vision Team wants cluster-admin to debug scheduling. The Recommender Team is on a different Kubeflow version. Nobody wants to share a kubectl context and accidentally break one another’s environments.

Using vCluster, each team gets their own isolated control plane, RBAC, namespaces, and CRDs. They can each have cluster-admin access. Underneath, all tenant clusters share the same nodes and GPUs.

Demo environment

This demo runs on an NVIDIA Brev GPU instance on Nebius with:

Note: For different setups, the cluster-creation and GPU Operator install steps will differ on GKE/EKS/AKS/vanilla k8s/k3s. Step 3 (KAI Scheduler) onward is identical on any Kubernetes that has the NVIDIA GPU Operator running with Container Device Interface (CDI) enabled.

Figure 1. KAI Scheduler and vCluster architecture on a shared Nebius host cluster

Install standalone kubectl and helm to avoid prefixing every command with microk8s. Then wire up a kubeconfig and pin MicroK8s so snap doesn’t auto-upgrade the control plane during the demo.

Next, make sure the required MicroK8s addons are enabled:

Then verify: 

Step 2: Add the Helm repo 

Add the NVIDIA Helm repo:

Step 3: Confirm the GPU Operator

If you’re using an older GPU Operator, upgrade in place to 26.3.x:

If needed, install NVIDIA GPU Operator directly:

Step 4: Install KAI Scheduler

Next, install the KAI Scheduler:

Then verify:

Step 5: Define team queues

KAI Scheduler uses a Queue CRD to model an org → team hierarchy. This step involves creating one parent (ml-org, with a total budget of one GPU) and three child queues. Each is guaranteed 0.33 of the GPU and allowed to increase to the full GPU when others are idle. 

Save the following as create-queues.yaml:

Here, quota is the guaranteed minimum, limit is the maximum allowed, and overQuotaWeight controls how surplus is split.

Next, apply and list:

Note that default-parent-queue and default-queue are created automatically by KAI Scheduler on first install. They are the fallback queue for any pod that doesn’t specify one.

Step 6: Spin up a vCluster per team

Next, install the vCluster CLI:

Define the vCluster config. The critical setting is setOwner: false. The KAI Scheduler pod-grouper walks ownership chains (Job → Pod, Deployment → ReplicaSet → Pod) to auto-group workloads. Disabling vCluster owner rewriting allows KAI Scheduler to see the real hierarchy.

Then create one vCluster per team:

Verify that all three vClusters are running:

Each team sees the real nodes, including the GPU node:

Step 7: Deploy a workload from each team

Each team deploys their GPU workload from their own vCluster. The pod spec is simple, with three fields telling KAI Scheduler what to do:

Do the same to deploy from each team, changing the queue name. 

Now, verify:

Three pods, three different vcluster-team-* namespaces, all running on the same physical node brev-8dq0cch1j. 

Each team sees only their own pod from inside their vCluster:

Finally, the queue confirms the allocation. You can check all three at the same time:

You will see the same result for all:

Note that KAI Scheduler handles scheduling—which pods land on which GPU and in what proportion. It does not enforce GPU memory isolation at the hardware level when GPU sharing is used. Applications need to respect their memory amount (for example, setting –gpu-memory-utilization in vLLM).

Under the hood, the GPU time-slices between CUDA contexts from each pod at kernel boundaries. For hard memory isolation on supported hardware, NVIDIA Multi-Instance GPU (MIG) provides hardware-level partitioning, which can be scheduled by KAI Scheduler as well.

Get started running isolated tenant Kubernetes clusters

KAI Scheduler decides fairly who gets GPU slices and schedules them as a group using GPU sharing and DRA Driver support, hierarchical queues with guaranteed quotas plus over-quota capabilities, gang scheduling, and topology awareness. Once AI workload scheduling is solved, teams want their own clusters. vCluster gives each team its own isolated Kubernetes control plane without the cost of separate infrastructure. 

Together, KAI Scheduler and vCluster provide a dedicated cluster experience for three teams with zero waste on a single GPU. The answer isn’t always more GPUs, but better utilization of infrastructure.

Ready to get started? Check out KAI Scheduler, vCluster, and NVIDIA GPU Operator on GitHub.

Learn more about KAI Scheduler and vCluster integration at KubeCon 2026 North America, November 9-12.