Minimize idle accelerators: Native RL job interleaving with co-operative time-slicing in llm-d
The math behind reinforcement learning (RL) post-training for large language models (LLMs) is notoriously unforgiving. As frontier AI labs push the boundaries of reasoning and coding models using RL post-training algorithms like Group Relative Policy Optimization (GRPO), they routinely hit hard architectural and infrastructure constraints. While much of the industry’s focus remains on acquiring raw accelerator capacity, infrastructure efficiency is equally critical for achieving the high velocity needed to run multiple RL jobs and drive models to higher levels of intelligence. At scale, distributed RL suffers from severe resource bottlenecks because synchronous sampling and training run as strictly sequential phases, causing trainer and sampler resources to alternate sitting idle. Meanwhile, asynchronous architectures attempt to overlap these phases, but trainers still experience frequent idle gaps while waiting for specific trajectory batches to finish before starting the next cycle. Today, we are introducing a solution to this structural waste: co-operative time-slicing through the llm-d project. By treating discrete RL steps — such as sampling rollouts and gradient training — as dynamic, schedulable entities, we can interleave independent RL jobs onto shared physical hardware. Our initial benchmarks show that this platform-level multiplexing increases aggregate accelerator duty cycles from a ~40% baseline up to 70% without impacting model convergence or accuracy. This improves price-performance and lowers TCO significantly by eliminating wasted compute accrued over time. For synchronous setups, the platform interleaves both samplers and trainers to minimize alternating idle windows, while asynchronous workloads leverage time-slicing to dynamically reclaim and utilize the fragmented idle gaps between RL-trainer iterations. Throughout this blog, we will describe the time-slicing solution, detailing the technical flows, current release and future roadmap. llm-d for RL infrastructure efficiency (the bigger picture) From the get-go, we anticipated the severe infrastructure bottlenecks of large-scale RL post-training and invested in addressing infrastructure inefficiency for RL workloads. We have built llm-d into a highly composable infrastructure stack for inference, agentic and RL workloads focused on eliminating accelerator idle time. The llm-d stack for RL features: Throughput-driven inference (llm-d-router): A mature, production-tested engine deployed across RL workloads and focused on maximizing rollout generation throughput to continuously saturate the pipeline. High-velocity Agent Sandbox (recipe): Tested for scale and density, and helping deliver secure, sub-second tool-use and isolated code execution during rollout generation and evals. Agent Sandbox serves as the high-speed intake manifold for reward signal generation, helping ensure the Sandbox never becomes the latency bottleneck that starves your time-sliced NVIDIA GPUs. Core pipeline primitives: To combat reliability and speed in weight transfer, we are building Weight Propagation Interface (WPI), as well as focusing on improving overall observability and reliability for RL. The efficiency problem with RL loops Distributed RL post-training operates as a fragmented, continuous cycle alternating between generation (sampling rollouts) and optimization (gradient updates). Because traditional cloud infrastructure is designed for continuous, steady-state workloads, standard Kubernetes clusters can’t adapt to this alternating cadence. At scale, this structural cadence introduces two massive systemic inefficiencies: Idle accelerators: Because these phases occur sequentially, GPU clusters sit completely idle (0% utilization) for 40% to 60% of their lifecycle. Trainers sit idle waiting for sampling rollouts to finish; samplers sit idle during gradient updates and weights distribution. This could represent millions in wasted capital annually. Locked-in context: RL training and samplers hold their accelerator allocations for the entirety of their runtime even during idle phases because the NVIDIA CUDA context and all device memory needs to remain resident. Standard schedulers treat these pods as static, siloed allocations rather than aligning them to the alternating, phase-level states of the live RL loop, leaving valuable hardware locked up even during inactive phases. Importantly, this is not just a synchronous RL problem. Asynchronous variants overlap generation and training, but they do not fully mitigate idle time. Generation remains the inherent bottleneck of the RL loop, meaning trainer accelerators still starve while waiting for rollout data to accumulate. The closer an asynchronous job runs to on-policy, the larger those idle windows become — bounded staleness limits how far generation and training can drift apart, stalling the pipeline whenever fresh rollouts are not ready. How co-operative time-slicing (RL job interleaving) helps To eliminate idle accelerators during RL jobs, co-operative time-slicing under the llm-d project allows the infrastructure to dynamically interleave independent RL jobs onto shared hardware blocks rather than forcing hardware to wait on upstream phases. This helps drive aggregate accelerator utilization up without altering the underlying model convergence or accuracy. When Job A goes idle at a phase boundary in synchronous RL (or stalls on fresh rollout data in asynchronous RL), the infrastructure time-slices the physical accelerators, swapping in the active sampling or training phase of Job B. Under the hood, a swap is a checkpoint/restore: Job A’s entire device state is checkpointed out of accelerator memory into host DRAM, and Job B’s previously saved state is restored in its place. Because only one job’s state ever occupies the accelerator at a time, steps alternate safely without framework-level interference or out-of-memory (OOM) faults. Time-slicing: High-level architecture The time-slicing system architecture is organized into three layers: workload-scoped (application logic), cluster-scoped (coordination), and node-scoped (hardware management). Workload-scoped layer (application runtime) This is where the user’s code runs — training loops, inference servers, and RL frameworks. The new addition is the time-slice client library, which exposes two gRPC APIs on the time-slice orchestrator: acquire() to request exclusive accelerator access, and yield() to release it. The user wraps any accelerator-touching phase with these calls to signal phase boundaries to the orchestrator. Everything else — the ML framework (PyTorch FSDP, vLLM, etc.), the CUDA context, the accelerator memory allocations — runs unmodified. Cluster-scoped layer (control and orchestration plane) This layer decides which job gets accelerator access, and when. Jobs that share the same physical accelerators — for example, two RL jobs interleaving on the same set of GPU nodes — are placed into a group. For each group, the time-slice orchestrator maintains a lock queue — an ordered list of jobs waiting for exclusive access to that group’s accelerators. Only the job at the head of the queue holds the lock and runs on the hardware; all the other jobs wait, blocked on their acquire() call. When the running job calls yield(), the orchestrator passes the lock to the next job in the queue and triggers a coordinated context switch across every node in the group. In the future, a workload placement optimizer will be able to profile workload phase patterns and automatically pair jobs with complementary idle phases, removing the need for the user to explicitly indicate job groupings. Node-scoped layer (hardware and data plane isolation) This layer performs the checkpoint/restore swap on each accelerator node. The snapshot agent, a privileged DaemonSet, receives directives from the orchestrator and translates them into hardware-level operations — pausing accelerator processes, serializing device state to host DRAM, and restoring it when the job regains access. The agent is built around a pluggable backend interface, with cuda-checkpoint as the first implementation (more to come). Future backends will introduce faster snapshot mechanisms and more selective approaches, such as offloading specific memory addresses like LoRA adapters instead of full device state. The agent itself is designed to run standalone outside Kubernetes for bare metal and Slurm environments. The flow: How it all comes together When a workload finishes its current accelerator phase, its time-slice client library calls yield() to the time-slice orchestrator to release access. The orchestrator initiates the context switch by sending directives to the snapshot agent on each node in the group. The agent freezes the yielding workload’s processes and moves its device state from accelerator memory into host DRAM. With the accelerators vacated, the orchestrator grants the group lock to the next workload waiting in the queue. It directs the Snapshot Agents on those nodes to restore that workload’s previously saved state from host DRAM back into accelerator memory, then unblocks the workload’s pending acquire() call. The workload resumes execution exactly where it left off — no container restart, no framework reinitialization, no model reload from storage. The yielding workload remains warm in host DRAM. When the orchestrator grants it the lock again, the Snapshot Agents perform the same swap in reverse. Developer experience (client-side) Researchers want to focus on core modeling logic rather than wrestling with low-level CUDA context switching or custom scheduling loops. If you use Ray or a similar platform to orchestrate your RL job, using time-slicing will have a minimal impact on the client side. In fact, there may not be any impact on the client side at all if you are queuing the training and sampling jobs separately at the platform level. code_block <ListValue: [StructValue([(‘code’, ‘from timeslice import TimeSliceOrchestratorClient\r\n\r\norchestrator = TimeSliceOrchestratorClient(target=“orchestrator:50051”)\r\n\r\n@orchestrator.on_accelerators(group_id=“trainer-group”)\r\ndef train_phase(model, trajectories):\r\n return model.update(trajectories)\r\n\r\n@orchestrator.on_accelerators(group_id=“sampler-group”)\r\ndef generate_phase(model, prompts):\r\n return model.generate(prompts)\r\n\r\n# Standard sequential loop — interleaved with other jobs under the hood\r\nfor epoch in range(EPOCHS):\r\n trajectories = generate_phase(policy, dataset)\r\n rewards = compute_rewards(trajectories)\r\n train_phase(policy, rewards)’), (’language’, ‘’), (‘caption’, <wagtail.rich_text.RichText object at 0x7efbe88eb430>)])]> Current release and future outlook Today we are releasing the full time-slicing stack: the Snapshot Agent, the Accelerator Orchestrator, and the Python client libraries, each with a user guide for integrating time-slicing into your RL workloads. Key roadmap highlights include: Latency and state optimization: Expanding the Snapshot Agent with faster checkpoint/restore backends to minimize context-switch overhead, alongside application-aware backends for selective memory region snapshotting (e.g., swapping LoRA adapters instead of full model weights). Automated scheduling and onboarding: Introducing an automated scheduler to profile running processes, identify time-sliceable structures, and handle job placement dynamically. Cross-hardware compatibility: Extending data plane support beyond GPUs to TPUs and custom accelerator architectures. Get started Building robust, highly optimized RL infrastructure requires tight collaboration with the engineers and researchers running these workloads at scale. If you are currently wrestling with low GPU utilization, synchronization stalls, or complex scheduling logic in your post-training pipelines, time-slicing can help. To get started, check out the following resources, and don’t forget to leave us your feedback! Start using time-slicing during your RL run immediately with these user guides. Try llm-d-router (kubernetes native) or the RL Scheduler (python library) user-guide for improved sampling throughput during the RL generation phase. Explore the Weight Propagation Interface repo. Join the discussion in the #sig-rl channel in the llm-d Slack. Contribute by sharing your reference implementations, benchmarks, and edge cases to help us refine this path. Thank you to Dolev Ish Am and Bogdan Berce for their contributions to this blog post.
