The Memory Wall in Deep Learning
Training deep neural networks has always been a memory-hungry affair.
End-to-end backpropagation demands that every intermediate activation be stored until the backward pass.
As networks grow deeper — especially with large Transformers — the memory footprint balloons linearly with depth.
This creates a “memory wall” that limits model size and accessibility.
Imagine a long hike where you must carry all the food and water you’ll need for the return trip at the very start.
That is exactly how standard training operates: the entire forward pass must remain in memory, blocking further scaling.
Researchers have long sought ways to train networks in smaller, independent pieces — but until now, most attempts have sacrificed performance for memory savings.
Block-wise Training: A Promising but Flawed Idea
Block-wise training breaks a network into separately trainable chunks, each requiring memory only for its own activations.
In theory, this could dramatically reduce peak memory usage.
Past methods, however, consistently underperform end-to-end training.
They rely on ad-hoc local objectives — like greedy layer-wise pre-training or synthetic gradient signals — without a principled way to coordinate the blocks.
Moreover, existing approaches have been largely confined to classification tasks and custom architectures.
Their applicability to modern Transformer models and generative AI remained largely unexplored.
These two core challenges — lack of theoretical grounding and limited architectural scope — have kept block-wise training from becoming a practical tool for the deep learning mainstream.
A Surprising Connection: Residual Networks as Diffusion Processes
A crucial insight bridges the gap: residual connections inside Transformers can be interpreted through the lens of score-based diffusion models.
In a diffusion model, data is gradually destroyed by adding noise, and a reverse process learns to remove noise step by step.
Each denoising step can be optimized independently of the others.
Networks with residual layers, it turns out, closely resemble the Euler discretization of a continuous-time probability flow ODE.
Each residual block acts like one infinitesimal denoising step in a continuous diffusion trajectory.
This correspondence was previously known in the context of neural ODEs, but DiffusionBlocks harnesses it for a novel purpose: reframing a deep Transformer as a stack of independent denoising stages.
The DiffusionBlocks Framework
DiffusionBlocks partitions a Transformer network into blocks, each assigned a contiguous range of noise levels.
During training, only the block corresponding to the current noise level is active — the rest can be completely ignored.
Gradients flow only within that single block.
Memory consumption drops in proportion to the number of blocks, making the training of very deep models feasible on modest hardware.
Crucially, the objective for each block is derived directly from score matching theory, not from a hand-crafted loss.
Every block learns to denoise its assigned noise range, and together they faithfully approximate the global reverse diffusion process.
Techniques from established diffusion training recipes (e.g., Karras et al., 2022) can be seamlessly integrated to further improve performance.
Equi-probability Partitioning for Balanced Learning
A naive partitioning that simply chops the noise timeline into equal segments risks underutilizing some blocks.
The density of noise states varies, and some intervals would see far more training signal than others.
DiffusionBlocks introduces equi-probability partitioning: noise levels are split so that each block accounts for an equal amount of cumulative probability mass under the forward noising kernel.
This ensures that every block receives a balanced workload during training.
No block is starved for signal, and no block is overwhelmed.
The result is a symmetric parameter utilization across the full network, which promotes stable convergence and consistently strong end-to-end equivalence.
Putting DiffusionBlocks to the Test
The framework was evaluated across a broad family of architectures: vision Transformers, diffusion Transformers, autoregressive Transformers, recurrent-depth Transformers, and masked diffusion Transformers.
In all cases, DiffusionBlocks training matched the performance of standard end-to-end training while enabling true block-wise independence.
The experiments go well beyond small-scale classification; they demonstrate viability on practical generative AI tasks for the first time in block-wise training literature.
This is a sobering rebuttal to the long-held belief that block-wise methods inevitably lag behind joint optimization.
The key lay not in a trickier local loss, but in a solid theoretical reinterpretation of the network itself.
From Training Efficiency to Inference Gains
Block-wise training is not only about saving memory during optimization.
In diffusion models, inference requires only the block assigned to the current denoising timestep, further shrinking the memory footprint at deployment time.
For recurrent-depth models — where a block of layers is applied repeatedly — DiffusionBlocks replaces costly iterative training with a single-pass procedure, completely eliminating backpropagation through time.
This dual benefit illustrates how a diffusion-centric perspective reshapes both training and inference.
The network structure is no longer a monolithic chain, but a modular stack of specialized denoisers that can be swappted or reused on demand.
A New Direction for Scalable Learning
DiffusionBlocks delivers two foundational contributions.
First, it reinterprets Transformer-based networks as discretized instances of a continuous-time diffusion process, enabling genuinely independent block training with gradients required for only one block at a time.
Second, the equi-probability partitioning strategy balances parameter utilization across blocks, ensuring no part of the network is left under-trained.
By marrying block-wise training with rigorous diffusion theory, the method opens the door to training ever-larger models without hitting the memory wall.
The code is publicly available at https://github.com/SakanaAI/DiffusionBlocks, inviting the community to explore and extend this new paradigm.
