Zero-Bubble Pipeline Parallelism

Pipeline parallelism splits a model into stages and processes micro-batches across devices. The problem is that devices often wait for each other, creating "bubbles."

Zero-Bubble improves scheduling and backward-pass arrangement so devices spend less time idle and large-model training throughput rises.

Training efficiency is a hidden battleground in foundation-model competition. Less idle time means the same compute can train more tokens, larger models, or more iterations.

This kind of systems paper and open implementation has practical value for large-model labs.

Sea AI Lab’s work on training systems shows that a Singapore homegrown corporate lab is entering lower-level model infrastructure, not only applications.

Together with Colossal-AI, it forms the "training systems" line in Singapore’s open-source ecosystem.