One-Step Gradient Delay is Not a Barrier for Large-Scale Asynchronous Pipeline Parallel LLM Pretraining

Preprint available on OpenReview. The work studies asynchronous pipeline parallel training for large language models and analyzes the impact of gradient delay on large-scale optimization.