Standard estimation techniques omit vital cluster bottlenecks, creating severe scheduling challenges on enterprise infrastructure. An investigation of hundreds of production runs showed that theoretical FLOP formulas miscalculate actual durations by double-digit hours on average. By factoring in operational metadata like tokenization, storage contention, and checkpoint cadences before dispatch, machine learning predictors drastically cut forecasting errors. A CatBoost model successfully shrank the mean absolute error to under two hours without needing access to real-time cluster logs.
Key points
Theoretical compute formulas fail because they capture only the core optimization loop wh…: Theoretical compute formulas fail because they capture only the core optimization loop while overlooking cluster-level bottlenecks.
Non-compute overheads such as environment setup, checkpointing, tokenization, and network…: Non-compute overheads such as environment setup, checkpointing, tokenization, and network contention dramatically expand job runtimes.
Machine learning models using pre-dispatch metadata can accurately anticipate job duratio…: Machine learning models using pre-dispatch metadata can accurately anticipate job durations without requiring runtime execution logs.
Sources checked by our analysis pipeline: 1
Predicting end-to-end fine-tuning time on shared GPU clusters, from 272 production runs.
A preview of our upcoming paper, Predicting End-to-End Fine-Tuning Time for Generative Language Models. The full version will be on arXiv soon.
GPUs are among the most expensive and scarcest resources in AI today. Anyone running a shared cluster has to decide, every day, which jobs go on which GPUs and in what order. That plan is only as good as the estimate of how long each job will take. When a job finishes hours early, GPUs sit idle until the next one is dispatched. When it runs hours late, everything queued behind it slips, experiments wait and delivery dates move.
Better duration estimates turn this into planning instead of guesswork: what runs, when, and on how many GPUs, with less idle capacity in between. Yet the estimate most teams use is a back-of-the-envelope FLOP calculation. It says nine hours, and the job finishes the next afternoon.
We wanted to know how far off that math really is, and whether we could do better before a job is…
Addressing widespread workforce anxiety surrounding automated systems, leadership offered voluntary buyouts to tens of thousands of staff to facilitate an internal reorganization around artificial intelligence. The strategy grants employees personal autonomy during sweeping workflow changes rather than imposing forced terminations. Ultimately, management expects all internal roles to transition from manual execution to directing and overseeing autonomous digital tools.
Capital secured from backers in Japan marks a major milestone for the artificial intelligence company as it completes its public listing. However, the final proceeds settled at the bottom of the targeted expectations outlined in earlier projections. The resulting total reflects the minimum threshold management had sought to achieve during this institutional offering phase.
The aggressive buildout of artificial intelligence facilities is placing unprecedented strain on natural resources and local municipal budgets across the country. Power providers and military developers are facing intense pushback over fossil fuel expansion and resource consumption. At the same time, communities and educational institutions are grappling with the societal friction caused by imperfect automated tools.