Artificial Intelligence1 min read

Why your fine-tuning job takes longer than the FLOPs say

By · Published by Everything Blog

In short

Standard estimation techniques omit vital cluster bottlenecks, creating severe scheduling challenges on enterprise infrastructure. An investigation of hundreds of production runs showed that theoretical FLOP formulas miscalculate actual durations by double-digit hours on average. By factoring in operational metadata like tokenization, storage contention, and checkpoint cadences before dispatch, machine learning predictors drastically cut forecasting errors. A CatBoost model successfully shrank the mean absolute error to under two hours without needing access to real-time cluster logs.

Key points

  • Theoretical compute formulas fail because they capture only the core optimization loop wh…: Theoretical compute formulas fail because they capture only the core optimization loop while overlooking cluster-level bottlenecks.
  • Non-compute overheads such as environment setup, checkpointing, tokenization, and network…: Non-compute overheads such as environment setup, checkpointing, tokenization, and network contention dramatically expand job runtimes.
  • Machine learning models using pre-dispatch metadata can accurately anticipate job duratio…: Machine learning models using pre-dispatch metadata can accurately anticipate job durations without requiring runtime execution logs.

Sources checked by our analysis pipeline: 1

Predicting end-to-end fine-tuning time on shared GPU clusters, from 272 production runs.

A preview of our upcoming paper, Predicting End-to-End Fine-Tuning Time for Generative Language Models. The full version will be on arXiv soon.

GPUs are among the most expensive and scarcest resources in AI today. Anyone running a shared cluster has to decide, every day, which jobs go on which GPUs and in what order. That plan is only as good as the estimate of how long each job will take. When a job finishes hours early, GPUs sit idle until the next one is dispatched. When it runs hours late, everything queued behind it slips, experiments wait and delivery dates move.

Better duration estimates turn this into planning instead of guesswork: what runs, when, and on how many GPUs, with less idle capacity in between. Yet the estimate most teams use is a back-of-the-envelope FLOP calculation. It says nine hours, and the job finishes the next afternoon.

We wanted to know how far off that math really is, and whether we could do better before a job is…

Artificial Intelligence