Our first machine learning engineering project cost more to operate in year two than it cost to build. Nobody had modelled that, including me, and the figure only surfaced when somebody asked why a particular infrastructure line had tripled.

Here is where the money actually goes, in the order the lines appear.

It is also the one every business case contains.

A training run has a start and an end, a measurable compute cost, and a number somebody can put in a slide.

Ours accounted for under a fifth of the total over three years. Everything else is continuous, which is exactly why it does not appear in a project budget.

The line that grows with success.

Every prediction costs something. A model serving a thousand requests a day is inexpensive. The same model succeeding and serving two million is not, and nothing about the architecture changed.

We now model serving cost at expected volume before building, and again at three times expected volume. The second figure has changed two decisions.

The mistake to avoid is sizing infrastructure for launch. Ours was provisioned for the first month and repeatedly resized reactively for a year.

The line that appears before anything reaches production.

Training runs during development, hyperparameter searches, and the environments that sit idle between them.

Ours ran unattended for a quarter before anybody looked. One search had been left running and consumed more than the eventual production serving cost for that model's first year.

We now cap experiment spend per team and report it weekly. Machine learning engineering work is exploratory by nature, and exploratory spend needs a ceiling rather than a conversation afterwards.

The cost that surprised our finance team most.

Feature computation, training data assembly, and the movement between storage and compute. None of it is interesting and all of it is metered.

Our largest single line in year two was not inference. It was reading training data across a region boundary, because the storage and the compute had been provisioned by different people at different times.

Fixing that took a day and saved more than any optimisation we made to the models.

The one least often in the business case.

A production model needs somebody watching drift, investigating degradation, retraining, and handling incidents. That is machine learning engineering as an ongoing function. That is a standing allocation rather than a project cost.

Across our estate it is roughly one engineer for every six to eight models, depending on how much the domains move. A model in a stable domain needs a few days a quarter. One in a moving domain needs weekly attention.

Budget for that before the first model ships, because it arrives whether or not you planned for it.

Four changes over two quarters, roughly in order of what they returned.

Co-locating storage and compute, which removed the data transfer line entirely.

Batching predictions where latency allowed. A meaningful share of our inference did not need to be real time and had been built that way by default.

Caching repeated predictions. Our recommendation model was being asked the same question for the same user several times per session.

And retiring two models that were no longer used by any product. Nobody had switched them off because nobody owned the question.

Together those reduced our spend by roughly half, and no model performed worse.

Worth being clear, because the saving is often overstated.

Almost all of our stack comes from open source development. Orchestration, feature storage, serving, monitoring. Open source development removes the licence line and adds an operational one.

That removes licence cost and adds operational cost. Somebody patches it, upgrades it, and fixes it when a version moves. At our scale the trade is favourable and it is a trade rather than a saving.

The decision I would question is a small team self hosting everything. Below a certain size, hosted products cost less in total once the attention is counted.

●      What does serving cost at three times expected volume?

●      Where does the data live relative to the compute?

●      How much of this inference actually needs to be real time?

●      Who watches this model, and how much of their week is it?

●      What would we switch off if this works?

The last question is the one most often unanswered. A model that runs alongside the process it was meant to replace has produced cost rather than saving.

We used a partner for the serving architecture. Narrow scope, a reference implementation, pairing with our engineers.

The generative AI consulting proposed alongside it was declined, mostly because our brief for it would have been vague. That work needs a defined task and a baseline, and we had neither at the time.

Scoping a partner against a specific problem has worked for us. Scoping one against a technology has not.

Where does the cost surprise usually come from?

Serving at scale and data movement. Machine learning engineering budgets built around training compute miss both, and both grow with success.

How should we forecast this?

Three year total including people, not a build estimate. Ours was wrong by a factor of four the first time because it contained only training and infrastructure.

Is a smaller model worth it?

Frequently. We route most traffic to a smaller model and escalate when it struggles, which cut inference cost substantially with no measurable quality change.

When does self hosting win?

At high volume, or where residency requires it. Below that, the operational commitment usually outweighs the licence saving.

What grows fastest?

Serving, with usage. It is the line to watch monthly rather than annually.

Machine learning engineering costs roughly what the build costs, every year, indefinitely.

That is a rough rule and it has been closer than anything else I have tried. It includes serving, data movement, the tooling, and the person watching it.

Presenting that at the start makes the first model look expensive and the fourth look reasonable, which is an accurate picture and not the one a project business case produces.