When is training or fine-tuning your own model worth it, and when is it a waste?
Worth it when you have proprietary data a general model has never seen, a task narrow enough to evaluate precisely, and volume high enough that inference cost matters. A waste when a better prompt, better retrieval, or a larger general model would have solved it, which is most of the time. We will tell you which before you commit to the expensive path.
Model Development & Training
Most of the time the answer is no
We would rather run a two-week assessment that ends in “do not do this” than take an eight-week fine-tuning engagement that a better prompt would have replaced. Roughly speaking, the question is whether a general model plus good retrieval already does the task acceptably, and it usually does.
The honest test has four parts. Do you have data a general model has never seen? Is the task narrow enough to evaluate precisely? Is volume high enough that inference cost or latency is a real constraint? And can you produce consistently labelled examples?
Fewer than four yeses and the assessment ends with a recommendation not to proceed, in writing, for the same fee.
When it does make sense
Domain language and output formats a general model handles inconsistently. High-volume classification where a small specialized model is an order of magnitude cheaper to serve. Latency-sensitive paths where a large model is too slow. Air-gapped environments where a hosted frontier model is not available at all.
What the work actually is
Mostly data. Assembly, cleaning, and the labelling process consume most of the budget on every engagement of this kind, and a team that has not done it before consistently underestimates it. The training run is comparatively short and comparatively easy.
Then evaluation, which decides whether any of it worked, and a serving and cost model at your real volume rather than at demo volume.
After launch
Held-out evaluation on a schedule, drift monitoring on the inputs, and retraining triggers agreed during the build. A model nobody evaluates after launch decays quietly until a customer finds it.
What you get
- A feasibility assessment that compares fine-tuning against prompting, retrieval, and a larger general model on your actual task
- Training data assembly, cleaning, and the labelling process, which is usually most of the effort
- Fine-tuning and small task-specific models where the economics justify them
- An evaluation harness with held-out cases, so quality is measured rather than asserted
- Deployment, serving, and the cost model at your real volume
- Retraining triggers and drift monitoring, so the model does not quietly decay
Questions we get
- Everyone says fine-tuning is unnecessary now. Is it?
- Usually, and the two-week assessment exists to check rather than assume. General models with good retrieval have absorbed most of what fine-tuning used to be for. The cases that survive are narrow, high-volume, latency-sensitive, or involve output formats and domain language a general model genuinely has not seen.
- How much data do we need?
- Fewer examples than people expect, and far more consistency than people expect. A few thousand clean, consistently labelled examples usually beats a hundred thousand inconsistent ones. The labelling process is where most of the budget and most of the risk sits.
- What stops the model degrading over time?
- Held-out evaluation that runs on a schedule, plus drift monitoring on the inputs. We set the retraining triggers during the build. A model that nobody evaluates after launch is a liability with a launch date.
How it starts
Two-week feasibility assessment, then eight to sixteen weeks if it proceeds. You keep everything produced, whether or not there is a next part.