Inference

Inference is the process of using a trained model to produce outputs from new inputs — generating a response to a prompt, classifying a document, or making a prediction. It is distinct from training, which is the earlier, one-off process of building the model by adjusting its weights on data. Training creates the model; inference is what happens every time the model is subsequently used.

The distinction matters operationally because training and inference have very different resource profiles. Training a large model is computationally intensive and typically done once (or periodically), whereas inference runs continuously in production, once per request, and its cost and latency determine what the deployed system feels like to use and what it costs to operate at scale. Capacity planning for a deployment is largely about sizing inference.

Where inference happens is central to secure deployment. When a model is used through a hosted service, inference takes place on the provider’s infrastructure, which means the input data is sent there to be processed. Running inference on-premises or in a controlled private environment keeps that processing — and therefore the sensitive data in each request — inside the organisation’s own perimeter, which is the crux of why deployment location matters for confidentiality and data residency.