Model risk

Model risk is the potential for adverse consequences arising from a model being wrong, being used incorrectly, or being relied upon beyond what it can support. The term originates in regulated finance, where model risk management is a well-established discipline, but the concept applies to any consequential use of a model — including AI systems — where errors or misuse can cause real harm.

For AI systems, model risk includes the model producing inaccurate, unsupported or biased outputs; behaving unpredictably on inputs unlike its training data; being applied to a task it was not suited to; and being trusted by users who do not understand its limitations. Because language models can produce fluent, confident-sounding output regardless of correctness, the risk of over-reliance is particularly acute, and it grows as a model’s outputs are wired into automated decisions.

Managing model risk means understanding what a model can and cannot do, validating its performance for the specific use, keeping human oversight where decisions carry weight, monitoring behaviour in production, and being deliberate about which use cases warrant automation and which do not. A common and prudent stance is to use AI to inform rather than to decide where regulatory or reputational exposure is high, keeping a person accountable for the outcome.