In theory, a poor model should be quickly discovered and either eliminated or tweaked so that it delivers more accurate prices. However, for various reasons, including intellectual arrogance and in some cases a crooked desire to book fictitious profits, this does not always occur.
A recent report entitled Model validation best practices: Achieving value-added from research consultancy Celent, now part of the Oliver Wyman group, suggests the implementation of a framework that will help ensure models perform as they should and actually add, rather than subtract, value. Authors Umit Kaya and Il-Dong Kwon point out that "the upsurge in usage of models, sophistication of underlying theories, and complexity of the IT environment have firmly placed model risk on regulator and senior executive agendas. Building a model governance framework to contain this risk has become a prevalent risk management practice. Model validation, essentially a peer review system with the ultimate goal of providing efficient quality control, has been an integral part of this effort."
They add: "Accounting troubles and tight regulatory controls around risk measurement models led the first model validation wave, primarily as a compliance exercise. The next wave will be characterized by repositioning beyond compliance towards a streamlined, value-adding model validation process. Completely eliminating model risk through controls is impossible; therefore institutions need to strike the right balance between costs and benefits. Achieving a value-adding model validation process through such a balance presents a number of conceptual, organizational and process design challenges."
One of the key points that the authors make is that institutions should have in place an independent model validation unit. "Independence of the review is the first principle to ensure that validation provides a genuine second line of defence," they argue.
|
Establishing the necessary corporate framework for robust model validation |
 |
|
Source: Celent |
Troy Rohrbaugh, global head of FX trading at JPMorgan, agrees. "Model validation should be a matter of common sense," he says. "We feel that at JPMorgan our process is very rigorous. We have checks and balances in place so that the person validating the model is not the same person who designed it or who is using it to trade."
Obvious
Celent adds another obvious point. "Models that are poorly designed, incorrectly deployed, inappropriately used, or improperly maintained can result in direct financial losses, inaccurate financial or regulatory reporting, and subsequent restatement, poor business decisions, or reputational damage," the report says.
According to Rohrbaugh, even though some risk will always remain, financial institutions have a natural incentive to have a solid validation process in place. "While having a rigorous process in place doesn’t categorically prevent a problem, it at least makes sure you have done as much as you can to stop one from developing," he argues.
The use of third-party models presents further challenges. "Whether the model is externally purchased or internally built affects the extent of model validation," say Kaya and Kwon. "In principle, treatment of vendor models should not be much different from in-house models. However, in practice, limited validation is possible since the vendors do not necessarily provide sufficient information on development and methodology."
They add: "Focus should shift to performance and usage of the model. On the whole, vendor models should not be exempt due to a lack of transparency. Otherwise, lower validation standards could bias the buy-or-build decision toward vendors for future models, and institutions could lose edge in exploiting their comparative advantages. This criterion has possible ramifications for model vendors as well. As validation becomes an integral part of model governance, financial institutions will start asking for higher disclosure levels from vendors, who will need to start competing on transparency of their models."
Many vendors will shudder at the thought of having to disclose the secrets their companies are built on. No doubt they will argue that the market continuously stress-tests their products and that the market ultimately is always right. Whether their customers agree, though, remains to be seen.