For those who cannot read the article due to the stupid FT paywall:
Don’t blame luck when your models misfire
By John Kay Published: March 1 2011 22:31 | Last updated: March 1 2011 22:31
When the financial crisis broke in August 2007, David Viniar, chief financial officer of Goldman Sachs, famously commented that 25-standard deviation events had occurred on several successive days. If you marked your position to market every day for a million years, there would still be a less than one in a million chance of experiencing a 25-standard deviation event. None had occurred. What had happened was that the models Goldman used to manage risk failed to describe the world in which it operated.
If the water in your glass turns to wine, you should consider more prosaic explanations before announcing a miracle. If your coin comes up heads 10 times in a row – a one in a thousand probability – it may be your lucky day. But the more likely reason is that the coin is biased, or the person who flips the penny or reports the result is cheating. The source of most extreme outcomes is not the fulfilment of possible but improbable predictions within models, but events that are outside the scope of these models.
Sixty years ago, a French economist described the Allais paradox, based on the discovery that most people treat very high probabilities quite differently from certainties. Not only do normal people think this way, but they are right to do so. There are no 99 per cent probabilities in the real world. Very high and very low probabilities are artifices of models, and the probability that any model perfectly describes the world is much less than one. Once you compound the probabilities delivered by the model with the unknown but large probability of model failure, the reassurance you crave disappears.
Techniques such as value at risk modelling – the principal methodology used by banks and pressed on them by their regulators – may be of help in monitoring the day-to-day volatility of returns. But they are useless for understanding extreme events, which is, unfortunately, the main purpose for which they are employed. This is what Mr Viniar and others learnt, or should have learnt, in 2007.
Yet the use of risk models of this type is one of many areas of finance in which nothing much has changed. The European Union is ploughing ahead with its Solvency II directive for insurers, which – incredibly – is explicitly modelled on the failed Basel II agreements for monitoring bank solvency. Solvency II requires that businesses develop models that show the probability of imminent collapse is below 0.5 per cent.
Insurance companies do fail, but not for the reasons described in such models. They fail because of events that were unanticipated or ignored, such as the long-hidden danger from asbestos exposure, or the House of Lords judgment on Equitable Life. They fail because underwriters misunderstood the risk characteristics of their policies, as at AIG, or because of fraud, as at Equity Fundings.
Multiple sigma outcomes do not happen in real life. When all the Merchant of Venice’s ships are lost at sea during the interval, we know that we are watching a play, not an account of history. Shakespeare, no fool, knew that too. In Act V Antonio was able to write back his loss provisions in full even if it was too late to fulfil his banking covenant to Shylock.
But today the modellers are in charge, not the poets. Like practitioners of alchemy and quack medicine, these modellers thrive on our desire to believe impossible things. But the search for objective means of controlling risks that can reliably be monitored externally is as fruitless as the quest to turn base metal into gold. Like the alchemists and the quacks, the risk modellers have created an industry whose intense technical debates with each other lead gullible outsiders to believe that this is a profession with genuine expertise.
We will succeed in managing financial risk better only when we come to recognise the limitations of formal modelling. Control of risk is almost entirely a matter of management competence, well-crafted incentives, robust structures and systems, and simplicity and transparency of design.
Facebook
Twitter