IRB – Page 2 – From the outside

Distinguishing luck and skill

Quantifying Luck’s Role in the Success Equation

“… we vastly underestimate the role of luck in what we see happening around us”

This post is inspired by a recent read of Michael Mauboussin’s book “The Success Equation: Untangling Skill and Luck in Business, Sports and Investing”. Mauboussin focuses on the fact that much of what we experience is a combination of skill and luck but we tend to be quite bad at distinguishing the two. It may not unlock the secret to success but, if you want to get better at untangling the contributions that skill and luck play in predicting or managing future outcomes, then this book still has much to offer.

“The argument here is not that you can precisely measure the contributions of skill and luck to any success or failure. But if you take concrete steps toward attempting to measure those relative contributions, you will make better decisions than people who think improperly about those issues or who don’t think about them at all.”

Structure wise, Mauboussin:

Starts with the conceptual foundations for thinking about the problem of distinguishing skill and luck,
Explores the analytical tools we can use to figure out the extent to which luck contributes to our achievements, successes and failures,
Finishes with some concrete suggestions about how to put the conceptual foundations and analytical tools to work in dealing with luck in decisions.

Conceptual foundations

It is always good to start by defining your terms; Mauboussin defines luck and skill as follows:

“Luck is a chance occurrence that affects a person or a group.. [and] can be good or bad [it] is out of one’s control and unpredictable”
Skill is defined as the “ability to use one’s knowledge effectively and readily in execution or performance.”

Applying the process that Mauboussin proposes requires that we first roughly distinguish where a specific activity or prediction fits on the continuum bookended by skill and luck. Mauboussin also clarifies that:

Luck and randomness are related but not the same: He distinguishes luck as operating at the level of the individual or small group while randomness operates at the level of the system where more persistent and reliable statistical patterns can be observed.
Expertise does not necessarily accumulate with experience: It is often assumed that doing something for a long time is sufficient to be an expert but Mauboussin argues that in activities that depend on skill, real expertise only comes about via deliberate practice based on improving performance in response to feedback on the ways in which the input generates the predicted outcome.

Mauboussin is not necessarily introducing anything new in his analysis of why we tend to bad at distinguishing skill and luck. The fact that people tend to struggle with statistics is well-known. The value for me in this book lies largely in his discussion of the psychological dimension of the problem which he highlights as exerting the most profound influence. The quote below captures an important insight that I wish I understood forty years ago.

“The mechanisms that our minds use to make sense of the world are not well suited to accounting for the relative roles that skill and luck play in the events we see taking shape around us.”

The role of ideas, beliefs and narratives is a recurring theme in Mauboussin’s analysis of the problem of distinguishing skill and luck. Mauboussin notes that people seem to be pre-programmed to want to fit events into a narrative based on cause and effect. The fact that things sometimes just happen for no reason is not a satisfying narrative. We are particularly susceptible to attributing successful outcomes to skill, preferably our own, but we seem to be willing to extend the same presumption to other individuals who have been successful in an endeavour. It is a good story and we love stories so we suppress other explanations and come to see what happened as inevitable.

Some of the evidence we use to create these narratives will be drawn from what happened in specific examples of the activity, while we may also have access to data averaged over a larger sample of similar events. Irrespective, we seem to be predisposed to weigh the specific evidence more heavily in our intuitive judgement than we do the base rate averaged over many events (most likely based on statistics we don’t really understand). That said, statistical evidence can still be “useful” if it “proves” something we already believe; we seem to have an intuitive bias to seek evidence that supports what we believe. Not only do we fail to look for evidence that disproves our narrative, we tend to actively suppress any contrary evidence we encounter.

Analytical tools for navigating the skill luck continuum

We need tools and processes to help manage the tendency for our intuitive judgements to lead us astray and to avoid being misled by arguments that fall into the same trap or, worse, deliberately exploit these known weaknesses in our decision-making process.

One process proposed by Mauboussin for distinguishing skill from luck is to:

First form a generic judgement on what the expected accuracy of our prediction is likely to be (i.e. make a judgement on where the activity sits on the skill-luck continuum)
Next look at the available empirical or anecdotal evidence, distinguishing between the base rate for this type of activity (if it exists) and any specific evidence to hand
Then employ the following rule:
- if the expected accuracy of the prediction is low (i.e. luck is likely to be a significant factor), you should place most of the weight on the base rate
- if the expected accuracy is high (i.e. there is evidence that skill plays the prime role in determining the outcome of what you are attempting to predict), you can rely more on the specific case.
use the data to test if the activity conforms to your original judgement of how skill and luck combine to generate the outcomes

Figuring out where the activity sits on the skill-luck continuum is the critical first step and Mauboussin offers three methods for undertaking this part of the process: 1) The “Three Question” approach, 2) Simulation and 3) True Score Theory. I will focus here on the first method which involves

First ask if you can easily assign a cause to the effect you are seeking to predict. In some instances the relationship will be relatively stable and linear (and hence relatively easy to predict) whereas the results of other activities are shaped by complex dependencies such as cumulative advantage and social preference. Skill can play a part in both activities but luck is likely to be a more significant factor in the latter group.
Determining the rate of reversion to the mean: Slow reversion is consistent with activities dominated by skill, while rapid reversion comes from luck being the more dominant influence. Note however that complex activities where cumulative advantage and social preference shape the outcome may not have a well-defined mean to revert to. The distribution of outcomes for these activities frequently conform to a power law (i.e. there are lots of small values and relatively few large values).
Is there evidence that expert prediction is useful? When experts have wide disagreement and predict poorly, that is evidence that luck is a prime factor shaping outcomes.

One of the challenges with this process is to figure out how large a sample size you need to determine if there is a reliable relationship between actions and outcome that evidences skill. Another problem is that a reliable base rate may not always be available. That may be because the data has just not been collected but also because a reliable base rate simply may not even exist.

The absence of a reliable base rate to guide decisions is a feature of activities that do not have simple linear relationships between cause and effect. These activities also tend to fall into Nassim Taleb’s “black swan” domain. The fundamental lesson in this domain of decision making is to be aware of the risks associated with naively applying statistical probability based methods to the problem. Paul Wilmott and David Orrell use the idea of a “zone of validity” to make the same point in “The Money Formula”.

The need to understand power laws and the mechanisms that generate them also stands out in Mauboussin’s discussion of untangling skill and luck.

The presence of a power law depends in part on whether events are dependent on, or independent of, one another. In dependent systems, initial conditions matter and come to matter more and more as time goes on. The final outcomes are (sometimes surprisingly) sensitive to both minor variations in the initial conditions and to the path taken over time. Mauboussin notes that a number of mechanisms are responsible for this phenomenon including preferential attachment, critical points and phase transitions are also crucial.

“In some realms, independence and bell-shaped distributions of luck can explain much of what we see. But in activities such as the entertainment industry, success depends on social interaction. Whenever people can judge the quality of an item by several different criteria and are allowed to influence one another’s choices, luck will play a huge role in determining success or failure.”
“For example, if one song happens to be slightly more popular than another at just the right time, it will tend to become even more popular as people influence one another. Because of that effect, known as cumulative advantage, two songs of equal quality, or skill, will sell in substantially different numbers. … skill does play a role in success and failure, but it can be overwhelmed by the influence of luck. In the jar model, the range of numbers in the luck jar is vastly greater than the range of numbers in the skill jar.”
“The process of social influence and cumulative advantage frequently generates a distribution that is best described by a power law.”
“The term power law comes from the fact that an exponent (or power) determines the slope of the line. One of the key features of distributions that follow a power law is that there are very few large values and lots of small values. As a result, the idea of an “average” has no meaning.”

Mauboussin’s discussion of power laws does not offer this specific example but the idea that the average is meaningless is also true of loan losses when you are trying to measure expected loss over a full loan loss cycle. What we tend to observe is lots of relatively small values when economic conditions are benign and a few very large losses when the cycle turns down, probably amplified by endogenous factors embedded in bank balance sheets or business models. This has interesting and important implications for the concept of Expected Loss which is a fundamental component of the advanced Internal Rating Based approach to bank capital adequacy measurement.

Mauboussin concludes with a list of ten suggestions for untangling and navigating the divide between luck and skill:

Understand where you are on the luck skill continuum
Assess sample size, significance and swans
Always consider a null hypothesis – is there some evidence that proves that my base belief is wrong
Think carefully about feedback and rewards; High quality feedback is key to high performance. Where skill is more important, then deliberate practice is essential to improving performance. Where luck plays a strong role, the focus must be on process
Make use of counterfactuals; To maintain an open mind about the future, it is very useful to keep an open mind about the past. History is a narrative of cause and effect but it is useful to reflect on how outcomes might have been different.
Develop aids to guide and improve your skill; On the luck side of the continuum, skill is still relevant but luck makes the outcomes more probabilistic. So the focus must be on good process – especially one that takes account of behavioural biases. In the middle of the spectrum, the procedural is combined with the novel. Checklists can be useful here – especially when decisions must be made under stress. Where skill matters, the key is deliberate practice and being open to feedback
Have a plan for strategic interactions. Where your opponent is more skilful or just stronger, then try to inject more luck into the interaction
Make reversion to the mean work for you; Understand why reversion to the mean happens, to what degree it happens, what exactly the mean is. Note that extreme events are unlikely to be repeated and most importantly, recognise that the rate of reversion to the mean relates to the coefficient of correlation
Develop useful statistics (i.e.stats that are persistent and predictive)
Know your limitations; we can do better at untangling skill and luck but also must recognise how much we don’t know. We must recognise that the realm may change such that old rules don’t apply and there are places where statistics don’t apply

All in all, I found Maubossin’s book very rewarding and can recommend it highly. Hopefully the above post does the book justice. I have also made some more detailed notes on the book here.

Tony

Looking under the hood – The IRB formula

This post is irredeemably technical so stop here if that is not your interest. If you need to understand some of the mechanics of the formula used to calculate credit risk weighted assets under the advanced Internal Ratings Based (IRB) approach then the BCBS published a paper in 2005 which offers an explanation:

describing the economic foundations
as well as the underlying mathematical model and its input parameters.

While a lot has changed as a result of Basel III, the models underlying the calculation of Internal Rating Based Capital (IRB) requirements are still based on the core principles agreed under Basel II that are explained in this BCBS paper.

The notes in the linked page below mostly summarise the July 2005 paper with some emphasis (bolded text) and comments (in italics) that I have added. The paper is a bit technical but worth reading if you want to understand the original thinking behind the Basel II risk weights for credit risk.

I initially found the paper useful for revisiting the foundation assumptions of the IRB framework as background to considering the regulatory treatment of Expected Loss as banks transition to IFRS9. The background on how the RW was initially intended to cover both Expected and Unexpected Loss, but was revised such that capital was only required to cover Unexpected Loss, is especially useful when considering the interaction of loan loss provisioning with capital requirements.

Reading the BCBS paper has also been useful for thinking through a range of related issues including:

The rationale for, and impact of, prudential conservatism in setting the risk parameters used in the IRB formula
The cyclicality of a risk sensitive capital requirement (and potential for pro cyclicality) and what might be done to mitigate the risk of pro-cyclical impacts on the economy

If you have read this far then my summary of the BCBS paper and my comments /observations can be found here (and thank you).

I am not a credit risk model expert, so the summary of the paper and my comments must be read with that in mind. I did this to help me think through some of the issues with bank capital adequacy. Hopefully others will find the notes useful. If you see something wrong or something you disagree with then let me know.

APRA’s proposed revisions to capital requirements for residential mortgages

… there is a lot to like in what APRA have proposed but also some issues that would benefit from further thought

Many readers will be aware that APRA released a Discussion Paper (DP) last week titled “Revisions to the capital framework for authorised deposit-taking institutions”. The paper sets out APRA’s proposed changes to ADI capital requirements defined by the Internal Ratings Based Approach (IRB) and Standardised Approach to Credit Risk, Interest Rate Risk in the Banking Book (IRRBB) and Operational Risk. The focus of this post will be the proposals impacting credit risk capital requirements for residential mortgage lending. This post presupposes that the reader is familiar with the detail of what APRA has proposed. For those of you who have not yet got around to reading the whole paper I have added a short summary of the proposals below (see “APRA’s proposals – in more detail”).

My gut reaction is that there is a lot to like in what APRA have proposed but there are also issues that deserve further consideration in order to address the risk of unintended consequence and to better deliver on the objectives of consistency, transparency and competitive neutrality.

Proposals which make sense to me:

The increased risk sensitivity of the proposed standardised RWs for residential mortgages is, I believe, a material enhancement of the capital adequacy framework
There are arguments (and indeed evidence) for why investor property loans can be as low risk as owner occupier loans (most of the time) but APRA’s desire to address the systemic tail risk of this form of lending is I think an understandable policy objective for a prudential regulator to pursue
Continuing to pursue higher IRB RW via changes to the correlation factor also looks to be a better approach than the 20% floor on LGD currently applied and thankfully also up for revision
Applying a higher correlation factor to low PD loans also makes intuitive sense, especially if your primary concern is the systemic risk associated with the residential mortgage lending that dominates the balance sheets of your banking system
In addition, the potential for the correlation adjustment to reduce the sensitivity of residential mortgage RWA to the economic cycle (and hence reduce the risk of pro-cyclical stress on capital ratios) is particularly welcome though I believe there is much more to do on this general issue
The support for Lender’s Mortgage Insurance (LMI) is also welcome

Areas where I believe the proposed revised capital framework could be improved (or at least benefit from some more thought):

The discussion of relative standardised and IRB RW does not address the fact IRB banks are required to hold additional capital to cover any shortfall between loan loss provisions and Regulatory Expected Loss (REL)
Residential mortgage portfolios subject to the standardised approach should be subject to a minimum average RW in the same way that IRB portfolios are currently constrained by the 25% floor
Applying a fixed scalar to Credit RWA can be problematic as the composition of the loan portfolio continues to evolve

The discussion of comparative IRB and Standardised RW you typically encounter seems to assume that the two approaches are identical in every aspect bar the RW but people working at the coal face know that the nominal RW advantage the IRB banks have has been partly offset by a higher exposure measure the RW are applied to. It appears that APRA’s proposed revisions will partly address this inconsistency by requiring banks using the Standardised Approach to apply a 100% Credit Conversion Factor (CCF) to undrawn loan limits. IRB banks are also required to take a Common Equity Tier 1 deductions for the shortfall between their loan loss provisions and REL. The proposed revisions do nothing to address this area of inconsistency and in fact the Discussion Paper does not even acknowledge the issue.

Residential mortgage portfolios subject to the standardised approach should be subject to a minimum average RW in the same way that IRB portfolios are constrained. The majority of new residential mortgages are originated at relatively high LVR (most at 70% plus and a significant share at 80% plus), but the average LVR will be much lower as principal is repaid (and even more so if you allow for the appreciation of property values). The introduction of a 20% RW bucket for standardised banks poses the question whether these banks will have an advantage in targeting the refinancing of seasoned loans with low LVR’s. The IRB banks would seek to retain these customers but they will still be constrained by the 25% average RW mandated by the FSI while the standardised banks face no comparable constraint.

This is unlikely to be an issue in the short term but one of the enduring lessons learned during my time “on the inside” is that banks (not just the big ones) are very good at identifying arbitrages and responding to incentives. It is widely recognised that housing loans have become the largest asset on Australian bank balance sheets (The Royal Commission issued a background paper that cited 42% of assets as at September 2017) but the share was significantly less when I started in banking. There has been a collection of complex drivers at play here (a topic for another post) but the relatively low RW has not harmed the growth of this kind of lending. Consequently, it is dangerous to assume that the status quo will persist if incentives exist to drive a different outcome.

This competitive imbalance could be addressed quite simply if the standardised banks were also subject to a requirement that their average RW was also no lower than 25% (or some alternative floor ratio that adjusted for the differences in exposure and REL noted above).

Another lesson learned “on the inside” is that fixed scalars look simple but are often not. They work fine when the portfolio of assets they are scaling up is stable but will gradually generate a different outcome to what was intended as the composition of the loan book evolves over time. I don’t have an easy solution to this problem but, if you must use them, it helps to recognise the potential for unintended consequence at the start.

Read on below if you have not read the Discussion Paper or want more detail on the revisions APRA has proposed and how these changes are proposed to be reconciled with the FSI recommendation. This is my first real post so feedback would be much appreciated.

Above all, tell me what I am missing …

Tony

Note: The original version of this post published 22 February 2018 stated that inconsistent measurement of the exposures at default between the standardised and IRB approaches was not addressed by APRA’s proposed revisions. I believe now that the proposed application of a 100% CCF in the Standardised Approach would in fact address one of the areas of inconsistency. The treatment of Regulatory Expected Loss remains an issue however. The post was revised on 24 February to clarify these points.

APRA’s proposals – in more detail

Good quality loans fully secured by mortgages on occupied residential property (either rented or occupied by the borrower) have been assigned concessionary risk weights (RW) ever since risk weighted capital adequacy ratios were introduced under Basel I (1988). The most concessionary risk weight was initially set at 50% and reduced to 35% in the Basel II Standardised Approach (2006).

APRA currently applies the concessionary 35% RW to standard eligible mortgages with Loan Valuation Ratios (LVR) of 80% or better (or up to 90% LVR if covered by Lender’s Mortgage Insurance) while the best case scenario for a non-standard mortgage is a 50% RW. Progressively higher RW (50/75/100) are applied for higher risk residential mortgages.

Under the Standardised Approach, APRA proposes:

The classification of a Standard Eligible Mortgage will distinguish between lowest risk “Owner-occupied P&I” and a higher risk “Other residential mortgages” category which is intended to be conceptually similar to the “material dependence” concept employed by Basel III to distinguish loans where repayment depends materially on the cash flows generated by the property securing the loan
6 RW bands for each of these two types of residential mortgage (compared to 5 bands currently)
Standard Eligible Mortgages with lower LVR loans to be assigned lower RW but these loans must also meet defined serviceability, marketability and valuation criteria to qualify for the concessionary RW
The higher RW applied to “Other residential mortgages” may take the form of a fixed risk-weight schedule (per the indicative RW in Table 3 of the Discussion Paper) but might also be implemented via a multiplier, applied to the RW for owner-occupied P&I loans, which might vary over time “… depending on prevailing prudential or financial stability objectives or concerns”
Relatively lower capital requirements to continue to apply where loans are covered by LMI but its preferred approach is to apply a RW loading to loans with LVR in excess of 80% that are not insured (i.e. the indicative RW in Table 3 assume that LMI covers the high LVR loans)
Non-Standard residential mortgages should no longer benefit from any RW concession and be assigned a flat 100% RW irrespective of LVR and LMI

While the IRB requirements impacting residential mortgages are largely unchanged under Basel III, APRA proposes the following changes to the Australian IRB Approach to reflect local requirements and conditions:

Increased capital requirements for investment and interest-only exposures; to be implemented via a higher correlation factor for these loans
The (currently fixed) correlation factor applied to residential mortgages to be amended to depend on probability of default (PD); reflecting empirical evidence that “… the default risk of lower PD exposures is more dependent on the economic cycle and can consequently increase at a relatively higher rate in a downturn”
A reduction in the minimum Loss Given Default (LGD) from 20% to 10% (subject to APRA approval of the LGD model); in order to facilitate “… better alignment of LGD estimates to key drivers of loss such as LVR and LMI”
Capital requirements for non-standard mortgages use the standardised approach; increasing consistency between the IRB an standardised approaches

APRA’s proposals seek to strike a balance between risk sensitivity and simplicity but must also take account of the FSI recommendations that ADI capital levels be unquestionably strong while also narrowing the difference between standardised and IRB RWs for residential mortgages. APRA is undertaking a Quantitative Impact Study (QIS) to better understand the impact of its proposals but the DP flagged that APRA does not expect the changes to correlation factors to meet its objectives for increased capital for residential mortgage exposures.

APRA could just further ramp up the correlation factor to generate the target IRB RW (which I assume continues to be 25%) but the DP notes that this would create undesirable inconsistencies with the correlation factors applied to other asset classes. Consequently, the DP indicates that the target increase in IRB RWA will likely be pursued via

A fixed multiplier (scalar) applied to total Credit RWA (i.e. althoughBasel III removes the 1.06 Credit RWA scalar, APRA is considering retaining a scalar with a value yet to be determined); and
If necessary, by applying additional specific RWA scalars for residential (and commercial) property.

These scalars will be subject to consultation with the industry and APRA has committed to review the 10.5% CET1 benchmark for unquestionably strong capital should the net result of the proposed revisions result in an overall increase in RWA’s relative to current methodologies.