1

Credit Risk Model Validation

Selected Model Validation Challenges

Dr Christian Kaiser
Head of Independent Review HSBC Brazil 23 October 2012
2

Contents 1. 2. 3. 4. Materiality of Defaults Rating System Granularity Rating System Dimensionality Multicollinearity and Omitted Variable Bias

Materiality of Defaults

Materiality of Defaults (1)

Background
Central point for model validation is check if default definition used for modelling is correct 2 criteria according to regulation (1) defaults due to 90 dpd; (2) unlikeliness to pay Contested question: Should there be a loss materiality threshold to consider defaults (in application and/or modelling)?

FSA default definition

[BIPRU 4.3.56 01/01/2007] A default must be considered to have occurred with regard to a particular obligor when either or both of the two following events has taken place: the firm considers that the obligor is unlikely to pay its credit obligations to the firm, the parent undertaking or any of its subsidiary undertakings in full, without recourse by the firm to actions such as realising security (if held); and the obligor is past due more than 90 days on any material credit obligation to the firm, the parent undertaking or any of its subsidiary undertakings

Default definition generally very similar across regulators

[aligned with BCBS, 2006, art. 452)]

Materiality of Defaults (2)

Materiality of 90 dpd default discussion
FSA (CP06/3): Initial consideration - Specification of prescriptive exposure thresholds (exclude smaller than 100 for retail, 1000 for non-retail) After industry feedback (PS06/6) Thresholds can be defined by firms themselves, (CRSG 2008) but must be in relation to total exposures, and not in relation to overdue amounts In practice threshold generally considered to be optional

Materiality of pre 90 dpd (qualitative) default discussion

Unlikeliness to pay (extracts of indicators) FSA Firm selling credit obligation at a material credit-related economic loss Firm consenting to a distressed restructuring likely to result in a diminished financial obligation caused by the material forgiveness or postponement of principal, interest or fees. A instituo vende, transfere ou renegocia com perda econmica relevante os direitos de crdito relativos obrigao, devido a deteriorao significativa da qualidade to crdito do tomador ou contraparte
6

Bacen

Materiality of Defaults (3)

Regulatory concern
Cherry Picking, if defaults with low materiality are chosen conservative PD, but lower LGD could reduce RWA
LGD 10%

wider default definition K values

PD 10% 20% 30% 50%

Basically possible to choose wider default definition (increase PD, decrease LGD reduce K/RWA) But opposite also possible: taking out defaults will reduce RWA (inverse cherry picking) Total effect of wide/narrow default definition on RWA depends on PD/LGD and PD/LGD combination
LGD

3,6% 4,5% 4,7% 4,0% 20% 7,3% 9,0% 9,3% 8,1% 30% 10,9% 13,5% 14,0% 12,1% 50% 18,2% 22,5% 23,4% 20,1%
PD 5% 20% 25% 30% 35% 10% 15% 20%

5,3% 7,3% 8,4% 9,0% 6,6% 9,1% 10,5% 11,2% 7,9% 10,9% 12,6% 13,5% 9,2% 12,7% 14,7% 15,7%

Likelihood that wide default definition reduces RWA. But effect uncertain (particularly for low default portfolios the opposite can occur)
7

Materiality of Defaults (4)

Perception Brazil
General insecurity of model developers in industry about correct default definition regarding materiality Model redevelopments have been/are being undertaken to address issue Perception - alternative considered in industry: Exclude all or specific restructurings for (retail) default modelling but no application of quantitative materiality filter

Materiality of Defaults (5)

Issues to be considered for validation
1. Removing all (or specific) types of restructurings in general without individual materiality check a. Unclear if this fully answers materiality question analyses suggest that restructurings are on average associated with losses (average LGD may be lower than for 90dpd defaults, but can still be material) b. Potential conflict with regulatory views in other countries (e.g. FSA) c. Model will not be strong in predicting defaults due to restructurings Recommendations Questions: Are qualitative defaults unlikeliness to pay cases, are they (in parts) material? 1. 2. Perform studies on significance and materiality: Percentage of restructured companies that default 90 dpd, calculate LGD and absolute losses) Sensitivity analysis of model parameter estimates and RWA with/without inclusion of questionable cases
9

Materiality of Defaults (6)

2. Potential mismatch if materiality filter is only applied for pre 90 dpd (qualitative defaults) Case Unlikeliness to pay e.g. at 60 dpd triggers specific loss recoveries case not considered for LGD model (as not 90 dpd) But the same facility defaults at 90 dpd default -- then considered for LGD model Potential inconsistencies From which point of time should recoveries be counted? Recommendations 1. 2. Consider alternative: Use same materiality filter by level of loss for 90 dpd and qualitative defaults Get specific regulatory guidance on how to treat these cases

10

Materiality of Defaults (7)

3. Question whether full model redevelopments are necessary and useful

Recommendations 1. Simple alternative solutions would in principle be sufficient to address RWA (cherry picking) concern for modelling: a. b. 2. a. b. Only LGD model filtering (most conservative solution: wide default definition for PD/Score
model; narrow default definition for LGD model )

For PD models (if needed): Filtering at calibration stage only Obtain certainty that applied default definition is acceptable by regulator before model development/enhancement Desirable to have explicit regulatory guidance communicated to all industry (defensible to other foreign regulators and equal level playing field)

Ask for specific regulatory guidance

11

Rating Systems - Granularity

12

Rating Systems Granularity (1)

Rating System
In Brazil Regulatory Requirement to use a rating system Need to segment population into buckets of PD model scores and LGD bands:
GRUPO_LGD LGD Default LGD No Default 10% 30% 50% 80% 5% 10% 20% 30% 40% 70% 80% 90% 1 2 3 4 5 6 7 8 9 10 11 12 Pool 1 Pool 5

GRUPO_PD

0 1 2 3 4 5 6 7 8 9 S/ Behavior Default

Pool 6 Pool 2 Pool 7 Pool 3 Pool 4 Pool 8 Pool 9 Pool 10

13

Rating Systems Granularity (2)

Pools

Requirements for rating systems and

18 Pontos de Observaes

Pool 1 p1 Pool 2 p2

Safra 1 Safra 2 Safra 3 Safra 4 p11 p12 p13 p14

...

Safra n p1n

Exemplo Pool 1:
p11 p1n p12 p1 p14 p13

examples of suitable tests:

1 Ni N M = i 1 1 M
2

Pool 3 p3 Pool 4 p4 Pool 5 p5 Onde i = numero do pool e j = indicador da safra C.V. <= 10% - Verde by buckets Avg. Default Rates 10% < C.V. <= 20% - Amarelo C.V. > 20% - Vermelho

HI

ajustado

Pool 6 p6 Pool 7 p7

Concentration: Adjusted Herfindahl

Pools Pool1 p1
Safra 1 Safra 2 Safra 3 Safra 4 p11 p12 p13 p14 ... Safra n p1n

Homogeneity: Coefficient of Variation

Exemplo Pool 1:

18 Pontos de Observaes

Pool2 p2

p11 p1n p12 p1 p14 p13

Pool3 p3 Pool4 p4 Pool5 p5

Pool6 p6 Pool7 p7

Stability: Confidence Intervals

Heterogeneity: Tukey Test

14

Rating Systems Granularity (3)

Rating System Granularity
Difficulty to prove all criteria in some portfolios To fulfil all requirements can require a rating system with low granularity Essential reason for problems: same model score is associated with a different DR (typical for instance for TTC models, for structural data changes, or insufficient number of variables,) Also more likely in low default portfolios (large random volatility)!
12% 10% 8% 6% 4%

95% CI

Same score different DRs over time DR 300

Avg. DR by buckets, 12 months forward; generic example

2% 0%
20 08 20 01 08 20 03 08 20 05 08 20 07 08 20 09 08 20 11 09 20 01 09 20 03 09 20 05 09 20 07 09 20 09 09 20 11 10 20 01 10 20 03 10 20 05 10 20 07 10 20 09 10 11

IC 1 sup IC 2 sup bucket 1

IC 1 inf IC 2 inf bucket 2

15 t

Rating Systems Granularity (4)

Low granularity PD RS
RS with 3 buckets
Min 0 0.33 0.67 Max 0.33 0.67 1 Midpoint 0.165 0.5 0.835 RS PD cust A

Case 1: PD stays within bucket limits Significant smoothing but over-, or underestimation Case 2: PD crosses bucket limits Rating System increases volatility over time

PD cust A

Comparison Individual PD vs RS PD (Bucket Mid-Points) Choice of 2 typical cases

RS PD cust B RS PD cust A

PD cust B PD cust A

Non granular rating systems can have very negative consequences! Average RS PD can be far away from individual best estimate PDs PD and RWA volatility increases in parts

RS PD cust B t

PD cust B

16

Rating Systems Granularity (5)

Recommendations?
Required number of buckets according to BCBS (2006, Art. 404): For corporate, sovereign and bank exposures [] a bank must have a minimum of seven borrower grades for non-defaulted borrowers and one for those that have defaulted. [ ] supervisors may require banks, which lend to borrowers of diverse credit quality, to
have a greater number of borrower grades

Build model and rating system jointly include stability considerations already in score model development (choice of model factors, consider pooling models) If a satisfactory improvement of the model stability is impossible despite all efforts Validate with more tolerance to accept expectations for rating system (?) Recommend monitoring and frequent re-alignment until better data vailable (?) disadvantages of low granularity can outweigh advantages of meeting all criteria

RS with few buckets may fulfil regulatory requirement for retail, but should be avoided due to severe disadvantages
17 t

Rating Systems - Dimensionality

18

Rating Systems Dimensionality (1)

Dimensionality (Inclusion of LGD part)
IRB-A idea to have estimates of PD and LGD, independently of each other, such that the model allows flexible combinations of PD and LGD
GRUPO_PD GRUPO_LGD LGD No Default LGD Default 10% 30% 50% 80% 5% 10% 20% 30% 40% 70% 80% 90% 1 2 3 4 5 6 7 8 9 10 11 12 Pool 1 Pool 5 Pool 6 Pool 2 Pool 7 Pool 3 Pool 4 Pool 8 Pool 9 Pool 10

In case of rating system (RS) this implies a bidimensional RS in matrix form Complexity: It may often be difficult to prove all criteria for this bi-dimensional rating system Solution: Regulation can be understood that in retail it is permitted to have only a PD RS system, with an average LGD attached to each PD bucket RS becomes one dimensional Art. 75. (Bacen Circular 3581) 3 Para as exposies classificadas na categoria "varejo", o valor do parmetro LGD deve ser estimado para cada grupo homogneo de risco, podendo ser obtido a partir das taxas de perdas observadas no longo prazo e do parmetro PD.

0 1 2 3 4 5 6 7 8 9 S/ Behavior Default

generic example

Pool ID PD long run LGD long run 1 0,5% 59% 2 4% 59% 3 8% 60% 4 15% 59% 5 30% 60% 6 60% 61% 7 100% 62%
19 t

Rating Systems Dimensionality (2)

Negative consequences: LGD model granularity is completely lost RS becomes a PD RS only, with fixed LGD factors multiplied In case of low or no PD-LGD correlation: LGD will not even differ significantly between rating classes Similar to IRB-F

Recommendations
1. Test rating systems separately Build 2 separate one dimensional rating systems (PD, LGD) Test PD and LGD rating system requirements separately (but not necessarily for joint bidimensional system together) 2. Request guidance from regulator if LGD RS really needed Internationally common to have only a PD RS (if at all), but not an LGD RS PD RS and individual LGD estimates
20 t

Multicollinearity and Omitted Variable Bias

21

Multicollinearity and Omitted Variable Bias (1)

Background
Multicollinearity is effect that predictor variables are themselves highly correlated. With full correlation between the predictor variables the OLS estimator cannot be calculated This is unusual in practice, but a high correlation between the predictors is common

Consequences
OLS estimator remains unbiased and is still the best linear unbiased estimator (BLUE) R2 statistic is unaffected However, high variance in the parameter estimate Effect similar to small sample - having low variability in regressors

X2

X1 Y=X1+aX2

Low confidence in the parameter estimate/low power of hypothesis testing Parameter estimate may not be precise (e.g. in chart 2nd dimension uncertain) Potential out of sample problems if correlation structure between variables changes

22 t

Multicollinearity and Omitted Variable Bias (2)

Detecting Multicollinearity
1. Common to investigate Correlation Matrix: values > 0.8 or 0.9 are considered high in literature Disadvantage: only captures bi-variate correlation Multicollinearity can still be present
1 2 3 4
1 2 4 6 8 2 2 1 2 0 3 0 3 1 0 4 1 0 4 0 3 2 2 1

1 1 0,6 0,4 0,6

2 0,6 1 -0,4 0,4

3 0,4 -0,4 1 0,1

4 0,6 0,4 0,1 1

1 vs 2-4 Correl (Sum) R2 VIF

0,9 0,85 6,8

3.

Other alternative: Condition Index: Square root of the ratio of the largest to the smallest characteristic root of XX Rule of thumb: Value > 30 is harmful
23

Multicollinearity and Omitted Variable Bias (3)

Omitted Variable Bias
Omitted Variable Bias occurs when a model is created which incorrectly leaves out one or more important causal factors Bias: model compensates for the missing factor by over- or underestimating one of the other factors Consequences Presence of omitted variable bias violates OLS assumption: Error and regressors are correlated OLS estimator biased and inconsistent Direction of the bias depends on estimators and covariance between regressors and omitted variables Positive estimator and positive covariance value OLS estimator will overestimate true

Omitted variable bias can be a consequence if variables are too generously discarded for modelling (e.g. due to collinearity considerations)
24 t

Multicollinearity and Omitted Variable Bias (4)

Recommendations
1. From a forecast point of view, multicollinearity issue is often overrated Essential for hypothesis testing, not very concerning for prediction quality (estimator of dependent variable remains BLUE) In practice common to exclude variables already if bivariate correlation is > 0.5 Conservative exclusion of variables ok if variance of dependent variable can be explained with alternative variables. Otherwise, need to be cautious about automatically discarding variables by trading this against other problems (too few variables, reduced prediction power, higher risk of influence of data errors in single variables, use of variables with many missing values, lower business acceptance) 2. 3. 4. 5. Check multicollinearity measures such as VIF; bivariate correlations alone are insufficient Severe multicollinearity should not be accepted. Recommend to reduce multicollinearity (variable transformations, alternative variables) More conservative variable exclusion recommended if expectation that correlation structure changes out of sample/time Test alternative models, accepting higher levels of collinearity at least to benchmark
25