Validating AMA frameworks - A Regulators Experience in Japan

2nd International Conference on Operational Risk Sao Paulo, Brazil, June 5, 2009 Tsuyoshi Nagafuji Financial Services Agency, Japan

This presentation does not necessarily express established views or policies of the FSA.

Overview : Objective
Sharing my experience in validating and approving Japanese banks AMA applications. Presenting what we have done or what we are actually doing in Japan, rather than what we hope to do. Focusing on the factors that remain until the final stage for application, which banks find difficult and time consuming to address. 9 Model sensitivity analysis / stress testing: Do you know all the possibilities for strange behavior? 9 Scenarios rules and documentation: Have you done your best to exclude subjectivity? 9 Use test: Are you actually using the framework? Is it really working?
2

Overview : Method
I am using a realistic model of AMA models" as an example, in consideration of anonymity. I am using The model of AMA models that I presented at the Operational Risk Scenario Analysis Workshop held at Bank of Japan, the central bank, in 2006*.
* The model presented here is the same as the one I presented in 2006, but the description is simplified. Please see Quantification of Operational Risk Using Scenario Data (Nagafuji, 2006) for the details.

The model is extremely simplified but still retains some aspects of typical AMA models used by Japanese banks. 9 The model is based on real internal data and real scenario data from major Japanese banks. 9 The model has a similar structure to typical Japanese models.

Overview : Outline
Presentation Overview (5 minutes) 1. Context (5 minutes) 2. Sample Model (10 minutes) 3. Validation of the Sample Model (15 minutes) Concluding Remarks (5 minutes) Q & A (5 minutes) (Total: 45 minutes)
4

Overview 1. Context 2. Sample Model 3. Validation Concluding Remarks Appendix

Context: Japanese Banking Industry

Consists of three "Mega banks and many smaller banks. Foreign banks play a very small role.
World banks by Tier I capital ($billion) (Source: The Banker, July 2008) MUFG
120 100 80 60 40 20 0

( $82 bil)

Mizuho 6 15

($49 bil)

18

SMFG ($44
28 38 48 50 66
co

81 89 94
co u Ita es an

SB C C R iti B JP S BOM A C re di tA IC g B SCC H PaBO C Ba rib rc as C lays C H BC BO S U ni C re d IN it G W ac ho R via ab o

Br ad

Top 20 Banks
<Japanese banks> 98 banks in the Top 1,000 list.

Top 21 100
<Brazilian banks> 14 banks in the Top 1,000 list.

ni b

(Now merged)

BB

Context: Op Risk in Japanese Banks

Operational risk losses are extremely small. Total annual loss amount
(Average Dollar Amount by year, percentages of total assets)

Loss frequencies
(# of losses greater or equal to $20,000, per year, per total assets of $1 billion)

US banks Japanese banks

0 0.01

US banks Japanese banks

0.04 0.05 0.06 0.07 0 0.5

About

1/40
0.03

About

1/20
1 1.5 2

0.02

* Both figures are medians of the banks that participated in the exercise (Source) 2004 U.S. LDCE, 2007 Japan LDCE (See Appendix: References about Japanese AMA implementation" for detail).

Context: Application Timetable

Typical validation process
1. Preparation

1. Preparation
Banks are encouraged to develop their framework to a practical level and use it for their internal purposes before going into the parallel run.

Ready?

2. Parallel Run (At least one year)

2. Parallel Run Two capital calculations are verified through visits and regular discussions.

OK?

3. Approval
Banks that do not meet the requirement stay at stage 2 or go back to stage 1.

3. Approval

Currently one banking group has been approved for the AMA.
8

Overview 1. Context 2. Sample Model 3. Validation Concluding Remarks Appendix

Sample Model: Overview

Several major banks are treated as if they were a single big bank. 9 Their internal loss data and scenario data are put into a very simple loss distribution model (LDA).
Bank A Bank B Bank C Scenario Data Internal Loss Data LDA Model Results

Bank X

10

Sample Model: Quantification Model

Monte Carlo simulation (100,000 simulations)
Frequency: Poisson distribution = Frequency of Scenarios + Frequency of Real Loss Data Severity: Empirical distribution

Single unit of measure (= top of the house calculations)

11

Sample Model: Internal Loss Data

Banks internal loss data are used as if they were from one big bank.
Bank A Bank B Bank C Scenario Data Internal Loss Data LDA Model Results

Bank X

Each data point is assumed to have a frequency of once in 10 years, because the observation period is 10 years
12

Sample Model: Scenarios (1/4)

Each banks scenarios for their quantification are used. 1) Independent scenarios:
Scenarios that hit each bank independently Each banks scenarios are put into the model as they are.

2) Common scenarios:
Scenarios that could hit all banks at the same time: earthquakes and inter-bank settlement system failures. Each banks scenarios are aggregated to a single scenario and then put into the model.

13

Sample Model: Scenarios (2/4)

Common Scenarios 1: Earthquakes
Losses by historical earthquakes are estimated for each bank and then aggregated.
Description (year , Frequency Severity Details magnitude of earthquake) (once in X (largest years) =100) Earthquake in Tokyo 1,200 100 Earthquake greater than any of those below is assumed. Keian (1649, 7.1) 49 (Frequency) 8 large-scale earthquakes between 1600 and Genroku (1703, 8.2) 85 1925 in Tokyo, Nagoya and Osaka are listed, assuming each Ansei Edo (1855, 6.9) 55 will occur once every 400 years. 400 each Meiji Tokyo (1894, 7.0) 47 (Severity) Great Kanto (1923, 7.9) 82 <Buildings> The damage to the building, furniture and the opportunity cost due to interruption of business are Hoei (1707, 8.4) 57 calculated based on the earthquake intensity and quake Ansei (1854, 8.4) 50 resistance of the buildings. Nobi (1881, 8.0) 55 <Systems> Extra work cost, damage to the machines and equipment and the opportunity cost due to business interruption are calculated. Damage to the computer center and paralysis of the head office functions are assumed. <Other> Declines in the value of the loans (including impairment of the value of collateral) are not factored in. Tokyo (1926) - Aichi (1997) (61 earthquakes) Average (Frequency) 61 earthquakes occurred between 1926 and 97 77 each 0.4 (of intensity 5- or higher) are listed, assuming each will occur once every 77 years. (Severity) as shown above.

14

Sample Model: Scenarios (3/4)

Common Scenario 2: Failure in the settlement system
A scenario was created where a failure occurs in the computer systems commonly used by the banks once every twenty years, causing total damage of JPY 20 billion. The following scenarios from a bank were referred to in creating this scenario.
Frequency Once in several decades Once in several decades Severity Details JPY several billions A failure in the accounting system or in the domestic ($US tens of millions) network, which would take 12 hours for full recovery. JPY several 1) A failure occurs in the communication infrastructure, or, 2) hundred millions there is a flaw in the emergency handling procedures, ($US several millions) causing interruption of the settlement operation for half a day. The compensation for damage paid to securities exchanges as clearing agents in charge of settlement of the government bonds is included. JPY several billions Foreign exchange / settlement operations are not performed ($US tens of millions) for a full day due to a system failure JPY several A failure occurs in the Zengin System just after 9:00 am. The hundred millions system recovers at around noon. However, the settlement ($US several millions) operation is erratic during that day.

Once in several decades Once every several years

15

Sample Model: Scenarios (4/4)

Independent Scenarios
<a> Actual scenarios collected from banks are used as they were.
<Scenarios Used> BIS event Major scenarios (scenarios for larger amounts of losses) types # of scenarios Examples Internal Fraud External Fraud Employment Clients, Products Physical assets Systems Process Total 30 3 5 30 Fraud in the market trading functions, withdrawal of customer funds Swindles, compromised online banking Discrimination Lenders liability, inappropriate advice to customers, failure to explain the risks, etc 11 Terrorist attacks

12 Failure in the accounts transfer system, including interruption of the accounting system 38 Failure in bond settlement (overseas), improper identity verification, error in cash transfer, etc 129

<b> Made-up scenarios for banks that did not have scenarios are also used. Some scenarios from <a> scaled by the total assets of each bank. 16

Sample Model: Results

Risks at the confidence levels of 99% and 99.9% are quantified.
The risk at a confidence level of 99.9% is about one third of the Basic Indicator Approach (BIA) amount. <Results as percentages of BIA amount>

99% 23%

99.9% 34%

EL 7%

(BIA) (100%)

Big scenarios, especially earthquakes, contribute much to the results.

17

Sample Model: Recap

The model retains some aspects of AMA models used by Japanese banks.
Sample Model Internal loss data and scenarios are directly input into the model. Internal Losses Model
External Loss Data
9

External loss data and BEICFs inform scenarios

Risk Amount

Scenarios
BEICFs*
9

Scenarios are estimated as individual data points. Number of scenarios are large.

9 9

Number of cells is relatively small. Empirical distribution is used by some banks.

*Business Environment and Internal Control Factors

18

Overview 1. Context 2. Sample Model 3. Validation Concluding Remarks Appendix

19

Validation of the Sample Model: Overview

Three major points that banks have trouble with in completing their AMA application.
(1) Model: (2) Scenarios: (3) Use test: Sensitivity analysis / stress testing Rules and documentations Are you actually using the framework?
Sensitivity analysis/ stress testing: Do you know all the possibilities for strange behavior?

Internal Losses

(1) Model
Risk Amount

External Loss Data

(2) Scenarios Scenarios

BEICFs

Rules and documentations: Have you done your best to exclude subjectivity?

(3) Use test

Are you actually using the framework? Is it really working?

20

Validation: Model (1/3)

Statistical integrity of the model is essential, but has not been a determining factor at the last stage.
<The sample model> 9 LDA model Independence between frequency and severity should be accounted for. 9 Choice of distributions and estimation methods Those may not be great discussion points as the sample model uses empirical distribution. 9 Granularity Independence among the data points should be accounted for.

As long as assumptions in the model are clarified and accounted for, this factor is not a decisive one.

21

Validation: Model (2/3)

Instead, sensitivity analysis / stress testing has been a great challenge at the last stage. 9 No model is free from strange (counter-intuitive) behavior. 9 Comprehending all the possibilities of strange behavior is time consuming, especially when the model is complex.
<The sample model> Massive losses have a large impact. A small change in frequency for a massive loss may have a large impact. <Simplified example> (Frequency: Poisson, Severity: Empirical) Data set 1) and data set 2) give completely different risk, although the only difference is the frequency of a single big loss!
Losses One JPY100 billion Loss (Once in 999 years) + 100 JPY 10,000 Losses (Each once in 10 years) One JPY100 billion Loss (Once in 1000 years) + 100 JPY 10,000 Losses (Each once in 10 years) EL 0.1 billion 0.1 billion 99.9 100 billion 0.0003 billion

1) 2)

3 / 1,000,000

22

Validation: Model (3/3)

Comprehending unusual (counter-intuitive) behavior of a model is essential for regulatory and internal purposes. We request banks: 9 To comprehend possible strange behaviors of their model. 9 To address those strange behaviors. Accept them (Management should fully understand the consequences). Revise and reconstruct their models. May take a lot of time.

23

Validation: Scenarios (1/3)

Estimation of frequency and severity is essential, but not a determining factor at the last stage. Scenarios for the sample model must include once in 1000 years events, as the model uses empirical distribution for severity. We verify this through
9 Checking the logic. 9 Checking facts that scenarios are based on. 9 Benchmarking scenarios between banks. <Examples> Earthquakes: Is the use of past earthquakes appropriate? What is included as losses from earthquakes? Is the latest seismological knowledge utilized? System failures: Are the statistics on computer failures utilized? What is the accuracy of the statistics? Are the statistics used appropriately?

As it is impossible and inappropriate to press one specific view, this factor has not been a decisive one at the last stage.
24

Validation: Scenarios (2/3)

Instead, setting rules in making scenarios and making thorough documentation has played a determining role at the last stage. (A hypothetical example)
External events that could happen to the bank are made into scenarios. The bank sets up a rule in judging whether a particular event can happen to it.

External Loss Data

Scenarios
BEICFs

The bank sets the frequency of scenarios based on the RCSA scores (ex. How effective are controls?).

The bank sets up a rule in judging whether a particular event can happen to it. 25

Validation: Scenarios (3/3)

We request banks to: 9 Do their best to set rules that ensure the same estimate regardless of who the estimator is. 9 Fully document and account for subjective judgments that remain. This looks easy to accomplish, but turns out to be very challenging, because: 9 Rules can be set only after experience is accumulated. 9 This is often neglected until the last stage.

26

Validation: Use test (1/3)

Use test is a strong tool to improve an AMA framework. Thus, it is very challenging and often becomes a determining factor at the last stage. Our use test in Japan is not special. Banks should show that they use their AMA framework in their day-to-day risk management (= The framework is not exclusively for regulatory purposes). Use test is based on the idea that supervisors can be more confident with an AMA framework that is really used.

27

Validation: Use test (2/3)

Banks decide how to demonstrate their compliance with the use test (= Banks decide how to use their AMA framework). Many Japanese banks choose to base their risk management cycle on their AMA model.
Evaluate risk reduction measures based on the model. Implement risk reduction measures based on the model.

.Plan

2. Do

Risk reduction measures Introducing double checking Computerize operations Restricting operations

Model/ Risk
3. See Verify the results using the model. 28

Validation: Use test (3/3)

Use test imposes improvement on their overall AMA framework. It imposes improvement on the model.
9 The model needs to be practically free from counter-intuitive behavior 9 The model needs to be sensitive enough.

Understanding by management and business units is essential. Thus, it often becomes a determining factor at the final stage. When the AMA framework does not meet the use test requirement, it needs modification. When the modification is drastic, banks are required to take the use test again, which needs at least half a year to complete.

29

Overview 1. Context 2. Sample Model 3. Validation Concluding Remarks Appendix

30

Concluding Remarks: In Theory

In theory, capturing 99.9% risk is the single most important requirement of AMA models. 9 Do distribution assumptions capture 99.9% risk? 9 Are scenarios representing once-in-1000-year events? However, this requirement does not turn out to be the remaining factor at the last stage. 9 Choice of distribution or estimation of scenarios boils down to subjective judgments, which are argumentative, but not decisive. 9 Banks that cannot address these issues cannot enter the parallel run.

31

Concluding Remarks: In Practice

The following practical factors have often played a decisive role at the last stage. 1. Comprehending unusual (counter-intuitive) behavior of the model and preparing for it. 2. Setting rules and perfecting documentation to minimize the subjectivity of scenarios. 3. Meeting the Use test requirement Those factors ensure workable framework both for internal and regulatory purposes.

32

Concluding Remarks: Challenges for Banks

We do not press banks to have an ideal or very sophisticated framework. Rather, we ask them to have a practical, reliable framework to meet requirements for regulatory purposes. After all, it is up to banks to build an AMA framework that truly enhances their risk management.

33

 Questions?
For further questions, feel free to contact:

34

Appendix: References about Japanese AMA implementation

Sample model used in this presentation 9 Quantification of Operational Risk Using Scenario Data (July, 2006) http://www.boj.or.jp/en/type/release/zuiji_new/data/fsc0608be3.pdf Losses of Japanese banks (Page 7 of this presentation) 9 Results of the 2007 Operational Risk Data Collection Exercise (August, 2007) http://www.fsa.go.jp/en/news/2007/20070810-2.pdf 9 A research paper comparing the loss data between the U.S. and Japan (April, 2008) http://www.boj.or.jp/en/type/release/adhoc/data/risk0804a.pdf (For the results of U.S. LDCE, see http://www.bos.frb.org/bankinfo/qau/papers/pd051205.pdf) Others 9 Use of External Data for Operational Risk Management Workshop (April, 2008) http://www.boj.or.jp/en/type/release/adhoc/fsc0804a.htm 9 The Effect of the Choice of the Loss Severity Distribution and the Parameter Estimation Method on Operational Risk Measurement (December, 2007) http://www.boj.or.jp/en/type/ronbun/ron/research07/ron0712c.htm 9 Discussions on Further Advancing Operational Risk Management (Part1: June 2006, Part2: August 2006) Part1: http://www.boj.or.jp/en/type/release/zuiji_new/fsc0608c.pdf Part2: http://www.boj.or.jp/en/type/release/zuiji_new/fsc0612a.pdf 9 Operational Risk Scenario Analysis Workshop (July 2006) http://www.boj.or.jp/en/type/release/zuiji_new/fsc0608a_add.htm

35