Financial institutions across the globe depend on mathematical models for data-driven business decisions and regulatory compliance. The types of models used in the industry have come a long way; starting from simple business rules to statistically and mathematically derived complex algorithms for assisting business decisions.
With an explosion in available and captured data, and technological enhancements, there has been a marked shift among banks to treat customers individually rather than as a group. This has increased the relevance of analytical models exponentially. For a variety of reasons, this process of building and implementing analytical models is fraught with risk, widely termed as “model risk”.
By definition, model risk is, “the potential for adverse consequences from decisions based on incorrect or misused model outputs and reports”1. Failure to take appropriate steps to mitigate this risk has resulted in financial and reputational losses to banks.
The ~$2.7 million loss incurred by The Commonwealth Bank of Australia over the period of June 2011 to September 2015 is one such example2. The bank used an ‘Automated Serviceability Calculator’ to approve personal overdraft applications. The calculator received incorrect input values, which caused the loss. Such examples establish a need for regulation in the development and use of models.
A regulation to mitigate model risk began to take shape with the BCBS mandate of 1996, where a consensus emerged on the need for model validation as a tool to counter model risk. SR 11-07, the most recent guidance issued by the Federal Reserve/OCC in 2011, takes a broader view of model risk management (MRM), suggesting risk mitigation across all stages of a model’s lifecycle. As a part of a bank’s Comprehensive Capital Analysis and Review (CCAR) submissions, banks are required to submit documentation regarding their model risk management policy and practices. Also, banks now actively publish their MRM practices in their annual reports. With the guideline turning from voluntary to mandatory last year, banks are taking steps to become compliant.
This first layer of model risk is inherent and implicit in all models. We call this systematic model risk. The second layer of model risk is the risk associated with a specific model.
This paper and the SR 11-07 guidelines exclusively focus on the risk associated with the specific models and the industry best practices in mitigating this type of risk.
Sources of model risk and SR 11-07 guidelines:
Model errors primarily arise in the form of models containing mathematical errors or assumptions that are misleading or inappropriate. The following are the major sources of model risk:
- Model errors
- Data errors
- Implementation errors
- Usage errors
SR 11-07 is aimed at ensuring the reduction in all these above mentioned errors and this paper presents a practitioner’s point of view for banks looking to be compliant. Following are the summarized interpretation of the SR 11-07 guideline across its three pillars:
- Model development, implementation and use function: Best practices and consideration of alternate models should be explored. The model development team should have solid business knowledge, and the choice of model should be based on objective criterion. Banks should maintain model change policy with clear severity assessment of the model changes.
- Model validation function: This deals with the compliance for one-time and ongoing model validation. As per SR 11-07, model validation prior to implementation and the monitoring post implementation are essential to keep the MRM process approved even before the model is put into production. The frequency of ongoing monitoring of a bank’s models is usually dependent on the risk-tiering of the models, and SR 11-07 is okay with it.
- Governance, policies and control function: This calls for a complete analysis of existing model governance policies to make them compliant with the expectations of regulators. It involves identifying the gaps, with an action plan around their remediation in the subsequent release of the model governance policy. In the end, this calls for improving the standard of documentation for each model.
This paper elaborates on the above presented framework and the related SR 11-07 guideline, supported by examples based on our experience of working with many of the top US banks. The structure of the paper is as follows:
- Core tenets of SR 11-07 and their implications for the banks
- EXL’s responsibility allocation plan for effective MRM
- List of key MRM tasks/objectives for the model user, development, implementation, and validation teams
- Design principles for model inventory
- Recommended structure for comprehensive model documentation
- SR 11-07 compliance of pre-existing models
The primary focus of the paper is risk mitigation during development, implementation, use and validation. The duties of governance, policies and controls
function have only been touched upon briefly.
Core tenets of SR 11-07 and their implications
The table on the right summarizes the core tenets of SR 11-07 and their implications for banks.
Responsibility allocation plan for effective MRM
SR 11-07 requires a clear allocation of responsibilities between the three stakeholders in the MRM process. Prior to SR 11-07, banks followed a reactive approach towards model risk management function. With the advent of SR 11-07, banks are pushed to take a proactive stance towards model risk management.
Based on EXL’s extensive experience across different stages of a model lifecycle, we propose a responsibility allocation plan that is in-line with the SR 11-07 guideline. EXL’s responsibility allocation plan lists the key tasks that need to be covered for effective MRM across the model lifecycle. These tasks have been distributed among the following teams: model development team, model implementation, model validation team, governance and controls aligned with the functions mentioned in the regulation. This is followed by a detailed discussion on the role of model development,implementation and use and model validation functions.
Bank risk teams can use this plan to validate their existing MRM checklist and assign responsibilities.
Risk Mitigation During Development:
The best approach to mitigate model risk is to make sure that it has a clear purpose and is developed in-line with its intended use. The modeler has to:
- Use robust model methodologies based on sound design, theory, and logic after trying out alternatives using championchallenger models
- Perform rigorous data quality checks, including benchmarking data used for modeling
- Apply only well-supported conservatism adjustments to model output, and improve existing models wherever possible
- Ensure model robustness and stability through sensitivity analysis, stress testing and back testing on the role of model development, implementation and use and model validation functions.
- Maintain complete documentation across the entire development phase of the model
Risk Mitigation During Implementation:
Error-free implementation of an effectively developed model is of paramount importance to mitigate model risk. The following is a checklist for the implementer to ensure correct implementation:
- IT systems are adequate to maintain data and reporting integrity
- Production code is of good quality and has a proper change control process in place
- Results generated using the production code are in agreement to the model outputs generated by the developer
Risk Mitigation During Model Use:
The inappropriate use of a well-developed and correctly implemented model is also a major source of model risk. Some practical examples of incorrect model usage are as follows:
- The use of a legacy Loss Given Default model even after significant changes in the recovery procedures of a bank
- The use of an application scorecard developed on a particular geography on another geography having considerably different demographics
- The use of a pre-existing model as an input for another model without accuracy testing and calibration
Such instances of model misuse can be prevented by the model user through:
- Questioning assumptions of a model, assessing limitations and ensuring that the model is appropriate for their use
- Choosing the correct set of outputs as per the prevailing scenario from the different options provided by the model development team in their report
- Maintaining proper inventory entry along with model linkages and dependencies, and communicating changes to the internal audit. In the case of changes to the model, it should be logged in as new entry in the inventory
- Scrutinizing vendor models according to the hiring standards provided by the governance function, and preparing a contingency plan in case the model is no longer available
Risk Mitigation During Model Validation:
Validation has always featured as one of the important tenets across every major guideline in the MRM space. Effective validation helps ensure that models have
been properly built and are performing as per expectations. Validation can be broadly divided into two sub-functions:
- First validation, which is required to do a critical and impartial review of the work done by the developer before the model goes into production
On-going monitoring, which needs to be done for all models already in production to ensure good performance in changing conditions, proper use and correct implementation
First Validation should ensure that:
Documentation is complete and contains evidence in favor of all model choices.
Decisions are based on a variety of statistical tests so that shortcoming of individual tests don’t bias choices
Testing covers all aspects of model quality
including accuracy, precision, robustness, stability
- Benchmarking of implementation and development outputs
The following needs to be done for ongoing monitoring:
Run a series of tests designed at the time of testing/first validation to check data and outputs
Look at the change log of implementation code and verify that no unauthorized changes have been made
Recommend a rebuild/calibrate for the model if the model is found to underperform, or in the case of better techniques or data emerging
Post the OCC guideline, the number of models that need to undergo monitoring is expected to steeply increase. It is recommended that partner banks adopt a high degree of automation for on-going monitoring of models to ensure efficiency, accuracy and timeliness. Our proprietary tool SMART® (Statistical Model Assessment Review Tool) can assist with this task, as it helps drives automations in data extraction, reporting and internal review to release bandwidth for a stronger review and analysis process. This proprietary tool by EXL helps the senior management of a bank to understand the portfolios where model risk is building up, identify areas of policy violation, and how all these changes in different analytical models can impact most cases these dashboards are manually prepared. Automating the production and distribution of reports can help banks figure out issues earlier and increase efficiency. By
automating the above process using tools such as SMART ® can potentially save 20-25% of costs in the first year of steady state operations. Higher cost savings can be achieved through efficiencies of scale.
Other key areas of Model
Model Inventory Management
The OCC guidance requires setting up of a firm-wide inventory of models to help assess aggregate model risk. A comprehensive inventory should contain the following information and model user inputs:
In the eventuality of a change request for a model, the audit team should cross reference the model against others in the inventory, and commission changes in all related models so that the aggregate risk introduced by the change is accounted for. An example would be a decision to change the bureau used to obtain credit scores. When this change is flagged by a particular model owner, the Internal Audit Team has to ensure that all other models that have this as an input component are refit.
Comprehensive Model Documentation
In-line with the mandate of SR 11-07, internal audit teams have to ensure sufficiently detailed documentation to allow parties unfamiliar with a model to understand how the model operates, as well as its limitations and key assumptions.
Model documentation is of key importance across model development, validation, and on-going monitoring of the model. One of the most important areas requiring proper model documentation is model validation. Independent reviews require the model documentation to be comprehensive. In the model validation process the model documentation is compared against the existing model documentation standards, and the reviewer determines whether each section of the model documentation is complete, not applicable, or requires additional information. If the model documentation is found to be incomplete, it is sent for revisions citing the sections that need attention.
As the use of models becomes more prevalent in the financial services industry, so does the risk arising out of incorrect model development and model misuse. As highlighted in this paper, we believe financial institutions must approach this by using a risk management framework and not threat this purely as a compliance initiative. This paper has laid down a practical approach to manage model risk based on the SR 11-07 guidance from the OCC and the Federal Reserve. Adopting the aforementioned risk management framework will not only aid banks towards
SR 11-07 compliance but will also provide a safety net against mismanaged model risk leading to reputational and financial losses.
This means that the policy should be guided in equal measure by business and aspects of risk, and should seek to understand and then mitigate the different sources of model risk. This means that every model is developed in-line with its intended use, implemented error free, used in appropriate situation and performing as per expectations. The extent of application is left to the discretion of individual banks based on the extent of their business activities, magnitude of their risk exposures, complexity and materiality of their models.