What is the PRA’s SS1?
The Bank of England has recently released a supervisory statement SS1/23 that will go into effect on May 17th, 2024 that outlines stringent requirements for Model Risk Management (MRM) for banks and financial institutions operating within the UK. The statement describes what to classify as a model, principles for risk classification, having proper governance & controls, model validation, and assigning proper roles and responsibilities for key MRM functions such testing, documentation, and reporting.
Why is SS1/23 Important?
SS1/23 going into effect marks a major milestone in AI Governance. While the SS1/23 statement covers the full range of how to govern all models within a firm and defines what a model is quite broadly, it also explicitly calls out AI as a sub-principle of the regulation on the Bank of England’s website.
The statement also specifically calls out Senior Management Function (SMF) accountability for carrying out the proper AI and Model Risk Management responsibilities. As evidence of further commitment to AI Safety, the European Artificial Intelligence Office has also been established within the European Commission earlier this year. Therefore now, more than ever, is it important to proactively implement proper governance for the proliferating models within your organization.
The Journey to SS1/23 Compliance
CIMCON Software has 25+ years of experience in helping firms with EUC, Model, and AI Risk Management and aims to greatly reduce the friction and headache for firms as they move towards SS1/23 Compliance as well as helping firms reduce errors that could be incredibly costly. Below are the technological solutions CIMCON provides to specific principles outlined in SS1:
- Automated Model Identification: CIMCON is uniquely able to take a model agnostic approach to identifying and risk assessing EUCs such as Excel files within an organization, as well as Models created in Python or R, and even 3rd party executables. This is especially important as these all could be considered models under the SS1/23 definition of a model: “A model is a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into output."
- Self Organizing Model Inventory: Scans can be scheduled to occur regularly in order to uncover hidden areas of risk and automatically keep the Model Inventory up to date. Firms can also easily maintain inventories that are firm wide as well as department specific. This also directly aligns with SS1/23 Principle 1.3 Model Tiering: Firms should implement a consistent, firm-wide model tiering approach that assigns a risk-based materiality and complexity rating to each of their models.
- Powerful, Yet Flexible Risk Assessment: CIMCON’s powerful risk assessment, or model tiering, functionality leverages a comprehensive view of risk, but firms can also create custom Risk Profiles, or algorithms to classify risk from different sources. This can make testing different classes of models differently and generating the corresponding documentation and reports seamless.
- Interdependency Map: A model’s level of risk is highly dependent on the models and data sources that serve as inputs to that model. Models that feed into high impact models can pose a major danger of hidden risk. With an interdependency map, you can easily visualize these relationships as well as adjust risk assessment scores for a model based on its relationships to other models and data. SS1/23 Principle 1.2 Model Inventory: Firms should maintain a firm-wide model inventory which would help to identify all direct and indirect model inter-dependencies in order to get a better understanding of aggregate model risk.
- AI Model Testing & Documentation Generation: CIMCON’s AIValidator tool provides a comprehensive testing suite for models that includes tests such as Data Drift, Validity & Reliability, Fairness, Interpretability, Privacy, the use of GenAI, Security Vulnerability, Code Quality among many others. Risk teams can run these tests through a no-code environment or Model Developers can automatically add in these testing protocols right in their python code through CIMCON’s Python Package. The results of these tests can be seamlessly documented within EUC Insight or exported according to the needs of the firm as well. SS1/23 Principle 1.3 Model tiering: Complexity assessment may also consider risk factors related to measures of a model's interpretability, explainability, transparency, and the potential for designer or data bias to be present.
- Comprehensive Documentation Generation & Management: Qualitative Model Information such as Purpose, Owner, Impact as well as the most recent quantitative risk score and documentation on the results of model testing are all conveniently kept up to date in one place across the firm. SS1/23 Principle 3.5 Model development documentation: Firms should have comprehensive, and up-to-date documentation on the design, theory, and logic underlying the development of their models.
- 3rd Party Risk Management: CIMCON’s product suite also includes comprehensive 3rd party risk management identifying the use of AI within 3rd Party Applications, 3rd Party Model Identification and Risk Assessment, Security Vulnerability analysis for 3rd party libraries, the testing of 3rd party data sources and models, and more. SS1/23 Principle 2.6 Use of externally developed models, third-party vendor products. Firms should:(i) satisfy themselves that the vendor models have been validated to the same standards as their own internal MRM expectations.
- Proper Controls and Accountability: Through the Change Management module, firms can restrict and track who makes changes to what models and when. This adds security and accountability as outlined in SS1/23 Principle 2.3 Policies and procedures: At a minimum, the policies and procedures should cover:[...] processes for restricting, prohibiting, or limiting a model’s use. It also allows firms to measure the frequency and extensiveness of use of models, which is also outlined in SS1/23 Principle 1.3 Model Tiering: The assessment of a model's complexity should consider the risk factors that impact a model’s inherent risk, eg [...] frequency and/or extensiveness of use of the model.
- Approval Workflows: The Change Management module also allows firms to create approval workflows automatically sending alerts and notifications to the proper approval authorities as well as allowing for tracking the model approval status. This helps you identify bottlenecks within the organization and areas of process improvement. SS1/23 Principle 2.3 Policies and procedures: At a minimum, the policies and procedures should cover: the model approval process and model change, including clear roles and responsibilities of dedicated model approval authorities.
Setting Up For Success
A survey from The Economist shows that 77% of bankers believe that AI will be a key differentiator for banks, but at the same time, according to Gartner, 85% of AI Projects will deliver results. Managing a proliferation not just of models, but of the different types of models and which departments may be generating them (GenAI Chatbots, Fraud Detection, Credit Reporting, Report Generation Software, IT Operations, Candidate Prospecting, etc.) will be a critical element of success within banks. We at CIMCON are here to leverage our decades of experience as well as a product suite that has been put through every test by every kind of financial institution to help you manage the risk from these models, whatever form they may be in (EUCs, Models, or AI).
AI Risk Management Framework
Explore the realm of Artificial Intelligence (AI) with our AI Risk Management Policy. This concise guide covers the spectrum of AI models, including supervised, unsupervised, and deep learning, and emphasizes making AI trustworthy based on the NIST AI Risk Management Framework.
Learn to assess and manage AI Risk, cultivate a culture of risk awareness, and utilize periodic testing with tools like ours. This policy is your essential toolkit for responsible and effective AI utilization in your organization.