Artificial intelligence (AI) is a rapidly evolving technology that has the potential to transform businesses and society. However, AI also poses significant challenges and risks for auditors, who need to provide assurance over its governance, design, implementation and use. How can auditors prepare themselves to audit AI effectively and efficiently?
In this blog, we will summarize the "Auditing Artificial Intelligence" white paper by ISACA that provides some practical guidance and insights for auditing AI, based on the COBIT 2019 framework. The white paper covers the following topics:
The definition and scope of AI, and why auditors should care about it
The potential impact of AI on organizations and their stakeholders
The challenges and solutions for the AI auditor, and the keys to success
The mapping of COBIT 2019 to strategy, and how to apply it in the auditing of AI
The resources and references for further learning and exploration
What is AI and Why Should Auditors Care?
AI is a broad term that encompasses machines carrying out tasks based on algorithms in an “intelligent” manner, such as learning, reasoning, perceiving, and decision making. AI can be classified into different types, such as machine learning, deep learning, natural language processing, computer vision, speech recognition, and robotics.
AI has many benefits and opportunities for organizations, such as improving efficiency, productivity, innovation, customer satisfaction, and competitive advantage. However, AI also has many challenges and risks, such as ethical, legal, social, security, quality, and performance issues. AI may also have unintended or adverse consequences, such as bias, discrimination, errors, fraud, or harm.
Therefore, auditors need to care about AI and understand its implications for their profession and their clients. Auditors need to provide assurance that AI is aligned with the business strategy, objectives, and values, and that it is governed, managed, and controlled effectively and appropriately. Auditors also need to assess the risks and impacts of AI on the organization and its stakeholders, and ensure that they are mitigated and monitored adequately.
What is the Impact of AI on Organizations?
AI has a significant impact on many areas in the business world, such as operations, products, services, processes, functions, and roles. AI can enhance, augment, or replace human capabilities and tasks, depending on the level of automation, autonomy, and intelligence. AI can also create new value propositions, business models, and markets, or disrupt existing ones.
The impact of AI on organizations depends on several factors, such as the type, scope, scale, and maturity of AI applications, the industry, sector, and domain of the organization, the culture, readiness, and adoption of the organization, and the external environment, regulations, and standards.
The impact of AI on organizations can be positive or negative, intended or unintended, direct or indirect, short-term or long-term, and certain or uncertain. Therefore, organizations need to evaluate the potential benefits and costs of AI, and balance the trade-offs and dilemmas involved. Organizations also need to monitor and measure the actual outcomes and performance of AI, and adjust and improve accordingly.
What are the Challenges and Solutions for the AI Auditor?
Auditing AI is not an easy task, as it involves many complexities, uncertainties, and unknowns. Some of the challenges for the AI auditor include:
Defining and scoping AI, and understanding its design and architecture
Identifying and assessing the relevant stakeholders, risks, and controls of AI
Obtaining and analyzing the appropriate evidence and data of AI
Explaining and communicating the results and recommendations of the audit
Keeping up with the fast pace and dynamic nature of AI
However, there are also some solutions that can help the AI auditor overcome these challenges, such as:
Becoming informed and educated about AI, and its implications for auditing
Involving and collaborating with all the relevant parties, such as AI experts, developers, users, and regulators
Adopting and adapting existing frameworks and standards, such as COBIT 2019, to guide and structure the audit
Focusing on transparency and accountability of AI, and ensuring its traceability and explainability
Adopting and leveraging AI itself as a tool to support and enhance the audit
How to Apply COBIT 2019 in the Auditing of AI?
COBIT 2019 is a comprehensive and flexible framework for the governance and management of information and technology (I&T) in organizations. COBIT 2019 provides the auditor with tools, such as process descriptions, desired outcomes, base practices, and work products, to enable the auditor to provide assurance over the AI initiative for any organization.
The white paper provides a visual representation of how to map COBIT 2019 to strategy, and how to apply it in the auditing of AI. The mapping consists of four steps:
Step 1: Define the AI strategy and objectives, and align them with the organizational strategy and objectives
Step 2: Identify the AI governance and management objectives, and select the relevant COBIT 2019 processes
Step 3: Assess the AI risks and controls, and determine the audit scope and objectives
Step 4: Perform the audit procedures and tests, and report the audit findings and recommendations
The white paper also provides some examples of how to use COBIT 2019 processes and practices to address specific AI risks and controls, such as:
Ensuring the alignment of AI plans and business needs
Defining the target digital capabilities and conducting a gap analysis
Assessing the potential of emerging technologies and innovative ideas
Ensuring traceability and accountability for information events
Managing performance and conformance monitoring
Managing system of internal control
Managing compliance with external requirements
What are the Resources and References for Auditing AI?
The white paper concludes with a list of resources and references for further learning and exploration of auditing AI, such as:
The Association for the Advancement of Artificial Intelligence, Digital Library, Conference Proceedings
ISACA, COBIT 2019 Framework: Introduction and Methodology
ISACA, COBIT 2019 Framework: Governance and Management Objectives
ISACA, The Institute of Internal Auditors, Artificial Intelligence: The Future for Internal Auditing
ISACA, The Institute of Internal Auditors, Global Perspectives and Insights Series, Artificial Intelligence—Considerations for the Profession of Internal Auditing
ISACA, The Institute of Internal Auditors, Global Perspectives and Insights Series, The IIA’s Artificial Intelligence Auditing Framework—Practical Applications, Part A
ISACA, The Institute of Internal Auditors, Global Perspectives and Insights Series, The IIA’s Artificial Intelligence Auditing Framework—Practical Applications, Part B
ISACA, The Institute of Internal Auditors, Internal Audit Foundation, Artificial Intelligence: The Data Below
ISACA, The Institute of Internal Auditors, Internal Audit Foundation, Request for Proposals, Artificial Intelligence Research Project
International Standards Organization (ISO), ISO/IEC 27000:2018(en), Information technology—Security techniques—Information security management systems—Overview and vocabulary
Tegmark, M., Life 3.0: Being Human in the Age of Artificial Intelligence
We hope this blog has given you a brief overview of the white paper and some useful tips and insights for auditing AI. If you are interested in reading the full white paper, you can download it from the link below. Happy auditing!