Artificial intelligence (AI) is a rapidly evolving technology that has the potential to transform businesses and society. However, AI also poses significant challenges and risks for auditors, who need to provide assurance over its governance, design, implementation and use. How can auditors prepare themselves to audit AI effectively and efficiently?
In this blog, we will summarize the "Auditing Artificial Intelligence" white paper by ISACA that provides some practical guidance and insights for auditing AI, based on the COBIT 2019 framework. The white paper covers the following topics:
-
The definition and scope of AI, and why auditors should care about it
-
The potential impact of AI on organizations and their stakeholders
-
The challenges and solutions for the AI auditor, and the keys to success
-
The mapping of COBIT 2019 to strategy, and how to apply it in the auditing of AI
-
The resources and references for further learning and exploration
What is AI and Why Should Auditors Care?
AI is a broad term that encompasses machines carrying out tasks based on algorithms in an “intelligent” manner, such as learning, reasoning, perceiving, and decision making. AI can be classified into different types, such as machine learning, deep learning, natural language processing, computer vision, speech recognition, and robotics.
AI has many benefits and opportunities for organizations, such as improving efficiency, productivity, innovation, customer satisfaction, and competitive advantage. However, AI also has many challenges and risks, such as ethical, legal, social, security, quality, and performance issues. AI may also have unintended or adverse consequences, such as bias, discrimination, errors, fraud, or harm.
Therefore, auditors need to care about AI and understand its implications for their profession and their clients. Auditors need to provide assurance that AI is aligned with the business strategy, objectives, and values, and that it is governed, managed, and controlled effectively and appropriately. Auditors also need to assess the risks and impacts of AI on the organization and its stakeholders, and ensure that they are mitigated and monitored adequately.
What is the Impact of AI on Organizations?
AI has a significant impact on many areas in the business world, such as operations, products, services, processes, functions, and roles. AI can enhance, augment, or replace human capabilities and tasks, depending on the level of automation, autonomy, and intelligence. AI can also create new value propositions, business models, and markets, or disrupt existing ones.
The impact of AI on organizations depends on several factors, such as the type, scope, scale, and maturity of AI applications, the industry, sector, and domain of the organization, the culture, readiness, and adoption of the organization, and the external environment, regulations, and standards.
The impact of AI on organizations can be positive or negative, intended or unintended, direct or indirect, short-term or long-term, and certain or uncertain. Therefore, organizations need to evaluate the potential benefits and costs of AI, and balance the trade-offs and dilemmas involved. Organizations also need to monitor and measure the actual outcomes and performance of AI, and adjust and improve accordingly.
What are the Challenges and Solutions for the AI Auditor?
Auditing AI is not an easy task, as it involves many complexities, uncertainties, and unknowns. Some of the challenges for the AI auditor include:
-
Defining and scoping AI, and understanding its design and architecture
-
Identifying and assessing the relevant stakeholders, risks, and controls of AI
-
Obtaining and analyzing the appropriate evidence and data of AI
-
Explaining and communicating the results and recommendations of the audit
-
Keeping up with the fast pace and dynamic nature of AI
However, there are also some solutions that can help the AI auditor overcome these challenges, such as:
-
Becoming informed and educated about AI, and its implications for auditing
-
Involving and collaborating with all the relevant parties, such as AI experts, developers, users, and regulators
-
Adopting and adapting existing frameworks and standards, such as COBIT 2019, to guide and structure the audit
-
Focusing on transparency and accountability of AI, and ensuring its traceability and explainability
-
Adopting and leveraging AI itself as a tool to support and enhance the audit
How to Apply COBIT 2019 in the Auditing of AI?
COBIT 2019 is a comprehensive and flexible framework for the governance and management of information and technology (I&T) in organizations. COBIT 2019 provides the auditor with tools, such as process descriptions, desired outcomes, base practices, and work products, to enable the auditor to provide assurance over the AI initiative for any organization.
The white paper provides a visual representation of how to map COBIT 2019 to strategy, and how to apply it in the auditing of AI. The mapping consists of four steps:
-
Step 1: Define the AI strategy and objectives, and align them with the organizational strategy and objectives
-
Step 2: Identify the AI governance and management objectives, and select the relevant COBIT 2019 processes
-
Step 3: Assess the AI risks and controls, and determine the audit scope and objectives
-
Step 4: Perform the audit procedures and tests, and report the audit findings and recommendations
The white paper also provides some examples of how to use COBIT 2019 processes and practices to address specific AI risks and controls, such as:
-
Ensuring the alignment of AI plans and business needs
-
Defining the target digital capabilities and conducting a gap analysis
-
Assessing the potential of emerging technologies and innovative ideas
-
Ensuring traceability and accountability for information events
-
Managing performance and conformance monitoring
-
Managing system of internal control
-
Managing compliance with external requirements
-
Managing assurance
What are the Resources and References for Auditing AI?
The white paper concludes with a list of resources and references for further learning and exploration of auditing AI, such as:
-
The Association for the Advancement of Artificial Intelligence, Digital Library, Conference Proceedings
-
ISACA, COBIT 2019 Framework: Introduction and Methodology
-
ISACA, COBIT 2019 Framework: Governance and Management Objectives
-
ISACA, The Institute of Internal Auditors, Artificial Intelligence: The Future for Internal Auditing
-
ISACA, The Institute of Internal Auditors, Global Perspectives and Insights Series, Artificial Intelligence—Considerations for the Profession of Internal Auditing
-
ISACA, The Institute of Internal Auditors, Global Perspectives and Insights Series, The IIA’s Artificial Intelligence Auditing Framework—Practical Applications, Part A
-
ISACA, The Institute of Internal Auditors, Global Perspectives and Insights Series, The IIA’s Artificial Intelligence Auditing Framework—Practical Applications, Part B
-
ISACA, The Institute of Internal Auditors, Internal Audit Foundation, Artificial Intelligence: The Data Below
-
ISACA, The Institute of Internal Auditors, Internal Audit Foundation, Request for Proposals, Artificial Intelligence Research Project
-
International Standards Organization (ISO), ISO/IEC 27000:2018(en), Information technology—Security techniques—Information security management systems—Overview and vocabulary
-
Tegmark, M., Life 3.0: Being Human in the Age of Artificial Intelligence
We hope this blog has given you a brief overview of the white paper and some useful tips and insights for auditing AI. If you are interested in reading the full white paper, you can download it from the link below. Happy auditing!