How do we govern an LLM we did not build and cannot inspect?

When direct inspection of an LLM's internal mechanisms is not possible, organisations must pivot their governance strategy towards comprehensive vendor due diligence and stringent contractual frameworks. This includes establishing clear service level agreements (SLAs) that define expected performance, accuracy, bias mitigation, and data privacy commitments from the LLM provider. Furthermore, the organisation must implement robust internal processes for validating the LLM's outputs, monitoring its behaviour in production, and managing any identified risks or incidents in collaboration with the vendor. This approach ensures accountability and control over the AI system's impact, even without direct access to its underlying code or training data.

ISO/IEC 42001 provides a structured approach to govern externally acquired AI systems by integrating them into the organisation's AI Management System (AIMS). Clause 6.1 mandates an AI system risk management process applicable to all AI systems, regardless of origin, requiring identification, assessment, and treatment of risks associated with their use. Clause 6.2, concerning the AI system lifecycle, explicitly includes acquisition, necessitating controls for due diligence and integration. Annex A control A.7.1, specifically on AI system acquisition, guides the establishment of requirements for external providers, ensuring the LLM's fitness for purpose and ongoing oversight of its performance and compliance, thereby extending governance to third-party AI.