In the A.I. era, traditional approaches to software procurement are no longer fit for purpose. Typically considered a commercial function—secure the best software for the best price—procurement teams are now making decisions about highly complex technical systems that can shape an organization’s governance posture, regulatory exposure, security and operational resilience.
Enterprises interact with A.I. in many forms, from standalone software to features embedded in existing platforms. Regardless of how A.I. enters the enterprise, its presence immediately raises governance questions. Who owns the data? How was the system trained? Who is liable for errors? Most procurement teams lack the technical expertise to parse this complexity and audit the systems before making purchasing decisions, creating governance risks from the outset.
Exacerbating these complexities is an asymmetry of power between procurement teams and A.I. vendors. A small number of dominant A.I. providers can set the terms of use and implement top-down changes without negotiating with clients. Executives urging organizational A.I. adoption only intensify this pressure, leaving procurement officials to navigate out-of-date processes without the benefit of A.I.-specific training or guidance.
Together, these factors have created a new governance frontier. Procurement has transformed from a commercial function into one of the most consequential—and least mature—components of enterprise A.I. governance.
The emerging risks of A.I. procurement
Unlike traditional software procurement, A.I. procurement introduces governance risks that continue to evolve after a contract is signed. A.I. supply chains span models, infrastructure providers, APIs and application layers, making it difficult to pinpoint who is liable for what if things go wrong. Even if liability is decided at the time of procurement, rolling updates and feature releases can complicate accountability structures and introduce novel risks after contracts are signed.
The pace of technological and regulatory change is also shortening procurement and contract review cycles. The upside is that shorter cycles offer flexibility; the downside is that they introduce ongoing procurement and governance demands that organizations must manage continuously.
These challenges are exacerbated by market concentration. OpenAI, Anthropic and Google collectively account for 88 percent of enterprise LLM usage, leaving buyers exposed to top-down changes in pricing, product features and contractual terms. On top of this, the nature of A.I. drives “lock-in” to individual providers because models improve through interaction with user data and workflows. Switching providers can therefore become operationally disruptive, expensive and technically difficult.
Data management is one of the most underestimated areas of risk in A.I. procurement. Core due diligence questions dealing with data collection, storage and model training are often left unanswered at the point of contract. As a result, organizations may inadvertently expose confidential information, proprietary business data or customer records to external model training processes.
A.I. presents IP risks, too; models trained on data scraped from the web may generate outputs that include copyrighted or unauthorized material, exposing organizations to downstream legal and reputational risks. Other IP considerations, such as ownership of A.I. outputs and metadata, should be proactively addressed during procurement.
Agentic A.I. is the next frontier of procurement risk. Capable of independently traversing multiple platforms and datasets, these systems introduce risks of a fundamentally different order of magnitude. Increasingly, vendors are excluding key A.I.-related harms from liability clauses within agentic A.I. contracts, leaving buyers exposed. As A.I. agents grow in sophistication and popularity, all these concerns—liability, accountability, dependency and data management—will have to be continuously addressed and renegotiated.
Four pillars for responsible A.I. procurement
Because purchasing A.I. introduces unique risks that are not associated with conventional software, organizations need procurement frameworks designed specifically for A.I. Based on our experience consulting enterprises on responsible A.I. adoption, we recommend building such frameworks on four pillars.
First, teams should be upskilled. Procurement teams should be literate in A.I. functionalities, safety and compliance. This doesn’t mean every procurement official should become a technical expert, but they do need enough understanding to evaluate governance implications.
Organizations should support this through certification programs, cross-functional procurement models and closer collaboration between procurement, legal, compliance, cybersecurity and technical teams. Broader workforce training on responsible A.I. use is also important, particularly as employees increasingly adopt A.I. tools independently.
Emerging vendor engagement models, in which procurement personnel collaborate with A.I. vendors’ forward deployment engineers to tailor tools to operational needs, ensure a deeper understanding of technical elements. While this approach can improve outcomes, organizations also need to invest time in internal change management, governance reviews and process design.
Second, tailor procurement processes to the type of A.I. system being acquired. Procurement teams face a range of A.I. products, including A.I.-powered tools, A.I.-enabled features and foundation models. Each category introduces novel governance, compliance and operational risks that require tailored procurement approaches.
For A.I.-powered products, procurement teams should focus on the use case fit, data sovereignty and hosting arrangements. Consider an adverse example: when Workday released its A.I.-powered Applicant Tracking System, it was billed as a ready-to-use tool for H.R. teams. However, the product violated the Age Discrimination in Employment Act by favoring applicants under 40. Organizations that adopted the tool without sufficient procurement scrutiny faced compliance exposure under employment law.
When vendors introduce A.I. features into existing software products, procurement teams should renegotiate contracts that lack A.I.-specific clauses. This is an increasingly common situation that presents complex governance and compliance risks. For example, when GitHub updated its training requirements early in 2026, organizations with lower-tier subscriptions found that their private data was being used to train A.I. models. Situations like this undermine organizations’ privacy, data protection and security controls.
When procuring a foundation model or platform, organizations should focus on technical capability and strategic implications. For example, the U.K.’s NHS recently faced criticism for a procurement process that allowed U.S. firm Palantir to access identifiable patient data while developing a federated data platform. The platform is intended to deploy A.I. across patient records to improve efficiency. However, the procurement approach has undermined public trust in the NHS and its services.
Third, A.I. procurement should be anchored in established governance frameworks, standards and emerging regulatory requirements. Even in jurisdictions where A.I. regulation remains underdeveloped, aligning with the wider regulatory system allows organizations to demonstrate that their A.I. use is safe, responsible and trustworthy. It also creates defensible evidence that governance obligations were considered before deployment.
Standards such as ISO 42001 for A.I. management systems and ISO 23894 for A.I. risk management can help organizations establish documented governance processes and create auditable evidence trails from the outset. Other mechanisms, like IEEE standards and conformity assessment and IAPP AI Governance Vendor Reports, can further support procurement due diligence.
Finally, procurement should be embedded within ongoing A.I. governance processes. When an organization purchases an A.I. system, it is making governance choices about data sovereignty, liability, regulatory compliance and long-term vendor dependency. These choices require continuous review with governance stakeholders as models evolve, regulations change and vendors update products or terms of service.
Organizations should therefore integrate procurement directly into broader A.I. governance structures through recurring audits, compliance reviews, performance assessments and cross-functional oversight processes. Where possible, organizations should pursue shorter contracts and procurement cycles to avoid making long-term commitments in the face of rapidly changing technology.
A.I. governance starts at procurement
For many organizations, procurement remains one of the least mature dimensions of A.I. governance. Yet procurement decisions increasingly determine how data is managed, where accountability sits, which vendors gain influence over operations and how resilient organizations remain as A.I. systems evolve.
As A.I. adoption accelerates, procurement can no longer operate as a purely commercial function. Every A.I. contract now embeds decisions about governance, risk, compliance, security and strategic dependency. It has become a governance mechanism, one that will play a defining role in whether enterprise A.I. systems are deployed responsibly, compliantly and effectively.
Amelia Williams is a Senior Research Impact Officer at Trilateral Research with expertise in scientific communication at the intersection of emerging technologies, environmental issues, ethics, and policy. At Trilateral, she supports the development and implementation of research projects alongside policy, media and industry engagement.
