The announcement goes out on a Thursday. The company is launching an AI Center of Excellence. There's a budget — somewhere between $1.5M and $2M for year one. There are hires — five, maybe eight people, a mix of data scientists and AI strategists and a program manager who came from a Big Four firm. The executive team is aligned. The board is excited. The slide deck is excellent.
Eighteen months later, the CoE has produced three white papers, a vendor matrix that's already out of date, and a governance framework that lives in a SharePoint folder nobody can find. The business units are still making their own AI purchases. The pilots are still failing at the same rate. The CoE team, hired to accelerate AI adoption, has become the primary obstacle to it — a new approval layer in a process that already had too many.
The problem wasn't the people. It was the model. And most companies are still copying it.
What an AI Center of Excellence Actually Is (and Isn't)
An AI center of excellence is not a team. This is the foundational mistake. When you organize it as a team — with headcount, a budget line, a reporting structure, and a mandate to “drive AI strategy” — you have created a department that now has organizational incentives to justify its own existence. Departments produce deliverables. They write frameworks. They conduct reviews. They hold meetings. None of that is the job.
The job is a function, not a headcount. A CoE exists to do four things: set standards so every business unit isn't reinventing evaluation criteria from scratch, assess tools so you're not buying the same category of software three times over, share learnings across the organization so a pilot that fails in finance doesn't fail again in operations six months later, and make AI deployment repeatable so the tenth initiative doesn't take as long as the first.
That function requires coordination and authority. It does not require a dedicated team. The distinction matters because it changes everything about how you staff it, what you expect it to produce, and how you measure whether it's working. A team that isn't delivering headcount-justifying output gets reorganized. A function that's producing speed and standards gets embedded into how the company operates.
The goal is speed. Not process. Not governance for its own sake. If your AI governance structure is slowing down deployment, something is wrong.
The Lean Model: 4 Roles, Not 4 Departments
The lean CoE model runs on four roles, each filled by someone who already works at your company, each requiring roughly 15–20% of their time. No new hires. No new org chart boxes.
The AI Sponsor
Sits at C-suite or VP level and owns two things: budget authority and organizational air cover. When a business unit resists a new standard, the Sponsor resolves it. When IT and operations disagree on a vendor security requirement, the Sponsor decides. This role is not optional. A CoE without executive sponsorship is a suggestion box. Companies that try to run this as a committee of peers — no one with clear authority above the others — find that every contested decision either stalls or escalates to someone who wasn't in the room.
The AI Program Lead
Owns the roadmap and runs the operation. This is typically an ops director, a chief of staff, or a senior strategy manager — someone who already knows how to coordinate across business units and track initiative ROI. They don't need to be technical. They need to be organized, credible with the business units, and comfortable telling stakeholders when a use case isn't ready to pilot. Most companies already have this person. They're currently managing something else.
The Technical Steward
Comes from IT or the data team and owns the technical side of vendor evaluation: security review, integration standards, data handling requirements, and API documentation review. This role does not need to be an AI engineer. They need to know what questions to ask, what red flags to escalate, and how to assess whether a vendor's architecture is compatible with your existing stack. A solid IT architect with vendor management experience can do this job entirely.
Business Unit Liaisons
One per major business unit — sales, operations, finance, whatever divisions are most likely to deploy AI tools. They surface use cases from their teams, own adoption in their area, and report back on what's working and what isn't. This is the most important role for catching failure early. Rotate them every 12–18 months so institutional knowledge stays fresh and you're not building a liaison layer that starts to look like a permanent department.
Free Resource
Benchmark Your Organization for Free
Before any AI initiative, you need an honest read on where you stand. The Fulcrum AI Readiness Scorecard — 25 questions, 5 minutes — tells you exactly what's ready and what will block you.
Get the Free Scorecard →The Three Things a CoE Must Produce
Structure without output is just a committee. The lean CoE model is defensible only if it produces three specific artifacts, consistently.
A use case intake process
A standardized way for anyone in the organization to submit, evaluate, and prioritize an AI initiative. Without this, what you have is informal requests, executive-sponsored pet projects, and business units quietly buying tools that weren't on anyone's roadmap. The intake process doesn't have to be complex. A structured form, a scoring rubric, and a monthly review cycle is enough to stop the random tool purchases that create the fragmented AI landscape most mid-market companies are now trying to untangle.
A vendor evaluation framework
A consistent set of criteria for assessing any AI tool, regardless of category. Security posture, integration requirements, pricing model, data retention policies, customer reference quality. Every vendor gets evaluated against the same criteria. This builds institutional memory. The team that evaluated a contract intelligence tool eighteen months ago learned something. That knowledge should be available to the team evaluating a proposal automation tool today. Without a documented framework, it isn't.
A lessons learned register
A living document that captures what worked, what failed, and why, for every AI initiative the organization runs. This is the single most valuable artifact a CoE can produce, and the one most companies skip. Pilots fail for predictable reasons: data wasn't ready, change management was underfunded, the vendor's onboarding was weak, the use case was poorly scoped. If those lessons aren't documented and shared, you will run the same failure pattern again. The lessons learned register is what turns a failed pilot into organizational learning instead of organizational amnesia.
When to Formalize (and When Not To)
A formal AI center of excellence is overhead if your organization isn't generating enough AI activity to justify the coordination cost. If you're under 50 employees, or running fewer than three active AI initiatives, you don't need a CoE. You need a working group — two or three people meeting monthly to compare notes, share vendor findings, and flag use cases that look promising. That's it.
Formalize when the coordination problem is real: when you're managing five or more concurrent AI tools, when your business units are making independent purchasing decisions without informing each other, or when AI governance failures — data handled incorrectly, a vendor who wasn't properly vetted, a rollout that caused compliance exposure — have started to create actual business risk. At that point, the informal working group isn't enough. You need the structure, the defined roles, and the artifacts.
The mistake most companies make is formalizing too early — creating the CoE as an ambition rather than a response to a real coordination problem. The ambition-driven CoE produces white papers. The problem-driven CoE produces speed.
Related Reading
The AI Strategy Roadmap
A phased roadmap for building an AI strategy that scales beyond the pilot.
The Organizational Readiness Checklist
A detailed checklist for assessing whether your organization is structurally ready for AI.
The Change Management Playbook for AI Rollouts
How to manage the human side of AI deployment — the part that determines whether it sticks.
Next Step
Get the governance architecture right before you start
If you've crossed the threshold where a formal structure makes sense, the first question isn't who to hire. It's how to design the governance model and the stakeholder structure before you start spending. Fulcrum AI's Implementation Advisory works with companies at exactly this stage. Getting the structure right in week one is cheaper than restructuring in month six.
Start with the AssessmentFulcrum AI is a strategic AI consultancy working with COOs, CMOs, and Heads of Ops at mid-market companies. We help operators cut through the noise and build AI strategies that actually work.