You are six months into your most visible AI initiative. The demo worked. The steering committee is satisfied. And somewhere in your system, something is very wrong. Nobody flagged it because nobody knew to look for it. This is not a technology failure. It is a talent failure. And it is happening inside organizations that believe they hired well.
McKinsey’s 2025 State of AI report found that 88 percent of organizations are now using AI in at least one business function. The number that rarely gets mentioned is right next to it: only 6 percent of those organizations are actually seeing meaningful returns on that investment. Six percent. That gap is not a strategy problem or a budget problem. In most cases, it is a talent gap that nobody priced in.
The Problem Nobody Wants to Name
When an AI initiative falls short, the diagnosis usually points to data, infrastructure, governance, or executive alignment. Those are real factors. But more often than not, hiring decisions made under pressure, by well-intentioned people working with limited tools, produced outcomes that cost far more than anyone budgeted for.
The wrong AI hire rarely announces itself right away. There is usually an early stretch of real-looking progress where documents get produced, demos get built, and stand-ups sound good. The problems come later, when the architecture cannot scale, when outputs break down at the edges, when a data pipeline quietly corrupts results in ways that take weeks to trace back.
We recently supported a financial services firm that built an internal AI system to surface regulatory guidance for compliance teams across multiple business lines. The project ran for nearly a year before a legal review flagged that the system was surfacing outdated or misapplied guidance in a significant share of its responses. It had been built without proper evaluation frameworks, fine-tuned on data that was never adequately governed, by a team that did not have the depth to recognize what they did not know. By the time the problem was caught, the system had touched hundreds of compliance reviews. Fixing it, not counting the potential regulatory exposure, far exceeded the original project budget.
The reason nobody caught it is structural. A certification earned eighteen months ago may already reflect a curriculum that was outdated when it was written. A portfolio shows what someone chose to publish, not how they hold up under the pressure of production. A title at a recognizable company tells you where someone sat, not what they built or whether it lasted.
So organizations fall back on shortcuts. And shortcuts are failing them.
The Real Numbers Behind the Risk
The financial exposure is larger than most organizations formally track. According to SHRM, replacing an employee costs between 50 and 200 percent of their annual salary depending on seniority and role. For a senior AI architect or machine learning engineer, that replacement cost alone can reach several hundred thousand dollars before you count a single delayed milestone or inherited technical debt.
Gartner now reports that 50 percent of GenAI projects are being abandoned after proof of concept, with poor data quality, weak risk controls, and unclear business value as the primary reasons. The consistent thread across every failed initiative is not the technology falling short. It is the implementation failing — and implementation quality comes down to the people doing the work. Which brings it back to that 6 percent. The organizations capturing real returns are not using better tools. They are deploying people who know how to build AI systems that hold up.
Why Standard Hiring Keeps Missing It
The people asked to evaluate AI candidates are often not in a position to assess AI depth. A VP of Engineering with a strong traditional software background can evaluate code quality, system design, and technical judgment with real confidence. Evaluating whether someone truly understands how a retrieval system will behave under real query volume, or whether their approach to model evaluation will catch the right failure modes before deployment, requires a different kind of fluency that most hiring processes are not built to test.
The result is a structural gap. Organizations know they need AI talent, they move quickly because the pressure from above is real, and a portion of the time they end up with someone who is excellent at talking about the work without being excellent at doing it.
Moving fast with the wrong team is almost always slower in the end. A well-built AI system deployed a few weeks later than planned creates value for years. A poorly built one creates rework, eroded trust, and organizational skepticism that can set a program back by eighteen months or more.
What Rigorous Actually Looks Like
The answer is not to slow down or treat every AI candidate as a suspect. It is to build evaluation processes that test what someone can actually do, not just what they can talk about.
That means moving past the resume review and the standard technical screen into assessments that mirror the real work.
-
- Can this person design a retrieval system for your specific data environment and walk through the tradeoffs clearly?
-
- Can they find the weaknesses in an evaluation framework you hand them?
-
- Can they show you where their knowledge ends, which in this field is one of the most important signals of all?
It also means treating AI talent evaluation as something your organization builds and owns, not something improvised each time a role opens up. The companies doing this well have structured assessment approaches that work consistently across candidates and roles, and the internal knowledge to recognize what strong actually looks like.
The Competitive Advantage Nobody is Talking About
The organizations that pull ahead in AI over the next several years will not win because they had access to better models. They will win because they deployed AI competently, with people who knew what they were doing, building systems that held up. That advantage starts at the point of hire and it compounds over time.
At ConsultNet, we believe the only way to close the gap between AI adoption and AI performance is to fundamentally change how AI talent gets validated before it ever reaches a client engagement. Not better resumes. Not more credentials. Actual proof of what someone can do under conditions that look like real work.
In the meantime, I want to hear from the people doing this work every day. Where are you seeing the evaluation gap show up, and what have you found that actually makes a difference?
