A procurement team books eleven demos in three weeks, and by the ninth one every pitch sounds the same. Every vendor says adaptive. Every vendor says personalized. Every deck carries the same upward curve, and the screenshots blur together until nobody can remember which product did the thing somebody liked. The hard part was never finding a platform that claims to use artificial intelligence. The hard part is working out which of those claims changes what a learner actually experiences on a Tuesday afternoon.
That shift matters because AI has become a default feature line rather than a differentiator, so the burden of proof has quietly moved onto the buyer. You have to know which questions separate a recommendation engine that genuinely reads skill gaps from one that reshuffles the catalog and calls it intelligence. Vendors are rarely dishonest about this. They are usually answering the question you asked instead of the one you meant.
The checks below come from where these rollouts tend to break: personalization that turns out to be filtering, analytics nobody can act on, integrations that become somebody's manual Monday, accessibility treated as a later phase, and data handling nobody raised until legal did. None of it requires a technical background. It requires asking for a demonstration rather than a description.
Personalization That Reflects Real Skill Gaps
Ask the vendor to run the same course through two different learner profiles while you watch. A genuine adaptive engine branches: a learner who passes the diagnostic skips ahead, and a learner who stumbles on one concept gets reinforcement on that concept rather than the whole module again. A filtering engine shows both people the same linear sequence with a different thumbnail on the homepage.
The inputs matter as much as the branching. Recommendations built on role, assessment results, and observed behavior tend to land, while recommendations built on what other people in your department happened to click drift toward whatever is popular. A capable AI learning platform should be able to tell you, in plain terms, which signals it reads and which it ignores.
Test the unglamorous case too. What happens to someone who joins mid-quarter with no assessment history, or someone whose role changed last week? Cold-start behavior is where personalization claims usually fall apart, and it is the first thing a new hire will hit.
Transparency in How the AI Makes Its Recommendations
A learner who cannot see why a course was suggested will assume it was arbitrary, and an administrator who cannot override a suggestion will stop trusting the system within a month. Look for a visible rationale, even a short one, plus the ability for a manager to pin or suppress specific content.
Transparency also covers generated material. Many platforms now draft quizzes, summaries, and translations automatically, which is useful right up until a confidently wrong answer key lands in a compliance course. Ask who reviews that output, whether review is enforced before publication, and how corrections propagate to learners who already finished. The NIST AI Risk Management Framework is a reasonable yardstick here, since it treats explainability, validity, and documented accountability as properties you can test rather than adjectives a vendor supplies.
Analytics That Point to a Decision
Completion rates are table stakes and they tell you almost nothing. The useful questions are sharper. Which skills are weakest across a team, how long competency now takes compared with last quarter, and which modules people abandon halfway and at which slide.
So look at what the reporting lets you do next. Can you compare cohorts without rebuilding the data in a spreadsheet? Can you export raw event data, or reach it through an API, so your analytics team can join it to performance figures you already hold? A dashboard that only exists inside the vendor's interface will eventually be screenshotted into a slide and forgotten.
Integration, Accessibility, and the Day-Two Experience
Check single sign-on, automated user provisioning, HRIS sync, and whether course assignment can be triggered by a change in someone's job record. Every gap there becomes recurring manual work, and manual work is what kills adoption quietly. Learners increasingly expect software to behave like the AI-assisted apps already woven into their daily routines, so a platform that lives in its own lonely tab competes badly against one that surfaces inside the chat tools people already keep open.
Accessibility deserves its own line in the evaluation, not a paragraph in the appendix. Request a current conformance report against WCAG 2.2, then test the AI features specifically, because recommendation widgets, chat assistants, and auto-generated captions are usually the newest code and the least audited. Keyboard navigation and screen reader behavior there tell you how carefully the vendor builds.
Data Privacy and the Vendor Behind the Model
Learning data is unusually sensitive because it records what people got wrong. Establish where it lives, how long it is kept, who inside the vendor can see it, and whether your learners' activity feeds model training that benefits other customers. Get those answers in the contract, not the sales call.
Then look past the product at the company. Ask about subprocessors, breach notification timelines, security certifications with current dates, and what happens to your content and your records if you walk away in two years. A clean exit path is a good proxy for confidence.
Making the Decision With Evidence
Run a short pilot with one real team and one real objective, and resist the urge to pilot everything at once. Pick a group whose skill gap you can already describe, set a measure you would defend to a finance director, and give it six to eight weeks before anyone writes a verdict.
The point is not to find the smartest model. It is to find the platform your people keep opening after the novelty wears off, which is a far less exciting question and the only one that predicts whether any of this pays for itself.
