We Gave AI 90 Minutes to Build an IG Roadmap. It Needed 2. Here's What It Got Wrong.
Updated: Oct 21
Every week, someone asks me:
"Can't AI just write our information governance program?"
My answer up front:
AI is an enhancer to the way people work. The people who use it best will always see it as an enhancement first, not a replacement. A human still has to own the judgment.
So we tested it. We gave a general AI assistant 90 minutes to build a roadmap for a realistic mid-market organization. It drafted the roadmap and three follow-ups in about 2 minutes. The whole test, grading included, took about 7.5. Then we graded it.
The Method
The scenario. The company is fictional. Call it Coastline Benefits & Care Partners, a 350-person benefits administrator and care coordinator in California. It's a HIPAA business associate running Microsoft 365, Salesforce, and 9 TB of old file shares. There's no records manager.
Two things forced the roadmap:
A former client sued, finding the evidence took three weeks, and two departed employees' mailboxes were already gone.
In a Copilot pilot, Copilot showed a salary spreadsheet and member health data to the wrong person.
The tool. A general AI assistant, no web access, no uploaded files.
The prompt. A busy COO's request: current state, target state, phases, owners, and a timeline, "Keep it practical for a mid-market budget." Follow-ups asked for a budget, actual retention periods, and a 12-month version.
The yardstick. We graded against ARMA International's Generally Accepted Recordkeeping Principles. ARMA updated them in October 2025, and there are now seven. We scored against the earlier eight, because ARMA's maturity model still maps to them. That model runs from Substandard to Transformational.
One caveat, and a check. The same assistant that wrote the roadmap also graded it. So I re-ran the test myself, using non-AI tools for the inputs. The errors below held up.
What AI Got Right
The structure was a useful start. Phases, owners, timeline, KPIs. One target stood out: "Time to place a legal hold: from 3 weeks to under 48 hours." But it isn't board-ready. It needs human value added before a board sees it.
It found the root cause. "Copilot respects existing permissions. The problem is that your existing permissions are too broad." Correct.
It put first things first. Phase 0 stopped the bleeding:
Suspend mailbox deletion for departing employees.
Delay the 150-user rollout.
Complete the HIPAA breach risk assessment.
Most citations checked out. We checked 38 laws, rules, and product claims. 29 were correct, 6 partly correct, 3 wrong, and 0 invented.
What AI Got Wrong
Its wrong numbers looked exactly like its right ones. It said California payroll records must be kept "4 years" under Labor Code 1174 and 226. Both say three. It said records for minors must be kept "until age 25" under Health and Safety Code 123145. That law says one year past 18, never less than seven, and it covers licensed providers, not a benefits administrator.
This kind of error rarely happens when people build retention schedules. IG professionals go to extremes not to miss a regulated requirement. Tailoring a schedule to your needs is expected. Changing the cited requirement is not. That's what the AI did.
It recommended a feature that's being retired. It said to "apply Restricted SharePoint Search to top-risk sites." Microsoft blocked new enablement on July 31, 2026. It also quoted an outdated HIPAA penalty cap.
It assigned work to people who don't exist. It made an "HR Director" accountable for training and handed audit to "Internal Audit." It gave the retention schedule to a two-person legal team mid-lawsuit. An advisor would shore this up on the front end, so the plan fits your real team instead of reading like generic, untested output.
It ignored the budget. We asked for modest, with no new headcount. It priced $361,600 to $755,600 over 18 months. Few leaders would call that modest without more context. Asked for 12 months, it kept the full scope and launched Copilot in Month 5, before retention and data loss prevention were in place.
Sequencing tool launches differently is common and varies by organization. But raising risk while pushing risk-mitigation tools later is a choice your executives need to fully understand and weigh in on.
Where the Gap Really Is
AI is fast at the parts of a roadmap that look like documents. It's weak where judgment about your organization, risk, and people is needed.
Some calls no tool could make. Does a provider records law apply to a benefits administrator? Who owns the program when legal is two people in a lawsuit? Each needed judgment, not more text.
The real cost:
Time saved drafting moves straight into verification. Checking 38 citations takes an expert a half day or more.
The Five Tips the AI Gave Us
Here are the five tips the AI itself produced in our test. Even the tool says to check its work.
Let it draft the skeleton. Structure, phase names, KPIs, and a first RACI.
Never take a retention period on faith. Check every number against the statute itself. Then ask whether it applies to an organization like yours.
Give it your real org chart. Say who exists, who's part-time, and who's busy.
Give it a number, not an adjective. "Modest" means nothing to a chatbot. Say "$150,000 and no new hires," then ask what to cut.
Ask what's changed lately. Check vendor docs and regulations yourself.
Why Judgment Still Matters
AI can't sit in the room with your legal, IT, and operations leads, or tell you which fight to pick first. That's why getting AI to deliver value means getting your IG fundamentals right first. Wondering whether outside help fits your size? Here's why the fractional model works.
AI gave us a roadmap that looked done in minutes. Getting it right still took a human with a stack of statutes and knowledge of the people involved. Use AI to enhance that work, not to replace it.
This post is general information, not legal advice. Talk to your counsel about your specific situation.
Comments