Build vs. buy: assembling an AI evidence binder an examiner can read
Author
Audrey
NobleCloak's AI Correspondent · AI-drafted, fact-checked against our sourced evidence before publishing.
Date Published
You can build an AI evidence binder yourself. Nothing in it requires a product. It needs a list of the AI your institution uses, a dated record of what you checked about each vendor, your policies, and a trail that shows the work continues. A careful compliance officer can produce all of it.
The real question is hours. If you run compliance with one or two people, is this the best use of a week? Below: what goes in the binder, what it costs, what can honestly be automated, and when to build it yourself.
What an examiner may ask for
On November 4 the OCC published a framework examiners may use to review how banks use AI. The OCC is clear that it "does not establish supervisory expectations for banks." It still shows the questions an examiner is now equipped to ask. Section 2.14, on documentation, lists what a record may cover: "purpose, ownership, design, data, providers, versions, prompts, permissions, assumptions, testing, validation, monitoring, decisions, approvals, incidents, and changes."
Its first example is decision traceability: "Inputs, outputs, context, retrieved sources, tool calls, approvals, overrides, and actions may provide evidence needed to reconstruct a material decision or event."
It also says documentation "may be proportionate to the use case." A small institution with a few AI tools needs a small binder. It still needs one. Our summary of the OCC framework covers the rest.
What goes in the binder
A tabbed folder, a shared drive or a PDF all work. Every entry should be dated and say where it came from.
- A one-page summary. How many AI tools you found, how many nobody approved, which touch sensitive data, your top risks, and what you did about each. Name the owner and the review date. An examiner reads this first, so write it last.
- The AI inventory. One row per tool: vendor, users, data it touches, how it got in (approved, a vendor switched on a feature, or nobody approved it), and a risk flag. Put the date it was run and the date it's next due at the top.
- Connected-app findings. The OAuth grants your identity provider shows: which apps, which scopes, how many users, and what you did about each. A finding with a dated resolution beside it shows the program works. The same grant found by the examiner first is a finding against you. Our post on shadow AI as an access problem walks through reading them.
- Vendor due-diligence records. One dated record per vendor that matters: questionnaire answers, the attestation you reviewed or a note that none exists, the sub-processor list, and a risk rating. Keep the refusals. "Vendor declined to confirm training-data use in writing" is evidence that you asked.
- Policies and procedures. Your AI-use policy and your service-provider oversight policy, each with an approval date. For RIAs and broker-dealers, Reg S-P requires written policies and procedures for "oversight, including through due diligence and monitoring, of service providers." See the Reg S-P AI vendor file.
- Decision records for agent actions. If any AI agent can act in your systems, this is the tab section 2.14 points at. For each action that needed a person: who asked, which agent, what it tried to do, who approved or denied it, and when. Our treasury example shows what one looks like.
- The monitoring trail. Short dated entries showing the inventory was re-run, grants were reviewed again, and renewals triggered fresh diligence. A few lines a quarter is enough. Without it, ongoing oversight looks like oversight that stopped after onboarding.
- Coverage limits. A short, honest statement of what the program doesn't see. For example: it covers AI reachable through your identity provider and your vendor list, and it misses personal devices and AI a vendor runs on its own servers.
An optional extra: a short crosswalk that maps your tabs to a recognized framework. The NIST AI RMF has four functions: "govern, map, measure, and manage." Policies map to govern, the inventory to map, risk ratings to measure, and remediation and monitoring to manage. See mapping AI findings to your exam vocabulary.
What it costs to build
Here's our rough estimate for a small program doing this the first time:
- Inventory: 8 to 16 hours. Reading OAuth grants in Google Workspace or Entra, then reconciling them with what departments say they use. Everyone underestimates this step. In Ncontracts' 2026 third-party risk survey of 173 financial services professionals, "72% admit they are only partially aware of which vendors use AI."
- Vendor diligence: 2 to 5 hours per material vendor. The answers are spread across a DPA, a trust page and a sub-processor list, and they don't always agree. A small stack still has 8 to 15 vendors that matter.
- Crosswalk to your exam vocabulary: 6 to 12 hours the first time.
- Assembly and the summary: 4 to 8 hours.
That adds up to roughly 40 to 80 hours. For most small programs, that's one or two working weeks from a person who also owns BSA, complaints and audit response. The same survey found that "Nearly two-thirds (63%) of TPRM programs operate with just one or two dedicated full-time employees."
The binder goes stale
The 40 to 80 hours is the first number. Vendors add AI features on their own schedule, sub-processors change, and people grant new apps every week. Each change makes a row wrong until someone checks it again. The re-run work never reaches zero, and few teams budget for it.
What can honestly be automated
Be skeptical of a binder that builds itself. Most of the binder is judgment, and judgment doesn't automate.
What a tool can capture for you:
- The connected-app list. A read-only scan of your identity provider can list the AI apps people authorized, the scopes each holds, and who uses them. It can't see past what SSO and OAuth expose.
- Agent decision records. If agents route their actions through an approval step, the record of who asked, who approved and when can be written at the moment it happens. Reconstructing it later from logs is slower and less complete.
- Dates. Anything generated by a system carries its own timestamp. That's most of what makes a monitoring trail credible.
What stays with you:
- Whether a finding is acceptable for your institution.
- Whether to revoke a grant or live with it.
- How to weigh a vendor that won't answer.
- Writing and approving the policies.
- The crosswalk, the summary and what you tell your board.
- The coverage-limits statement.
The useful principle: capture evidence when the work happens. Scraping it together the week before an exam is where the 40 to 80 hours go.
When building it yourself is the right call
- You have the hours. A quiet quarter or an analyst with room on their plate. The work builds knowledge that stays with your institution.
- Your AI footprint is small and stable. A few vendors you chose on purpose, no agents acting in your systems, little unapproved use. The binder won't go stale fast.
- You want to learn it. Building it once is the best way to understand your AI vendor risk. Our guide to AI vendor risk for a 1–2 person compliance program has the question set we'd use.
If two of those fit, build it.
When to get help
- Your hours are the scarce resource. If the binder means BSA work slips, the hours aren't free.
- Your footprint is large or moving. Many vendors, unapproved apps, or agents that act in customer or financial systems. The binder goes stale faster than one person can refresh it.
- An exam is already scheduled. A self-build runs at the speed of your calendar.
Where NobleCloak fits
NobleCloak doesn't produce an evidence binder or a framework crosswalk. You assemble those. It can supply two tabs.
Connected apps (tab 3). Our read-only scan reads Google Workspace and Entra and shows every AI app your people authorized, the data scopes it holds, who's using it, and since when. Coverage is limited to what SSO and OAuth can see. Our team runs the first report with you.
Agent decision records (tab 6). Policy marks each tool an agent can use as allowed, needs approval, or denied. Approval goes to a named person or role, and one approval covers one action. The Record of Decisions keeps who asked, which agent, which tool, the decision, who approved, and when. Read how human-in-the-loop works.
It doesn't do vendor diligence, write your policies, validate models, or capture model inputs and outputs. Mapping scanned apps to tools you can govern, and drafting policies from the scan, are coming soon.

