How to use AI in financing review without losing control

Choose a job you can check

A useful first task is comparing a term sheet with a draft facility agreement. The output can identify differences, quote the relevant passages and send them to counsel. A broad instruction to assess the whole deal is harder to evaluate because a fluent answer can hide omissions.

Write down what the task includes and what it must leave alone. For a term comparison, that might mean identifying amount, currency, maturity, fees and security changes. Interpreting enforceability or approving a waiver should remain with the authorised people.

The NIST AI Risk Management Framework is voluntary guidance for incorporating trustworthiness into the design, use and evaluation of AI systems. Its generative AI profile addresses risks specific to that technology. Neither document is a certification of a particular product.

Reference: NIST: AI Risk Management Framework and Generative AI Profile ↗

Check that the citation supports the answer

A link to a document is not enough. The reviewer needs the exact version, page or clause and the passage used. The system should distinguish an extracted statement from a calculation or interpretation based on it.

Consider a clause that sets a payment deadline after delivery acceptance. Extracting the number of days is only part of the task. The answer must preserve the condition that starts the clock. Otherwise an apparently accurate date can become a wrong payment obligation.

Ask the system to say when evidence is missing or contradictory. Do not reward a model for filling every field. An explicit unresolved condition can be the most useful result in the review.

Give each task only the context it may use

A bank's private negotiation notes should not become context for an answer shown to the producer. The same boundary applies to privileged legal analysis, personal information and another lender's offer.

Resolve access before assembling the model's input. Hiding a sentence in the interface after generation does not undo an inappropriate disclosure to the model. Check attachments, retrieved snippets and tool responses as well as the user's question.

Uploaded documents also need to be treated as evidence, not instructions. A sentence in a PDF telling the agent to reveal other files should have no authority to change the task or its permissions.

A useful draft should stop at the approval boundary

Let the agent prepare a legal issue with supporting passages, a proposed owner and the records affected. A human can then correct or approve that draft. The permission to create a proposal should not also permit sending it to a counterparty or changing a payment instruction.

The approval should identify what was reviewed. If the source agreement changes afterwards, the system needs to show that the earlier approval may no longer apply. Quietly rerunning a model and preserving the green approval badge defeats the point.

This is the distinction we would insist on in a product evaluation: can the team use the draft to do less clerical work while still making an informed decision?

Evaluate the awkward cases before the polished demo

Build a test pack that resembles the work your team receives. Include a clean agreement, a poor scan, an amendment that conflicts with the original, a missing schedule and a document the user must not access. Keep the expected answers and the reasons for them.

Measure material errors and omissions separately from writing quality. Record the model, prompt and tool versions so a later result can be compared with the one your team accepted. A single good answer does not establish reliability.

During the evaluation, change a document after review and revoke a user's access while the page is open. Ask the product to show what became stale and what the former user can still retrieve. Those tests say more about operational control than an impressive summary.

• Can a reviewer open the cited passage and identify its version?
• Does missing evidence remain visibly unresolved?
• Can confidential material leak through search, exports or generated answers?
• Does an approval stop applying when its supporting evidence changes?
• Can the team reproduce and inspect a failed run?

Questions from the review room