A revised mortgage commitment arrives. The agent reads it, updates the loan amount, and answers the broker's next question correctly. An older draft still contains the previous amount.
The answer is right. The work is now inconsistent.
Our first article explored how agents should handle changing facts. The next question is what happens to work already built on those facts. A correction matters wherever the old value was used, including places the agent has not been asked to revisit.
This is where provenance becomes practical: a record of which evidence supported a claim, which decision accepted it, and which output relied on it.
One revision, two pieces of work
Consider a fictional file. An initial lender commitment states a loan amount of $600,000. The broker reviews the document, records that amount in the working file, and asks the agent to prepare a client update. The update is saved as a draft.
Later, a revised commitment states $575,000 for the same loan amount field. These are invented figures used to illustrate a system design problem, not a client story or a description of every capability available in Clara today.
The new document creates two questions. Which commitment applies to the file? If the amount changes, what existing work needs attention?
Original
Working record
$600,000, linked to the original commitment and the broker's review.
Client update
Draft A uses that accepted amount.
Revision
Working record
$575,000 is extracted as a proposed change. Whether this revision applies is unresolved.
Client update
Draft A is flagged for review before use.
Review
Working record
The broker confirms the applicable revision. $575,000 becomes the accepted amount, with history retained.
Client update
Draft A remains out of date. Accepting a new fact does not repair an old draft.
Redraft
Working record
The accepted amount points to the revised commitment and review decision.
Client update
Draft B uses $575,000 and awaits review and authorization to send.
Extraction is a proposal
The agent can identify the changed amount and show the relevant passages from both documents. That gives the broker a specific difference to inspect.
A later upload time or a filename containing “revised” is not enough to establish which commitment applies. The document's identity, terms, and relationship to the existing file still matter. If that relationship is unclear, the open question needs to remain visible.
In this example, the broker checks the evidence and confirms the applicable revision. That decision records which lender terms the working file should use. It does not give the broker authority to alter those terms.
The system should distinguish the model's proposed extraction from the accepted record. A useful acceptance record identifies the old and new values, their source versions and passages, the reviewer, and when and why the change was accepted. Superseding the old amount should preserve the earlier decision's history.
A correction has dependents
Updating the working record is only part of the job. Draft A still says $600,000. Its text does not change merely because the agent can now retrieve $575,000.
To surface that problem reliably, the system needs a connection between the draft and the specific claim version it used. A link to the entire file is too broad to show what changed beneath the draft.
When conflicting evidence arrives, the draft can be marked for review while the conflict is resolved. Once the new amount is accepted, that status should explain the dependency: this draft used a loan amount that has been superseded.
Simply finding and replacing every occurrence of $600,000 would be a poor substitute. Some occurrences may describe the original commitment accurately. Other figures may be calculations that require a separate update. The goal is to identify affected work and review its meaning.
A sent message also cannot be repaired by changing the current record. The system should preserve what was sent, surface the discrepancy, and help an authorized person decide on a follow-up. Provenance needs to reach beyond the latest chat response.
Approval belongs to a specific output
Suppose the broker had already approved Draft A, but it had not been sent when the revision arrived. Does that approval authorize sending Draft B with a different loan amount?
For this workflow, it should not. Approval needs to identify the output being approved and the relevant facts it relied on. A material change should require another review before sending.
This also exposes a timing problem. The evidence can change between approval and execution. The sending step should check that the approved draft and its relevant dependencies are still current. Otherwise, the system can perform an authorized action using information that has since been superseded.
A record a reviewer can inspect
A reviewer should be able to move from the amount in Draft B to the accepted claim, the revised commitment's relevant passage, and the recorded review decision. They should also be able to see why Draft A stopped being suitable for use.
These are records of evidence, decisions, and actions. They do not require reconstructing a model's private reasoning. A plausible explanation generated after the fact is not a substitute for the records created when the work happened.
A source link is useful only if the passage supports the claim. Traceability makes that judgment inspectable; it does not make the judgment automatically correct.
Test the work left behind
For a brokerage evaluating an agent, this makes a useful demonstration. Start with the original commitment, prepare a draft, introduce the revised evidence, and resolve which version applies. Then inspect:
- Did the new extraction remain a proposal until the review decision?
- Can you trace the accepted amount to the correct document version and passage?
- Was the earlier draft flagged, and does the revised draft use the accepted amount?
- Did approval stay tied to the output actually reviewed?
- If an earlier version was already sent or exported, is that history still visible?
These are design requirements shaping how we think about Clara, and behaviours that need testing as the product develops. A correct answer to “What is the loan amount?” tests only one part of the workflow.
The more revealing question is: what happened to everything that used the old answer?
Continue reading
AI Agents Need More Than a Context Window