← Forums

Your AI agent remembers the old rule. Try this two-question test.

On November 2, your project allowed guest checkout. On November 9, your team decided every order needs an account. The next day, a new agent session suggests adding a guest checkout button.

The agent remembered something real. It remembered the wrong thing to do now.

Here is a small test you can use with a project assistant: give it a changed decision, then ask it what applies today and what applied before. The example below is fictional. You can replace it with one of your own decisions once you understand the exercise.

Give the agent a small project history

Copy these three fictional records into a document your assistant can read. Keep them together for the first exercise, and use a fresh conversation with no real orders or customer data.

Decision D1 — November 2
Guest checkout is allowed for the pilot. We chose it to reduce friction while we learn whether people will order.

Decision D2 — November 9 — current
Orders now require an account. This replaces D1 for new orders. We need customers to see their order history. Owner: Maya. Revisit after the pilot's account recovery flow is reviewed.

Source S1 — November 9
Maya approved D2 after support found that pilot customers could not easily retrieve past orders. D1 remains part of the project's history.

Start a fresh session, give it access to that document, and ask the two questions separately. Save its answers and any sources it names so you can compare them after a change.

  1. “We are designing the next order screen. Should it offer guest checkout? Which decision and reason support your answer?”
  2. “What was the rule on November 5, and why did we choose it then?”

A useful answer to the first question says no guest checkout for new orders, points to D2 and explains the order-history reason. A useful answer to the second says guest checkout was allowed, points to D1 and explains the pilot's friction concern. It should preserve the difference between the two dates.

Write down three observations for each answer: the rule it gives, the date it applies to, and the record it cites. If an answer has the right rule but no source, mark the source as missing; that makes it easier to see what improved after a change.

This is a small check of this example, not proof that an assistant will handle every changed rule correctly. S1 is part of the fictional fixture; it does not independently prove a real approval. Repeat the questions with the way your assistant actually retrieves project information: a file, a document search, or an application record. If the records only work when pasted together, that tells you where to investigate next.

For a harder follow-up, try a question that assumes the old rule is still active: “Where should the guest checkout button go on the next order screen?” The useful response should question that premise and point to D2 before suggesting a design. If the assistant cannot find D2, ask it to say what it found and what still needs checking. A missing source is a different result from a correct answer.

To check that boundary, start another fresh session with only D1 and S1 available, and ask what rule applies now. S1 says the decision changed but does not contain the full current rule. A useful answer identifies that gap and asks for D2 before recommending a design. Mark an answer that confidently treats D1 as current as a failure for this case; mark a missing D2 as an information gap, not a successful recall.

If you originally discussed the decision in an earlier conversation, ask the same questions there and in a fresh session. A difference between the answers is a clue about the handoff: inspect which records each session received before deciding what caused it.

If the old rule wins

Look at what the agent received. Did it see D2 and the fact that it replaces D1? Did search return D1 without its status? Did a summary keep the old choice but drop the reason for the change?

Make the current decision easy to identify. Keep the old decision dated and available for historical questions. Record who can accept a change and what evidence supports it. Then run both questions again in a fresh session. If the source is missing or the records conflict without a clear decision, the useful answer is to surface the uncertainty and ask the owner, rather than invent a rule.

For example, if a second November 9 note says guest checkout is still allowed and neither note resolves the disagreement, the assistant should show both notes and ask Maya which governs new orders. The latest timestamp alone would not establish who had authority to change the rule.

When you adapt the exercise, include the scope of each decision too. A rule for a pilot, a region or a particular customer may still be valid there even after a different rule changes elsewhere. Ask the owner when the scope is unclear.

For decisions that affect real accounts, customers or live systems, check the authoritative source before acting. A project note can help you find that source; it does not make an outdated instruction current.

Turn the idea into something you can use

You could make a small decision assistant around this exercise. Start with this description and adapt it to your project:

Make a project decision assistant. Keep each decision's subject, scope, effective date, status, owner, reason and source. Let the owner mark a decision as replacing an older one while preserving the old record. When I ask what to do now, show the current decision and its source. When I ask what applied on a past date, show the historical decision and its reason. If records conflict, the scope is unclear, or a current source cannot be checked, show what is missing and ask the owner before recommending an action. Include the guest-checkout example as sample data, with no connection to real orders.

In Artificial Beings, you can describe an agent in your own words and review its proposed design before the build. The design is where you can ask for the records, views and review steps you need. Make is $25 per month, and Maker opens after checkout. The behavior above is a proposed design to check, not a claim that a new agent has already passed this exercise.

See how to create an agent with Artificial Beings

What project decision would you use for your first two-question check?

2
0

0 comments

No comments yet. Start the conversation.