[ FM-FDE-03 ] // AIP

Think Like an FDE III: Agents That Act, Not Chat

This is the doctrine module for operationalizing LLMs in Foundry, and every claim in it is anchored in a lab session we ran and verified. The spine: chat is not the product, acted-on decisions are, and an agent that cannot write back is a search box with manners. The evidence runs both directions. AIP built a correct pipeline from one prompt (FM-PB-01), triaged three escalating notes correctly with a visible token meter (FM-AIP-01), and called its tool when it should while refusing to guess when it should not (FM-AIP-02). It also produced a case-sensitive filter that silently dropped 43 percent of closed rows, then repaired it with a correct condition appended where it could do nothing (FM-PB-02). AIP Assist flatly refuses to execute actions and says so verbatim (FM-ON-02), which fixes the division of labor: the app surface executes, the agent advises. The guardrails are real and checkable: submission criteria and parameter validation as the front door, the dry run as the verification instrument, the Tool description field as the AIP-facing contract, functions as where judgment lives in code. The doctrine that falls out: trust gradients. Read-only suggestions need the least verification, proposed writes need dry runs, autonomous writes need evals and audit trails.

VERIFIED 2026-09-07 // OPERATOR // 30 MIN // FM-FDE-03
Foundry build // Doctrine module. No new platform claims here; every claim cites a verified lab module, sessions dated 2026-08-29 through 2026-09-07.
Environment // Avinyr lab enrollment (Foundry cloud); evidence drawn from FM-PB-01, FM-PB-02, FM-AIP-01, FM-AIP-02, FM-AIP-03, FM-ON-02, and FM-ON-03
[ PREREQUISITES ]
  • FM-FDE-01. The decisions-not-data stance is assumed here. If you have not asked who decides what, the agent design question has no answer yet.

Field notes from the Avinyr lab. This module has no new screenshots because it is not a new build. It is the doctrine that fell out of eight lab sessions, and every claim below names the module where the evidence lives. When we say AIP failed, we show the failure. When we say AIP was right, we show that too. The lab honesty rule is the point: you cannot calibrate trust in a system you only describe at its best.

[ READ THIS FIRST: WHAT THIS MODULE IS ]

The manual has now run AIP across four surfaces: generating pipeline transforms (FM-PB-01, FM-PB-02), reasoning in a Logic function (FM-AIP-01), talking to a user with tools (FM-AIP-02), and being graded by an eval suite (FM-AIP-03). It has also watched AIP Assist refuse, verbatim, to execute an ontology action (FM-ON-02). Put those sessions side by side and a single doctrine emerges, and it is not "use AI more" or "use AI less". It is: decide what the agent is allowed to change, then match the verification to that permission.

This module is that doctrine, with the receipts.

[ THE CLAIM: CHAT IS NOT THE PRODUCT ]

Most AIP demos end at a good answer in a chat window. That is the demo, not the product. In an operation, value lands when a decision gets made and acted on: the work order reassigned, the priority changed, the shipment expedited. An agent that cannot write back is a search box with manners.

FM-FDE-01 gave you the first half: find the decision, find who makes it. This module is the second half: once you know the decision, the agent question is not "what should the model say" but "what is the model allowed to do about it, and who checks". Every design choice below, writeback, guardrails, evals, is a version of that one question.

[ WHAT THE LAB SHOWED: WHEN AIP WAS RIGHT ]

Report this side first, because the skepticism that follows only means something if the praise is earned.

FM-PB-01: one plain prompt ("Keep only closed work orders. Add a resolution_days column containing the number of days between reported_date and closed_date.") produced a correct three-board chain on the first try. We audited every board, deployed, and diffed the built output against source: 66 of 66 rows matched, zero mismatches. We published that as a success, because it was one.

FM-AIP-01: the triage Logic function handled three notes of escalating severity and was right all three times, including the one where we expected a different answer. A note about a leaking air handler got triaged ELECTRICAL, not HVAC, because the water was landing on an electrical panel. The model triaged the consequence, not the asset, and its rationale said why in one sentence. Each run showed its cost: 145 prompt tokens and 9.35 seconds on the long one, under 600 tokens for all three.

FM-AIP-02: the chatbot called its tool when it should, refused to guess when it should not, and carried context across turns. The tool trace on the emergency note contained the line we most wanted to see, the model's own reasoning: "I will not classify this myself." The instruction "Do not classify on your own" showed up in the chain of thought.

FM-AIP-03: the eval suite ran green 3 of 3, and when we forced a failure by demanding a keyword the agent never uses, the suite caught it in under a minute and the failure detail showed the agent had answered correctly and the test was wrong. The net works in both directions.

That is the honest headline: on scoped tasks with clear inputs, AIP is fast and mostly right. Which is exactly why the next section is dangerous.

[ WHAT THE LAB SHOWED: WHEN AIP WAS WRONG ]

FM-PB-02 reran the identical FM-PB-01 prompt against trap-laden data, 120 rows where we had computed ground truth before opening Foundry: 70 rows closed in some capitalization, 6 closed rows with null closed_date, 4 closed rows with closed_date before reported_date.

AIP generated a filter, status is equal to "closed", that matched only the 40 exact-lowercase rows and silently dropped the other 30. No warning, no error badge, and a generation summary that read as success: "Filters closed work orders and calculates the number of days it took to resolve each one, then saves the results." The summary described what we asked for, not what the chain does. 43 percent of the intended output was gone behind a green preview.

Then it got worse in a more instructive way. We asked AIP to fix it, and it wrote the correct condition, Uppercase(status) is equal to "CLOSED", and appended it as a new board after the broken filter. Boards execute in order. The correct condition received the 40 surviving rows and changed nothing. Content right, placement dead. The preview still read 40 rows.

And the failure was hiding a second one. All 4 negative-duration rows carried status Closed or CLOSED, so the bad filter excluded them by accident. The wrong pipeline looked cleaner than the truth. That is the failure mode to memorize: not wrong answers, but silent drift between what you asked and what the system did, with reassuring prose on top.

What caught it was not cleverness. It was ground truth computed before prompting, and the FM-PB-01 habit of reading boards instead of summaries. Hold onto that; it becomes the eval doctrine below.

[ THE CAPABILITY MAP: WHO EXECUTES ]

In FM-ON-02 we asked AIP Assist to execute the Reassign technician action on a real work order. Its answer, verbatim: "I cannot execute actions on your behalf. AIP Assist does not have the ability to interact with, modify, or trigger actions on objects in your ontology." Pressed on where execution should happen, it conceded: executing an action on a specific instance "is best done through an application built on top of the Ontology (such as a Workshop app) rather than Ontology Manager itself."

That refusal is not a bug. It is the capability boundary of the platform, and it fixes the division of labor you should design around:

  • The agent advises. It reads, reasons, classifies, drafts, and recommends.
  • The app surface executes. Actions run from Workshop or Slate apps, from merged ontology state, or through the OSDK and API.
  • The contract between them is the action type, and the action type is where the guardrails live.

Design consequence: when a stakeholder asks for "an AI that just does it", the honest architecture is an agent that proposes plus an app surface that disposes, with a human or a rule in between. The refusal is the platform telling you the same thing.

[ THE GUARDRAILS ARE REAL, AND YOU CAN WATCH THEM WORK ]

FM-ON-02 also showed what a governed write looks like before it ever touches data.

The front door is submission criteria: "Configure who can execute this action type", set at Organization, Group, or User granularity. That is an authorization decision made at build time, separate from datasource permissions, and you set both on purpose.

The verification instrument is the Test run: a full dry run that prefilled a real object's live state (technician S. Kowalski, matching ground truth to the character), computed the proposed edit to T. Nguyen, rendered a before/after diff ("Modifies 1 Object"), and wrote nothing. Banner, verbatim: "This output displays the edits that this action would make to the Ontology. These changes are not applied to your Ontology." The execution log itemized the run in timed phases, and two of those phases are the guardrails: Submission criteria 0.000s, Parameter validation 0.191s, total 0.386s. You can watch the front door do its job.

The AIP-facing contract is the Tool description field on the action editor's Overview tab. Ours read "No tool description", which means the action was invisible-in-spirit to AIP tooling. An empty tool description is not neutral; it is the agent having no way to know your action exists or when to reach for it. If you intend an agent to propose this action someday, the tool description is the field that teaches it when. Write it like a contract: what the action does, what inputs it needs, when it is the right tool and when it is not.

Judgment that outgrows a prompt moves into code. FM-ON-03 is the evidence: an ontology function in TypeScript with a typed signature, a linter that rejected a plain number field with the verbatim error ""number" is not a supported type." and accepted Long from @osdk/functions, a tagged version, and a published artifact whose runs we verified (41 rows for Hawaiian Airlines Inc., the same 41 after publish). A function is where business rules get an enforced contract: typed inputs, typed outputs, versioned, testable, callable by apps and agents alike. Prompts drift between runs; FM-AIP-03 watched the same case say "electrocution hazard" one run and "power hazard" the next. Code does not drift. The rule of thumb: if a wrong answer is expensive, the logic belongs in a function, and the model belongs upstream of it, deciding when to call it.

[ TRUST GRADIENTS: THE DOCTRINE ]

Put the sessions together and the verification burden sorts itself into three tiers, by what the agent is allowed to change.

Tier 1, read-only suggestions: the agent summarizes, classifies, drafts, recommends. Examples from the lab: AIP generating a pipeline chain, the Logic triage function returning JSON, the chatbot answering a technician. Least verification required, but not zero. The FM-PB-02 failure lived entirely in this tier: a generated chain that previewed green and was wrong. Minimum standard here: ground truth you computed before you asked, and an audit of what was actually produced. You do not need a fancy eval setup to start. You need known answers. That is the eval instinct, and FM-PB-02 proves it is a habit, not a product.

Tier 2, proposed writes: the agent or the operator stages a change that nothing has applied yet. Example from the lab: the action Test run, a complete before/after diff with zero bytes written. Standard here: dry runs before real runs, diffs against live state, guardrail phases observable in the log. The dry run is not the consolation prize for missing writeback; it is the step that makes real writeback boring.

Tier 3, autonomous or app-driven writes: the change lands in the ontology, and downstream state depends on it. The lab entered this tier in FM-WS-02 on 2026-09-08: a human pressed the button in a Workshop app, the edit landed, and it survived a full reload. FM-ON-02 ended at a proven proposed edit; FM-WS-02 made it real. What remains unentered is agent-driven Tier 3, where the agent proposes and the write lands without a human per transaction. Standard here, before you let anything write without a human per-transaction: evals that grade the agent on known cases (FM-AIP-03's suite, green 3 of 3, and it caught our deliberate break), an audit trail that survives the write (object edit history, the "Create action log objects" capability, still unexercised in our lab), and a regression habit, because the suite grades the last published version and a model or prompt change can degrade it silently.

Mapped to surfaces:

SurfaceTierWhat the lab verifiedVerification standard
AIP pipeline generation (FM-PB-01, FM-PB-02)1Correct chain on clean data, silent 30-row drop on trap dataPrecomputed ground truth, board-level audit, row-count check
Logic function (FM-AIP-01)13 correct triages, token meter visible on every runPreview against escalating inputs, read the rationale
Chatbot with tools (FM-AIP-02)1Tool called when it should be, follow-up question instead of a guessRead the tool trace; the reasoning line is the audit
Eval suite (FM-AIP-03)metaGreen 3 of 3, forced red caught, stale-save race caughtRun on every publish; read failures before fixing anything
Action Test run (FM-ON-02)2Full diff, guardrail phases timed, zero writesDry run every action before any app calls it
Applied action from an app3Exercised human-driven in FM-WS-02; agent-proposed writes still unexercisedEvals green, edit history on, action log objects considered

The gradient is the whole doctrine: verification cost scales with blast radius. Anyone who applies Tier 3 autonomy with Tier 1 verification is building the FM-PB-02 failure with a writeback attached.

[ EXPECTATION VS REALITY ]

What you might expectWhat the lab showed
AIP either works or it does notBoth, in the same week, on the same prompt. Data composition decided which (FM-PB-01 vs FM-PB-02)
A green preview means the output is rightA 43 percent data loss previewed green with a reassuring summary. Only row counts against ground truth caught it
A correct repair fixes the problemAIP's repair was semantically correct and placed where it could do nothing. Audit placement, not just content
The agent can execute the action it recommendsAIP Assist refuses, verbatim: "I cannot execute actions on your behalf." App surfaces execute; agents advise
Guardrails are paperworkThey ran in the dry run's execution log as timed phases: Submission criteria 0.000s, Parameter validation 0.191s
An empty Tool description field is harmlessIt makes the action invisible to agents. The field is the AIP-facing contract of the write
Evals need heavy infrastructureFM-PB-02 ran evals with a CSV and a known answer. Three keyword cases in FM-AIP-03 caught a real break in under a minute
A red eval means the agent brokeOur only planned red was the test's fault. Read the response text before touching the agent

[ FAQ ]

So should we trust AIP or not?

Wrong axis. Trust is per-tier, not per-tool. We trust AIP as a fast first draft in Tier 1 because we always hold ground truth. We trust the Test run in Tier 2 because it shows the full diff and writes nothing. We do not yet trust anything with Tier 3 autonomy, because we have not built the evals and audit trail that tier requires. That is a posture, not a mood.

What is the minimum viable eval setup?

Known answers, computed before you ask the model anything. In FM-PB-02 that was a ground truth script against a trap CSV. In FM-AIP-03 it was three test cases taken from verified live traffic, graded by one keyword evaluator. Both caught real failures. Start there; grow the setup when the cost of a miss justifies it.

Where does the model's judgment end and code's begin?

Where a wrong answer gets expensive. Severity labels and routing rules that must not drift belong in functions (FM-ON-03), with typed contracts a linter enforces. The model's job narrows to understanding messy input and choosing the right tool. FM-AIP-02's chatbot is the shape: the model decides when to call, the function decides what the answer is.

Does the agent ever get to write directly?

The platform's own answer today is no for AIP Assist (its refusal is quoted above) and yes-with-guardrails for actions from app surfaces. If you build agent-proposed writes, put a dry-run-equivalent and a human or rule gate in front of the apply step, and turn on edit history. Then write that design down, because you will be asked to defend it.

What do we do when AIP is wrong in production?

The FM-PB-02 answer: defects cluster in the rows that look nonstandard, which are the rows lazy filters drop. Keep anomalies visible instead of discarding them, alert on distribution shifts, and make sure someone can trace every number back through lineage. A system that hides its own errors is worse than one that makes them.

[ NOT TESTED / OPEN QUESTIONS ]

  • Agent-proposed applied writeback from an app surface (human-driven writeback proven in FM-WS-02; the agent-proposed variant is still unexercised)
  • Agent-triggered actions through the Apply action tool in Chatbot Studio (the catalog entry exists; we have not wired it)
  • The "Create action log objects" capability as an audit trail for Tier 3 writes (toggle exists, off, unexercised in FM-ON-02)
  • Whether an agent respects a well-written Tool description field measurably better than an empty one (our field reads "No tool description"; the experiment is designed but not run)
  • LLM-as-a-judge evaluators for semantic grading (FM-AIP-03 used keyword checks only)
  • Scheduled or CI-triggered eval runs as a release gate, including the run history dataset for trending
  • Cost telemetry for agent runs in evals (chatbot functions reported No value on every cost column in FM-AIP-03)

[ NET POSITION ]

Chat is not the product; acted-on decisions are. The lab record runs both ways: AIP built a verified pipeline in one prompt and triaged real-seeming emergencies correctly with a visible meter, and it also dropped 43 percent of rows behind a green preview and repaired its own mistake into dead code. The honest summary is not trust or distrust; it is gradients. Read-only suggestions get ground truth and an audit. Proposed writes get dry runs and diffs. Autonomous writes get evals, audit trails, and a human-shaped gate until the evals earn their keep. The platform agrees with the doctrine: its own assistant refuses to execute actions and points you at app surfaces, its action editor ships the dry run as the verification instrument, and its eval app only grades versions you have shipped. Agents that act are built out of these pieces, in this order, with the verification matched to the blast radius. The labs that prove each piece are cited inline; the ones we have not run yet are listed above, by name, as open questions.