Black-Box Agents: How to Govern What You Can't See

July 23, 20266 min read

A routine Salesforce platform update ships on a Tuesday. Buried in the release notes is a new AI feature: an embedded support agent that reads customer records to draft replies. Nobody on your security team approved it. Nobody scoped its access. It is running in production with access to customer data, and the first time anyone hears about it is when Discovery surfaces an unregistered agent calling an external model provider. No one chose to adopt it, but the moment it leaks a record, the incident is yours.

Most agent-governance advice assumes you built the agent. You can instrument it, read its prompts, trace its reasoning, and wrap it in your own controls. A growing share of agents arrive the other way: bought, embedded in vendor software, or switched on inside a tool you have run for a decade. You cannot see their prompts, their models, or their tool logic, yet they run with real access to your data and systems. Governing a black-box agent is a different discipline than governing one you wrote. It shifts the work from instrumentation to interrogation: what you demand from the vendor, what you can observe from the outside, and how discovery catches the agents nobody told you about.

This post covers why bought agents are the harder problem, the three ways they get in, the due-diligence checklist to run before you sign, how to govern an agent you can't open up, and why discovery is what finds the ones nobody told you about.

Why bought agents are the harder governance problem

There is an asymmetry at the center of this. You can instrument what you build. A vendor agent is opaque by default. The prompts, the model choices, the tool logic, and the reasoning steps all sit behind an interface you don't control, and often behind a contract that gives you no visibility into any of it.

The sneakiest part is that many of these agents arrive with no procurement decision at all. Applications your enterprise has run for years are now shipping agentic features under the hood through routine updates and patches. Salesforce, ServiceNow, ledger tools, and customer-service suites are all weaving agents into software that was deployed long before anyone thought about agent governance. You didn't buy an agent. You accepted an update, and an agent came with it.

"We didn't build it" is not the same as "we aren't responsible for it." When a vendor-embedded agent leaks data, misroutes a refund, or acts on customer records it should never have touched, the enterprise owns the incident. Regulators, customers, and your own board will not accept "the agent was in a third-party tool" as a defense. The accountability stays with you even when the code does not.

The three ways a bought agent gets in

Agents are entering the enterprise from every vector, and the bought ones arrive through three distinct paths.

New solutions. Startups and vertical tools built agent-first, for legal, finance, customer service, and dozens of other functions. These are procured deliberately, which at least gives you a moment to evaluate before signing. It is the one path where due diligence is naturally on the table.
Existing software. Long-deployed applications weaving agents in through updates. This is the sneakiest vector, because no one chose to adopt an agent. The Salesforce support agent from the opening is this case: a capability switched on inside software you already trusted, with no new contract to trigger a review.
Embedded subagents. A bought agent that itself calls other agents, models, or MCP servers you never evaluated. You may have vetted the vendor's headline agent, but not the chain of actors it delegates to behind the scenes. Each hidden hop is a piece of your risk surface you cannot see.

What to demand from the vendor

The most useful thing you can do before a bought agent enters your environment is ask the questions that reveal whether it can be governed at all. A vendor who can't answer these is a vendor whose agent you can't govern. Run this checklist during procurement and security review:

Models and providers. What models and providers does the agent call, and does our data leave our environment to reach them?
Access scope. What tools and systems can the agent access, and can we scope those permissions down to what the task requires?
Telemetry. Does the vendor emit telemetry we can ingest, ideally OTel traces, or is the agent a complete black box?
Guardrails. What guardrails run on inputs and outputs, covering PII, prompt injection, and toxicity, and can we configure them for our context?
Ownership and incident response. Who is the accountable owner on the vendor side, and what is their incident-response commitment when the agent misbehaves?
Independent evaluation. Can we run our own evals against it, or test it against our own behavior dataset before rollout?

Treat the ability to answer these as a procurement signal in itself. A vendor with clear answers has thought about governance and built for it. A vendor who deflects is telling you their agent will be an ungovernable black box inside your environment, and that belongs in the risk assessment before anyone signs.

Governing what you can't see

Suppose the vendor gives you little and the agent is already running. You can still govern it, from the outside, at the boundaries you control.

Observe at the boundary. Even without internal traces, you can monitor the agent's network traffic, MCP calls, and data access at the perimeter. The same network-layer and MCP-monitoring techniques that surface agents during discovery also let you watch a black-box agent's behavior from outside its walls.
Wrap it in your own guardrails. Put pre- and post-LLM guardrails around the agent's inputs and outputs at the boundary you own. You can't change the vendor's internals, but you can redact PII before it reaches their model and check their outputs for hallucinations or policy violations before they reach your users.
Constrain access, not internals. If you can't govern how the agent reasons, govern what it can touch. Scope its data and tool permissions tightly so a misbehaving agent can only reach the narrow slice its task requires. Opacity you can't fix is far less dangerous when the blast radius is small.
Evaluate behavior externally. Run supervised evals against the agent's outputs using your own behavior dataset, treating it as a black box you score from the outside. You don't need to see inside a system to hold its outputs to a standard and catch regressions when a vendor update quietly changes its behavior.

Arthur's role here is the boundary layer, not X-ray vision into a vendor's code. We give you the Discovery, boundary-level guardrails, external evaluation, and governance inventory that let you govern an agent you can't open up. We do not claim to see inside a vendor's black box, and no honest platform should.

Discovery: finding the ones nobody told you about

Every control above assumes you know the agent exists. The bought agent embedded in a patch, and the subagent your vendor's agent quietly calls, won't announce themselves. Automated Discovery across OTel telemetry, MCP monitoring, network analysis, and cloud-provider APIs is what surfaces these before they become a shadow-agent incident.

Governance starts with an inventory. For agents you built, the inventory is a record of decisions you made. For bought agents, the inventory is often the only thing standing between you and an agent you don't know exists. The Salesforce support agent from the opening only became governable the moment discovery caught it, assigned it an owner, and put it under boundary controls. Before that, it was pure exposure.

Takeaway

The agents you buy, inherit, or have switched on without asking are the hardest to govern precisely because you can't instrument them. The discipline shifts from looking inside to interrogating the vendor, watching the boundary, wrapping your own guardrails and evals around the outputs, and constraining what the agent can touch. Discovery is what makes the whole thing possible by surfacing the agents nobody registered.

Inventory your bought agents and score your vendors before the next incident does it for you. Book a demo with an AI expert or explore the Agent Development Toolkit.