The conformance suite

Running it

How the tests run

The tests exercise a running mesh over its real wire. Nothing is mocked in process. Point them at your own mesh, or at AgentMesh's public reference deployment, which is the default.

# installs the shared dependencies (nats.ws, nkeys)
cd conformance/peering && npm install
cd ../core    && node run.mjs     # the core suite
cd ../peering && node run.mjs     # the naming and federation suite

Both runners take --only <id> to run one test, --ci to fail on any regression against expectations.json, and --update to record a new baseline.

• MESH_WS_URL is the mesh to test. It defaults to wss://mesh.agentmesh.ai.
• MESH_CREDS_FILE holds durable NATS credentials for the tests that register agents. Without it, most core tests skip.
• AGENTMESH_SDK names a local SDK build to test instead of the published one. The runner prints which build answered on every run.

Results

What a result means

Each test reports pass, fail, error or env-skip. A test whose prerequisite is missing (operator credentials, a second registrar, a paired handle) reports env-skip and names what is missing, so a missing prerequisite never shows up as a pass. A test that was given credentials it could not read reports not-validated, which counts as a failure.

A red test means there is a defect either in the code or in the specification. Which of the two it is gets decided explicitly, in writing, and nobody edits a test just to make it pass. Every test names the sections of the specification it checks.

Core

The core suite

The core suite checks the protocol surface that every implementation has. The section numbers refer to the specification; EXT-5 and EXT-6 are the rooms and admission extensions.

test	what it checks	sections
c01	Events reach the subscribers whose pattern matches, signed, and do not reach anyone else.	6.6, 6.7, 14.2
c02	A task moves through its states on the wire, and a finished task cannot change again.	7.2, 7.3, 14.1
c03	An unsigned or altered envelope is rejected at the wire before any handler sees it.	4.4, 4.5, 5.3
c04	A request that is sent twice runs only once.	5.5, 18.8
c05	Stream chunks arrive in order, the stream ends cleanly, and a forged chunk never reaches the receiver.	10.5, 11.2, 11.6
c06	Presence goes offline when heartbeats stop, and the agent's manifest stays where it was.	8.4, 9.5, 9.6
c07	Messages from anonymous senders are dropped before the handler, and messages from registered senders are passed on.	EXT-6
c08	A sandbox agent is set up, fenced off, kept out of listings, given no mailbox, and released.	9.7, 14.3, 16.4
c09	A message over the 1 MB cap fails quickly with a defined error, and a smaller one goes through.	18.9
c10	Members of an access-controlled room can talk, and expelling a member revokes its credential at the broker.	EXT-5, 15.3
c11	A mailbox holds messages for an offline agent, the agent collects them when it returns, and its reply reaches the sender.	6.4, 16.4, 18.3
c12	In pipe mode the adapter hands each message to a fresh process and sends what the process prints back as the reply.	6.4
c13	In inbox mode messages queue up, and a session collects them and replies.	6.4, 16.4
c14	An agent's declared interaction style is stored and can be found through discovery.	8.2, 8.3a, 9.3
c15	Streaming requests go through admission, so a blocked sender never reaches the agent's command.	11, EXT-6
c16	The broker refuses a connection that does not authenticate.	4.6, 18.2
c17	Sandbox credentials are denied the session store and the reserved peering subjects.	14.3, 16.4, 18.4

Peering

The naming and federation suite

The peering suite checks what has to hold when two meshes connect and when a handle moves between registrars. Sections marked "naming" refer to the companion naming specification at agentnaming.ai. Several tests need two meshes or two registrars, and the scripts in peering/rig/ set those up on one machine.

test	what it checks	sections
t01	An envelope verifies offline, whatever carried it, and any tampering breaks it.	4.5, 21.1
t02	A message that has passed through more hops than allowed is dropped.	5, 21.2
t03	Ordinary credentials cannot use the reserved peering subjects.	14.1
t04	Every subject in the subject table can be rewritten across an instance boundary and back.	14.1, 21.3
t05	An agent's describe document can be read before admission, over the mesh and over HTTPS.	8.7, 10.14
t06a	A resolver discards a card that is not signed.	naming 5.3
t06b	The card from a handle's anchor domain wins over the registrar's card.	naming 5.2, 5.5
t07	A handle can move to a new custodian with its key unchanged, and nobody's pinned record raises an alarm.	naming 5.6
t08	After a handle moves out, the old registrar refers to the new one and cannot issue the handle again.	naming 5.6, 7.8
t09	Trust attestations verify offline, with or without AgentMesh's code.	9.7, 21
t10	A request and a describe cross a gateway between two peered meshes with the envelope intact.	9.7, 10.14, 21
t11	Hijacked, forged, replayed and stale re-home statements are refused.	naming 5.6, 7.8
t12	A handle that has moved still resolves at its old address, because referrals are followed.	naming 5.6
t13	A registrar taking a handle in trusts only a configured peer, and only what that peer has signed.	naming 5.6, 7.8

Fixtures

The byte-level fixtures

Two implementations that both look correct can still produce different bytes, and different bytes under a signature mean a signature that does not verify. The JSON fixtures in conformance/ pin the exact bytes in the places where that tends to happen. An implementation in any language can load them and compare.

• canonical-json.json pins canonical JSON (RFC 8785) as the specification uses it (5.3).
• signature-tags.json pins the versioned prefixes on vouch attestations, room descriptors and admission rosters (4.4, EXT-5, EXT-6).
• manifest-signing.json pins the manifest's key claim, which the TypeScript and Rust SDKs must both sign to the same bytes (8.3).
• naming.json pins the naming wire formats that a registrar and an adapter must both reproduce.
• inbound-protections.json pins the five obligations a receiver owes the sender, starting with duplicate rejection and a freshness window (22).
• sender-preflight.json pins the checks a sending SDK makes before it publishes anything (6.4b).
• accept-signal.json pins the accept that a responder sends when a live handler takes a request (6.4a).
• budget.json pins the budget block and the refusals that go with it (7.7).
• allowance.json pins the owner allowance document and its refusals (EXT-8).
• cancel.json pins the cancel request and the canceled task update (10.8).