Swarm & Execution Plans

Added in 0.8.0.

A 0.7 request picked one node and sent the prompt there. A 0.8 request becomes a plan first: a job, a set of work units, and targets chosen — and refused — before anything is dispatched.

Two things follow from that. Several models can work on one request. And you can ask what would happen without anything happening.

mycellm/swarm

Send it like any other model:

curl http://localhost:8420/v1/chat/completions \
  -H "Authorization: Bearer $MYCELLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mycellm/swarm",
    "messages": [{"role": "user", "content": "Why is the sky blue?"}],
    "mycellm": {"fanout": 3, "token_budget": 4096}
  }'

N models answer in parallel (proposers), then one merges the answers into a final response (synthesis). Proposers run at spread temperatures, because identical settings on one model produce N copies of the same answer — that is redundancy, not diversity.

mycellm/swarm appears in GET /v1/models only when the node can actually form a swarm — two or more distinct models reachable. A node that would always refuse does not list it.

Streaming

Supported since 0.8. The shape of a swarm decides what can stream: proposers run in parallel and their answers are an input to synthesis, so only the synthesis phase produces user-facing tokens.

→ chunk  delta:{}                 mycellm:{phase:"proposing", planned:3, targets:[…]}
→ chunk  delta:{}                 mycellm:{phase:"synthesizing", from_proposals:3}
→ chunk  delta:{content:"The "}
→ chunk  delta:{content:"answer"}
→ chunk  finish_reason:"stop"     mycellm:{…the full plan…}

Progress chunks carry empty content. That distinction is load-bearing: a client that simply concatenates delta.content gets exactly the answer and nothing else, while a client that wants to render “3 proposers on 2 nodes, now synthesising” has the facts to do it. Progress in content would make every naive client print the execution plan into the reply.

Degraded runs still stream. When synthesis is skipped — one surviving proposer, budget exhausted, no synthesiser planned — the salvaged answer arrives as a single chunk and mycellm.degradation says why. The wire shape does not change, so a client never has to care whether an answer was synthesised or rescued.

A refusal arrives as content

An SSE response cannot change its status code once headers are sent, so a refused swarm reports the reason as message content with the plan in mycellm — never as a silent empty stream, which a client would read as success.

The plan comes back with the answer

Every swarm response carries a mycellm object. You are paying for N proposers, so what ran is part of the response rather than a debug extra:

{
  "choices": [{ "message": { "content": "…" } }],
  "mycellm": {
    "strategy": "swarm",
    "units": [
      {"unit_id": "job_x-p0", "role": "proposer",    "target": "local:qwen2.5-7b"},
      {"unit_id": "job_x-p1", "role": "proposer",    "target": "group:grp_aurora:llama3-70b"},
      {"unit_id": "job_x-syn","role": "synthesizer", "target": "group:grp_aurora:llama3-70b",
       "depends_on": ["job_x-p0", "job_x-p1"]}
    ],
    "rejected": [
      {"target": "group:grp_public:mixtral", "reason": "sensitive data blocked for untrusted egress: AWS access key"}
    ],
    "reasons": ["swarm: 2 proposers across 2 distinct model(s) → synthesis on …"],
    "units_ok": 2, "units_failed": 0,
    "synthesized_by": "group:grp_aurora:llama3-70b",
    "completion_tokens_spent": 812,
    "elapsed_s": 9.4
  }
}

rejected matters as much as units. A target blocked by egress policy is otherwise indistinguishable from one that was never there.

Degradation is reported, never hidden

Situation	What happens
Fewer than two eligible proposers	Runs direct and sets strategy: "direct" with a reason saying so
All proposers share one model	Runs, and says self-consistency sampling, not a heterogeneous swarm
A proposer fails	Job continues; the failure is listed under failures
Only one proposer survives	That answer is returned; degraded: true
Synthesis fails	The best proposal is returned rather than discarding paid-for work
token_budget reached	Remaining units are cancelled; cancelled_for_budget is set

A swarm that quietly became a single call while still reporting "swarm" would be unfalsifiable, so it never does.

Egress refusal is a 4xx, not a fallback

If every candidate is refused, the request returns HTTP 422 with the plan attached. It does not fall back to an ordinary single-model answer — that would hide an egress block, which is the one thing that must never be silent.

POST /v1/node/plan — dry run

Same decision, nothing executed. The planner is a pure function, so this is exactly what the real request would do.

curl http://localhost:8420/v1/node/plan \
  -H "Authorization: Bearer $MYCELLM_API_KEY" \
  -d '{"model": "mycellm/swarm", "messages": [{"role":"user","content":"…"}]}'

Returns {"plan": …, "candidates": […]}. Use it to answer “why did that go there”, or to check what a prompt would be allowed to reach before sending it.

The mycellm request block

Field	Status
min_tier, min_params, required_tags	Enforced during model resolution
fallback	Enforced — "downgrade" or "reject"
trust	Enforced — "local", "trusted", "any". A ceiling you lower for yourself; it cannot raise a peer’s trust
token_budget	Enforced by the coordinator as results land
fanout	Swarm proposer count; 0 lets the planner decide
routing	Only "best" is implemented. Anything else returns HTTP 400
min_context, max_cost	Not implemented. Setting either returns HTTP 400

Until 0.8 the last two rows were accepted and silently ignored, so a caller who set a credit ceiling had none. Refusing is the smaller harm: silently dropping a constraint leaves you believing a limit is in force.

GET /v1/node/groups — serving groups

A serving group is a gateway this node fronts — an OpenAI-compatible endpoint, which may itself be a cluster. Each model it serves is a deployment with a stable id, so two groups serving the same model name stay distinguishable.

{
  "groups": [{
    "group_id": "grp_aurora",
    "name": "aurora",
    "endpoint": "http://aurora.lan:8080",
    "runtime": "external",
    "online": true,
    "healthy": true,
    "deployments": [
      {"deployment_id": "dep_grp_aurora_relay-llama3-70b",
       "model": "relay:llama3-70b", "upstream_model": "llama3-70b"}
    ]
  }],
  "count": 1, "healthy_count": 1, "deployment_count": 1
}

online and healthy are not the same field. Online means the endpoint answers. Healthy means it currently owns at least one registered deployment. They came apart in a real incident: after a restart an auto-loaded model blocked the relay’s claim on its own name, and the node kept serving the group while reporting zero deployments.

Fleet commands

Relayed to a peer through POST /v1/admin/fleet/command on a bootstrap. The bootstrap never stores or inspects the fleet admin key — it is verified on the target node.

Command	Effect
node.status	That peer’s role, models, peers, uptime, hardware
node.config	Its effective configuration (secrets redacted)
node.groups	Its serving groups and deployments
node.targets	Everywhere a work unit could run on that peer
node.plan	Dry-run a request on that peer, including its refusals
model.list	Models it has loaded and saved
model.load	Load a model there
model.unload	Unload one
model.scope	Change a model’s visibility (private / home / public)
relay.list	Its relay backends
relay.add	Attach a gateway to that peer
relay.remove	Detach one
relay.refresh	Re-discover models (one relay, or all)
train.status	Federated-training state, when enabled

curl http://bootstrap:8420/v1/admin/fleet/command \
  -H "Authorization: Bearer $MYCELLM_API_KEY" \
  -d '{"node_peer_id": "…", "command": "relay.add",
       "params": {"url": "http://aurora.lan:8080", "name": "aurora",
                  "api_key": "secret:aurora"},
       "fleet_admin_key": "…"}'

api_key accepts a secret reference resolved on the target node, so a command can name a credential instead of carrying one over the wire.

node.plan has no privacy override. The local HTTP route honours an X-Privacy-Override header because it is your own loopback surface; the fleet channel lets a remote admin submit a prompt of their choosing, and the key that manages models does not also disable egress scanning on someone else’s machine.