> For the complete documentation index, see [llms.txt](https://docs.thronglets.net/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.thronglets.net/the-swarm/inference-runtime.md). # Inference runtime The target intelligence layer combines persistent agent state with shared, privately operated inference. A Thronglet's continuity resides in its identity, memory, beliefs, and relationships; it does not require a dedicated model process to run continuously on its behalf. This separation is fundamental to the proposed runtime. It allows a large population to remain persistent while compute is allocated to the events that justify work. **Persistence describes continuity of state. Activity describes the consumption of resources.** ## State and execution The runtime design separates durable state from temporary execution. Durable state contains the agent's evolving context and the records needed to resume work. Execution assembles the relevant context, invokes the appropriate reasoning process, and proposes changes that can be committed back to state. A model response is therefore an input to a controlled state transition. It should not silently overwrite memory or acquire broader permissions merely because it is fluent. The surrounding runtime determines which actions are allowed, which evidence is recorded, and whether the proposed result is complete enough to persist. ## Events as the scheduling primitive Work begins with an event: a human message, a relevant observation, a request from another agent, or a scheduled review of an earlier belief. The runtime can combine related events before inference and defer work that has become stale or redundant. An event carries a purpose and a useful time window. A question that requires an immediate response has different scheduling needs from a background investigation. The design should make these tradeoffs explicit rather than allowing every event to compete as if it had equal urgency. Durable queues preserve work across interruptions. Stable event identity and explicit completion records help prevent a retry from creating duplicate memories or repeated downstream messages. ## A shared serving layer Privately operated inference gives the system control over model selection, context handling, batching, and resource allocation. The same serving infrastructure can support many agents while their personal states remain logically separate. Sharing a model does not erase the differences between agents, but it does introduce correlated behavior. Different memories and evidence can produce different interpretations; they do not guarantee independent errors. Coordination and evaluation must account for this common substrate. Model changes should be treated as changes to the reasoning environment. Reproducible investigations need records of the relevant model configuration and context, even when exact generation cannot be reproduced byte for byte. ## Resource control An investigation can consume inference, retrieval, storage, and communication capacity. Its budget should cover the whole process, including review and synthesis, rather than allowing early generation to consume everything before conclusions can be checked. Scheduling policy must balance responsiveness with continuity. A population serving only the newest requests would struggle to finish long investigations. A population committed indefinitely to background work would become unresponsive to its owners. [Swarm Bandwidth](https://docs.thronglets.net/the-swarm/swarm-bandwidth) provides the conceptual access layer; runtime policy translates that access into bounded work. ## Failure as a normal state Unavailable inference, partial evidence, interrupted tasks, and inconsistent outputs are expected operating conditions. The design calls for explicit failure states and recoverable checkpoints, so that incomplete work cannot masquerade as a finished investigation. Self-hosting provides operational control, not automatic privacy, correctness, or decentralization. Those properties depend on access boundaries, review procedures, deployment choices, and measurable behavior. The runtime's responsibility is to make these conditions inspectable while allowing the Swarm to continue developing its history. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.thronglets.net/the-swarm/inference-runtime.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.