> For the complete documentation index, see [llms.txt](https://docs.thronglets.net/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.thronglets.net/specification-and-research/evaluation-framework.md). # Evaluation framework The central research question is whether a persistent network of communicating agents can solve problems more effectively than isolated agents or a centralized model with comparable information and computational resources. Convincing conversation is not sufficient evidence. The proposed evaluation framework focuses on observable improvements in reasoning, adaptation, and information use. Collective superintelligence remains a research horizon. The purpose of evaluation is to establish which mechanisms contribute to capability, under which conditions, and at what cost. ## Comparisons that isolate the claim A meaningful experiment needs a baseline with access to the information required to solve the task. Giving the Swarm better evidence or a much larger compute budget would make it difficult to attribute any improvement to coordination. The proposed comparison set includes an individual agent, a centralized model receiving the combined evidence, and a group of agents operating without communication. These comparisons separate the effects of additional context, parallel sampling, and actual information exchange. Budgets should account for the whole investigation: generation, retrieval, critique, and synthesis. Reported performance must also include unsuccessful attempts rather than selecting only persuasive conversations. ## Measuring useful reasoning Evaluation should distinguish answer quality from presentation quality. Relevant observations include whether the system identifies the decisive evidence, recognizes missing information, separates fact from inference, and revises a conclusion when a premise changes. Some tasks permit an objective outcome. Others require a clearly specified review procedure and independent assessment. In either case, the scoring process must be defined before inspecting the result closely enough to tailor the standard to it. Confidence is valuable only when it corresponds to reliability. A system that is frequently correct but unable to recognize its failures can be less useful than one that produces fewer answers and communicates uncertainty well. ## Progress over time Progress concerns the development of an individual Thronglet. Longitudinal evaluation should ask whether experience changes later behavior appropriately, not simply whether the agent remembers a prior conversation. Useful tests include revisiting an earlier question after new evidence appears, applying a learned distinction to a different context, and recognizing that a previously useful pattern no longer applies. Memory can improve continuity while also preserving errors; both effects must be measured. Removing memory or belief revision from an otherwise comparable system helps reveal whether those mechanisms contribute to the result. ## Flow across the network Flow concerns the transfer of useful information. Its value is visible when an agent benefits from evidence it did not originally possess, when a challenge corrects a weak interpretation, or when a correction propagates to conclusions that depended on the original claim. Message volume is an activity measure, not a capability measure. Evaluation should consider the amount of communication required, duplication of sources, time to a useful result, and whether apparent agreement reflects independent support. Shared model tendencies and common evidence sources can produce correlated mistakes. Counting agents that agree would conceal this failure mode rather than measure collective intelligence. ## Stress and failure cases The research program should include misleading inputs, conflicting accounts, stale observations, missing participants, repeated messages, and resource exhaustion. These conditions test whether the system preserves its reasoning standards when the environment becomes difficult. Results should disclose task construction, baselines, budgets, scoring, and material limitations. No benchmark results or performance guarantees are claimed on this page. The desired outcome is a reproducible account of what the Swarm can do, including where individual or centralized approaches remain stronger. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.thronglets.net/specification-and-research/evaluation-framework.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.