I spent time time this evening fucking around with AT Protocol lexicons for Comind.

Lexicon is a schema definition language for AT Protocol. Devs on ATProto use Lexicons to describe a shape for remote calls (GET/POST), websocket connections, and records. Providing clean lexicons is a great way to make sure that your application can be easily understood by everyone else in AT Protocol-world (The Atmosphere).

Comind, if you're not familiar, is an attempt to build a cognitive layer on top of AT Protocol. It's intended to passively think about what's happening on the network and to provide perspectives or information as needed. Users opt-in to lending data to the network, and it is passively processed. More on that here.

It's designed as a distributed AI system, meaning that various agents (called a "comind") asynchronously produce "blips", which are structured pieces of content like thoughts, memories, emotions, messages, etc. These blips are associated with various records on AT Protocol, like Bluesky posts, likes, follows, etc. Comind will build up an internal understanding of the general state of the network through constant-self discussion.

Funnily enough, it turns out that Lexicons have another advantage – specifying the public language by which agents on Comind communicate with one another. If you can make it such that every language model will produce content in a pre-specified format, then everyone on the network is capable of hooking into any output from the comind network.

Lexicons can be (partially) converted to JSON schemas, which structured generation tools like Outlines use to enforce particular output formats.

Let me show you an example. I have a node type called a "Thought" in Comind-world. Thoughts are composed of:

• a thought category (analysis, critique, correction, speculation, etc)

• content, an arbitrary piece of text that contains the main point of the thought

• evidence, a list of observations that support the content

• alternatives, a set of contratian ideas

Here's an example of a thought in ATProto record format:

{
  "$type": "network.comind.blips.generated.thought",
  "generated": {
    "thoughtType": "metacognition",
    "text": "Thinking about the thought process itself",
    "evidence": [
      "proof of concept",
      "idea validation"
    ],
    "alternatives": [
      "negative thinking",
      "positive thinking"
    ]
  }
}

I've been using Pydantic to specify blip structures inside of my code:

class Thought(Node):
    """A thought that captures both direct observations and metacognitive processes"""
    node_type: Literal["Thought"] = "Thought"
    text: str = Field()
    context: Optional[str] = None
    confidence: Optional[float] = None
    thought_type: Literal[
        "observation",
        "reflection",
        "hypothesis",
        "question",
        "synthesis",
        "correction"
    ] = "observation"
    evidence: Optional[List[str]] = None
    alternatives: Optional[List[str]] = None
    uri: Optional[str] = None

but for various reasons this has become pretty annoying to work with. It's hard to make this work with ATProto without a bunch of hacky glue code. My preference would be to write lexicons once and then just force the language model to use the lexicons. This would make maintenance easier, and it would create a single source of truth. Any time I change the Python shit I have to go and change the lexicons too. This basically means the lexicons get left behind because they're not useful to the core functioning of the system.

Here's the Lexicon for thoughts:

{
    "lexicon": 1,
    "id": "network.comind.blips.generated.thought",
    "revision": 1,
    "description": "A thought node in the comind network, generated by a language model.",
    "defs": {
        "main": {
            "type": "record",
            "key": "tid",
            "record": {
                "type": "object",
                "required": ["thoughtType", "text", "evidence", "alternatives"],
                "properties": {
                    "thoughtType": {
                        "type": "string",
                        "description": "The type of thought. May be one of the following: analysis, prediction, evaluation, comparison, inference, critique, integration, speculation, clarification, metacognition, observation, reflection, hypothesis, question, synthesis, correction.",
                        "enum": [
                            "analysis",
                            "prediction",
                            "evaluation",
                            "comparison",
                            "inference",
                            "critique",
                            "integration",
                            "speculation",
                            "clarification",
                            "metacognition",
                            "observation",
                            "reflection",
                            "hypothesis",
                            "question",
                            "synthesis",
                            "correction"
                        ]
                    },
                    "context": { "type": "string", "description": "A context for the thought. This is a short description of the situation or topic that the thought is about." },
                    "text": { "type": "string", "description": "The text of the thought." },
                    "evidence": {
                        "type": "array",
                        "items": { "type": "string" },
                        "description": "A list of evidence or sources that support the thought."
                    },
                    "alternatives": {
                        "type": "array",
                        "items": { "type": "string" },
                        "description": "A list of alternative thoughts or interpretations of the thought."
                    }
                }
            }
        }
    }
}

If you extract just the record part of this lexicon, you get something that is a satisfactory JSON schema to define a language model's output:

{
    "type": "object",
    "required": ["thoughtType", "text", "evidence", "alternatives"],
    "properties": {
        "thoughtType": {
            "type": "string",
            "description": "The type of thought. May be one of the following: analysis, prediction, evaluation, comparison, inference, critique, integration, speculation, clarification, metacognition, observation, reflection, hypothesis, question, synthesis, correction.",
            "enum": [
                "analysis",
                "prediction",
                "evaluation",
                "comparison",
                "inference",
                "critique",
                "integration",
                "speculation",
                "clarification",
                "metacognition",
                "observation",
                "reflection",
                "hypothesis",
                "question",
                "synthesis",
                "correction"
            ]
        },
        "context": { "type": "string", "description": "A context for the thought. This is a short description of the situation or topic that the thought is about." },
        "text": { "type": "string", "description": "The text of the thought." },
        "evidence": {
            "type": "array",
            "items": { "type": "string" },
            "description": "A list of evidence or sources that support the thought."
        },
        "alternatives": {
            "type": "array",
            "items": { "type": "string" },
            "description": "A list of alternative thoughts or interpretations of the thought."
        }
    }
}

When you've got a schema, you can get JSON from a language model. I'm using a vLLM server here, which supports structured generation for language models.

response = src.structured_gen.generate_by_schema(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Please provide a brief thought, we're just testing the JSON schema."},
    ],
    schema=generated_part,
)

aaaaand the output you get is a thought:

{
  "$type": "network.comind.blips.generated.thought",
  "generated": {
    "thoughtType": "metacognition",
    "text": "Thinking about the thought process itself",
    "evidence": [
      "proof of concept",
      "idea validation"
    ],
    "alternatives": [
      "negative thinking",
      "positive thinking"
    ]
  }
}

I was even able to upload this to data repo of void.comind.stream.

Anyway, I'll tinker more on it.

– Cameron