Specification Algebra

Specifications and partial specifications that are used in py-mcmas support a kind of algebra with various operators.

Many operators are focused on quick and easy construction/combination of the main specification objects:

Other operators are more ambitious, and trying to lay the groundwork for evolving new specifications from existing ones.

Some quick links:

  1. Ellipsis Notation: ISPL(...)
  2. Constructor Notation: ..
  3. Shuffle Notation: ..
  4. Completion Notation: ..
  5. Ordering Notation: ..
  6. Boolean Algebra: ..
  7. Concatenation Algebra: ..

Ellipsis Notation

Since partial specifications are supported, one of the most common use-cases is to combine these fragments to get a ready-to-run specification that can actually be simulated.

Ellipsis notation helps here. It can be used to get a minimum viable object.

spec = ISPL(...)
agent = Agent(...)
environment = Environment(...)

Agents and Environments need only validate, but the minimum-viable ISPL specification should be able to validate and execute. From the command line:

# Returns raw ISPL source code
$ ispl --python --command 'ISPL(...)'

-- Minimal valid ISPL specification
Agent Environment
  Vars:
    ticking: boolean;
  end Vars
  Actions = { tick };
  Protocol:
    Other: {tick};
  end Protocol
  Evolution:
  end Evolution
end Agent
..

To confirm this runs, just pass the sim argument:

# Run the spec
$ ispl --python --command 'ISPL(...)' --sim

You can use extended ellipsis notation to add overrides on top of the minimal object. For example, to obtain a minimum-viable agent with custom actions:

agent = Agent(...)(name='card-player', actions=['hold-em', 'fold-em'])

As seen in the pythonic-ISPL demo you can compose specs in one shot this way, or compose them incrementally as seen here.

Constructor Notation

Constructor notation uses square brackets and looks like item-retrieval but has different semantics:

agent = Agent[obj]

Above, obj can be things like: an LLM prompt, a python function, or a pydantic agent, and you'll get back at least a partial ISPL agent specification. See the test-cases below for some concrete examples. This works with Specifications and Environments in theory, but if your AI backend is using small local LLM models, in practice it's probably most useful with Agents. For concrete examples, check out the test-cases below.

Summary 
import os
import sys

from mcmas import ISPL, Agent, Environment

import pytest

# required for even importing the pydantic examples
os.environ["GROQ_API_KEY"] = "fake"


def test_agent_from_prompt():
    prompt = (
        "a person named alice is playing a game of cards."
        'actions are available at each turn, and alice may "hold" or "fold"'
    )
    agent = Agent[prompt]
    assert agent.name == "alice"
    assert agent.actions == ["hold", "fold"]


def test_environment_from_prompt():
    prompt = "Variables are foo, bar, and baz."
    env = Environment[prompt]
    assert sorted(list(env.vars.keys())) == "bar baz foo".split()


@pytest.mark.xfail(reason="smaller LLMs choke on bigger json-schemas! :/")
def test_spec_from_prompt():
    prompt = "The title of the specification is Bridge Game."
    spec = ISPL[prompt]
    assert spec.title == "Bridge Game"


def test_spec_from_implementation():
    from pydantic_ai_examples import roulette_wheel

    agent_implementation = roulette_wheel.roulette_agent
    agent_spec = Agent[agent_implementation]
    expected = "load_from_pydantic_ai_agent"
    actual = agent_spec.metadata.parser
    assert expected in actual
    assert "square" in agent_spec.vars
    actual = agent_spec.vars["square"]
    expected = f"0..{sys.maxsize}"
    assert actual == expected, "integer->bounded integer type conversion failed"

Shuffle Notation

The mutation operator returns a "mutationd" version of a given object. A mutation has the same type as the object you started with, and is based on the original object, but the result has been mutated using a simple prompt.1 The mutation operator looks like using exponentiation, and the argument used as the exponent is passed through to pydantic-ai as top_p and used to control nucleas sampling2.

Shuffling is really just a proof of concept, and the prompt is very naive. But the idea is to pave the way for stuff like using genetic-programming, with LLMs as evolutionary operators. See the reference zoo for related papers / inspiration.

Completion Notation

Summary 
""" """

from mcmas import Agent, ai

# download model if necessary
ai.init()


def test_model_completion():
    a1 = Agent()
    assert not a1.concrete, "original agent is empty and should not be valid"

Ordering, Ranking, Sorts

The related concepts of length, absolute value, and ordering are defined for specification and agent objects.

Like can be compared with like, but e.g. Specifications cannot be compared to Agents.

Summary 
from mcmas import ISPL, Agent

import pytest

simple_expr = "p"
nested_expr = "GCK(<g1>, K(a1, AF(p)))"
very_nested_expr = "AG(EF(GCK(<g1>, K(a1, p))))"


def test_abs():
    value = abs(ISPL(formulae=[simple_expr]))
    assert isinstance(value, float)


def test_spec_ranking():
    x = ISPL(formulae=[simple_expr])
    y = ISPL(formulae=[nested_expr])
    z = ISPL(formulae=[very_nested_expr])
    assert x < y < z
    assert z > y > x
    assert sorted([x, z, y]) == [x, y, z]


def test_agent_ranking():
    a1 = Agent(vars={var: bool for var in "xyz"})
    a2 = Agent(vars={var: bool for var in "abcd"})
    assert a2 > a1


def test_incomparable():
    agent = Agent(...)
    spec = ISPL(...)
    with pytest.raises(TypeError):
        agent < spec

For Specification objects: absolute value, less-than, and greater-than work based on a heuristic approach for determining formulae complexity. The complexity of a specification is just the sum of the individual formulae complexity.

For Agents: length, less-than, and greater-than is based on the description length. Note that this is very different from say, state-space complexity! For example, a bounded-integer from 0..<maxint> could have huge implications for the exhaustive search of state-space, but still only has a score of 1 for the description length.

Addition, Subtraction, Equality, etc

Summary 
from mcmas.ispl import ISPL, Agent, Environment

from pytest import fail


def test_add():
    """adding agents to spec embeds those agents in the spec"""
    spec = (
        ISPL()
        + Agent(name="alice", actions="holdem foldem".split())
        + Agent(name="bob")
        + Environment()
    )
    assert "alice" in spec.agents
    assert "bob" in spec.agents


def test_subtract():
    """subtracting agents from spec removes it, if present"""
    alice = Agent(name="alice")
    bob = Agent(name="bob")
    spec = ISPL() + alice + bob
    assert all([alice in spec, bob in spec])
    spec -= alice
    assert alice not in spec


def test_iadd():
    """incremental addition also works as expected"""
    spec = ISPL()
    spec += Agent(name="alice")
    spec += Agent(name="bob")
    spec += Environment()
    assert "alice" in spec.agents
    assert "bob" in spec.agents


def test_agent_addition_undefined():
    try:
        Agent(name="alice") + Agent(name="bob")
    except (TypeError,) as exc:
        pass
    else:
        fail("agent addition is undefined")

Summary 
from mcmas.ispl import ISPL, Agent, Environment


def test_eq():
    alice1 = Agent(name="alice")
    alice2 = Agent(name="alice")
    assert alice1 == alice2
    bob = Agent(name="bob")
    assert alice1 != bob


def test_trivial_spec():
    """
    ellipsis notation returns 'trivial' objects,
    which are the concrete minimum that validates *and* runs.
    """
    spec = ISPL(...)
    assert spec.environment.concrete
    assert spec.agents
    assert not list(spec.agents.values())[0].advice
    assert spec.concrete


def test_trivial_env():
    """
    ellipsis notation returns 'trivial' objects,
    which are the concrete minimum that validates *and* runs.
    """
    spec = Environment(...)
    assert spec.concrete


def test_trivial_agent():
    """
    ellipsis notation returns 'trivial' objects,
    which are the concrete minimum that validates *and* runs.
    """
    spec = Agent(...)
    assert spec.concrete


def test_invert():
    """
    the inversion operation means 'autocomplete from ai'.
    since trivial specs are the minimum complete spec,
    the autocomplete operation returns the same object
    """
    spec = ISPL(...)
    assert spec.concrete
    assert ~spec == spec


def test_ellipsis():
    spec = ISPL(...)(agents=dict(p1=Agent(...)))

Other Usage Hints

Check out the test-suite for more usage hints:

References

  1. src/mcmas/templates/prompts ↩

  2. A concept similar to temperature. See pydantic_ai.settings.ModelSettings.top_p ↩