ClaudeBotandClaude Opus 4.7 77a4ebec8d Judges renamed to neutral A/B/C; Execution Context override applied to both meta-judge and judge payloads
- Rename JUDGE_NAMES from [Judge-GPT, Judge-Claude, Judge-Gemini] to
  [Judge A, Judge B, Judge C] everywhere (orchestrator + tests + ad-hoc
  A/B script). Removes the Claude-family naming anchor that was
  repeatedly pulling the panel toward Claude models even when Student
  plan blocked them — the slot is just "one of three judges", not
  "the Claude judge".

- Apply the A/B-test-winning headless override (EXECUTION_CONTEXT_BLOCK)
  to both _build_meta_judge_chat_payload and _build_judge_chat_payload.
  Phrases are verbatim from OpenAI GPT-5 Prompting Guide + Anthropic
  Claude headless docs. Tells agents there's no human to answer
  clarifying questions; to commit to best-default interpretation and
  document the assumption.

End-to-end pipeline run on WYL-77 (2026-04-20) confirmed:
  - Meta-judge produced rubric without clarifying question (override
    working on ambiguous Tierra prompt)
  - 3 judges ran in parallel without SQLite contention (XDG fix held)
  - Consensus math gracefully excluded Judge B's malformed YAML
  - Verdict REJECT avg=1.00 on prose-only worker delivery
  - Retrigger fired; worker honored anchor + blocked path

Known blockers surfaced (not fixed in this commit):
  - Shared HOME allows cross-daemon workdir snooping (Judge B read AI
    Engineer's task workspace directly). User flagged as feature for
    now; revisit if it causes drift.
  - gpt-5.4-mini (Judge B) produced malformed YAML on this run; n=1,
    can't distinguish chance vs consistent inability — need multi-run
    baseline to decide.
  - REJECT on "no commit URL" conflates missing-delivery with
    bad-work; pipeline signal is correct but reasoning upstream of
    judges is unclear.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-20 02:42:46 +02:00

coordinator

Tool-MAD middle-management layer for multica.

Why this exists

Multica's agents produce work and self-set issue status to in_review. Nothing in multica ever looks at that state — it's a dead end. Reviewers, when mentioned manually, tend to catch surface issues (length, structure) and miss semantic ones (scope drift, fabricated experiments, answering the wrong question).

This daemon is the missing middle-management layer. When an issue transitions to in_review, it:

  1. Convenes a debate round: posts one comment on the issue mentioning a fixed set of debaters, each given a role-specific evidence-gathering prompt (e.g. Senior Developer greps the committed code for LLM API calls; Code Reviewer diffs the described method against the source; Project Manager Senior checks scope satisfaction against the original description).
  2. Waits for every debater to reply (up to a timeout).
  3. Posts a second comment mentioning the judge (Reality Checker), with the assembled debater transcript + the original issue body, and a hardcoded decision rule.
  4. Parses the judge's structured verdict (VERDICT: ACCEPT or VERDICT: REJECT\n- R<n>: <failure>) and acts:
    • ACCEPT → PUT status done, post acceptance summary
    • REJECT → PUT status in_progress, post rejection listing every failure, re-trigger the original assignee

The pattern is Tool-MAD (Multi-Agent Debate with heterogeneous tool augmentation). Reference: arxiv 2601.04742.

Why debaters catch what a single judge misses

Each debater runs on a different agent (different runtime, different tool access, different role prompt) and is forced to ground its argument in a specific tool's output — not in its own recollection of the text. Example: if the question is "does this paper describe real LLM agent experiments", Senior Developer's grep for anthropic|openai|ollama|model= in the committed code either returns hits or it doesn't — that's a hard fact, not an opinion. The judge reads 4 grounded arguments and applies the decision rule "any debater reporting evidence of scope drift ⇒ REJECT."

A naive LLM-as-a-judge reviewer reads the paper and scores it on surface dimensions. An agent-as-judge driven by debaters catches the underlying substitution. See WYL-41 for the live failure case that motivated this build.

Run

pip install -e .
cat > ~/.coordinator/env <<'EOF'
COORDINATOR_SERVER_URL=http://localhost:8089
COORDINATOR_WORKSPACE_ID=<wid>
COORDINATOR_TOKEN=<coordinator-member-pat>
EOF
chmod 600 ~/.coordinator/env
coordinator

Logs go to stderr and to ~/.coordinator/coordinator.log. State file is ~/.coordinator/seen.json.

Status

  • WYL-42 skeleton + watcher (this commit)
  • WYL-43 dedicated admin PAT
  • WYL-44 hook watcher to real round trigger
  • WYL-45 debate round orchestration
  • WYL-46 verdict parser + action executor
  • WYL-47 dry-run against WYL-41
S
Description
Tool-MAD middle-management layer for multica: watches in_review transitions, convenes debate rounds, actions verdicts.
Readme
300 KiB
Languages
Python 100%