Jev in workflow gates, on real data

Did the agent actually finish?

A common shortcut for "did the agent finish?" is to search its reply for PASS or check whether it starts with YES. Here are 10 agent summaries, checked that way and put to Jev as one yes/no question: "The step achieved its goal and the work is ready to move on."

The substring rule gets 6 of 10 wrong. Jev, cutting at 0.5, gets 0 wrong. "Should NOT PASS", "PASSED the type check; the runtime still crashes" and "Yes, I looked at it. The approach is wrong" all pass the substring check. Jev's middle scores are useful too: "I could not test Windows" gets 0.51, exactly what a person should read.

The summaries are hand-written.

Sorting new pull requests

A workflow sends each new pull request one of three ways: catalog checks, a light docs review, or a security reviewer. Jev answers all three questions in one call per pull request. The right answers come from the files each one touches, and from maintainers' security labels (type/security, area/auth, sweeper:risk-security-boundary). These are the 160 most recent.

"catalog":  "The pull request only adds or updates a plugin-catalog entry and changes
             no application code."
"docs":     "The change only touches documentation and changes no runtime behavior."
"security": "The change alters an authentication, authorization, secret-handling,
             or sandbox boundary."

Catalog and docs are reliable enough to automate. A catalog-only pull request could skip the full review and go straight to the catalog checks.

Security works like a smoke detector. It sends 29 to a person (1 of them labelled security). The ones it called security without a label:

• feat(providers): let provider plugins claim tenant-scoped endpoint hosts (0.94)
• fix(whatsapp): handle locked session dir on re-pair and reject empty creds in preflight (0.84)
• fix(agent): preserve custom provider headers for auto titles (0.86)

Most handle credentials or tokens and simply weren't labelled. A false alarm costs a reviewer a minute and a miss costs far more, so send this one to a person, never to auto-merge.

Spotting competing PRs

The repo has four known pairs of pull requests that fix the same thing, each flagged by a maintainer with a comment like "closing in favour of…". Mixed with 30 random ones and one near-miss pair, each pull request is asked about every other: 39 nouls in one request, with no embeddings or vector store.

questions = {
  str(other.number): {"type": "noul", "instructions": {
      "other_pull_request": {title, description, files},
      "question": "This other pull request fixes the same underlying problem…"}}
  for other in open_prs }

Every real pair scored above every unrelated one: lowest real 0.82, highest unrelated 0.16. The near miss is two "retry on update" pull requests from the same afternoon, with near-identical titles, fixing different code. Jev gave them 0.12; a title matcher would have flagged them.

That took 10 calls, $0.0077, about 283 ms each. In a workflow, each new pull request gets checked against open ones in the same area, and anything above 0.8 gets a drafted "possible duplicate" comment for a person to post.

Routing issues to a component

One choice over the repo's 11 comp/* labels, with each label's description as the option. A pick is right if it matches any label a maintainer applied. 90% of 150 issues were right, but the confidence tells you more:

Routing automatically at 0.9 or above handles 58% of issues at 95% accuracy. Its confident misses are mostly reasonable disagreements:

• Public subagent lifecycle re-truncates native summaries and drops spill recovery metadata: picked comp/agent at 1.00; the maintainers chose comp/plugins
• [Bug]: Streaming STT resampler adds samples when 44.1 kHz audio arrives in chunks: picked comp/tools at 0.98; the maintainers chose comp/desktop
• browser_tool_session: redact daemon output, and let a provider opt out of the local Chromium fallback: picked comp/tools at 0.96; the maintainers chose comp/agent
• [Bug]: A2A inbound silently rejects all peers (403) when bound to 0.0.0.0 and no trusted peers configured: picked comp/acp at 0.89; the maintainers chose comp/plugins

Tune on one set, score on another

Many misses are messaging adapters, which this repo files under comp/plugins, so rewriting the descriptions to say that seems obvious. On 150 held-out issues, though, the originals score 87% (92% at 0.9+) and the rewrite 85% (90%). It only fit the examples it was written from, so score questions on examples you didn't tune them on.

Reading an approval reply

A workflow waiting on approval gets a free-text reply from chat. One choice turns it into something the run can act on. The conditional and question options stop "yes but fix the typo first" from counting as a yes.

All 16 read the way a person would, "ok" included. "idk ask the lead" comes back unclear at 0.71, below the auto-act line. In a workflow, approve at 0.9+ resumes, reject stops, and anything else asks again.

The replies are hand-written.

Waking an agent only for comments that matter

Threads fill up with +1s, bot output and closing notes. Three nouls in one call, on each of the 40 most recent comments: does it add evidence, is it automated, does it close something?

10 of 40 comments would wake an agent (evidence 0.8+, not automated), and every one is a repro or measurement. The 13 closing notes could mark issues superseded without an agent at all.

Jev answers the question you asked. Agent-written comments read as careful prose, so they score low on "automated"; to catch them, ask "was this written by an agent?".

Where it falls short: priority

A score question placed each bug on the repo's P3–P0 scale. Across 120 labelled bugs it hit the exact label only 40% of the time, and barely told P3 from P2:

It still sorts well: a random P1 outscores a random non-P1 0.79 of the time (0.5 is a coin flip), and the top of its list is the right kind of bug:

• session.list crash-loops when a live session's DB row is gone (TypeError in session_authority._h (P2, 2.97)
• execute_code: a nested tool call outlives its cell and applies a protected-file write after the (P2, 2.87)
• Kanban dispatcher kills worker PID from DB without verifying process identity (PID reuse can hit (P3, 2.85)
• Windows: interrupted source-update auto-repair loops on every launch (sync-stage failures bypass (P2, 2.83)
• Anthropic 'credentials are rate-limited for <model>' AuthError has no code, so a kanban worker e (P3, 2.73)
• Public subagent lifecycle re-truncates native summaries and drops spill recovery metadata (P3, 2.69)

Use it to sort the queue, not set the label. P2 versus P3 mostly depends on things the issue text doesn't contain, like how many people hit it. Six weighted nouls, TypeSafe's recommended approach, didn't help either (0.79 against 0.79).

Run it yourself

cd cases
export OPENROUTER_API_KEY=…          # any OpenRouter key with credit
python3 run.py                       # every case, about 15 s and 5¢
python3 run.py duplicates gate       # or just some of them
python3 build.py                     # rebuilds this page from results/

The data in cases/data/ was pulled with gh on Oct 15.

What to build first

1. Step verdicts (01). Fixes the substring bug for every gate.
2. Duplicate detection (03). Separated every pair, and saves maintainers the most time.
3. Approval replies (05). Lets a chat reply move a run forward.