Can OpenRE-Bench tell whether AI turned a GitHub issue into a usable requirement?
A ten-case benchmark lets teams compare how agent frameworks turn feature requests into requirements. A worked example shows what to score before any code is written.
A GitHub issue says, “Keep my filters when I come back.” A coding agent can implement something immediately. It may also choose the wrong interpretation. Does “come back” mean opening the list after viewing an item, refreshing the browser, signing in tomorrow, or moving to another workspace? Each answer produces a different feature and a different set of acceptance tests.
OpenRE-Bench examines what happens in the space between an issue and an implementation. A preprint submitted to arXiv on November 10, 2026, presents a common test harness for agent frameworks that produce requirements. Its first suite contains ten cases, with five derived from public GitHub feature requests and five written by the researchers. The authors provide three example adapters so the same inputs and scoring contract can be applied to different frameworks. OpenRE-Bench preprint
Think of the benchmark as a way to ask a sharper question about agent output. Would two people reading the proposed requirements build and test the same behavior? The filter example below shows why a trace of the decisions matters as much as the final prose, and how a team could check its own issue-to-requirement workflow.
Keep the decision visible before the code exists
Requirements engineering is the work of making goals, constraints and observable behavior clear enough for people to agree on them. A good issue may explain a user's problem well and still leave several implementation decisions open. Two stakeholders might agree that navigation should preserve work and disagree on whether preferences should follow a user across devices. A useful first draft records that distinction so it can be resolved by the people who own it.
Here is one illustrative interpretation of the filter request. When a signed-in user opens an item and returns to the list in the same workspace, the list restores the previous filter values. The scope excludes changes to preferences in other workspaces until the team makes a separate decision. The acceptance check has a clear starting state, an action and an observable outcome. It can run against a prototype or a finished feature.
A second possible interpretation stores filters after sign-out and on another device. That version needs an account-level storage decision and a rule for how two devices with different saved filters should interact. The one-line issue did not specify those behaviors. An agent that adds them because they sound convenient can turn a small navigation fix into a product change with a different cost and privacy impact.
| Part of the request | A reviewable first draft | What someone still needs to decide |
|---|---|---|
| Return from an item to the list | Restore the visible filters within the same workspace | Whether list position should return as well |
| Reload or sign in again | Record this as an open decision | How long preferences survive and where they are stored |
| Switch workspaces | Keep a workspace's filters from appearing in another | Whether each workspace keeps its own history |
| Share a filtered link | Treat link sharing as a separate use case | Which values can be represented in a shared URL |
A worked example for reviewing generated requirements before implementation. Original illustration by Telli.sh.
Notice that a reader can test the first row without a technical explanation of the storage layer. A reviewer can also see that other rows need decisions. The useful requirement is specific about the behavior it promises while remaining clear about the questions a team still owns.
The benchmark holds the input and evidence contract steady
The OpenRE-Bench preprint describes a shared CaseInput with a case name, description and requirement text. An adapter translates between this contract and each participating framework. Its job is to preserve what the harness needs for comparison while leaving the framework free to choose how its agents collaborate. The bundled examples are MARE, iReDev and QUARE. The authors describe MARE and iReDev as reimplementations based on the original papers. OpenRE-Bench paper, section II
The harness collects four phases of output. It records initial models, a negotiation record, an integrated model and a verification output. Each phase answers a different review question. What did the agents see at the start? Where did they disagree? Which choices survived the integration? What evidence is available for checking the result? This shared structure makes it easier to follow a decision even when different frameworks use different names for their internal roles.

Figure 1 from Cheng et al., OpenRE-Bench, November 10, 2026. Cropped from the authors' PDF and resized for display. Original figure · CC BY 4.0.
In the filter example, a Phase 1 output might propose both “preserve the current list on return” and “remember preferences across devices.” A Phase 2 record could mark these as two possible scopes. The integrated artifact could keep the first and leave the second unresolved. The final verification would then check the behavior that survived. A review based on all four phases can distinguish a legitimate scope decision from an accidental omission.
For someone comparing agent frameworks, adapters also make the limits of the comparison easier to state. A system with a sophisticated internal representation may lose some nuance when it exports to a common schema. Keep the native artifacts when they help explain a result. The standardized outputs are the consistent comparison surface and the native artifacts can explain where a framework performed differently.
The ten cases test the pipeline more than the field
Cheng and coauthors report starting with 3,000 fetched issues across five projects spanning Python, TypeScript and Java. Their filters left 1,396 issues. From that pool, 18 issues went into five mined cases, which joined five hand-built cases in the v1.0 suite. The pool and the evaluated cases are different quantities. A run on the bundled suite operates on ten cases, even though the upstream pipeline begins with thousands of issue records. OpenRE-Bench paper, table I
The process is part of the benchmark's contribution. A recorded fetch and filter sequence tells you how a real project issue reached a test case. That helps when you want to compare a newer run to a published result. It also lets reviewers see which kinds of requests the corpus tends to select, including the effect of labels, comments, aggregation periods and other filters.
Those choices shape which problems the benchmark represents. Public repositories frequently discuss feature scope where anyone can read it. A team planning an internal permission change may negotiate much of its scope in a customer call or private design review. A model that does well on the public cases may still need domain-specific tests for that setting. Add cases involving incomplete context, requirements spread across several documents, or contradictory comments when those are common in your backlog.
Five hand-written cases give the suite a different kind of starting point. A curated case can make a specific ambiguity easier to isolate, while a mined case can preserve how requests naturally arrive. Report the results on each slice. A framework that improves on a clean case while losing track of decisions in a mined thread may need different prompts or supporting evidence before it earns a place in an issue-triage process.
A shared scoring run gives reviewers a stable comparison and a place to add checks for their own work. Original illustration by Telli.sh.
A verifiability score needs a source check beside it
The paper describes a reproducible 1-to-5 structural verifiability score based on compliance signals associated with ISO/IEC/IEEE 29148. It also reports element counts, semantic preservation between a run's own initial and integrated outputs, and elapsed time. Together these measures can show whether a system produces structured artifacts consistently and preserves ideas through its workflow. They cannot settle whether a business assumption was correct if every phase inherited it. OpenRE-Bench paper, metrics
For example, assume the first model interprets “come back” as “sign in next week.” If the same interpretation appears in the integrated output, a semantic preservation metric may reflect consistency across phases. A human reading the issue might still flag that interpretation as unjustified. The metric and the review have answered different questions. Both are useful when their scope is clear.
Keep a small gold set for claims about the source. For each issue, identify the sentences that directly support a requirement and the choices the source leaves unresolved. Ask a reviewer to mark dropped constraints, plausible assumptions presented as facts, and contradictions between the proposed test and the issue. A checklist with five specific questions can reveal more about your deployment than a single average across many differently scoped cases.
Consider another example from a hypothetical export feature. An issue asks for reports to remain readable in a spreadsheet after a regional date format changes. One draft defines an actor, a data resource, a sequence and a verification record. The verification reads “Confirm the date is exported correctly.” Another draft simply says “After choosing a day-first locale and exporting a row dated 4 March 2026, verify that the exported spreadsheet still stores the date as 4 March 2026 when reopened in a month-first locale.” The second sentence is easy to turn into a test. The first draft is richer in formal elements, but its checking language leaves two readers free to disagree on the correct result.
There is a useful test of the review itself here. Ask two reviewers to write the minimum fixture required for the same proposed test. If one picks an ambiguous date and the other chooses the 28th of a month, their fixtures may reveal different classes of problem. A good acceptance check names the data that exposes the error. The agent can help produce that check if it sees the reason the original issue mattered.
Cost and time deserve the same care. Some frameworks make more calls as they reconcile competing perspectives. A slower workflow can earn that time if it surfaces conflicts that your team regularly misses. If it makes the same decisions as a simpler baseline while adding verbose intermediate artifacts, the difference belongs in the cost comparison. Run the baseline and framework on the same case before deciding that additional steps are worthwhile.
Run a small comparison on requirements you can judge
- Start with the paper's pinned artifact and an offline smoke run. The authors document a key-free
single_agentsetting. It is a way to check the runtime and artifact paths before paying for a model-backed comparison. OpenRE-Bench artifact instructions - Pick a small set of public or authorized issues. Include one simple request, one request with conflicting comments and one that leaves a policy decision open. Write the expected behavior and the open questions before you examine agent output.
- Run each candidate with the same input and record the framework version, model configuration and run setting. Save the intermediate artifacts. Use the same criteria for what counts as a missing requirement or an unsupported assumption.
- Compare the output at the decision level. Ask what could be verified by observing the product and which issue text justifies it. Record whether an issue needs a person to clarify scope before anyone writes code.
- Use the results to choose where an agent belongs. Some requests are ready for a draft acceptance test, while others benefit from a list of questions for the author. Keep those two outputs visible in the workflow.
A promising follow-up is to take a requirement through a real review meeting. Compare the agent draft to what the team ultimately agrees to ship. If reviewers rewrite the same category of claim every time, add a test for it. A requirement with a precise expected result and a traceable source is useful even when the rest of the document still needs editing.
OpenRE-Bench makes comparison practical before the first code change. The thing to judge is whether the next person can see which behavior the team agreed on, where the evidence came from and which decisions still need someone to make them.
Sources
- Cheng et al., OpenRE-Bench preprint, November 10, 2026
- Full paper with methods, data and original Figure 1
- Author-provided versioned artifact
- Creative Commons Attribution 4.0 International
To keep the decisions behind feature requests and reviews searchable, create a Telli.sh account.