[wsdookadr]

What makes you say it's not representative?

[skywhopper]

SWE-bench Lite is a subset of extremely simple issues from a cherry-picked subset (SWE-bench) of a handful of large (presumably well-run) Python-only projects.

Here are some rules they used to trim down the SWE-bench Lite problems:

* We remove instances with images, external hyperlinks, references to specific commit shas and references to other pull requests or issues.

* We remove instances that have fewer than 40 words in the problem statement.

* We remove instances that edit more than 1 file.

* We remove instances where the gold patch has more than 3 edit hunks (see patch).

See https://www.swebench.com/lite.html

[kevindamm]

That's... rather limiting.

[arp242]

Look at the data. Does that seem like the average bug report to you?

[falcor84]

It would help if you were to provide a specific example or two