SWEbench alternatives

HumanEval is a benchmark dataset developed by OpenAI specifically for evaluating large language models on code generation tasks, focusing on understanding programming tasks and producing syntactically correct and functionally accurate code.

LiveCodeBench evaluates LLMs on 400 problems from competitive programming platforms, focusing on code generation, self-repair, and test output prediction, with problems updated over time to reduce data contamination.

ClassEval is a manually constructed benchmark that measures how well LLMs can generate full classes of code, including tasks with library, field, or method dependencies, reflecting real-world software engineering scenarios.

APPS is a large-scale code generation benchmark comprising 10,000 problems collected from open-access competitive coding websites, ranging from one-line solutions to substantial algorithmic challenges.

This benchmark evaluates LLMs on real-world software engineering tasks sourced directly from Upwork freelance jobs, including both coding ability and engineering management decisions, with actual dollar values attached.