Replace tasks with 11,429 validated tasks from 10k batch#1
Replace tasks with 11,429 validated tasks from 10k batch#1self-supervisor wants to merge 1 commit intomainfrom
Conversation
Oracle-pass + nop-fail filtered dataset across 4 repos: facebook/docusaurus, Kong/insomnia, mantinedev/mantine, rollup/rollup Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
![mesa-dot-dev[bot]](https://avatars.githubusercontent.com/in/1050077?s=60&v=4){kind=small}
There was a problem hiding this comment.
Performed full review of 3542d8d...ddc25cc
Analysis
-
Repository Structure Transformation: The PR converts the repository into a large data store for 11k+ reproduction tasks using a rigid directory structure, shifting it from source code to a binary dataset without documentation on how to navigate this structure.
-
Git Scalability Concerns: Storing thousands of tasks directly in Git will lead to repository bloat, slow clone times, and painful history rewrites, with no strategy for sharding or external storage.
-
Inadequate Git LFS Implementation: Current Git LFS configuration only targets test.sh files (which are small) while leaving potentially larger Dockerfiles and artifacts in regular Git, delivering minimal benefits.
-
Data Quality Issues: At least one "solution" patch contains the exact bug it claims to fix, raising serious concerns about validation quality across all 11,429 tasks.
-
Missing Critical Infrastructure: No manifest file, metadata registry, documentation, CI/CD pipeline, programmatic interface, or licensing information to support discoverability and usage of the dataset.
Tip
Help
Slash Commands:
/review- Request a full code review/review latest- Review only changes since the last review/describe- Generate PR description. This will update the PR body or issue comment depending on your configuration/help- Get help with Mesa commands and configuration options
76 files reviewed | 1 comments | Edit Agent Settings • Read Docs
| + const model = mustGetModel(docToCopy.type); | ||
| + const overrides = { | ||
| + _id: generateId(model.prefix), | ||
| + modified: docToCopy.modified, |
There was a problem hiding this comment.
The fix changes modified: Date.now() to modified: docToCopy.modified, which preserves the original document's timestamp. While this might seem reasonable for maintaining metadata, combined with the bug at line 63 (where children incorrectly reference the original parent), this creates a compound issue where duplicated documents not only have broken relationships but also retain old timestamps. This makes debugging harder since the modified timestamp won't reflect when the duplicate was created.
Prompt for Agent
Task: Address review feedback left on GitHub.
Repository: VmaxAI/tasks#1
File: data/Kong__insomnia.44642a49.BaseModel.784f0635/solution/fix.patch#L45
Action: Open this file location in your editor, inspect the highlighted code, and resolve the issue described below.
Feedback:
The fix changes `modified: Date.now()` to `modified: docToCopy.modified`, which preserves the original document's timestamp. While this might seem reasonable for maintaining metadata, combined with the bug at line 63 (where children incorrectly reference the original parent), this creates a compound issue where duplicated documents not only have broken relationships but also retain old timestamps. This makes debugging harder since the `modified` timestamp won't reflect when the duplicate was created.
1 participant
Oracle-pass + nop-fail filtered dataset across 4 repos: facebook/docusaurus, Kong/insomnia, mantinedev/mantine, rollup/rollup
Now I have a clear understanding of the changes. Let me write the high-level PR description:
What changed?
Added 11,429 validated code repair tasks across 4 open-source repositories:
Each task includes:
instruction.md- Bug report with reproduction steps and expected behaviorsolution/solve.sh- Validated fix implementationtests/- Test configuration and validation scriptsValidation
Dataset filtered using oracle-pass + nop-fail criteria to ensure:
Description generated by Mesa. Update settings