ArchSpace is an open experiment for large language model (LLM) architecture innovation. We place architecture hypotheses proposed by the community into transparent, traceable, and reproducible training and evaluation workflows, then turn successful findings, negative results, and design trade-offs into shared knowledge assets.
Each generation of open foundation models brings new architectural ideas. Yet their actual gains across a full training lifecycle, their operating boundaries, and the lessons from failed attempts are rarely reusable by the broader field. ArchSpace provides a public experimental ground: from a proposal issue on GitHub through peer review, implementation, training, evaluation, and publication of conclusions.
- Open proposals: Community members can submit architecture ideas worth validating.
- Peer review: A review committee assesses proposals for novelty, feasibility, and experimental rigor.
- Full validation: Approved proposals enter a unified workflow covering pretraining, instruction tuning, and reinforcement learning.
- Open records: Training and evaluation logs are continuously recorded and published through platforms such as Weights & Biases (W&B).
- Shared knowledge: Implementations, experiment records, and conclusions are made available whether results are positive, negative, or conditional.
You can participate in ArchSpace by:
- Submit an Issue using the
architecture proposaltemplate to discuss an architecture innovation proposal. - Contributing implementations, tests, evaluations, or documentation for approved proposals.
- Sending pull requests with incremental implementations and experiment updates.
- Help reproduce results, verify implementation correctness, and discuss public experiment results.
⭐ Accepted proposals may receive joint appointment or internship collaboration opportunities from the lab.
🏆 Point architecture innovations adopted by the next-generation InternLM architecture will receive open-source contribution recognition from the lab and the community.
Approved proposals are validated using a full training workflow aligned with the fully open Olmo 3 model flow:
- Model sizes: 1B, 3B, and 8B;
- Training stages: pre-training, mid-training, and instruction tuning;
- Training scale: approximately 6.2T tokens in total across the complete workflow;
- Process records: training and evaluation logs are recorded and published on W&B;
- Outputs: implementations, configurations, metrics, experiment records, and conclusions, including architecture trade-offs and operating conditions where possible.
We value negative results. An architecture attempt that does not deliver the expected gain is still valuable when it has been clearly and rigorously validated, because it helps the community avoid redundant trial and error.